# 🔍 Description
## Issue References 🔗
Fix a ClassNotFound issue.
```
java.lang.NoClassDefFoundError: org/apache/kyuubi/shade/javax/ws/rs/core/Cookie
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Verified manually.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6723 from pan3793/6638-followup.
Closes#6638
56e9842e0 [Cheng Pan] [KYUUBI #6638] authz shaded should include jsr311-api
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- replace immutable empty list and map with mutable one for BatchRequest initialization, for easier BatchRequest construction for rest-client use
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6727 from bowenliang123/batchapi-emptylist.
Closes#6727
9d6a04c8a [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6720
## Describe Your Solution 🔧
If pod goes into OOMKilled state, application should be marked as KILLED, which is eventually identified as isFailed
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested locally, was able to launch new session
<img width="922" alt="kyuubi_new_session" src="https://github.com/user-attachments/assets/b003c86f-484d-40c5-b173-847374a45b1d">
---
**Be nice. Be informative.**
Closes#6721 from Madhukar525722/OOM.
Closes#6720
cd0bdf633 [madlnu] [KYUUBI #6720] K8s pod OOM Killed should be identified as Application failed state
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix a couple of typos
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6719 from pionCham/fix-typos.
Closes#6719
71409a875 [chengpeiming] fix violations in jvm-quake.md
de8f0d7b8 [chengpeiming] fix some typos
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6709
## Describe Your Solution 🔧
transfer the type of authTypes from seq to set before do equals check
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
test in our produce kyuubi env
#### Behavior Without This Pull Request ⚰️
<img width="1431" alt="image" src="https://github.com/user-attachments/assets/6fd40d49-8d6b-446a-8feb-70df8c92604d">
<img width="603" alt="image" src="https://github.com/user-attachments/assets/637e5788-bfe0-4bab-a2a5-f6a79fb93fa6">
always hint to login in all sub tabs of management tab in kyuubi web ui, but can not login in actually, while use confkyuubi.authentication NOSASL, and our java version is 1.8 scala version is 2.12.18
#### Behavior With This Pull Request 🎉
<img width="1422" alt="image" src="https://github.com/user-attachments/assets/211b76ee-d937-456b-bed4-f94dd41896f0">
#### Related Unit Tests
none, no need i think
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6711 from lifulong/authentication_filter_auth_type_bug_fix.
Closes#6709
850bda4f3 [lifulong] fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
Authored-by: lifulong <gglifulong@163.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump `pnpm` to v9.11, release note: https://github.com/pnpm/pnpm/releases/tag/v9.11.0
- updated pnpm lockfile by running `pnpm install --fix-lockfile`
- bump `nodejs` version from v18 to v20 (LTS version, released on 18 Apr 2023)
- update related github actions, aligning the pnpm and nodejs versions
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6712 from bowenliang123/pnpm9.
Closes#6712
ca6e425f4 [liangbowen] nit
a9ea92216 [liangbowen] bump pnpm to v9
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6666
## Describe Your Solution 🔧
Bump ranger version to 2.5.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.5.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6692 from Madhukar525722/ranger_upgrade.
Closes#6666
88e1e12c5 [madlnu] [KYUUBI #6666] Upgrade spark ranger plugin to 2.5.0
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Followup for #6681
We shall log the kill batch request before or inside `closeSession`, as it will close the operation log.
## Describe Your Solution 🔧
1. log the kill batch request before `closeSession`.
2. log the kill batch response in BatchJobSubmission::close.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
<img width="1571" alt="image" src="https://github.com/user-attachments/assets/dbb4049a-6cf3-4ae5-a2a7-4decba67cd76">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6697 from turboFei/kill_batch_log.
Closes#6681
13dbf4a75 [Wang, Fei] log
617cf54a6 [Wang, Fei] log
b3483c7e4 [Wang, Fei] log
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The server module does not consume the `spark-sql-engine` classes, but it supposes that
`spark-sql-engine`'s jar existed on the target folder, so here we declare it as a dependency
to make sure that Maven always processes `spark-sql-engine` module before the server module.
IntelliJ IDEA 2024.1 fixed the IDEA-93855, thus the relocated classes inside the
`spark-sql-engine`'s shaded jar are visible in the server module in IDEA, for example,
`org.apache.kyuubi.shaded.spark.connect.proto.ExecutePlanRequest`, which silently breaks
the IDEA code analysis and jumping capabilities.
## Describe Your Solution 🔧
Changing the dependency type from `jar`(default value) to `pom` seems to be a workaround.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
<img width="593" alt="image" src="https://github.com/user-attachments/assets/cc53709a-5f9d-4452-a24f-0c84e2342191">
#### Behavior Without This Pull Request ⚰️
<img width="1511" alt="image" src="https://github.com/user-attachments/assets/1c547c6d-a603-4c9d-92b4-8e2059b35fac">
#### Behavior With This Pull Request 🎉
<img width="1509" alt="image" src="https://github.com/user-attachments/assets/55f9e6e5-b9ea-4959-9142-ab7db21ab9b1">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6708 from pan3793/IDEA-93855.
Closes#6708
a73fd4235 [Cheng Pan] fix
e4a8d36e7 [Cheng Pan] nit
8c58d2ddc [Cheng Pan] IDEA-93855
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 4.0.0-preview2 RC1 passed the vote
https://lists.apache.org/thread/4ctj2mlgs4q2yb4hdw2jy4z34p5yw2b1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6699 from pan3793/spark-4.0.0-preview2.
Closes#6699
2db1f645d [Cheng Pan] 4.0.0-preview2
42055bb1e [Cheng Pan] fix
d29c0ef83 [Cheng Pan] disable delta test
98d323b95 [Cheng Pan] fix
2e782c00b [Cheng Pan] log4j-slf4j2-impl
fde4bb6ba [Cheng Pan] spark-4.0.0-preview2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- use `newDaemonSingleThreadScheduledExecutor` avoid `timer` thread leak
- reduce same status info out
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6696 from lsm1/branch-fix-trino-printer.
Closes#6696
01f917cb7 [senmiaoliu] fix style
0d20fd1f9 [senmiaoliu] fix trino info printer thread leak
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
# 🔍 Description
## Issue References 🔗
`paimon-spark-3.5` is a bundled jar but still pulls transitive deps.
## Describe Your Solution 🔧
Explicitly excluding transitive deps of paimon.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA and manually checked.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6698 from pan3793/exclude-paimon.
Closes#6698
6048b1156 [Cheng Pan] Exclude transitive deps from paimon
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump superlinter Github Action to v7
- skip linting `.env` files as the rules are too strict in unorder keys
- remove deprecated `ERROR_ON_MISSING_EXEC_BIT` option
- no major change
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
- pass the style CI workflow
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6682 from bowenliang123/superlinter-7.
Closes#6682
0ff321127 [Bowen Liang] skip linting env files
cab7aa810 [liangbowen] Remove deprecated ERROR_ON_MISSING_EXEC_BIT
67515df5b [liangbowen] bump superlinter to v7
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As title, format the batch command output.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6680 from turboFei/batch_error.
Closes#6680
f9de99b92 [Wang, Fei] ut
0562d2d23 [Wang, Fei] unused
d44947894 [Wang, Fei] pretty
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
To close#6686
The pod already in failed state, and the driver container is in waiting state.
We shall mark the application terminated and ignore the container state.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6690 from turboFei/pod_state.
Closes#6686
0d4c8a255 [Wang, Fei] comments
d60b901c1 [Wang, Fei] check pod terminated
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6668
## Describe Your Solution 🔧
1. when failed to kill the batch, check the current application info
2. if the application state is UNKNOWN(less than submit timeout) or NOT_FOUND, mark the batch state to CANCELED
3. If the k8s pod added after the batch marked as CANCELED, delete the pod
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6670 from turboFei/session_close_operation.
Closes#6668
068eaf216 [Wang, Fei] def
248c3e383 [Wang, Fei] check for onUpdate
695bb805d [Wang, Fei] clean up on add
9304f4605 [Wang, Fei] method
e2a15f8bc [Wang, Fei] batch
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- Bump `scala-collection-compat` to 2.12.0
- `scala-collection-compat` is used by the authz plugin testing Iceberg
- Currently Iceberg 1.6.1 uses `scala-collection-compat` 2.12.0,
according to: https://github.com/apache/iceberg/blob/apache-iceberg-1.6.1/gradle/libs.versions.toml#L78
- release note: https://github.com/scala/scala-collection-compat/releases/tag/v2.12.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
- pass the Authz tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6677 from pionCham/bump-scala-collection-compat-version.
Closes#6677
fd2738dce [chengpeiming] bump scala collection compat to 2.12.0
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
- Scala 2.12.19 release https://github.com/scala/scala/releases/tag/v2.12.19
## Describe Your Solution 🔧
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6676 from pionCham/bump-scala-version.
Closes#6676
7f5e17e98 [chengpeiming] change dependencyList
8736f2c57 [chengpeiming] bump scala version to 2.12.19
36f9984f5 [chengpeiming] bump scala version
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As title, log the delete batch request in operation log.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6681 from turboFei/audit_kill.
Closes#6681
8550868a6 [Wang, Fei] withOperationLog
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
As title, track the clientIp for internal kyuubi RESTful requests.
Likes
- getBatch
- getBatchLocalLog
- deleteBatch
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6669 from turboFei/additional_headers.
Closes#6669
87f144ec5 [Wang, Fei] headers
8dd7aca59 [Wang, Fei] track the client ip
afc78f2c2 [Wang, Fei] proxy ip
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes typos in the logging.md documentation file.
## Describe Your Solution 🔧
Fixed typos while reading through the page.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Not needed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6673 from MLNW/patch-1.
Closes#6673
02ea73113 [Bowen Liang] Update docs/monitor/logging.md
659580ed4 [Lucas Resch] Fix typos in logging.md
Lead-authored-by: Lucas Resch <lucas.resch@gmx.de>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- Apache Commons Lang2 is no longer actively maintained and not used by Kyuubi modules
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6672 from bowenliang123/remove-commonlang2.
Closes#6672
34cda170a [liangbowen] remove common lang2
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix the typo in ENGINE SHARE LEVEL docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6671 from pionCham/fix-typos.
Closes#6671
7dfbd4036 [chengpeiming] Fixed typo in ENGINE SHARE LEVEL docs
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6660
## Describe Your Solution 🔧
### Pull Request Description
This pull request improves the drag-and-resize functionality between the SQL Editor and the Result/Log panel. The updates ensure smoother resizing and better management of UI constraints, including the automatic adjustment of the Result/Log panel and the option to minimize/maximize it.
> drag-to-resize
> minimize
> maximize
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6665 from wangzhigang1999/master.
Closes#6660
6d659e05b [Zhigang] Remove unnecessary comment
df67c4c96 [Zhigang] Improve the display of sql editor and query result
Authored-by: Zhigang <wangzhigang1999@live.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- adding `maximumSize` to support size based eviction for server-side temp files cleanup in `TempFileService`
- size-based eviction is disabled by default , with `maximumSize` set to optional by default
- time-based eviction time is now extended from 14 days to 30 days by default
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6645 from bowenliang123/temp-file-size-evict.
Closes#6645
e1f166b6a [liangbowen] docs
0b2d5aa6e [liangbowen] increase default SERVER_TEMP_FILE_EXPIRE_TIME to 30 days
ee87da56a [liangbowen] make config optional
0607efcd7 [Bowen Liang] import
9cc777660 [liangbowen] update
f9e4de00e [Bowen Liang] docs
55bf238d3 [liangbowen] size
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump Maven from 3.8.x to 3.9.x
- Maven 3.9 release notes:
- 3.9.0: https://maven.apache.org/docs/3.9.0/release-notes.html
- 3.9.9: https://maven.apache.org/docs/3.9.9/release-notes.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6664 from bowenliang123/maven39.
Closes#6664
074b7ba00 [Bowen Liang] bump maven to 3.9
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6661
## Describe Your Solution 🔧
TColumnGenerator.getColumnToList should not access to non-IndexedSeq with Seq.apply(i), which will cause performance reduce, convert it to foreach loop will be good. see https://issues.apache.org/jira/browse/SPARK-47085 for more details.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6662 from hh-cn/KYUUBI-6661.
Closes#6661
4597e88c1 [hang.huang] improve column-based TRowSet generation
Authored-by: hang.huang <hang.huang@advancegroup.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.6.1 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1
## Describe Your Solution 🔧
In the project POM file, I have updated the Apache Iceberg version from 1.6.0 to 1.6.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6659 from pionCham/upgrate-iceberg-version.
Closes#6659
923019440 [chengpeiming] Update the docs for Spark connector
433981e66 [chengpeiming] Supplement other configurations
1617e36fe [chengpeiming] Upgrate iceberg.version in pom.xml
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- fix the typo in REST API docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6658 from pionCham/fix-typos.
Closes#6658
e8937f1e0 [chengpeiming] Fixed typos in rest_api.md
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As title, save the pod name conf into batch metadata, it is very meaningful.
## Describe Your Solution 🔧
For spark on k8s batch app, persist the spark pod name configs into batch metadata.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6653 from turboFei/pod_name_conf.
Closes#6653
2bfa0aa9e [Wang, Fei] save spark pod name conf into metadata
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5402
## Describe Your Solution 🔧
When facing out-of-control memory management in Spark engine, we typically use JVMkill as a remedy by killing the process and generating a heap dump for post-analysis. However, even with jvmkill protection, we may still encounter issues caused by JVM running out of memory, such as repeated execution of Full GC without performing any useful work during the pause time. Since the JVM does not exhaust 100% of resources, JVMkill will not be triggered.
So introducing JVMQuake provides more granular monitoring of GC behavior, enabling early detection of memory management issues and facilitating fast failure.
You can use the following configuration to enable jvmQuake plugins：
```
spark.plugins=org.apache.spark.kyuubi.jvm.quake.KyuubiJVMQuakePlugin
```
| configuration | default | comment |
| ---- | ---- | ---- |
| spark.driver.jvmQuake.enabled | false | when true, enable driver jvmQuake |
| spark.executor.jvmQuake.enabled | false | when true, enable executor jvmQuake |
| spark.driver.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.executor.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.jvmQuake.dumpThreshold | 100 | The number of seconds to dump memory |
| spark.jvmQuake.killThreshold | 200 | The number of seconds to kill process |
| spark.jvmQuake.exitCode | 502 | The exit code of kill process |
| spark.jvmQuake.heapDumpPath | /tmp/kyuubi_jvm_quake/apps | The path of heap dump |
| spark.jvmQuake.checkInterval | 3 | The number of seconds to check jvmQuake |
| spark.jvmQuake.runTimeWeight | 1.0 | The weight of rum time |
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6572 from yoock/features/kyuubi-jvm-quake.
Closes#5402
84361ce8f [王龙] add jvm quake
Authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Before we only support to list the batches in `ASC` ORDER. It is not user friendly.
## Describe Your Solution 🔧
Support the list the batches in `DESC` order.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6652 from turboFei/latest_batch.
Closes#6652
b3d80f5bd [Wang, Fei] ut
dce0b226d [Wang, Fei] doc
d815ec39c [Wang, Fei] ut
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This PR is to return the launch engine operation handle for thrift http frontend service when opening session.
So that, the kyuubi connection show the launch engine log.
Refer https://github.com/apache/kyuubi/pull/1377
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Just return the launch engine op handle for thrift http connection.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6648 from turboFei/engine_launch_log.
Closes#6648
6a3540775 [Wang, Fei] nit
0bd08d208 [Wang, Fei] Support to show launch engine log for thrift http mode
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
Follow up for #6619, I found that, there are many duplicate auth audit logs for thrift http.
We shall only log when make authentication with auth handler, and skip it for cookie auth.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6632 from turboFei/need_to_auth.
Closes#6619
3a25a3dd3 [Wang, Fei] do auth
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
We meet issue when restarting the kyuubi server:
```
2024-08-28 09:48:41.507 WARN org.apache.kyuubi.server.api.RestExceptionMapper: Error occurs on accessing REST API.
java.lang.NullPointerException: Cannot invoke "org.apache.kyuubi.server.KyuubiServer.getConf()" because the return value of "org.apache.kyuubi.server.KyuubiServer$.kyuubiServer()" is null
at org.apache.kyuubi.server.api.v1.BatchesResource.batchV2Enabled(BatchesResource.scala:72) ~[kyuubi-server_2.12-1.10.0.0.1.0.jar:1.10.0.0.1.0]
at org.apache.kyuubi.server.api.v1.BatchesResource.$anonfun$batchInfo$3(BatchesResource.scala:345) ~[kyuubi-server_2.12-1.10.0.0.1.0.jar:1.10.0.0.1.0]
at scala.Option.map(Option.scala:230) ~[scala-library-2.12.18.jar:?]
```
The root cause is that, the `KyuubiServer.kyuubiServer` is null until the kyuubi server start finished.
db57e9365d/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiServer.scala (L231-L234)
But the RESTful api initialization finished before than that, so, NPE will be thrown between REST front started to all the kyuubi service started.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
In this PR,
1. move the KyuubiBatchService into KyuubiRestFrontendService
2. move the TempFileService (introduced in #6587) into KyuubiSessionManager
<img width="872" alt="image" src="https://github.com/user-attachments/assets/a4c5fe38-1d34-4fee-933f-72511bc06f27">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6646 from turboFei/npe.
Closes#6646
42c1b7512 [Wang, Fei] common
42b305557 [Wang, Fei] fix
e0f58d234 [Wang, Fei] fix
84492f21a [Wang, Fei] move code
530eb21c1 [Wang, Fei] fix
bc206ab25 [Wang, Fei] unused
dd8908861 [Wang, Fei] prevent NPE
4e6d39ecf [Wang, Fei] npe
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump Paimon from 0.7.x-incubating to 0.8.2
- Paimon is currently used for Authz plugin testing only for Paimon Spark catalog
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6644 from bowenliang123/paimon08.
Closes#6644
6944099e4 [liangbowen] update paimon to 0.8
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
-
## Describe Your Solution 🔧
Fix the memory leak on server caused by `Files.deleteOnExit`.
For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook` by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue.
This PR fix this issue by:
1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list
2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup.
3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6587 from bowenliang123/file-expiration.
Closes#6587
e23b72e08 [liangbowen] change to P14D
acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime
6c7ddd527 [liangbowen] import
ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet
fbf73ccb4 [liangbowen] update
34d3fc71c [liangbowen] add guava to common module's dep
49c10e5ef [Bowen Liang] file expiration
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
Fix#6594.
This PR ports HIVE-26633(https://github.com/apache/hive/pull/3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6631 from pan3793/thrift-max-size.
Closes#6594
e4841c88e [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Backport https://github.com/apache/hive/pull/4819
Note: it's only the JDBC driver side change, to make it work, we also need to modify the engines.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
We can not test this feature so far because we don't have engine support that. Pass GHA to ensure it breaks nothing.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6639 from pan3793/HIVE-27815.
Closes#6639
4b3bcd65f [Cheng Pan] fix
c16dc28e8 [Cheng Pan] Port HIVE-27815: Support update numModifiedRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5314
## Describe Your Solution 🔧
To support Java 21, some dependency packages need to be upgraded.Here is the detailed explanation.
1. To build with Java 21, Scala binary version needs to be upgraded to 2.13.11 or above.
2. The versions of `spotless` and `googlejavaformat` need to be upgraded to higher versions. However, this will lose support for Java 8. In order to continue supporting Java 8, code style checks will be ignored in the Java 21 profile.
3. Since the Flink engine does not support Scala 2.13, the Scala version of the Flink module is fixed at 2.12.
Previous discussions were at https://github.com/apache/kyuubi/pull/6306.
---
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
**Be nice. Be informative.**
Closes#6597 from dupen01/issue-java21.
Closes#5314
0c5a9e455 [Cheng Pan] simplify
54cbadf79 [dupeng] java21+scala2.13.11+flink_2.12+CI(spark4,21,2,13)
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
SPARK-46257 (Spark 4.0.0) moves to Derby 10.16, `org.apache.derby.jdbc.AutoloadedDriver` has been moved to `org.apache.derby.iapi.jdbc.AutoloadedDriver`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually tested with Spark 4.0.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6640 from pan3793/authz-derby.
Closes#6640
46edb32be [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/util/AuthZUtils.scala
7eee47f0d [Cheng Pan] Adapt Derby 10.16 new JDBC driver package name
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
To support Java 21, the Scala version needs to be upgraded to 2.13.11 or above.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6603 from dupen01/scala213.
Closes#6603
541f86054 [Cheng Pan] exclude SparkLocalClusterTest
13c11bf52 [Perl Du] add ignoreClasses comments
d5b67971b [Cheng Pan] Update pom.xml
582de9b90 [dupeng] ignore org.jline.terminal.impl.ffm.*
364eea00c [dupeng] upgrade maven.plugin.silencer.version from 1.7.13 to 1.7.17
99a1cc40a [dupeng] upgrade scala version to 2.13.14
54b0f4d3f [dupeng] upgrade scala version to 2.13.11
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Perl Du <34719039+dupen01@users.noreply.github.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I faced the following error when trying to run authz with Spark 4.0
```
Cause: java.lang.NoClassDefFoundError: javax/ws/rs/core/Cookie
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.ranger.plugin.policyengine.RangerPluginContext.createAdminClient(RangerPluginContext.java:96)
at org.apache.ranger.plugin.util.PolicyRefresher.<init>(PolicyRefresher.java:90)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:251)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
```
The `javax.ws.rs:jsr311-api` is the transitive dep of `jersey-client`, we should shade and relocate it correctly.
Why does it work with Spark 3? Spark 3 provides `jakarta.ws.rs:jakarta.ws.rs-api:2.1.6` which provides `java.ws.rs.*` classes, but Spark 4 upgrades to `jakarta.ws.rs:jakarta.ws.rs-api:3.0.0` which changed package name to`jakarta.ws.rs.*`.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA and manually tested with Spark 4
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6638 from pan3793/jsr311.
Closes#6638
5699200cf [Cheng Pan] Shade jsr311-api in Authz
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The current Hive 2.3.10 integration tests are broken, the root cause is CALCITE-1224
```
2024-08-20T12:20:55,405 ERROR [HiveTBinaryFrontendHandler-Pool: Thread-49] org.apache.kyuubi.util.KyuubiUncaughtExceptionHandler - Uncaught exception in thread HiveTBinaryFrontendHandler-Pool: Thread-49
java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonGenerator.writeStartArray(Ljava/lang/Object;I)V
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:78) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4485) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3699) ~[jackson-databind-2.12.0.jar:2.12.0]
at org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:518) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:143) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.kyuubi.engine.hive.operation.HiveOperation.runInternal(HiveOperation.scala:62) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:175) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:103) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.engine.hive.session.HiveSessionImpl.runOperation(HiveSessionImpl.scala:59) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:133) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:84) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:130) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_422]
```
## Describe Your Solution 🔧
Unpack `avatica-1.8.0.jar` and delete Jackson classes to workaround CALCITE-1224
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This PR should recover CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6635 from pan3793/hive-2310-fix.
Closes#6635
334bc5c39 [Cheng Pan] nit
22ae945fb [Cheng Pan] fix
dec03196d [Cheng Pan] Fix Hive 2.3.10 integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#3897
## Describe Your Solution 🔧
enrich the description for GROUP Share Level.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6628 from futureltl/master.
Closes#6628
ba18bfce4 [futureltl] Improve docs for GROUP Share Level
3b19521e1 [futureltl] Improve docs for GROUP Share Level
da7d9b61e [Cheng Pan] Update docs/deployment/engine_share_level.md
674066a08 [Cheng Pan] Update docs/deployment/engine_share_level.md
c3a373370 [Cheng Pan] Update docs/deployment/engine_share_level.md
7389cedd2 [futureltl] Improve docs for GROUP Share Level
Lead-authored-by: futureltl <futureltl@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6583
## Background and Goals
Currently, kyuubi cannot perform operation level interrupts when executing Python code. When it is necessary to cancel an operation that has been running for a long time, the entire session needs to be interrupted, and the execution context will be lost, which is very unfriendly to users. Therefore, it is necessary to support operation level interrupts so that the execution context is not lost when the user terminal operates.
## Describe Your Solution 🔧
Refer to the implementation of Jupyter Notebook and let the Python process listen to Signel SIGINT semaphore, when receiving a signel When SIGINT, interrupt the current executing code and capture KeyboardInterrupt to treat it as cancelled
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6612 from yoock/features/support-operation-cancel.
Closes#6583
bf6334d8c [Wang, Fei] log error to do not break the cleanup process
ae7ad3f3c [Wang, Fei] comments
509627e65 [王龙] PySpark support operation cancel
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6619
## Describe Your Solution 🔧
This is a subtask of #6590
This PR makes the following changes:
1. Instead of using SessionManager, the ThriftHttpService will use AuthorizationFilter to handle http client parameters. The purpose is to pass correct client IP address for bearer token authentication.
2. Partial backport Partial backport: https://github.com/apache/hive/pull/3006
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6624 from George314159/thrift.
Closes#6619
9761d6ddf [Wang, Fei] Partial backport HIVE-25575: Add support for JWT authentication in HTTP mode
30ede05d8 [Wang, Fei] Revert "suffix"
39c945802 [Wang, Fei] suffix
b99420633 [Wang, Fei] Revert "Fix"
ea6236a8d [George314159] Fix
a75025be4 [George314159] Refine UT
945433583 [George314159] Refine
8f4eeff03 [George314159] Update UT
6192d4713 [Wang, Fei] revert unused
c7f037c66 [Wang, Fei] Support bearer token in kyuubi jdbc
f7725f761 [George314159] Refine
f865abf01 [George314159] Refine
9d4edd50d [George314159] Add http header config for thrift bearer token auth and UT
6d0f97355 [Wang, Fei] refine
a7e785bcd [Wang, Fei] save
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: George314159 <hua16732@gmail.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
Followup for #6335
Just use the session handle identifier to build the batchResourceUploadFolderPath, and do not rely on the BatchJobSubmission operation initialization.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6629 from turboFei/batch_id.
Closes#6335
28df03d4d [liangbowen] unused import
2eee2a64f [Wang, Fei] Move to KyuubiApplicationManager
f23ecae1f [liangbowen] update batchResourceUploadFolderPath method for removing batch prefix and checking sessionId
b30a7b996 [Wang, Fei] using session id
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
As mentioned in https://github.com/apache/kyuubi/pull/6626, the operation never idle because of periodical get operation status, but in my opinion, the operations should be closed after session closed, but it did not.
Currently, for session operation handles set, it does not use concurrent collections, I wonder there is concurrent update issue.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Not needed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6627 from turboFei/operation_set.
Closes#6627
2b80c25e1 [Wang, Fei] revert
57e4dbfa2 [Wang, Fei] using concurrent
bbd9ecf6a [Wang, Fei] use concurrent hash map
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
