# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.7.0 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.7.0
## Describe Your Solution 🔧
- Bump Apache Iceberg to 1.7.0
- As Apache Iceberg 1.7.0 drops support for Java 8 and building with Java 11, keep it in 1.6.x for Java 8
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6804 from pionCham/bump-iceberg-version.
Closes#6804
0896ac768 [Bowen Liang] keep iceberg 1.6.1 in playground
eba16ae6c [chengpeiming] Specify the iceberg version in java-8 profile
3b160ddd6 [chengpeiming] Bump iceberg version
Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Check the uploaded resource files when creating batch via REST API
- add config `kyuubi.batch.resource.file.max.size` for resource file's max size in bytes
- add config `kyuubi.batch.extra.resource.file.max.size` for each extra resource file's max size in bytes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6756 from bowenliang123/resource-maxsize.
Closes#6756
5c409c425 [Bowen Liang] nit
4b16bcfc4 [Bowen Liang] nit
743920d25 [Bowen Liang] check resource file size max size
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- build python dependencies for docs building to latest versions
- no display or behaviour changes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6752 from bowenliang123/doc-req.
Closes#6752
ffd8782bd [Bowen Liang] update
c328c7584 [Bowen Liang] bump doc build requirements
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- as titled
- add Spark 3.4 and 3.5 to the supported Spark list
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6728 from bowenliang123/doc-authz-build-am.
Closes#6728
f8254bc5c [Bowen Liang] doc
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- add authentication example in REST API docs
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6734 from bowenliang123/rest-doc-auth.
Closes#6734
f9ac9446d [Cheng Pan] Update docs/client/rest/rest_api.md
528e55e79 [Bowen Liang] update doc
371af8806 [Bowen Liang] update doc
e64a08245 [Bowen Liang] update doc
341e7e010 [Bowen Liang] update doc
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6704
## Describe Your Solution 🔧
if periodic gc is set to 0, there is no need to perform an explicit gc.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6725 from taylor12805/master.
Closes#6704
a52ddda62 [Bowen Liang] update doc
b84a32f35 [Bowen Liang] make periodic gc thead pool lazy
2d4bd7c05 [Bowen Liang] update doc in spark style
3e04604b0 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
bf20b134b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
c2b7c3078 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
6182075fc [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
52b1c078b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
ccf19cf24 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
affd67c88 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
d4ee164d1 [taylor.fan] disable periodic gc if set interval to 0
Lead-authored-by: taylor.fan <taylor.fan@vipshop.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix a couple of typos
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6719 from pionCham/fix-typos.
Closes#6719
71409a875 [chengpeiming] fix violations in jvm-quake.md
de8f0d7b8 [chengpeiming] fix some typos
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6666
## Describe Your Solution 🔧
Bump ranger version to 2.5.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.5.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6692 from Madhukar525722/ranger_upgrade.
Closes#6666
88e1e12c5 [madlnu] [KYUUBI #6666] Upgrade spark ranger plugin to 2.5.0
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
To close#6686
The pod already in failed state, and the driver container is in waiting state.
We shall mark the application terminated and ignore the container state.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6690 from turboFei/pod_state.
Closes#6686
0d4c8a255 [Wang, Fei] comments
d60b901c1 [Wang, Fei] check pod terminated
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes typos in the logging.md documentation file.
## Describe Your Solution 🔧
Fixed typos while reading through the page.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Not needed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6673 from MLNW/patch-1.
Closes#6673
02ea73113 [Bowen Liang] Update docs/monitor/logging.md
659580ed4 [Lucas Resch] Fix typos in logging.md
Lead-authored-by: Lucas Resch <lucas.resch@gmx.de>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix the typo in ENGINE SHARE LEVEL docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6671 from pionCham/fix-typos.
Closes#6671
7dfbd4036 [chengpeiming] Fixed typo in ENGINE SHARE LEVEL docs
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- adding `maximumSize` to support size based eviction for server-side temp files cleanup in `TempFileService`
- size-based eviction is disabled by default , with `maximumSize` set to optional by default
- time-based eviction time is now extended from 14 days to 30 days by default
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6645 from bowenliang123/temp-file-size-evict.
Closes#6645
e1f166b6a [liangbowen] docs
0b2d5aa6e [liangbowen] increase default SERVER_TEMP_FILE_EXPIRE_TIME to 30 days
ee87da56a [liangbowen] make config optional
0607efcd7 [Bowen Liang] import
9cc777660 [liangbowen] update
f9e4de00e [Bowen Liang] docs
55bf238d3 [liangbowen] size
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.6.1 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1
## Describe Your Solution 🔧
In the project POM file, I have updated the Apache Iceberg version from 1.6.0 to 1.6.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6659 from pionCham/upgrate-iceberg-version.
Closes#6659
923019440 [chengpeiming] Update the docs for Spark connector
433981e66 [chengpeiming] Supplement other configurations
1617e36fe [chengpeiming] Upgrate iceberg.version in pom.xml
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- fix the typo in REST API docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6658 from pionCham/fix-typos.
Closes#6658
e8937f1e0 [chengpeiming] Fixed typos in rest_api.md
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5402
## Describe Your Solution 🔧
When facing out-of-control memory management in Spark engine, we typically use JVMkill as a remedy by killing the process and generating a heap dump for post-analysis. However, even with jvmkill protection, we may still encounter issues caused by JVM running out of memory, such as repeated execution of Full GC without performing any useful work during the pause time. Since the JVM does not exhaust 100% of resources, JVMkill will not be triggered.
So introducing JVMQuake provides more granular monitoring of GC behavior, enabling early detection of memory management issues and facilitating fast failure.
You can use the following configuration to enable jvmQuake plugins：
```
spark.plugins=org.apache.spark.kyuubi.jvm.quake.KyuubiJVMQuakePlugin
```
| configuration | default | comment |
| ---- | ---- | ---- |
| spark.driver.jvmQuake.enabled | false | when true, enable driver jvmQuake |
| spark.executor.jvmQuake.enabled | false | when true, enable executor jvmQuake |
| spark.driver.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.executor.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.jvmQuake.dumpThreshold | 100 | The number of seconds to dump memory |
| spark.jvmQuake.killThreshold | 200 | The number of seconds to kill process |
| spark.jvmQuake.exitCode | 502 | The exit code of kill process |
| spark.jvmQuake.heapDumpPath | /tmp/kyuubi_jvm_quake/apps | The path of heap dump |
| spark.jvmQuake.checkInterval | 3 | The number of seconds to check jvmQuake |
| spark.jvmQuake.runTimeWeight | 1.0 | The weight of rum time |
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6572 from yoock/features/kyuubi-jvm-quake.
Closes#5402
84361ce8f [王龙] add jvm quake
Authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Before we only support to list the batches in `ASC` ORDER. It is not user friendly.
## Describe Your Solution 🔧
Support the list the batches in `DESC` order.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6652 from turboFei/latest_batch.
Closes#6652
b3d80f5bd [Wang, Fei] ut
dce0b226d [Wang, Fei] doc
d815ec39c [Wang, Fei] ut
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
-
## Describe Your Solution 🔧
Fix the memory leak on server caused by `Files.deleteOnExit`.
For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook` by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue.
This PR fix this issue by:
1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list
2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup.
3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6587 from bowenliang123/file-expiration.
Closes#6587
e23b72e08 [liangbowen] change to P14D
acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime
6c7ddd527 [liangbowen] import
ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet
fbf73ccb4 [liangbowen] update
34d3fc71c [liangbowen] add guava to common module's dep
49c10e5ef [Bowen Liang] file expiration
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
Fix#6594.
This PR ports HIVE-26633(https://github.com/apache/hive/pull/3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6631 from pan3793/thrift-max-size.
Closes#6594
e4841c88e [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#3897
## Describe Your Solution 🔧
enrich the description for GROUP Share Level.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6628 from futureltl/master.
Closes#6628
ba18bfce4 [futureltl] Improve docs for GROUP Share Level
3b19521e1 [futureltl] Improve docs for GROUP Share Level
da7d9b61e [Cheng Pan] Update docs/deployment/engine_share_level.md
674066a08 [Cheng Pan] Update docs/deployment/engine_share_level.md
c3a373370 [Cheng Pan] Update docs/deployment/engine_share_level.md
7389cedd2 [futureltl] Improve docs for GROUP Share Level
Lead-authored-by: futureltl <futureltl@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6618
## Describe Your Solution 🔧
It is a subtask of #6590
This PR is to support http bearer token authentication for REST protocol. In addition to BasicAuthenticationHandler, BearerAuthenticationHandler will be added to handle http bear token authentication. They will both support CUSTOM AuthType. In order to distinguish them, two new configurations are added: kyuubi.authentication.custom.basic.class and kyuubi.authentication.custom.bearer.class. For http bear token custom authentication, users could implement the new 'org.apache.kyuubi.service.authentication.TokenAuthenticationProvider', and specify it in the configuration.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6608 from George314159/authentication.
Closes#6618
d07a30f83 [Wang, Fei] fix UT
6499c9986 [George314159] Update Test Case
da519a9c6 [George314159] Update based on comments
f47160148 [Wang, Fei] Refine UT
544422399 [George314159] Add test suite for custom authentication
f2bbfbf7e [Wang, Fei] comments & refine
a733c0e8f [George314159] Remove unused val
6f669d46c [George314159] Fix
650b88d4e [George314159] Update based on comments
5bc2bac58 [George314159] Update based on comments
1893889db [George314159] Update based on Comments
ddee882e9 [George314159] Fix Style
379a563fa [George314159] Support http bearer token authentication
Lead-authored-by: George314159 <hua16732@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6615
## Describe Your Solution 🔧
Add a config item that controls whether Jetty should send its version in response.
Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Compiled and tested manually.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6616 from paul8263/KYUUBI-6615.
Closes#6615
c1567fdfa [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable
Authored-by: zhang_yao <xzhangyao@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 3.5.2 was released recently.
Release Notes is available at https://spark.apache.org/releases/spark-release-3-5-2.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6609 from pan3793/spark-3.5.2.
Closes#6609
587cf1dd3 [Cheng Pan] Bump Spark 3.5.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
The current FileSessionConfAdvisor only supports configuring a single file through kyuubi.session.conf.profiler. However, users may need to use multiple configuration files, and the existing setup does not support this. To address this need, we have updated the FileSessionConfAdvisor to support multiple configuration files.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6585 from lsm1/branch-file-advidor-support-chain.
Closes#6585
15b6524e0 [senmiaoliu] fix style
fcedc4fe3 [senmiaoliu] FileSessionConfAdvisor support multi files
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
Fixing some typos
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6596 from hyperbolic2346/hyperbolic2346-patch-1.
Closes#6596
ebbddf0a6 [Mike Wilson] Update architecture.md
Lead-authored-by: Mike Wilson <knobby@burntsheep.com>
Co-authored-by: Mike Wilson <hyperbolic2346@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Flink 1.20 is out, [Release Note](https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
CI is updated to cover Flink 1.20
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6557 from pan3793/flink-1.20.
Closes#6557
a414094b9 [Cheng Pan] remove rc
8ee4cf5cd [Cheng Pan] fix url
fbaf66071 [Cheng Pan] docs
ddbd10ffe [Cheng Pan] Support Flink 1.20
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
fix typos
## Describe Your Solution 🔧
anderror -> an error
deprected -> deprecated
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6566 from XorSum/fix/kyuubi-conf-typo.
Closes#6566
5c270ad0b [xorsum] fix typo in KyuubiConf
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Upgrade to latest Apache Iceberg 1.6.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass existing GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6558 from pan3793/iceberg-1.6.0.
Closes#6558
75e42938f [Cheng Pan] nit
ba508b6bb [Cheng Pan] nit
0180521aa [Cheng Pan] Bump Iceberg 1.6.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This pull request aims to remove building support for Spark 3.2, while still keeping the engine support for Spark 3.2.
Mailing list discussion: https://lists.apache.org/thread/l74n5zl1w7s0bmr5ovxmxq58yqy8hqzc
- Remove Maven profile `spark-3.2`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.5) still works well on Spark 3.2 runtime.
- Merge `kyuubi-extension-spark-common` into `kyuubi-extension-spark-3-3`
- Remove `log4j.properties` as Spark moves to Log4j2 since 3.3 (SPARK-37814)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6545 from pan3793/deprecate-spark-3.2.
Closes#6545
54c172528 [Cheng Pan] fix
f4602e805 [Cheng Pan] Deprecate and remove building support for Spark 3.2
2e083f89f [Cheng Pan] fix style
458a92c53 [Cheng Pan] nit
929e1df36 [Cheng Pan] Deprecate and remove building support for Spark 3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6524
## Describe Your Solution 🔧
Trino engine supports insecure configuration, just as trino client supports --insecure parameter
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6525 from jiaoqingbo/6524.
Closes#6524
b414b2e05 [jiaoqingbo] update settings.md
129d40742 [jiaoqingbo] [KYUUBI #6524] Trino engine supports insecure configuration
24f374b38 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
I'd like to introduce the feature that allows users to forcibly kill an engine through API.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6008 from zhaohehuhu/dev-0123.
Closes#6008
00c208a26 [Cheng Pan] fix
8721a2d2a [Cheng Pan] log
efc7587f7 [Cheng Pan] client
cd5129db3 [Cheng Pan] fix ut
5e1b6a161 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
72d7df357 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
6d5d08710 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
b013194d1 [zhaohehuhu] move the position of log
0cdeede7a [zhaohehuhu] restore ENGINE_SPARK_REGISTER_ATTRIBUTES
f826d0515 [zhaohehuhu] reformat
a13466e37 [zhaohehuhu] update doc and log string encoded
3a2f5970a [zhaohehuhu] refactor
ae24ea74d [zhaohehuhu] refactor UT
936a54e27 [Wang, Fei] register app mgr info
9bacc2c8b [hezhao2] fix UTs
11106d75b [Wang, Fei] comments
ba57c2c3f [hezhao2] refactor code to delete the node and then kill application
634ceb677 [hezhao2] reformat
ab31382ee [hezhao2] reformat
513bcdc57 [hezhao2] fix UT
506220654 [hezhao2] get refId by user, sharelevel and subdomain
3ad9577df [hezhao2] rename params to support multiple engines
632c56b88 [hezhao2] fix unused import
bd7bb45f0 [hezhao2] refactor
fb9b25176 [hezhao2] add default value for forceKill param
070aad06f [hezhao2] refactor
51827ecde [hezhao2] fix UT
f11e7657e [hezhao2] add an UT
8a65cf113 [hezhao2] refactor code
d6f82ff9a [hezhao2] refactor code
f3ab9c546 [hezhao2] new parameter added to decide whether to kill forcefully handle the result of killApplication
5faa5b54f [hezhao2] kill engine forcibly
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: zhaohehuhu <luoyedeyi459@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Canonicalize the words, and enrich the description for KSCH.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6512 from pan3793/kshc-doc.
Closes#6512
201c11341 [Cheng Pan] nit
1becc1ebb [Cheng Pan] nit
8d48c7c93 [Cheng Pan] fix
aea1e0386 [Cheng Pan] fix
5ba5094ab [Cheng Pan] fix
0c40de43d [Cheng Pan] fix
63dd21d11 [Cheng Pan] nit
1be266163 [Cheng Pan] Improve docs for KSHC
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
[Good First Issues](https://kyuubi.readthedocs.io/en/master/contributing/code/get_started.html) image links to the Github issues page while the image on [Contributing Documentation](https://kyuubi.readthedocs.io/en/master/contributing/doc/get_started.html) does not.
## Describe Your Solution 🔧
Added a link to https://github.com/apache/kyuubi/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Adocumentation to match the behaviour from good first issues
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6514 from artilexx/patch-1.
Closes#6514
3d54ce982 [artilexx] Add link to github issues on documentation issues image
Lead-authored-by: artilexx <arkhanlewis@gmail.com>
Co-authored-by: artilexx <97213042+artilexx@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6506
## Describe Your Solution 🔧
Set the alias `--conf` to `--hiveconf`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
```
# use --hiveconf or --hivevar to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--hiveconf kyuubi.operation.result.format=arrow \
--hiveconf kyuubi.operation.incremental.collect=true-f \
--hivevar app_name=xxx \
xxx.sql
```
#### Behavior With This Pull Request 🎉
```
# use --conf to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--conf kyuubi.operation.result.format=arrow \
--conf kyuubi.operation.incremental.collect=true-f \
--conf app_name=xxx \
xxx.sql
```
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6511 from BruceWong96/support-beeline-conf.
Closes#6506
7a81455d8 [Bruce Wong] fix code style.
8d1fec242 [Bruce Wong] delete some spaces.
d64505dd7 [Bruce Wong] [FEATURE] kyuubi-beeline supports --conf.
Authored-by: Bruce Wong <603334301@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Hive 2.3.0 (HIVE-7224) turns it on by default, while Kyuubi BeeLine is a fork from Hive BeeLine 3.1.3, we should update the help message to reflect the change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6507 from pan3793/beeline-incr.
Closes#6507
8b1ed655a [Cheng Pan] fix
7a872eeec [Cheng Pan] fix
88e751d3a [Cheng Pan] Correct Kyuubi BeeLine help message for incremental
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Update the outdated docs to mention new features:
1. Arrow serialization
2. Sava query results into files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Docs changes, review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6504 from pan3793/docs-large-result.
Closes#6504
77043bd4d [Cheng Pan] fix
e8ba1bac1 [Cheng Pan] grammar
198d00e33 [Cheng Pan] grammar
a37f631eb [Cheng Pan] Improve docs for Solution for Large Query Results
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
as title
This pull request closes #6255
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6339 from zhaohehuhu/dev-0426.
Closes#6339
38b36f260 [Cheng Pan] fix
b2e3fcf3d [Cheng Pan] rewrite
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes the issue related to incorrect casing in `kubectl` commands used for creating `serviceaccount`.
## Describe Your Solution 🔧
This PR corrects the casing for the Kubernetes `serviceaccount` creation and rolebinding command to ensure compatibility with `kubectl`. Previously, the commands used incorrect casing (`serviceAccount`), which could potentially lead to execution errors or compatibility issues with Kubernetes API expectations.
Reference: [Kubernetes Official Documentation for Creating a ServiceAccount](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_serviceaccount/)
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6440 from huangzhir/doc-fix.
Closes#6440
5c657034d [huangzhir] Fix casing in kubectl commands for serviceaccount creation
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
`master` and `branch-1.9` already added basic support for Spark 4.0.0-preivew1.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6432 from pan3793/docs-4.0.
Closes#6432
73177aab3 [Cheng Pan] Mention support for Spark 4.0 in docs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6394 from lllong33/patch-1.
Closes#6394
eb79375aa [lllong33] Update tpcds.rst, The order table removed
Lead-authored-by: lllong33 <704669594@qq.com>
Co-authored-by: lllong33 <29864358+lllong33@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
trino engine increment mode support fetch log
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
1. Trino engine supports outputting final status information, and we can also support outputting running information.
2. In increment mode, the operation status is set to FINISHED quickly, which prevents the client from fetching detailed operation logs. The client is unaware of the exact execution status of the operation. It should be possible to explicitly check if the ResultSet has more data (hasNext), and delay setting the status to output more operation information.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6387 from lsm1/branch-trino-print-info.
Closes#6387
b9b733a2f [senmiaoliu] trino engine output running info trino engine increment mode support fetch log
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
I found that, with saveToFile enabled with the default min size threshold, even I run a simple `set` command, It also save the result to file.
<img width="1718" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/5bcc0da1-201a-453a-8568-d1bfadd7adef">
I think we need to skip this kind of queries.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6377 from turboFei/check_is_DQL.
Closes#6377
da9c2a921 [Wang, Fei] ut
04e20db5f [Wang, Fei] conf
8f20ed84b [Wang, Fei] refine the check
f558dcca5 [Wang, Fei] ut
c81340333 [Wang, Fei] DQL
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6338
## Describe Your Solution 🔧
Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6340 from wForget/KYUUBI-6338.
Closes#6338
acd73a16b [wforget] fix
5e64ed3a5 [wforget] comments
48b8ec6cd [wforget] fix style and address comments
ca653492c [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
# 🔍 Description
## Issue References 🔗
Now for kubernetes application, there is no runtime AppURL return in the application report.
In fact, we can get the driver svc and get the spark-ui port, then build the Spark UI URL same with that of spark.
074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L67)074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L96-L102)
Service Selector example:
```
selector:
kyuubi-unique-tag: bf5fa281-0b2f-4096-aa0a-13a463c147a6
spark-app-selector: spark-bf074093a7994954a89101eb2831bd1e
```
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6318 from turboFei/k8s_spark_ui.
Closes#6321
f77999ef0 [Wang, Fei] with label selector does not work
4e8d5767b [Wang, Fei] typo
3f1eb545c [Wang, Fei] comments
6f0d01d94 [Wang, Fei] ut
9edfde034 [Wang, Fei] ut
65d5e8175 [Wang, Fei] using pattern
9b21d30df [Wang, Fei] Support to get kubernetes app url
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6299
## Describe Your Solution 🔧
when disabling web ui, return 404 page
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6311 from lsm1/branch-support-disable-webui.
Closes#6299
aa96c2737 [senmiaoliu] remove enable.html
998504710 [senmiaoliu] fix style
a2622cbbc [senmiaoliu] disable web ui
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6253
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
<img width="1251" alt="image" src="https://github.com/apache/kyuubi/assets/18713676/b654a300-8c79-4461-9fba-4ad1c913accc">
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6275 from lsm1/branch-jdbc-engine-on-yarn.
Closes#6253
5ed4af041 [senmiaoliu] fix style
86e032688 [senmiaoliu] fix style
b3e114445 [senmiaoliu] fix style
bf33de300 [senmiaoliu] fix style
c38404918 [senmiaoliu] jdbc engine on yarn
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6250
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass CI
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6273 from liuxiaocs7/issue-6250.
Closes#6250
c6ba1e88a [liuxiao] remove unused import
db887ef9b [liuxiao] inline method
769da013b [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
21dbd37a7 [liuxiao] remove unused import
e869d571e [liuxiao] update for miss
7d755a879 [liuxiao] Drop support for Spark 3.1
Lead-authored-by: liuxiao <liuxiao2103@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This pull request
- improves comments for SPARK-33832
- removes unused `spark.sql.analyzer.classification.enabled` (I didn't update the migration rules because this configuration seems never to work properly)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6260 from pan3793/nit.
Closes#6260
d762d30e9 [Cheng Pan] update comment
4ebaa04ea [Cheng Pan] nit
b303f05bb [Cheng Pan] remove spark.sql.analyzer.classification.enabled
b021cbc0a [Cheng Pan] Improve docs for SPARK-33832
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6216
## Describe Your Solution 🔧
Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
SessionLimiterSuite.test("test session limiter with ip deny list")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6217 from wForget/KYUUBI-6216.
Closes#6216
e0a058c1d [wforget] update ctl
f66426eda [wforget] typo
5c5308ad8 [wforget] typo
959242e9b [wforget] Support to deny some client ips to make connection
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
