# 🔍 Description
## Issue References 🔗
This pull request fixes operation never expired issue.
I found that, some Kyuubi operations never expired.
After investigation, the root cause:
1. our kyuubi console lookup the operations periodically
2. it retrieves the kyuubi operation events
3. the KyuubiOperationEvent::apply will get the operation status and update the operation the last access time, https://github.com/apache/kyuubi/pull/2452
4. then the operation never expired.
## Describe Your Solution 🔧
Just get the operation event without updating the operation last access time.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6626 from turboFei/operation_not_idle.
Closes#6626
14e6898da [Wang, Fei] Get operation event
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6623
## Describe Your Solution 🔧
This is a subtask of #6590
Followup of #6591
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6625 from George314159/swagger.
Closes#6623
d94f950c9 [George314159] retest
7c38c63fb [George314159] Support authorization on swagger UI
Authored-by: George314159 <hua16732@gmail.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6618
## Describe Your Solution 🔧
It is a subtask of #6590
This PR is to support http bearer token authentication for REST protocol. In addition to BasicAuthenticationHandler, BearerAuthenticationHandler will be added to handle http bear token authentication. They will both support CUSTOM AuthType. In order to distinguish them, two new configurations are added: kyuubi.authentication.custom.basic.class and kyuubi.authentication.custom.bearer.class. For http bear token custom authentication, users could implement the new 'org.apache.kyuubi.service.authentication.TokenAuthenticationProvider', and specify it in the configuration.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6608 from George314159/authentication.
Closes#6618
d07a30f83 [Wang, Fei] fix UT
6499c9986 [George314159] Update Test Case
da519a9c6 [George314159] Update based on comments
f47160148 [Wang, Fei] Refine UT
544422399 [George314159] Add test suite for custom authentication
f2bbfbf7e [Wang, Fei] comments & refine
a733c0e8f [George314159] Remove unused val
6f669d46c [George314159] Fix
650b88d4e [George314159] Update based on comments
5bc2bac58 [George314159] Update based on comments
1893889db [George314159] Update based on Comments
ddee882e9 [George314159] Fix Style
379a563fa [George314159] Support http bearer token authentication
Lead-authored-by: George314159 <hua16732@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Default Kyuubi version in the chart is not up to date with the latest release version.
## Describe Your Solution 🔧
Upgrade default version to the latest 1.9.2 to be up to date with the release version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6620 from DenineLu/helm-default-kyuubi-version-1.9.2.
Closes#6620
a101673ac [ludejiu] [K8S][HELM] Update default Kyuubi version to 1.9.2
Authored-by: ludejiu <ludejiu@xiaohongshu.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6615
## Describe Your Solution 🔧
Add a config item that controls whether Jetty should send its version in response.
Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Compiled and tested manually.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6616 from paul8263/KYUUBI-6615.
Closes#6615
c1567fdfa [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable
Authored-by: zhang_yao <xzhangyao@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 3.5.2 was released recently.
Release Notes is available at https://spark.apache.org/releases/spark-release-3-5-2.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6609 from pan3793/spark-3.5.2.
Closes#6609
587cf1dd3 [Cheng Pan] Bump Spark 3.5.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I found the [previous change](https://github.com/apache/kyuubi/pull/6606) breaks compatibility in some shell interpreters, for example `[ ! -f "$MVN_BIN" ]`, `[ $MVN_DETECTED_VERSION != $MVN_VERSION ]` are not supported in ubuntu dash, which fails the CI https://github.com/apache/kyuubi/actions/runs/10347069575/job/28636666290
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This change should recover the CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6610 from pan3793/mvn-followup.
Closes#6606
5381b60ac [Cheng Pan] 1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6584
## Describe Your Solution 🔧
It involves bumping the version from 1.0.2 to 1.1.1 for flip-tables
## Test Plan 🧪
Build locally
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6614 from Madhukar525722/flip_tables.
Closes#6584
dee9fd7a9 [madlnu] [KYUUBI #6584] Upgrade flip-tables from 1.0.2 to 1.1.1
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR upgrades gRPC, Protobuf, and Guava to the relatively up-to-date versions, to support the latest Netty 4.1.111.
https://groups.google.com/g/grpc-io/c/R4JB2h7y5N4
> For Netty 4.1.111 compat, it is best to use grpc-java 1.63.2, 1.64.2, 1.65.1, and later.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6607 from pan3793/grpc.
Closes#6607
a7c299629 [Cheng Pan] update dependencyList
c4cac4372 [Cheng Pan] Bump gRPC 1.65.1 Protobuf 3.25.4 Guava 32.1.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6485
## Describe Your Solution 🔧
Ignore uppercase and lowercase letters in table names when using regular expressions to match.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Added unit tests when table names have capital letters.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6605 from BruceWong96/fix-presto-regex.
Closes#6485
06f737f24 [Bruce Wong] Fix typos
93071754a [Bruce Wong] Added unit tests for table names with both upper and lower case letters
9837030a1 [Bruce Wong] fix table not found
Authored-by: Bruce Wong <603334301@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I found sometimes `build/mvn -version` is pretty slow even `build/apache-maven-${MVN_VERSION}` is already cached, and I found it may stuck at the following call when network quality is not good.
bdd91f4539/build/mvn (L86)
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested `build/mvn -version` in the following cases:
1. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is available, the command always finishes quickly.
2. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is unavailable but global installed `mvn` version matches `maven.version` defined in root `pom.xml`, the command always finishes quickly too.
3. otherwise, it automatically downloads the maven binary tarball from network.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6606 from pan3793/mvn.
Closes#6606
a801d793f [Cheng Pan] nit
3a9f35f2b [Cheng Pan] build/mvn checks local downloaded binary ahead before network access
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
The current FileSessionConfAdvisor only supports configuring a single file through kyuubi.session.conf.profiler. However, users may need to use multiple configuration files, and the existing setup does not support this. To address this need, we have updated the FileSessionConfAdvisor to support multiple configuration files.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6585 from lsm1/branch-file-advidor-support-chain.
Closes#6585
15b6524e0 [senmiaoliu] fix style
fcedc4fe3 [senmiaoliu] FileSessionConfAdvisor support multi files
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
# 🔍 Description
As title.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6601 from pan3793/delta-4.0.
Closes#6601
711f2e071 [Cheng Pan] Bump Delta Lake 4.0.0rc1 and enable Delta Lake test for Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
It was wrong applied to all goals previously, which caused all `test-compile`s are skipped.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Let's monitor what CI happens
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6600 from pan3793/scaladoc.
Closes#6600
86ea6fdd0 [Cheng Pan] scala213
2a76a93d5 [Cheng Pan] Correct usage of maven.scaladoc.skip
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR disables Javadoc/Scaladoc by default, while still enabling that on Nexus deploy and CI, to speed up the normal building by developers daily performed.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Apple M1 Max, Java 17
```
$ time build/mvn clean install -DskipTests
```
before: 1145.67s user 62.41s system 276% cpu 7:16.56 total
after: 530.33s user 35.82s system 214% cpu 4:23.69 total
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6599 from pan3793/javadoc.
Closes#6599
660f8dc02 [Cheng Pan] Disable Javadoc/Scaladoc by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
Fixing some typos
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6596 from hyperbolic2346/hyperbolic2346-patch-1.
Closes#6596
ebbddf0a6 [Mike Wilson] Update architecture.md
Lead-authored-by: Mike Wilson <knobby@burntsheep.com>
Co-authored-by: Mike Wilson <hyperbolic2346@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Support authorization on swagger UI.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6591 from turboFei/swagger_auth.
Closes#6591
28010bd99 [Wang, Fei] Support authorization on swagger UI
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
As title, for security concern.
Before, it skips to check the local dir access for resource upload use case, which is not expected.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6592 from turboFei/check_batch_access.
Closes#6592
80cbc7414 [Wang, Fei] do not check if is empty
a82fe8e87 [Wang, Fei] check batch access
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
In some cases, env var USER is unavailable, for example, docker container, we should evaluate it via `id -nu`
ref: https://github.com/apache/hadoop/blob/rel/release-3.4.0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L893-L896
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
```
$ docker run -t -i ubuntu:latest
root1dbeaefd6cd4:/# echo $USER
root1dbeaefd6cd4:/# id -nu
root
root1dbeaefd6cd4:/# exit
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6588 from pan3793/USER.
Closes#6588
a79786347 [Cheng Pan] Get effective USER if env var is unavailable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- support creating batch jobs with uploading extra resource files
- allow uploading extra resource when creating batch jobs via REST API
- support binding the subresources to configs by customed configs, eg.`spark.submit.pyFiles`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
+ new test
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6335 from bowenliang123/batch-subresource.
Closes#6335
57d43d26d [Bowen Liang] nit
d866a8a17 [Bowen Liang] warn exception
20d4328a1 [Bowen Liang] log exception when exception ignored
58c402334 [Bowen Liang] rename param to ignoreException
80bc21034 [Bowen Liang] cleanup the uploaded resource folder when handling files error
3e7961124 [Bowen Liang] throw exception when file non-existed
09ac48a26 [liangbowen] pyspark extra resources
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump apache parent pom from 31 to 33:
- 32: https://github.com/apache/maven-apache-parent/releases/tag/apache-32
- [[MPOM-264](https://issues.apache.org/jira/browse/MPOM-264)] - Parameterize maven-compiler-plugin with parameter "release" when running on JDK 9+
- 33: https://github.com/apache/maven-apache-parent/releases/tag/apache-33
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6571 from bowenliang123/mvn-parent-33.
Closes#6571
bb367efdf [Bowen Liang] bump parent pom to 33
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- Bump commons-io from 2.11.0(2021-07-09) to 2.16.1(2024-04-04) for accumulate updates, release note: https://commons.apache.org/proper/commons-io/changes-report.html#a2.16.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6586 from bowenliang123/commonsio-216.
Closes#6586
4fbb05650 [Bowen Liang] bump commons-io to 2.16.1
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6581
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
I modified `KyuubiSparkSQLAstBuilder#visitMultipartIdentifier` and implemented `KyuubiSparkSQLAstBuilder#visitQuotedIdentifier` to process the quoted identifiers.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
```
extensions/spark/kyuubi-extension-spark-3-3/src/test/scala/org/apache/spark/sql/ZorderSuiteBase.scala
test("optimize sort by backquoted column name")
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6582 from XorSum/features/zorder-backquote.
Closes#6582
16ffa1238 [xorsum] zorder by support quote
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Flink 1.20 is out, [Release Note](https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
CI is updated to cover Flink 1.20
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6557 from pan3793/flink-1.20.
Closes#6557
a414094b9 [Cheng Pan] remove rc
8ee4cf5cd [Cheng Pan] fix url
fbaf66071 [Cheng Pan] docs
ddbd10ffe [Cheng Pan] Support Flink 1.20
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6564
## Describe Your Solution 🔧
Remove the `columnDesc` for `InsertIntoHadoopFsRelationCommand ` and `InsertIntoHiveTable ` in `table_command_spec.json`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Insert into table will check the privilege of columns.
#### Behavior With This Pull Request 🎉
Insert into table will check the privilege of table.
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6570 from liujiayi771/insert-permission.
Closes#6564
d956aa916 [joey.ljy] Fix ut
d282f8ec5 [joey.ljy] insert into table check the privilege of table
Authored-by: joey.ljy <joey.ljy@alibaba-inc.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6574
## Describe Your Solution 🔧
Skip eagerly execute command in physical and execution plan only mode
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Related Unit Tests
added unit test
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6575 from wForget/KYUUBI-6574.
Closes#6574
6f79228c6 [wforget] fix
9aff4a803 [wforget] fix
839ea4a4f [wforget] fix
8a08c9fa7 [wforget] [KYUUBI #6574] Skip eagerly execute command in PlanOnly mode of Spark Engine
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
[KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
# 🔍 Description
## Issue References 🔗
This pull request fixes#6549
## Describe Your Solution 🔧
When obtaining configuration items, if it is null or empty, return none
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6556 from LiJie20190102/launch_engine.
Closes#6549
c57a08aff [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
642d807e2 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
67926094c [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
4ba9fb587 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
Authored-by: lijie0203 <lijie@qishudi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pr enhance the kyuubi batch logs for spark on k8s with kyuubi-ctl.
## Describe Your Solution 🔧
For spark on k8s, before, it does not show the app detail info until the log batch command finished.
In this PR, it will show the batch app details in app state change.
Especially show the app URL info.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
<img width="1326" alt="image" src="https://github.com/user-attachments/assets/2356af83-4422-4dfc-812c-d90b809ea724">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6579 from turboFei/print_batch_report.
Closes#6579
0651dad18 [Wang, Fei] show app details on app state change
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump `scala-maven-plugin` from 4.8.0 to 4.9.2, with `zinc` bumped from 1.8.0(Nov 11, 2022) to 1.10.0(May 6, 2024)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6364 from bowenliang123/scala-maven-490.
Closes#6364
554379702 [Bowen Liang] bump scala-maven-plugin to 4.9.2
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6567
## Describe Your Solution 🔧
With `setuptools==72.0.0`, the `test` command was removed after being deprecated for a long time.
This completely breaks the installation of any packages using the `test` command, including PyHive.
This PR fixes PyHive to be compatible with `setuptools` 72+ again.
It seems that `setup.py test` wasn't used anywhere at all, which is why I just removed it.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
```
$ pip install -e .
Obtaining file:///home/localstack/Repos/kyuubi/python
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/home/localstack/Repos/kyuubi/python/setup.py", line 4, in <module>
from setuptools.command.test import test as TestCommand
ModuleNotFoundError: No module named 'setuptools.command.test'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details
```
#### Behavior With This Pull Request 🎉
```
$ pip install --upgrade setuptools
Requirement already satisfied: setuptools in .../.pyenv/versions/3.11.5/lib/python3.11/site-packages (72.0.0)
$ pip install -e .
Obtaining file:///home/.../repos/kyuubi/python
Preparing metadata (setup.py) ... done
Collecting future (from PyHive==0.7.0)
Obtaining dependency information for future from ae30dadffc/future-1.0.0-py3-none-any.whl.metadata
Using cached future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
Requirement already satisfied: python-dateutil in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from PyHive==0.7.0) (2.8.2)
Requirement already satisfied: six>=1.5 in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from python-dateutil->PyHive==0.7.0) (1.16.0)
Using cached future-1.0.0-py3-none-any.whl (491 kB)
Installing collected packages: future, PyHive
Running setup.py develop for PyHive
Successfully installed PyHive-0.7.0 future-1.0.0
[notice] A new release of pip is available: 23.2.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
```
#### Related Unit Tests
None
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6568 from alexrashed/fix-pyhive-setuptools-72.
Closes#6567
acfb80903 [Alexander Rashed] remove usage of setuptools.command.test
Authored-by: Alexander Rashed <alexander.rashed@localstack.cloud>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
fix typos
## Describe Your Solution 🔧
anderror -> an error
deprected -> deprecated
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6566 from XorSum/fix/kyuubi-conf-typo.
Closes#6566
5c270ad0b [xorsum] fix typo in KyuubiConf
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6541
## Describe Your Solution 🔧
Fix an issue where DataSourceV2RelationTableExtractor#table could not fetch the ‘database’ attribute causing the Ranger checks to fail when using the Paimon Catalog.
If the database attribute is not resolved, use DataSourceV2RelationTableExtractor#identifier to complete it.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6544 from promising-forever/issues/6541.
Closes#6541
6549f8528 [caoyu] Fix test failure, paimon-spark run on Scala 2.12.
c1a09214a [caoyu] Optimising the 'database' capture logic
69fb0bc7e [caoyu] PolicyJsonFileGenerator#genPolicies add paimonNamespace
c89c70bad [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
77f121b0d [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
9cfb5847b [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
Authored-by: caoyu <caoy.5@jifenn.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
Upgrade to latest Apache Iceberg 1.6.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass existing GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6558 from pan3793/iceberg-1.6.0.
Closes#6558
75e42938f [Cheng Pan] nit
ba508b6bb [Cheng Pan] nit
0180521aa [Cheng Pan] Bump Iceberg 1.6.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR contains various improvement for Kyuubi release scripts:
- update known_translations
- improve `pre_gen_release_notes.py` to generate the copy-paste contributor list
- update vote and announcement mail templates
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually tested and self reviewed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6559 from pan3793/rel-scripts.
Closes#6559
78aad0500 [Cheng Pan] Various improvments for release scripts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6554
## Describe Your Solution 🔧
- Delete `/kyuubi/extensions/spark/kyuubi-extension-spark-3-x/src/main/scala/org/apache/kyuubi/sql/zorder/InsertZorderBeforeWritingBase.scala` file
- Rename `InsertZorderBeforeWriting33.scala` to `InsertZorderBeforeWriting.scala`
- Rename `InsertZorderHelper33, InsertZorderBeforeWritingDatasource33, InsertZorderBeforeWritingHive33, ZorderSuiteSpark33` to `InsertZorderHelper, InsertZorderBeforeWritingDatasource, InsertZorderBeforeWritingHive, ZorderSuiteSpark`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6555 from huangxiaopingRD/6554.
Closes#6554
26de4fa09 [huangxiaoping] [KYUUBI #6554] Delete redundant code related to zorder
Authored-by: huangxiaoping <1754789345@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
StatefulSet's `updateStrategy.rollingUpdate.maxUnavailable` is being used, but this feature is still in the alpha stage. It should not be used by default.
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features:~:text=%E2%80%93-,MaxUnavailableStatefulSet,-falsehttps://kubernetes.io/blog/2022/05/27/maxunavailable-for-statefulset/
## Issue References 🔗
None
## Describe Your Solution 🔧
I simply removed it. The behavior should not change even after the removal.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6550 from be-hase/polish-updateStrategy.
Closes#6550
ab31779fd [ryosuke-hasebe] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
Authored-by: ryosuke-hasebe <ryosuke.hasebe@linecorp.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6551
## Describe Your Solution 🔧
Update `canInsertZorder` to allow insert zorder when global sort is `false` and the plan is `Repartition` or `RepartitionByExpression`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
/kyuubi-extension-spark-common/src/test/scala/org/apache/spark/sql/ZorderSuiteBase.scala
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6552 from huangxiaopingRD/6551.
Closes#6551
b597443c3 [huangxiaoping] Fix code style
618594667 [huangxiaoping] [KYUUBI #6551] Allow insert zorder when when the plan is Repartition or RepartitionByExpression
Authored-by: huangxiaoping <1754789345@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
# 🔍 Description
This pull request aims to remove building support for Spark 3.2, while still keeping the engine support for Spark 3.2.
Mailing list discussion: https://lists.apache.org/thread/l74n5zl1w7s0bmr5ovxmxq58yqy8hqzc
- Remove Maven profile `spark-3.2`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.5) still works well on Spark 3.2 runtime.
- Merge `kyuubi-extension-spark-common` into `kyuubi-extension-spark-3-3`
- Remove `log4j.properties` as Spark moves to Log4j2 since 3.3 (SPARK-37814)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6545 from pan3793/deprecate-spark-3.2.
Closes#6545
54c172528 [Cheng Pan] fix
f4602e805 [Cheng Pan] Deprecate and remove building support for Spark 3.2
2e083f89f [Cheng Pan] fix style
458a92c53 [Cheng Pan] nit
929e1df36 [Cheng Pan] Deprecate and remove building support for Spark 3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Bump Kyuubi Shaded 0.4.1
The only change is #6496
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6543 from pan3793/shaded-041.
Closes#6543
3b674af95 [Cheng Pan] Remove staging repo
cae175517 [Cheng Pan] fix
549310067 [Cheng Pan] Bump Kyuubi Shaded 0.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- this bump helps to recognize Java 21-23 class formats: https://github.com/mojohaus/extra-enforcer-rules/releases
- reduce warning message when checking java bytecode versions
```
Warning: Unknown bytecodeVersion for com.fasterxml.jackson.core:jackson-core:jar:2.15.4:compile : META-INF/versions/21/com/fasterxml/jackson/core/io/doubleparser/FastDoubleSwar.class: got null class-file-version
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6548 from bowenliang123/mojo-rules-180.
Closes#6548
f53e4725d [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
This is similar to https://github.com/apache/kyuubi/pull/5310.
We need to wait for Jetty Server started before picking jobs, otherwise, it may get the wrong local HTTP server port -1.
This only affects Batch V2
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested in internal workloads.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6542 from pan3793/batch-wait-started.
Closes#6542
1f62debfe [Cheng Pan] fix
ee1d05d07 [Cheng Pan] KyuubiBatchService should wait for HTTP server started before picking jobs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6523
## Describe Your Solution 🔧
`KyuubiServer` respects configuration properties provided as `--conf` cli options with highest priority.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
REST endpoint starts at `10099` port:
```shell
bin/kyuubi run --conf kyuubi.frontend.rest.bind.port=9999
...
2024-07-10 17:57:24.998 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is started.
2024-07-10 17:57:24.999 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Exposing REST endpoint at: http://0.0.0.0:10099
2024-07-10 17:57:24.999 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started.
```
#### Behavior With This Pull Request 🎉
REST endpoint starts at `9999` port:
```shell
bin/kyuubi run --conf kyuubi.frontend.rest.bind.port=9999
...
2024-07-10 17:49:41.659 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is started.
2024-07-10 17:49:41.660 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Exposing REST endpoint at: http://0.0.0.0:9999
2024-07-10 17:49:41.660 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started.
```
#### Related Unit Tests
`kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala`
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6530 from dnskr/kyuubi-supports-conf-args.
Closes#6523
c7dbb305b [dnskr] Parse command and shift args
4977af168 [dnskr] [KYUUBI #6523] Support configuration with --conf options
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6534
## Describe Your Solution 🔧
check null ahead.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6540 from pan3793/client-npe.
Closes#6534
6976cd3f2 [Cheng Pan] [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6537
## Describe Your Solution 🔧
Upgrade Flink from 1.19.0 to 1.19.1.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6538 from SteNicholas/KYUUBI-6537.
Closes#6537
b39d308d6 [SteNicholas] [KYUUBI #6537] Upgrade Flink from 1.19.0 to 1.19.1
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This pull request fixes #
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6529 from avishnus/kubernetes.
Closes#6529
2a82df557 [Cheng Pan] Update dev/dependencyList
a4855467c [Cheng Pan] Update pom.xml
1ef0c40aa [Cheng Pan] Update dev/dependencyList
4b45b0038 [Cheng Pan] Update dev/dependencyList
d0c5174e8 [Cheng Pan] Update dev/dependencyList
bc2f6b8eb [Cheng Pan] Update dev/dependencyList
f76a7f487 [avishnus] Upgrading kubernetes-client
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6531
## Describe Your Solution 🔧
Fix SPARK-EngineTab stop/gracefulstop not work
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6533 from promising-forever/patch-1.
Closes#6531
66be0fdfd [promising-forever] [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
Authored-by: promising-forever <79634622+promising-forever@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
