# 🔍 Description
Fix#6594.
This PR ports HIVE-26633(https://github.com/apache/hive/pull/3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6631 from pan3793/thrift-max-size.
Closes#6594
e4841c88e [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Backport https://github.com/apache/hive/pull/4819
Note: it's only the JDBC driver side change, to make it work, we also need to modify the engines.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
We can not test this feature so far because we don't have engine support that. Pass GHA to ensure it breaks nothing.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6639 from pan3793/HIVE-27815.
Closes#6639
4b3bcd65f [Cheng Pan] fix
c16dc28e8 [Cheng Pan] Port HIVE-27815: Support update numModifiedRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5314
## Describe Your Solution 🔧
To support Java 21, some dependency packages need to be upgraded.Here is the detailed explanation.
1. To build with Java 21, Scala binary version needs to be upgraded to 2.13.11 or above.
2. The versions of `spotless` and `googlejavaformat` need to be upgraded to higher versions. However, this will lose support for Java 8. In order to continue supporting Java 8, code style checks will be ignored in the Java 21 profile.
3. Since the Flink engine does not support Scala 2.13, the Scala version of the Flink module is fixed at 2.12.
Previous discussions were at https://github.com/apache/kyuubi/pull/6306.
---
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
**Be nice. Be informative.**
Closes#6597 from dupen01/issue-java21.
Closes#5314
0c5a9e455 [Cheng Pan] simplify
54cbadf79 [dupeng] java21+scala2.13.11+flink_2.12+CI(spark4,21,2,13)
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
SPARK-46257 (Spark 4.0.0) moves to Derby 10.16, `org.apache.derby.jdbc.AutoloadedDriver` has been moved to `org.apache.derby.iapi.jdbc.AutoloadedDriver`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually tested with Spark 4.0.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6640 from pan3793/authz-derby.
Closes#6640
46edb32be [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/util/AuthZUtils.scala
7eee47f0d [Cheng Pan] Adapt Derby 10.16 new JDBC driver package name
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
To support Java 21, the Scala version needs to be upgraded to 2.13.11 or above.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6603 from dupen01/scala213.
Closes#6603
541f86054 [Cheng Pan] exclude SparkLocalClusterTest
13c11bf52 [Perl Du] add ignoreClasses comments
d5b67971b [Cheng Pan] Update pom.xml
582de9b90 [dupeng] ignore org.jline.terminal.impl.ffm.*
364eea00c [dupeng] upgrade maven.plugin.silencer.version from 1.7.13 to 1.7.17
99a1cc40a [dupeng] upgrade scala version to 2.13.14
54b0f4d3f [dupeng] upgrade scala version to 2.13.11
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Perl Du <34719039+dupen01@users.noreply.github.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I faced the following error when trying to run authz with Spark 4.0
```
Cause: java.lang.NoClassDefFoundError: javax/ws/rs/core/Cookie
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.ranger.plugin.policyengine.RangerPluginContext.createAdminClient(RangerPluginContext.java:96)
at org.apache.ranger.plugin.util.PolicyRefresher.<init>(PolicyRefresher.java:90)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:251)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
```
The `javax.ws.rs:jsr311-api` is the transitive dep of `jersey-client`, we should shade and relocate it correctly.
Why does it work with Spark 3? Spark 3 provides `jakarta.ws.rs:jakarta.ws.rs-api:2.1.6` which provides `java.ws.rs.*` classes, but Spark 4 upgrades to `jakarta.ws.rs:jakarta.ws.rs-api:3.0.0` which changed package name to`jakarta.ws.rs.*`.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA and manually tested with Spark 4
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6638 from pan3793/jsr311.
Closes#6638
5699200cf [Cheng Pan] Shade jsr311-api in Authz
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The current Hive 2.3.10 integration tests are broken, the root cause is CALCITE-1224
```
2024-08-20T12:20:55,405 ERROR [HiveTBinaryFrontendHandler-Pool: Thread-49] org.apache.kyuubi.util.KyuubiUncaughtExceptionHandler - Uncaught exception in thread HiveTBinaryFrontendHandler-Pool: Thread-49
java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonGenerator.writeStartArray(Ljava/lang/Object;I)V
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:78) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4485) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3699) ~[jackson-databind-2.12.0.jar:2.12.0]
at org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:518) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:143) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.kyuubi.engine.hive.operation.HiveOperation.runInternal(HiveOperation.scala:62) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:175) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:103) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.engine.hive.session.HiveSessionImpl.runOperation(HiveSessionImpl.scala:59) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:133) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:84) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:130) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_422]
```
## Describe Your Solution 🔧
Unpack `avatica-1.8.0.jar` and delete Jackson classes to workaround CALCITE-1224
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This PR should recover CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6635 from pan3793/hive-2310-fix.
Closes#6635
334bc5c39 [Cheng Pan] nit
22ae945fb [Cheng Pan] fix
dec03196d [Cheng Pan] Fix Hive 2.3.10 integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#3897
## Describe Your Solution 🔧
enrich the description for GROUP Share Level.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6628 from futureltl/master.
Closes#6628
ba18bfce4 [futureltl] Improve docs for GROUP Share Level
3b19521e1 [futureltl] Improve docs for GROUP Share Level
da7d9b61e [Cheng Pan] Update docs/deployment/engine_share_level.md
674066a08 [Cheng Pan] Update docs/deployment/engine_share_level.md
c3a373370 [Cheng Pan] Update docs/deployment/engine_share_level.md
7389cedd2 [futureltl] Improve docs for GROUP Share Level
Lead-authored-by: futureltl <futureltl@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6583
## Background and Goals
Currently, kyuubi cannot perform operation level interrupts when executing Python code. When it is necessary to cancel an operation that has been running for a long time, the entire session needs to be interrupted, and the execution context will be lost, which is very unfriendly to users. Therefore, it is necessary to support operation level interrupts so that the execution context is not lost when the user terminal operates.
## Describe Your Solution 🔧
Refer to the implementation of Jupyter Notebook and let the Python process listen to Signel SIGINT semaphore, when receiving a signel When SIGINT, interrupt the current executing code and capture KeyboardInterrupt to treat it as cancelled
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6612 from yoock/features/support-operation-cancel.
Closes#6583
bf6334d8c [Wang, Fei] log error to do not break the cleanup process
ae7ad3f3c [Wang, Fei] comments
509627e65 [王龙] PySpark support operation cancel
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6619
## Describe Your Solution 🔧
This is a subtask of #6590
This PR makes the following changes:
1. Instead of using SessionManager, the ThriftHttpService will use AuthorizationFilter to handle http client parameters. The purpose is to pass correct client IP address for bearer token authentication.
2. Partial backport Partial backport: https://github.com/apache/hive/pull/3006
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6624 from George314159/thrift.
Closes#6619
9761d6ddf [Wang, Fei] Partial backport HIVE-25575: Add support for JWT authentication in HTTP mode
30ede05d8 [Wang, Fei] Revert "suffix"
39c945802 [Wang, Fei] suffix
b99420633 [Wang, Fei] Revert "Fix"
ea6236a8d [George314159] Fix
a75025be4 [George314159] Refine UT
945433583 [George314159] Refine
8f4eeff03 [George314159] Update UT
6192d4713 [Wang, Fei] revert unused
c7f037c66 [Wang, Fei] Support bearer token in kyuubi jdbc
f7725f761 [George314159] Refine
f865abf01 [George314159] Refine
9d4edd50d [George314159] Add http header config for thrift bearer token auth and UT
6d0f97355 [Wang, Fei] refine
a7e785bcd [Wang, Fei] save
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: George314159 <hua16732@gmail.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
Followup for #6335
Just use the session handle identifier to build the batchResourceUploadFolderPath, and do not rely on the BatchJobSubmission operation initialization.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6629 from turboFei/batch_id.
Closes#6335
28df03d4d [liangbowen] unused import
2eee2a64f [Wang, Fei] Move to KyuubiApplicationManager
f23ecae1f [liangbowen] update batchResourceUploadFolderPath method for removing batch prefix and checking sessionId
b30a7b996 [Wang, Fei] using session id
Lead-authored-by: Wang, Fei <fwang12@ebay.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
As mentioned in https://github.com/apache/kyuubi/pull/6626, the operation never idle because of periodical get operation status, but in my opinion, the operations should be closed after session closed, but it did not.
Currently, for session operation handles set, it does not use concurrent collections, I wonder there is concurrent update issue.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Not needed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6627 from turboFei/operation_set.
Closes#6627
2b80c25e1 [Wang, Fei] revert
57e4dbfa2 [Wang, Fei] using concurrent
bbd9ecf6a [Wang, Fei] use concurrent hash map
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes operation never expired issue.
I found that, some Kyuubi operations never expired.
After investigation, the root cause:
1. our kyuubi console lookup the operations periodically
2. it retrieves the kyuubi operation events
3. the KyuubiOperationEvent::apply will get the operation status and update the operation the last access time, https://github.com/apache/kyuubi/pull/2452
4. then the operation never expired.
## Describe Your Solution 🔧
Just get the operation event without updating the operation last access time.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6626 from turboFei/operation_not_idle.
Closes#6626
14e6898da [Wang, Fei] Get operation event
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6623
## Describe Your Solution 🔧
This is a subtask of #6590
Followup of #6591
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6625 from George314159/swagger.
Closes#6623
d94f950c9 [George314159] retest
7c38c63fb [George314159] Support authorization on swagger UI
Authored-by: George314159 <hua16732@gmail.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6618
## Describe Your Solution 🔧
It is a subtask of #6590
This PR is to support http bearer token authentication for REST protocol. In addition to BasicAuthenticationHandler, BearerAuthenticationHandler will be added to handle http bear token authentication. They will both support CUSTOM AuthType. In order to distinguish them, two new configurations are added: kyuubi.authentication.custom.basic.class and kyuubi.authentication.custom.bearer.class. For http bear token custom authentication, users could implement the new 'org.apache.kyuubi.service.authentication.TokenAuthenticationProvider', and specify it in the configuration.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6608 from George314159/authentication.
Closes#6618
d07a30f83 [Wang, Fei] fix UT
6499c9986 [George314159] Update Test Case
da519a9c6 [George314159] Update based on comments
f47160148 [Wang, Fei] Refine UT
544422399 [George314159] Add test suite for custom authentication
f2bbfbf7e [Wang, Fei] comments & refine
a733c0e8f [George314159] Remove unused val
6f669d46c [George314159] Fix
650b88d4e [George314159] Update based on comments
5bc2bac58 [George314159] Update based on comments
1893889db [George314159] Update based on Comments
ddee882e9 [George314159] Fix Style
379a563fa [George314159] Support http bearer token authentication
Lead-authored-by: George314159 <hua16732@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Default Kyuubi version in the chart is not up to date with the latest release version.
## Describe Your Solution 🔧
Upgrade default version to the latest 1.9.2 to be up to date with the release version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6620 from DenineLu/helm-default-kyuubi-version-1.9.2.
Closes#6620
a101673ac [ludejiu] [K8S][HELM] Update default Kyuubi version to 1.9.2
Authored-by: ludejiu <ludejiu@xiaohongshu.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6615
## Describe Your Solution 🔧
Add a config item that controls whether Jetty should send its version in response.
Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Compiled and tested manually.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6616 from paul8263/KYUUBI-6615.
Closes#6615
c1567fdfa [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable
Authored-by: zhang_yao <xzhangyao@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 3.5.2 was released recently.
Release Notes is available at https://spark.apache.org/releases/spark-release-3-5-2.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6609 from pan3793/spark-3.5.2.
Closes#6609
587cf1dd3 [Cheng Pan] Bump Spark 3.5.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I found the [previous change](https://github.com/apache/kyuubi/pull/6606) breaks compatibility in some shell interpreters, for example `[ ! -f "$MVN_BIN" ]`, `[ $MVN_DETECTED_VERSION != $MVN_VERSION ]` are not supported in ubuntu dash, which fails the CI https://github.com/apache/kyuubi/actions/runs/10347069575/job/28636666290
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This change should recover the CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6610 from pan3793/mvn-followup.
Closes#6606
5381b60ac [Cheng Pan] 1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6584
## Describe Your Solution 🔧
It involves bumping the version from 1.0.2 to 1.1.1 for flip-tables
## Test Plan 🧪
Build locally
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6614 from Madhukar525722/flip_tables.
Closes#6584
dee9fd7a9 [madlnu] [KYUUBI #6584] Upgrade flip-tables from 1.0.2 to 1.1.1
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR upgrades gRPC, Protobuf, and Guava to the relatively up-to-date versions, to support the latest Netty 4.1.111.
https://groups.google.com/g/grpc-io/c/R4JB2h7y5N4
> For Netty 4.1.111 compat, it is best to use grpc-java 1.63.2, 1.64.2, 1.65.1, and later.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6607 from pan3793/grpc.
Closes#6607
a7c299629 [Cheng Pan] update dependencyList
c4cac4372 [Cheng Pan] Bump gRPC 1.65.1 Protobuf 3.25.4 Guava 32.1.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6485
## Describe Your Solution 🔧
Ignore uppercase and lowercase letters in table names when using regular expressions to match.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Added unit tests when table names have capital letters.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6605 from BruceWong96/fix-presto-regex.
Closes#6485
06f737f24 [Bruce Wong] Fix typos
93071754a [Bruce Wong] Added unit tests for table names with both upper and lower case letters
9837030a1 [Bruce Wong] fix table not found
Authored-by: Bruce Wong <603334301@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I found sometimes `build/mvn -version` is pretty slow even `build/apache-maven-${MVN_VERSION}` is already cached, and I found it may stuck at the following call when network quality is not good.
bdd91f4539/build/mvn (L86)
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested `build/mvn -version` in the following cases:
1. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is available, the command always finishes quickly.
2. when `build/apache-maven-${MVN_VERSION}/bin/mvn` is unavailable but global installed `mvn` version matches `maven.version` defined in root `pom.xml`, the command always finishes quickly too.
3. otherwise, it automatically downloads the maven binary tarball from network.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6606 from pan3793/mvn.
Closes#6606
a801d793f [Cheng Pan] nit
3a9f35f2b [Cheng Pan] build/mvn checks local downloaded binary ahead before network access
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
The current FileSessionConfAdvisor only supports configuring a single file through kyuubi.session.conf.profiler. However, users may need to use multiple configuration files, and the existing setup does not support this. To address this need, we have updated the FileSessionConfAdvisor to support multiple configuration files.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6585 from lsm1/branch-file-advidor-support-chain.
Closes#6585
15b6524e0 [senmiaoliu] fix style
fcedc4fe3 [senmiaoliu] FileSessionConfAdvisor support multi files
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
# 🔍 Description
As title.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6601 from pan3793/delta-4.0.
Closes#6601
711f2e071 [Cheng Pan] Bump Delta Lake 4.0.0rc1 and enable Delta Lake test for Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
It was wrong applied to all goals previously, which caused all `test-compile`s are skipped.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Let's monitor what CI happens
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6600 from pan3793/scaladoc.
Closes#6600
86ea6fdd0 [Cheng Pan] scala213
2a76a93d5 [Cheng Pan] Correct usage of maven.scaladoc.skip
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR disables Javadoc/Scaladoc by default, while still enabling that on Nexus deploy and CI, to speed up the normal building by developers daily performed.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Apple M1 Max, Java 17
```
$ time build/mvn clean install -DskipTests
```
before: 1145.67s user 62.41s system 276% cpu 7:16.56 total
after: 530.33s user 35.82s system 214% cpu 4:23.69 total
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6599 from pan3793/javadoc.
Closes#6599
660f8dc02 [Cheng Pan] Disable Javadoc/Scaladoc by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
Fixing some typos
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6596 from hyperbolic2346/hyperbolic2346-patch-1.
Closes#6596
ebbddf0a6 [Mike Wilson] Update architecture.md
Lead-authored-by: Mike Wilson <knobby@burntsheep.com>
Co-authored-by: Mike Wilson <hyperbolic2346@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Support authorization on swagger UI.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6591 from turboFei/swagger_auth.
Closes#6591
28010bd99 [Wang, Fei] Support authorization on swagger UI
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
As title, for security concern.
Before, it skips to check the local dir access for resource upload use case, which is not expected.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6592 from turboFei/check_batch_access.
Closes#6592
80cbc7414 [Wang, Fei] do not check if is empty
a82fe8e87 [Wang, Fei] check batch access
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
In some cases, env var USER is unavailable, for example, docker container, we should evaluate it via `id -nu`
ref: https://github.com/apache/hadoop/blob/rel/release-3.4.0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L893-L896
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
```
$ docker run -t -i ubuntu:latest
root1dbeaefd6cd4:/# echo $USER
root1dbeaefd6cd4:/# id -nu
root
root1dbeaefd6cd4:/# exit
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6588 from pan3793/USER.
Closes#6588
a79786347 [Cheng Pan] Get effective USER if env var is unavailable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- support creating batch jobs with uploading extra resource files
- allow uploading extra resource when creating batch jobs via REST API
- support binding the subresources to configs by customed configs, eg.`spark.submit.pyFiles`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
+ new test
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6335 from bowenliang123/batch-subresource.
Closes#6335
57d43d26d [Bowen Liang] nit
d866a8a17 [Bowen Liang] warn exception
20d4328a1 [Bowen Liang] log exception when exception ignored
58c402334 [Bowen Liang] rename param to ignoreException
80bc21034 [Bowen Liang] cleanup the uploaded resource folder when handling files error
3e7961124 [Bowen Liang] throw exception when file non-existed
09ac48a26 [liangbowen] pyspark extra resources
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump apache parent pom from 31 to 33:
- 32: https://github.com/apache/maven-apache-parent/releases/tag/apache-32
- [[MPOM-264](https://issues.apache.org/jira/browse/MPOM-264)] - Parameterize maven-compiler-plugin with parameter "release" when running on JDK 9+
- 33: https://github.com/apache/maven-apache-parent/releases/tag/apache-33
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6571 from bowenliang123/mvn-parent-33.
Closes#6571
bb367efdf [Bowen Liang] bump parent pom to 33
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- Bump commons-io from 2.11.0(2021-07-09) to 2.16.1(2024-04-04) for accumulate updates, release note: https://commons.apache.org/proper/commons-io/changes-report.html#a2.16.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6586 from bowenliang123/commonsio-216.
Closes#6586
4fbb05650 [Bowen Liang] bump commons-io to 2.16.1
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6581
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
I modified `KyuubiSparkSQLAstBuilder#visitMultipartIdentifier` and implemented `KyuubiSparkSQLAstBuilder#visitQuotedIdentifier` to process the quoted identifiers.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
```
extensions/spark/kyuubi-extension-spark-3-3/src/test/scala/org/apache/spark/sql/ZorderSuiteBase.scala
test("optimize sort by backquoted column name")
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6582 from XorSum/features/zorder-backquote.
Closes#6582
16ffa1238 [xorsum] zorder by support quote
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Flink 1.20 is out, [Release Note](https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
CI is updated to cover Flink 1.20
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6557 from pan3793/flink-1.20.
Closes#6557
a414094b9 [Cheng Pan] remove rc
8ee4cf5cd [Cheng Pan] fix url
fbaf66071 [Cheng Pan] docs
ddbd10ffe [Cheng Pan] Support Flink 1.20
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6564
## Describe Your Solution 🔧
Remove the `columnDesc` for `InsertIntoHadoopFsRelationCommand ` and `InsertIntoHiveTable ` in `table_command_spec.json`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Insert into table will check the privilege of columns.
#### Behavior With This Pull Request 🎉
Insert into table will check the privilege of table.
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6570 from liujiayi771/insert-permission.
Closes#6564
d956aa916 [joey.ljy] Fix ut
d282f8ec5 [joey.ljy] insert into table check the privilege of table
Authored-by: joey.ljy <joey.ljy@alibaba-inc.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6574
## Describe Your Solution 🔧
Skip eagerly execute command in physical and execution plan only mode
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Related Unit Tests
added unit test
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6575 from wForget/KYUUBI-6574.
Closes#6574
6f79228c6 [wforget] fix
9aff4a803 [wforget] fix
839ea4a4f [wforget] fix
8a08c9fa7 [wforget] [KYUUBI #6574] Skip eagerly execute command in PlanOnly mode of Spark Engine
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
[KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
# 🔍 Description
## Issue References 🔗
This pull request fixes#6549
## Describe Your Solution 🔧
When obtaining configuration items, if it is null or empty, return none
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6556 from LiJie20190102/launch_engine.
Closes#6549
c57a08aff [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
642d807e2 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
67926094c [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
4ba9fb587 [lijie0203] [KYUUBI #6549] Fix 'Could not find or load main class when launching engine'
Authored-by: lijie0203 <lijie@qishudi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pr enhance the kyuubi batch logs for spark on k8s with kyuubi-ctl.
## Describe Your Solution 🔧
For spark on k8s, before, it does not show the app detail info until the log batch command finished.
In this PR, it will show the batch app details in app state change.
Especially show the app URL info.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
<img width="1326" alt="image" src="https://github.com/user-attachments/assets/2356af83-4422-4dfc-812c-d90b809ea724">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6579 from turboFei/print_batch_report.
Closes#6579
0651dad18 [Wang, Fei] show app details on app state change
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump `scala-maven-plugin` from 4.8.0 to 4.9.2, with `zinc` bumped from 1.8.0(Nov 11, 2022) to 1.10.0(May 6, 2024)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6364 from bowenliang123/scala-maven-490.
Closes#6364
554379702 [Bowen Liang] bump scala-maven-plugin to 4.9.2
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6567
## Describe Your Solution 🔧
With `setuptools==72.0.0`, the `test` command was removed after being deprecated for a long time.
This completely breaks the installation of any packages using the `test` command, including PyHive.
This PR fixes PyHive to be compatible with `setuptools` 72+ again.
It seems that `setup.py test` wasn't used anywhere at all, which is why I just removed it.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
```
$ pip install -e .
Obtaining file:///home/localstack/Repos/kyuubi/python
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/home/localstack/Repos/kyuubi/python/setup.py", line 4, in <module>
from setuptools.command.test import test as TestCommand
ModuleNotFoundError: No module named 'setuptools.command.test'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details
```
#### Behavior With This Pull Request 🎉
```
$ pip install --upgrade setuptools
Requirement already satisfied: setuptools in .../.pyenv/versions/3.11.5/lib/python3.11/site-packages (72.0.0)
$ pip install -e .
Obtaining file:///home/.../repos/kyuubi/python
Preparing metadata (setup.py) ... done
Collecting future (from PyHive==0.7.0)
Obtaining dependency information for future from ae30dadffc/future-1.0.0-py3-none-any.whl.metadata
Using cached future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
Requirement already satisfied: python-dateutil in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from PyHive==0.7.0) (2.8.2)
Requirement already satisfied: six>=1.5 in /home/.../.pyenv/versions/3.11.5/lib/python3.11/site-packages (from python-dateutil->PyHive==0.7.0) (1.16.0)
Using cached future-1.0.0-py3-none-any.whl (491 kB)
Installing collected packages: future, PyHive
Running setup.py develop for PyHive
Successfully installed PyHive-0.7.0 future-1.0.0
[notice] A new release of pip is available: 23.2.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
```
#### Related Unit Tests
None
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6568 from alexrashed/fix-pyhive-setuptools-72.
Closes#6567
acfb80903 [Alexander Rashed] remove usage of setuptools.command.test
Authored-by: Alexander Rashed <alexander.rashed@localstack.cloud>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
fix typos
## Describe Your Solution 🔧
anderror -> an error
deprected -> deprecated
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6566 from XorSum/fix/kyuubi-conf-typo.
Closes#6566
5c270ad0b [xorsum] fix typo in KyuubiConf
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6541
## Describe Your Solution 🔧
Fix an issue where DataSourceV2RelationTableExtractor#table could not fetch the ‘database’ attribute causing the Ranger checks to fail when using the Paimon Catalog.
If the database attribute is not resolved, use DataSourceV2RelationTableExtractor#identifier to complete it.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6544 from promising-forever/issues/6541.
Closes#6541
6549f8528 [caoyu] Fix test failure, paimon-spark run on Scala 2.12.
c1a09214a [caoyu] Optimising the 'database' capture logic
69fb0bc7e [caoyu] PolicyJsonFileGenerator#genPolicies add paimonNamespace
c89c70bad [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
77f121b0d [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
9cfb5847b [caoyu] [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor#table can't get the 'database' attribute if it's a Paimon plan.
Authored-by: caoyu <caoy.5@jifenn.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
Upgrade to latest Apache Iceberg 1.6.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass existing GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6558 from pan3793/iceberg-1.6.0.
Closes#6558
75e42938f [Cheng Pan] nit
ba508b6bb [Cheng Pan] nit
0180521aa [Cheng Pan] Bump Iceberg 1.6.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR contains various improvement for Kyuubi release scripts:
- update known_translations
- improve `pre_gen_release_notes.py` to generate the copy-paste contributor list
- update vote and announcement mail templates
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually tested and self reviewed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6559 from pan3793/rel-scripts.
Closes#6559
78aad0500 [Cheng Pan] Various improvments for release scripts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6554
## Describe Your Solution 🔧
- Delete `/kyuubi/extensions/spark/kyuubi-extension-spark-3-x/src/main/scala/org/apache/kyuubi/sql/zorder/InsertZorderBeforeWritingBase.scala` file
- Rename `InsertZorderBeforeWriting33.scala` to `InsertZorderBeforeWriting.scala`
- Rename `InsertZorderHelper33, InsertZorderBeforeWritingDatasource33, InsertZorderBeforeWritingHive33, ZorderSuiteSpark33` to `InsertZorderHelper, InsertZorderBeforeWritingDatasource, InsertZorderBeforeWritingHive, ZorderSuiteSpark`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6555 from huangxiaopingRD/6554.
Closes#6554
26de4fa09 [huangxiaoping] [KYUUBI #6554] Delete redundant code related to zorder
Authored-by: huangxiaoping <1754789345@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
