### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 3.5.4 is released https://spark.apache.org/releases/spark-release-3-5-4.html
### How was this patch tested?
Pas GHA
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6842 from pan3793/spark-3.5.4.
Closes#6842
0fb7ad8a0 [Cheng Pan] ga
8eacc9c97 [Cheng Pan] Spark 3.5.4 RC2
0721fa401 [Cheng Pan] fix
49e98a201 [Cheng Pan] maven repo
951db0c82 [Cheng Pan] Spark 3.5.4
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The GHA cache `engine-archives` seems frequently to be evicted because of exceeding the project cache capacity(10GiB), and I found the K8s IT job produces fresh large cache objects frequently, we should disable that to let `engine-archives` survive, to avoid downloading from the fragile `archive.apache.org` each time.
```
Connect to archive.apache.org:443 [archive.apache.org/65.108.204.189, archive.apache.org/2a01:4f9:1a:a084:0:0:0:2] failed: Network is unreachable (connect failed)
```
<img width="1306" alt="Xnip2024-12-04_17-36-03" src="https://github.com/user-attachments/assets/7e00ecad-8492-4a3e-b669-03b56a05747c">
PS: I deleted bunches of `buildkit-blob-*` manually to save `engine-archives`
### How was this patch tested?
Monitor GHA after merging.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6837 from pan3793/k8s-it.
Closes#6837
8bab514dc [Cheng Pan] [INFRA] Disable K8s CI image GHA cache
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- skip explicitly set the parameter `check-latest` of Github action `actions/setup-java` to false, which is its default value.
- docs: https://github.com/actions/setup-java?tab=readme-ov-file#check-latest
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6768 from bowenliang123/action-java-latest-false.
Closes#6768
a0ecd4688 [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
`Spark Engine On Kubernetes Integration Test` always fails after #6739
The default java version of Ubuntu 24.04 is OpenJDK 21: https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat-release-notes/39890#openjdk , this is a version higher than that supported by spark 3.5.X.
## Describe Your Solution 🔧
Specify JDK 11 for spark engine on kubernetes it.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
test ga: https://github.com/wForget/kyuubi/actions/runs/11441231714/job/31828818049
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6763 from wForget/minor.
Closes#6763
aee39339f [wforget] Specify JDK 11 for spark engine on kubernetes it
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- as titled
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6741 from bowenliang123/gha-upload-artifact-v4.
Closes#6741
83a0ee90e [Bowen Liang] update GHA actions/upload-artifact from v3 to v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- The Ubuntu 24.04 image for Actions is now generally available since Sep 2024 : https://github.blog/changelog/2024-09-25-actions-new-images-and-ubuntu-latest-changes/
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6739 from bowenliang123/ubuntu-2404.
Closes#6739
2d120d1f6 [Bowen Liang] bump ci image from ubuntu 22.04 to 24.04
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- inspired by comments from #6664 : https://github.com/apache/kyuubi/pull/6664#issuecomment-2344351119
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6687 from bowenliang123/mvn-args.
Closes#6687
6a9170b60 [Cheng Pan] Update .github/workflows/master.yml
3573131d8 [liangbowen] export MAVEN_ARGS
36ced0532 [liangbowen] update
a93fa28ee [liangbowen] MAVEN_ARGS
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- update GHA docker/build-push-action from v5 to v6 to support build mmary
https://docs.docker.com/build/ci/github-actions/build-summary/
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6740 from bowenliang123/docker-push-v6.
Closes#6740
674cc15fc [Bowen Liang] update GHA docker/build-push-action from v5 to v6 to support build summary
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Upgrade default flink version to 1.20.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6755 from wForget/minor.
Closes#6755
e968502a6 [wforget] comment
c0094c760 [wforget] comment
b74a0bb75 [wforget] comment
1aea2216c [wforget] Upgrade default flink version to 1.20.0
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 4.0.0-preview2 RC1 passed the vote
https://lists.apache.org/thread/4ctj2mlgs4q2yb4hdw2jy4z34p5yw2b1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6699 from pan3793/spark-4.0.0-preview2.
Closes#6699
2db1f645d [Cheng Pan] 4.0.0-preview2
42055bb1e [Cheng Pan] fix
d29c0ef83 [Cheng Pan] disable delta test
98d323b95 [Cheng Pan] fix
2e782c00b [Cheng Pan] log4j-slf4j2-impl
fde4bb6ba [Cheng Pan] spark-4.0.0-preview2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5314
## Describe Your Solution 🔧
To support Java 21, some dependency packages need to be upgraded.Here is the detailed explanation.
1. To build with Java 21, Scala binary version needs to be upgraded to 2.13.11 or above.
2. The versions of `spotless` and `googlejavaformat` need to be upgraded to higher versions. However, this will lose support for Java 8. In order to continue supporting Java 8, code style checks will be ignored in the Java 21 profile.
3. Since the Flink engine does not support Scala 2.13, the Scala version of the Flink module is fixed at 2.12.
Previous discussions were at https://github.com/apache/kyuubi/pull/6306.
---
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
**Be nice. Be informative.**
Closes#6597 from dupen01/issue-java21.
Closes#5314
0c5a9e455 [Cheng Pan] simplify
54cbadf79 [dupeng] java21+scala2.13.11+flink_2.12+CI(spark4,21,2,13)
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
To support Java 21, the Scala version needs to be upgraded to 2.13.11 or above.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6603 from dupen01/scala213.
Closes#6603
541f86054 [Cheng Pan] exclude SparkLocalClusterTest
13c11bf52 [Perl Du] add ignoreClasses comments
d5b67971b [Cheng Pan] Update pom.xml
582de9b90 [dupeng] ignore org.jline.terminal.impl.ffm.*
364eea00c [dupeng] upgrade maven.plugin.silencer.version from 1.7.13 to 1.7.17
99a1cc40a [dupeng] upgrade scala version to 2.13.14
54b0f4d3f [dupeng] upgrade scala version to 2.13.11
Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Perl Du <34719039+dupen01@users.noreply.github.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The current Hive 2.3.10 integration tests are broken, the root cause is CALCITE-1224
```
2024-08-20T12:20:55,405 ERROR [HiveTBinaryFrontendHandler-Pool: Thread-49] org.apache.kyuubi.util.KyuubiUncaughtExceptionHandler - Uncaught exception in thread HiveTBinaryFrontendHandler-Pool: Thread-49
java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonGenerator.writeStartArray(Ljava/lang/Object;I)V
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:78) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4485) ~[jackson-databind-2.12.0.jar:2.12.0]
at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3699) ~[jackson-databind-2.12.0.jar:2.12.0]
at org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:518) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:143) ~[hive-service-2.3.10.jar:2.3.10]
at org.apache.kyuubi.engine.hive.operation.HiveOperation.runInternal(HiveOperation.scala:62) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:175) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:103) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.engine.hive.session.HiveSessionImpl.runOperation(HiveSessionImpl.scala:59) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:133) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:84) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:130) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[kyuubi-hive-sql-engine_2.12-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_422]
```
## Describe Your Solution 🔧
Unpack `avatica-1.8.0.jar` and delete Jackson classes to workaround CALCITE-1224
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This PR should recover CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6635 from pan3793/hive-2310-fix.
Closes#6635
334bc5c39 [Cheng Pan] nit
22ae945fb [Cheng Pan] fix
dec03196d [Cheng Pan] Fix Hive 2.3.10 integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 3.5.2 was released recently.
Release Notes is available at https://spark.apache.org/releases/spark-release-3-5-2.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6609 from pan3793/spark-3.5.2.
Closes#6609
587cf1dd3 [Cheng Pan] Bump Spark 3.5.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
I found the [previous change](https://github.com/apache/kyuubi/pull/6606) breaks compatibility in some shell interpreters, for example `[ ! -f "$MVN_BIN" ]`, `[ $MVN_DETECTED_VERSION != $MVN_VERSION ]` are not supported in ubuntu dash, which fails the CI https://github.com/apache/kyuubi/actions/runs/10347069575/job/28636666290
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
This change should recover the CI.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6610 from pan3793/mvn-followup.
Closes#6606
5381b60ac [Cheng Pan] 1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR disables Javadoc/Scaladoc by default, while still enabling that on Nexus deploy and CI, to speed up the normal building by developers daily performed.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Apple M1 Max, Java 17
```
$ time build/mvn clean install -DskipTests
```
before: 1145.67s user 62.41s system 276% cpu 7:16.56 total
after: 530.33s user 35.82s system 214% cpu 4:23.69 total
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6599 from pan3793/javadoc.
Closes#6599
660f8dc02 [Cheng Pan] Disable Javadoc/Scaladoc by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
Flink 1.20 is out, [Release Note](https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
CI is updated to cover Flink 1.20
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6557 from pan3793/flink-1.20.
Closes#6557
a414094b9 [Cheng Pan] remove rc
8ee4cf5cd [Cheng Pan] fix url
fbaf66071 [Cheng Pan] docs
ddbd10ffe [Cheng Pan] Support Flink 1.20
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This pull request aims to remove building support for Spark 3.2, while still keeping the engine support for Spark 3.2.
Mailing list discussion: https://lists.apache.org/thread/l74n5zl1w7s0bmr5ovxmxq58yqy8hqzc
- Remove Maven profile `spark-3.2`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.5) still works well on Spark 3.2 runtime.
- Merge `kyuubi-extension-spark-common` into `kyuubi-extension-spark-3-3`
- Remove `log4j.properties` as Spark moves to Log4j2 since 3.3 (SPARK-37814)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6545 from pan3793/deprecate-spark-3.2.
Closes#6545
54c172528 [Cheng Pan] fix
f4602e805 [Cheng Pan] Deprecate and remove building support for Spark 3.2
2e083f89f [Cheng Pan] fix style
458a92c53 [Cheng Pan] nit
929e1df36 [Cheng Pan] Deprecate and remove building support for Spark 3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6537
## Describe Your Solution 🔧
Upgrade Flink from 1.19.0 to 1.19.1.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6538 from SteNicholas/KYUUBI-6537.
Closes#6537
b39d308d6 [SteNicholas] [KYUUBI #6537] Upgrade Flink from 1.19.0 to 1.19.1
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Bump minikube and kubernetes version of integration test to fix failure of integration test in CI.
- minikube: v1.29.0->v1.33.1
- kubernetes: v1.26.1->v1.30.0
Docker version 25 loads images into various distributions of Kubernetes has been impossible due to what appears to be mismatching hashes of manifests/images as follows:
```
X Exiting due to GUEST_IMAGE_LOAD: save to dir: caching images: caching image "/home/runner/.minikube/cache/images/amd64/apache/kyuubi_latest": write: unable to calculate manifest: blob sha256:5bbc241b2d6dcc55b5a63c080556ad458ba1395e93af3204d12e72c9a192eb06 not found
```
Minikube and kubernetes version should be bumped to fix failure of integration test which refers to https://github.com/moby/moby/issues/47207.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6466 from cfmcgrady/bump-minikube.
Closes#6466
c4bf4b56e [Fu Chen] Bump minikube and kubernetes version of integration test
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
# 🔍 Description
This PR makes KSHC support Spark 4.0, and also makes sure that the KSHC jar compiled against Spark 3.5 is binary compatible with Spark 4.0.
We are ready to enable CI for Spark 4.0, except for authZ module.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6453 from pan3793/spark4-ci.
Closes#6453
695e3d7f7 [Cheng Pan] Update pom.xml
2eaa0f88a [Cheng Pan] Update .github/workflows/master.yml
b1f540a34 [Cheng Pan] cross test
562839982 [Cheng Pan] fix
9f0c2e1be [Cheng Pan] fix
45f182462 [Cheng Pan] kshc
227ef5bae [Cheng Pan] fix
690a3b8b2 [Cheng Pan] Revert "fix"
87fe7678b [Cheng Pan] fix
60f55dbed [Cheng Pan] CI for Spark 4.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR adds Spark 4.0 cross-test for the Spark SQL engine - built against Spark 3.5, run with Spark 4.0.
As Spark 4.0 requires Java 17 or 21, the cross test uses Java 17.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6452 from pan3793/xtest-4.0.
Closes#6452
c49883756 [Cheng Pan] Add cross test for Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Kyuubi uses the Hudi Spark bundle jar in authZ module for testing, Hudi 0.15 brings Spark 3.5 and Scala 2.13 support, it also removes hacky for profile `spark-3.5`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6451 from pan3793/hudi-0.15.
Closes#6451
98d6e97c5 [Cheng Pan] fix
2d31307da [Cheng Pan] remove spark-authz-hudi-test
8896f8c3f [Cheng Pan] Enable hudi test
7e9a7c7ae [Cheng Pan] Bump Hudi 0.15.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request adds cross-version tests for Kyuubi Spark TPC-DS Connector and TPC-H Connector.
## Describe Your Solution 🔧
Add TPC-DS Connector and TPC-H Connector into GitHub Actions job `spark-connector-cross-version-test`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6441 from zhouyifan279/tcp-ds/h-cross-version.
Closes#6441
c2abc468a [zhouyifan279] Kyuubi Spark TPC-DS/H Connector cross version test
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request closes#6247
This also closes#6431
## Describe Your Solution 🔧
Add a job `spark-connector-cross-version-test` in GitHub Actions to:
1. Build KSHC package with maven opt `-Pspark-3.5`
2. Run KSHC tests with maven opt `-Pspark-3.3` and `-Pspark-3.4` and KSHC package built in step 1
3. Fix the binary-compatible issue via reflection.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6436 from zhouyifan279/kshc-cross-version-test.
Closes#6247
d3ac2ef47 [zhouyifan279] Tune the KSHC code to fix binary-compatible issues
4e14edcb5 [zhouyifan279] Fix invalid unit-tests-log name
56ca45d18 [zhouyifan279] Fix invalid unit-tests-log name
4c5ab7b9e [zhouyifan279] Update test log name
8a84e8812 [zhouyifan279] Add matrix scala
17cb67155 [zhouyifan279] [KYUUBI #6247] KSHC cross-version test
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
`DeltaTest` was duplicated, while the second one should be `PaimonTest`
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review and Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6423 from pan3793/excl-tag.
Closes#6423
c94218a5d [Cheng Pan] Fix Scalatest exclusion tags
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Hive 2.3.10 is available now, and it will be the latest version of the 2.3 serial.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6328 from pan3793/hive-2310.
Closes#6328
82a9fb70c [Cheng Pan] Update .github/workflows/master.yml
10af5e13b [Cheng Pan] hive-2.3.10-rc-1
2dbb2b54e [Cheng Pan] test Hive 2.3.10 RC0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Previously, we were asked to limit the job concurrency to 10, while the new policy is more relaxed.
> All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20.
https://infra.apache.org/github-actions-policy.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Monitor CI
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6363 from pan3793/ci-con.
Closes#6363
101121aa9 [Cheng Pan] [INFRA] Increase CI concurrency
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6323
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6333 from lsm1/branch-bump-spark-3.4.3.
Closes#6323
cc324fd77 [senmiaoliu] bump spark 3.4.3
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6250
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass CI
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6273 from liuxiaocs7/issue-6250.
Closes#6250
c6ba1e88a [liuxiao] remove unused import
db887ef9b [liuxiao] inline method
769da013b [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
21dbd37a7 [liuxiao] remove unused import
e869d571e [liuxiao] update for miss
7d755a879 [liuxiao] Drop support for Spark 3.1
Lead-authored-by: liuxiao <liuxiao2103@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6249.
## Describe Your Solution 🔧
According to the plan, we will no longer support Flink 1.16, and this PR will try to remove Flink 1.16.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6259 from slfan1989/drop_flink1.16_support.
Closes#6249
808acc2fb [Cheng Pan] Update docs/deployment/migration-guide.md
fc5ecf6fd [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
8d8f9ded4 [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
7a1d974b3 [Shilun Fan] [KYUUBI #6249] Drop support for Flink 1.16.
Lead-authored-by: Shilun Fan <slfan1989@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6134
## Describe Your Solution 🔧
Add profile and ci for flink 1.19.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6190 from wForget/KYUUBI-6134-FOLLOWUP.
Closes#6134
6c5c8824f [wforget] [KYUUBI #6134][FOLLOWUP] Add profile and ci for flink 1.19
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
`kyuubi-gluten-it` is always enabled and uses a different Spark version, which mess the code jumping in IDEA.
## Describe Your Solution 🔧
- conditional enable `kyuubi-gluten-it` module via profile `gluten-it`
- refactor POM and GHA workflow to reduce duplicated definition
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review, and wait for daily GHA results.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6178 from pan3793/gluten-it.
Closes#6178
77b7bb809 [Cheng Pan] fix
c62ce40f8 [Cheng Pan] Improve Gluten intergartion test
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
We already marked the support for Flink 1.16 as deprecated, this PR drops the building support for Flink 1.16, while we still keep the cross-version verification to ensure the Flink engine still works on Flink 1.16 runtime.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6171 from pan3793/drop-flink-116-profile.
Closes#6171
2dd74eca7 [Cheng Pan] Drop building support for Flink 1.16
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Kyuubi fully supports Spark 3.5 now, this pull request aims to set the default Spark to 3.5 in Kyuubi 1.9
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6163 from pan3793/spark-3.5-default.
Closes#6163
f386aeb7a [Cheng Pan] Set default Spark version to 3.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
We are asked by ASF Infra to limit the max-parallel up to 10 for each workflow.
Merge JDBC Trino TPC jobs in GitHub Actions CI to reduce concurrency of workflow, also increase max-parallel of Spark jobs from 1 to 3
Also merge "shell check" job into "style check" job
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6170 from pan3793/ci-2.
Closes#6170
9bf505321 [Cheng Pan] merge shell check
d0e7ece54 [Cheng Pan] nit
b17169573 [Cheng Pan] Tune CI
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Drop JDK 11 on GitHub Actions to save CI resources.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6167 from pan3793/ci-drop-java11.
Closes#6167
880af7f0e [Cheng Pan] Drop JDK 11 on GitHub Actions
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
We are asked by ASF Infra to set a max-parallel limit for each workflow.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Let's see the CI results.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6164 from pan3793/max-parallel.
Closes#6164
beb528159 [Cheng Pan] Limit max parallel for each GitHub Actions workflow
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Maven profile `spark-3.1` is removed.
## Describe Your Solution 🔧
Remove invalid matrix spark 3.1 in GA. Note, that the cross version verify still exists to ensure that the Spark SQL engine works well on the Spark 3.1 runtime
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6151 from pan3793/ci-3.1.
Closes#6151
f50de2efd [Cheng Pan] Remove invalid matrix spark 3.1 in GA
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request closes#6094
## Describe Your Solution 🔧
Upgrade Spark from 3.5.0 to 3.5.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6103 from GGHWS/features/kyuubi-6094.
Closes#6094
c44f2b879 [XH Qian] Upgrade Spark from 3.5.0 to 3.5.1
Authored-by: XH Qian <16678125728@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6098, https://github.com/apache/kyuubi/issues/6097
## Describe Your Solution 🔧
Upgrade flink from 1.16.2 to 1.16.3 and 1.18.0 to 1.18.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6105 from Deepali1211/master.
Closes#6097Closes#6098
0d3516b24 [Deepali Gupta] Update flink version
1543443d3 [Deepali Gupta] Update flink version
c95e3bfc0 [Deepali Gupta] Update flink version
Lead-authored-by: Deepali Gupta <g.deepali1211@gmail.com>
Co-authored-by: Deepali Gupta <90255953+Deepali1211@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Historically, we provided a few standalone YAML files to allow users to deploy Kyuubi on K8s, and later a Helm Chart was introduced, which is a simple and mature way for production use cases.
## Describe Your Solution 🔧
Remove the legacy standalone YAML files way and update docs to guide user to use Helm chart to deploy Kyuubi on K8s.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6080 from pan3793/k8s-cleanup.
Closes#6080
3a9fa9be3 [Cheng Pan] comment
8f681d28c [Cheng Pan] Improve instruction for deploy Kyuubi on K8s
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
I saw [the K8s IT](https://github.com/apache/kyuubi/actions/runs/7776268590/job/21203214430?pr=6045) failed with
```
Error: No space left on device : '/home/runner/runners/2.312.0/_diag/pages/131e7057-723d-4281-b296-b876237f4318_21774d93-c106-57cd-810b-f0e6bf96d865_1.log'
```
## Describe Your Solution 🔧
Borrow the solution from Apache Spark, just uninstall the un-used large pre-installed packages.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Wait CI report
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6047 from pan3793/ci-space.
Closes#6047
272df2790 [Cheng Pan] nit
8c78115a4 [Cheng Pan] nit
a9c8b035c [Cheng Pan] nit
ab4c3e461 [Cheng Pan] keep jdk8
9d1ec4e44 [Cheng Pan] extend
47f90b95e [Cheng Pan] nit
30b82cdf0 [Cheng Pan] fix
352689f89 [Cheng Pan] [CI] Free up disk space for K8s IT
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request canonicalizes Trino IT in GitHub Action workflow.
`mvn -am` means "also make", so that `mvn integration-tests/kyuubi-trino-it -am` would trigger compiling of `kyuubi-server`, `externals/kyuubi-spark-sql-engine`, `externals/kyuubi-download` modules automatically.
## Describe Your Solution 🔧
CI part change in 6688b3dacf is unnecessary.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#5978 from pan3793/trino-ci.
Closes#5978
1b8ef94d1 [Cheng Pan] set KYUUBI_HOME in trino-it
18e72d355 [Cheng Pan] Canonicalize Trino IT in GitHub Action workflow
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5856
## Describe Your Solution 🔧
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5859 from zml1206/KYUUBI-5856.
Closes#5856
872fd06d2 [zml1206] Revert changes in SparkProcessBuilderSuite
bc4996f90 [zml1206] Bump spark 3.4.2
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Currently, we always enable Jacoco, which would add additional building time and produce lots of HTML files, also increase the IDEA indexing burden.
<img width="1065" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/6276f309-8ba4-4563-96d6-91cb53ce20a5">
## Describe Your Solution 🔧
Add a new maven profile `codecov`, only enable it in CI with codecov enabled.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
See CI result.
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5837 from pan3793/codecov.
Closes#5837
97d0b22de [Cheng Pan] fi
c60367dd1 [Cheng Pan] fix
a0e8910bc [Cheng Pan] Only enable Jacoco with explicit profile
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- `setup-java` Github Action v4 release note: https://github.com/actions/setup-java/releases/tag/v4.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5839 from bowenliang123/actions-java-4.
Closes#5839
af76bf44c [Bowen Liang] Bump actions/setup-java from v3 to v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- `setup-python` Github Action v5 release note: https://github.com/actions/setup-python/releases/tag/v5.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5838 from bowenliang123/actions-python-5.
Closes#5838
f1703d690 [Bowen Liang] Bump actions/setup-python from v4 to v5
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
The CDH Hive 2.1.1-cdh6.3.2 is the latest free version of CDH, which is adopted widely, it would be great if the Kyuubi Hive engine could support this version.
## Describe Your Solution 🔧
Actually, the Kyuubi Hive engine could work with CDH Hive 2.1.1-cdh6.3.2, this pull request just adds an integration test and fixes one test case.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5830 from pan3793/hive-211-cdh6.
Closes#5830
afa80bd82 [Cheng Pan] nit
89c9ae96e [Cheng Pan] nit
e402e49cd [Cheng Pan] nit
3c126a2a6 [Cheng Pan] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
