# 🔍 Description
StatefulSet's `updateStrategy.rollingUpdate.maxUnavailable` is being used, but this feature is still in the alpha stage. It should not be used by default.
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features:~:text=%E2%80%93-,MaxUnavailableStatefulSet,-falsehttps://kubernetes.io/blog/2022/05/27/maxunavailable-for-statefulset/
## Issue References 🔗
None
## Describe Your Solution 🔧
I simply removed it. The behavior should not change even after the removal.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6550 from be-hase/polish-updateStrategy.
Closes#6550
ab31779fd [ryosuke-hasebe] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
Authored-by: ryosuke-hasebe <ryosuke.hasebe@linecorp.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6551
## Describe Your Solution 🔧
Update `canInsertZorder` to allow insert zorder when global sort is `false` and the plan is `Repartition` or `RepartitionByExpression`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
/kyuubi-extension-spark-common/src/test/scala/org/apache/spark/sql/ZorderSuiteBase.scala
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6552 from huangxiaopingRD/6551.
Closes#6551
b597443c3 [huangxiaoping] Fix code style
618594667 [huangxiaoping] [KYUUBI #6551] Allow insert zorder when when the plan is Repartition or RepartitionByExpression
Authored-by: huangxiaoping <1754789345@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
# 🔍 Description
This pull request aims to remove building support for Spark 3.2, while still keeping the engine support for Spark 3.2.
Mailing list discussion: https://lists.apache.org/thread/l74n5zl1w7s0bmr5ovxmxq58yqy8hqzc
- Remove Maven profile `spark-3.2`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.5) still works well on Spark 3.2 runtime.
- Merge `kyuubi-extension-spark-common` into `kyuubi-extension-spark-3-3`
- Remove `log4j.properties` as Spark moves to Log4j2 since 3.3 (SPARK-37814)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6545 from pan3793/deprecate-spark-3.2.
Closes#6545
54c172528 [Cheng Pan] fix
f4602e805 [Cheng Pan] Deprecate and remove building support for Spark 3.2
2e083f89f [Cheng Pan] fix style
458a92c53 [Cheng Pan] nit
929e1df36 [Cheng Pan] Deprecate and remove building support for Spark 3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Bump Kyuubi Shaded 0.4.1
The only change is #6496
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6543 from pan3793/shaded-041.
Closes#6543
3b674af95 [Cheng Pan] Remove staging repo
cae175517 [Cheng Pan] fix
549310067 [Cheng Pan] Bump Kyuubi Shaded 0.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- this bump helps to recognize Java 21-23 class formats: https://github.com/mojohaus/extra-enforcer-rules/releases
- reduce warning message when checking java bytecode versions
```
Warning: Unknown bytecodeVersion for com.fasterxml.jackson.core:jackson-core:jar:2.15.4:compile : META-INF/versions/21/com/fasterxml/jackson/core/io/doubleparser/FastDoubleSwar.class: got null class-file-version
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6548 from bowenliang123/mojo-rules-180.
Closes#6548
f53e4725d [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
This is similar to https://github.com/apache/kyuubi/pull/5310.
We need to wait for Jetty Server started before picking jobs, otherwise, it may get the wrong local HTTP server port -1.
This only affects Batch V2
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested in internal workloads.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6542 from pan3793/batch-wait-started.
Closes#6542
1f62debfe [Cheng Pan] fix
ee1d05d07 [Cheng Pan] KyuubiBatchService should wait for HTTP server started before picking jobs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6523
## Describe Your Solution 🔧
`KyuubiServer` respects configuration properties provided as `--conf` cli options with highest priority.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
REST endpoint starts at `10099` port:
```shell
bin/kyuubi run --conf kyuubi.frontend.rest.bind.port=9999
...
2024-07-10 17:57:24.998 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is started.
2024-07-10 17:57:24.999 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Exposing REST endpoint at: http://0.0.0.0:10099
2024-07-10 17:57:24.999 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started.
```
#### Behavior With This Pull Request 🎉
REST endpoint starts at `9999` port:
```shell
bin/kyuubi run --conf kyuubi.frontend.rest.bind.port=9999
...
2024-07-10 17:49:41.659 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is started.
2024-07-10 17:49:41.660 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Exposing REST endpoint at: http://0.0.0.0:9999
2024-07-10 17:49:41.660 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started.
```
#### Related Unit Tests
`kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala`
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6530 from dnskr/kyuubi-supports-conf-args.
Closes#6523
c7dbb305b [dnskr] Parse command and shift args
4977af168 [dnskr] [KYUUBI #6523] Support configuration with --conf options
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6534
## Describe Your Solution 🔧
check null ahead.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6540 from pan3793/client-npe.
Closes#6534
6976cd3f2 [Cheng Pan] [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6537
## Describe Your Solution 🔧
Upgrade Flink from 1.19.0 to 1.19.1.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6538 from SteNicholas/KYUUBI-6537.
Closes#6537
b39d308d6 [SteNicholas] [KYUUBI #6537] Upgrade Flink from 1.19.0 to 1.19.1
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This pull request fixes #
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6529 from avishnus/kubernetes.
Closes#6529
2a82df557 [Cheng Pan] Update dev/dependencyList
a4855467c [Cheng Pan] Update pom.xml
1ef0c40aa [Cheng Pan] Update dev/dependencyList
4b45b0038 [Cheng Pan] Update dev/dependencyList
d0c5174e8 [Cheng Pan] Update dev/dependencyList
bc2f6b8eb [Cheng Pan] Update dev/dependencyList
f76a7f487 [avishnus] Upgrading kubernetes-client
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6531
## Describe Your Solution 🔧
Fix SPARK-EngineTab stop/gracefulstop not work
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6533 from promising-forever/patch-1.
Closes#6531
66be0fdfd [promising-forever] [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
Authored-by: promising-forever <79634622+promising-forever@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6522
## Describe Your Solution 🔧
Remove comments in dispatch commands only execute command
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6526 from zcx513566/issues_6522.
Closes#6526
3dfd94364 [axiangzheng] [Bug] Use kyuubi beeline connect kyuubi server error if url contains '--' string
6677497ee [axiangzheng] [Bug] Use kyuubi beeline connect kyuubi server error if url contains '--' string
Authored-by: axiangzheng <axiangzheng@tencent.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Upgrade the Netty version to fix the CVEs
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
GHAs
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6528 from Madhukar525722/netty_upgrade.
Closes#6528
6c083c1fa [madlnu] Bump Netty 4.1.108.Final
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6446
## Describe Your Solution 🔧
Add tests for Spark saveToFile function
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6467 from lsm1/branch-kyuubi-6446.
Closes#6446
433f78cb4 [senmiaoliu] fix style
f16821c97 [senmiaoliu] add ut for spark engine save result
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
# 🔍 Description
## Issue References 🔗
SPARK-48238 replaced YARN AmIpFilter with a forked implementation, the code should be changed too.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6518 from pan3793/spark-4-url.
Closes#6518
c5026500b [Cheng Pan] Support extracting URL for Spark 4 on YARN
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6524
## Describe Your Solution 🔧
Trino engine supports insecure configuration, just as trino client supports --insecure parameter
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6525 from jiaoqingbo/6524.
Closes#6524
b414b2e05 [jiaoqingbo] update settings.md
129d40742 [jiaoqingbo] [KYUUBI #6524] Trino engine supports insecure configuration
24f374b38 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6515
## Describe Your Solution 🔧
Bumped up dropwizard metrics from 4.2.23 to 4.2.26
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6527 from avishnus/metrics.
Closes#6515
253036ca3 [avishnus] Bumping up dropwizard metrics to 4.2.26
Authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6516
## Describe Your Solution 🔧
1. Using `buildStaticChecked` instead of `build` for static method `fromUri`
2. Using `Array.empty[Object]` instead of empty argument when invoking the build method
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Throw error like when running the unit test
```
java.lang.RuntimeException
at org.apache.kyuubi.util.reflect.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.buildURI(KyuubiSparkUtil.scala:143)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtilSuite.$anonfun$new$1(KyuubiSparkUtilSuite.scala:33)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.scalatest.funsuite.AnyFunSuite.run(AnyFunSuite.scala:1564)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
```
#### Behavior With This Pull Request 🎉
Test passed
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6517 from jeanlyn/issue-6516.
Closes#6516
c6bf8b622 [jeanlyn] adjust
e50e58531 [jeanlyn] adjust
8afdc0c97 [jeanlyn] fix
a61ae7012 [jeanlyn] fix buildURI error and adding unit tests
Authored-by: jeanlyn <me@jeanlyn.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
I'd like to introduce the feature that allows users to forcibly kill an engine through API.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6008 from zhaohehuhu/dev-0123.
Closes#6008
00c208a26 [Cheng Pan] fix
8721a2d2a [Cheng Pan] log
efc7587f7 [Cheng Pan] client
cd5129db3 [Cheng Pan] fix ut
5e1b6a161 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
72d7df357 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
6d5d08710 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
b013194d1 [zhaohehuhu] move the position of log
0cdeede7a [zhaohehuhu] restore ENGINE_SPARK_REGISTER_ATTRIBUTES
f826d0515 [zhaohehuhu] reformat
a13466e37 [zhaohehuhu] update doc and log string encoded
3a2f5970a [zhaohehuhu] refactor
ae24ea74d [zhaohehuhu] refactor UT
936a54e27 [Wang, Fei] register app mgr info
9bacc2c8b [hezhao2] fix UTs
11106d75b [Wang, Fei] comments
ba57c2c3f [hezhao2] refactor code to delete the node and then kill application
634ceb677 [hezhao2] reformat
ab31382ee [hezhao2] reformat
513bcdc57 [hezhao2] fix UT
506220654 [hezhao2] get refId by user, sharelevel and subdomain
3ad9577df [hezhao2] rename params to support multiple engines
632c56b88 [hezhao2] fix unused import
bd7bb45f0 [hezhao2] refactor
fb9b25176 [hezhao2] add default value for forceKill param
070aad06f [hezhao2] refactor
51827ecde [hezhao2] fix UT
f11e7657e [hezhao2] add an UT
8a65cf113 [hezhao2] refactor code
d6f82ff9a [hezhao2] refactor code
f3ab9c546 [hezhao2] new parameter added to decide whether to kill forcefully handle the result of killApplication
5faa5b54f [hezhao2] kill engine forcibly
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: zhaohehuhu <luoyedeyi459@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6508
## Describe Your Solution 🔧
Add the key-value pairs in optimizedConf to session conf，to make the Kyuubi.env.SPARK_HOME take effect
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
When I configure the following parameters in kyuubi-env.sh
Then submit a spark batch task through the restful interface and pass the kyuubi.engineEnv.SPARK_HOME parameter to change the default value
```curl -H "Content-Type: application/json" -X POST -d '{"batchType": "SPARK", "resource":"xxxx/spark-examples_2.12-3.3.2.jar", "name": "Spark-PI", "conf": {"spark.master":"yarn","hive.server2.proxy.user":"XXXX","kyuubi.engineEnv.SPARK_HOME":"XXXXX/cluster114/spark","kyuubi.engineEnv.HADOOP_CONF_DIR":"XXXXX/conf"}, "args": [10],"className": "org.apache.spark.examples.SparkPi"}' http://XXXXX:XXXX/api/v1/batches```
Observe the log and find that SPARK_HOME is not set and takes effect. It still uses the value in kyuubi-env.sh.
#### Behavior With This Pull Request 🎉
After this PR, the update was successful
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6510 from jiaoqingbo/6508.
Closes#6508
e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Canonicalize the words, and enrich the description for KSCH.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6512 from pan3793/kshc-doc.
Closes#6512
201c11341 [Cheng Pan] nit
1becc1ebb [Cheng Pan] nit
8d48c7c93 [Cheng Pan] fix
aea1e0386 [Cheng Pan] fix
5ba5094ab [Cheng Pan] fix
0c40de43d [Cheng Pan] fix
63dd21d11 [Cheng Pan] nit
1be266163 [Cheng Pan] Improve docs for KSHC
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
[Good First Issues](https://kyuubi.readthedocs.io/en/master/contributing/code/get_started.html) image links to the Github issues page while the image on [Contributing Documentation](https://kyuubi.readthedocs.io/en/master/contributing/doc/get_started.html) does not.
## Describe Your Solution 🔧
Added a link to https://github.com/apache/kyuubi/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Adocumentation to match the behaviour from good first issues
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6514 from artilexx/patch-1.
Closes#6514
3d54ce982 [artilexx] Add link to github issues on documentation issues image
Lead-authored-by: artilexx <arkhanlewis@gmail.com>
Co-authored-by: artilexx <97213042+artilexx@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6506
## Describe Your Solution 🔧
Set the alias `--conf` to `--hiveconf`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
```
# use --hiveconf or --hivevar to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--hiveconf kyuubi.operation.result.format=arrow \
--hiveconf kyuubi.operation.incremental.collect=true-f \
--hivevar app_name=xxx \
xxx.sql
```
#### Behavior With This Pull Request 🎉
```
# use --conf to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--conf kyuubi.operation.result.format=arrow \
--conf kyuubi.operation.incremental.collect=true-f \
--conf app_name=xxx \
xxx.sql
```
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6511 from BruceWong96/support-beeline-conf.
Closes#6506
7a81455d8 [Bruce Wong] fix code style.
8d1fec242 [Bruce Wong] delete some spaces.
d64505dd7 [Bruce Wong] [FEATURE] kyuubi-beeline supports --conf.
Authored-by: Bruce Wong <603334301@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Hive 2.3.0 (HIVE-7224) turns it on by default, while Kyuubi BeeLine is a fork from Hive BeeLine 3.1.3, we should update the help message to reflect the change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6507 from pan3793/beeline-incr.
Closes#6507
8b1ed655a [Cheng Pan] fix
7a872eeec [Cheng Pan] fix
88e751d3a [Cheng Pan] Correct Kyuubi BeeLine help message for incremental
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Update the outdated docs to mention new features:
1. Arrow serialization
2. Sava query results into files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Docs changes, review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6504 from pan3793/docs-large-result.
Closes#6504
77043bd4d [Cheng Pan] fix
e8ba1bac1 [Cheng Pan] grammar
198d00e33 [Cheng Pan] grammar
a37f631eb [Cheng Pan] Improve docs for Solution for Large Query Results
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Default Kyuubi version in the chart is not up to date with the latest release version.
## Describe Your Solution 🔧
Upgrade default version to the latest 1.9.1 to be up to date with the release version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### The chart deploys Kyuubi 1.9.1
Install the chart:
```sh
helm install kyuubi charts/kyuubi
```
Check version:
```sh
kubectl exec kyuubi-0 -- cat /opt/kyuubi/RELEASE
Kyuubi 1.9.1 (git revision c232a4b) built for
Java 1.8.0_412
Scala 2.12
Flink 1.17.2
Spark 3.5.1
Kyuubi Hadoop 3.3.6
Hive 3.1.3
Build flags:
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6493 from dnskr/helm-default-kyuubi-version-1.9.1.
Closes#6493
e9b74067c [dnskr] [K8S][HELM] Update default Kyuubi version to 1.9.1
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
# 🔍 Description
## Issue References 🔗
as title
This pull request closes #6255
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6339 from zhaohehuhu/dev-0426.
Closes#6339
38b36f260 [Cheng Pan] fix
b2e3fcf3d [Cheng Pan] rewrite
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
It was fixed in https://github.com/apache/kyuubi-shaded/pull/44
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6500 from pan3793/rm-hms-dt-identifiler.
Closes#6500
f535e3f1b [Cheng Pan] fix
47b32e3b7 [Cheng Pan] Remove org.apache.kyuubi.shaded.hive.metastore.security.DelegationTokenIdentifier
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR rewrites some utility methods in Java, specifically,
```
Utils.isWindows
Utils.isMac
Utils.findLocalInetAddress
```
and moves them from `kyuubi-common`'s `Utils` to the `kyuubi-util`'s `JavaUtils`, so that they could be used in other modules that do not depend on `kyuubi-common`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6499 from pan3793/javautils.
Closes#6499
565936def [Cheng Pan] fix
f06a85e9f [Cheng Pan] Move some untiliy methods in Java
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Small notice change to include that pyhive was originally developed at Dropbox.
Closes#6434 from bkyryliuk/bkyryliuk-patch-1.
Closes#6434
d2ed97b42 [Bogdan] Add footnote about pyhive origin
Authored-by: Bogdan <b.kyryliuk@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6489
## Describe Your Solution 🔧
After my investigation, I found the bug and solution.
The function get_table_names returns an incorrect value when I used Superset to connect to Kyuubi for Spark SQL.
[get_table_names](https://github.com/apache/kyuubi/blob/master/python/pyhive/sqlalchemy_hive.py#L380)
The following code is used to connect to hive directly.
`return [row[0] for row in connection.execute(text(query))]`
Because The following value is returned when the Hive is connected.
show tables in default :
[('student',), ('student_scores',)]
The following code is used to connect to Kyuubi.
`return [row[1] for row in connection.execute(text(query))]`
Because The following value is returned when the Kyuubi is connected.
show tables in default :
[('default', 'employees', False), ('default', 'student', False), ('default', 'student_scores', False)]
So, for the difference in return value, I modified the code.
And I test them in Superset. The code works.
Hive
<img width="1214" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/9048b21d-053e-4b5d-be35-ba29d3bd6848">
Kyuubi
<img width="1085" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/d600dfed-1127-41ea-a0bf-ca662a5487df">
Spark SQL also works properly.
<img width="1199" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/7026e39e-6d63-473d-9e43-eeab580719ea">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6490 from BruceWong96/branch-kyuubi-6489.
Closes#6489
94a52c0e5 [wenjie.wang01] add else branch.
8ab20becf [wenjie.wang01] fix bug for function get_table_names.
136c7b795 [wenjie.wang01] fix bug for function get_table_names.
Authored-by: wenjie.wang01 <wenjie.wang01@liulishuo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6469
## Describe Your Solution 🔧
Instead of initializing all RecordReaderIterator when create OrcFileIterator，we can lazily initialize the RecordReaderIterator to make sure that there is only one RecordReaderIterator which reads file current fetching by client in driver memory.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
Closes#6470 from Z1Wu/bugfix-fetch-big-resultset-lazily.
Closes#6469
83208018c [吴梓溢] update
56284e68e [吴梓溢] update
Authored-by: 吴梓溢 <wuziyi02@corp.netease.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6476 : spark historyserver -> Show incomplete applications -> kyuubi query engine ui error（java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long）.
The reason:
it's related to https://github.com/FasterXML/jackson-module-scala/issues/62
## Describe Your Solution 🔧
add JsonDeserialize(contentAs = classOf[java.lang.Long]) annotation
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6479 from felixzh2020/issues/6476.
Closes#6476
034bfe53c [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
b7b0db278 [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
a66163a5a [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
Authored-by: felixzh <felixzh2020@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Bumps [ws](https://github.com/websockets/ws) from 8.8.1 to 8.17.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p>
<blockquote>
<h2>8.17.1</h2>
<h1>Bug fixes</h1>
<ul>
<li>Fixed a DoS vulnerability (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>).</li>
</ul>
<p>A request with a number of headers exceeding the[<code>server.maxHeadersCount</code>][]
threshold could be used to crash a ws server.</p>
<pre lang="js"><code>const http = require('http');
const WebSocket = require('ws');
<p>const wss = new WebSocket.Server({ port: 0 }, function () {
const chars = "!#$%&'*+-.0123456789abcdefghijklmnopqrstuvwxyz^_`|~".split('');
const headers = {};
let count = 0;</p>
<p>for (let i = 0; i < chars.length; i++) {
if (count === 2000) break;</p>
<pre><code>for (let j = 0; j &lt; chars.length; j++) {
const key = chars[i] + chars[j];
headers[key] = 'x';
if (++count === 2000) break;
}
</code></pre>
<p>}</p>
<p>headers.Connection = 'Upgrade';
headers.Upgrade = 'websocket';
headers['Sec-WebSocket-Key'] = 'dGhlIHNhbXBsZSBub25jZQ==';
headers['Sec-WebSocket-Version'] = '13';</p>
<p>const request = http.request({
headers: headers,
host: '127.0.0.1',
port: wss.address().port
});</p>
<p>request.end();
});
</code></pre></p>
<p>The vulnerability was reported by <a href="https://github.com/rrlapointe">Ryan LaPointe</a> in <a href="https://redirect.github.com/websockets/ws/issues/2230">websockets/ws#2230</a>.</p>
<p>In vulnerable versions of ws, the issue can be mitigated in the following ways:</p>
<ol>
<li>Reduce the maximum allowed length of the request headers using the
[<code>--max-http-header-size=size</code>][] and/or the [<code>maxHeaderSize</code>][] options so
that no more headers than the <code>server.maxHeadersCount</code> limit can be sent.</li>
</ol>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="3c56601092"><code>3c56601</code></a> [dist] 8.17.1</li>
<li><a href="e55e5106f1"><code>e55e510</code></a> [security] Fix crash when the Upgrade header cannot be read (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>)</li>
<li><a href="6a00029edd"><code>6a00029</code></a> [test] Increase code coverage</li>
<li><a href="ddfe4a804d"><code>ddfe4a8</code></a> [perf] Reduce the amount of <code>crypto.randomFillSync()</code> calls</li>
<li><a href="b73b11828d"><code>b73b118</code></a> [dist] 8.17.0</li>
<li><a href="29694a5905"><code>29694a5</code></a> [test] Use the <code>highWaterMark</code> variable</li>
<li><a href="934c9d6b93"><code>934c9d6</code></a> [ci] Test on node 22</li>
<li><a href="1817bac06e"><code>1817bac</code></a> [ci] Do not test on node 21</li>
<li><a href="96c9b3dedd"><code>96c9b3d</code></a> [major] Flip the default value of <code>allowSynchronousEvents</code> (<a href="https://redirect.github.com/websockets/ws/issues/2221">#2221</a>)</li>
<li><a href="e5f32c7e1e"><code>e5f32c7</code></a> [fix] Emit at most one event per event loop iteration (<a href="https://redirect.github.com/websockets/ws/issues/2218">#2218</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.8.1...8.17.1">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6484 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/ws-8.17.1.
Closes#6484
58a7da865 [dependabot[bot]] ⬆️ Bump ws from 8.8.1 to 8.17.1 in /kyuubi-server/web-ui
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/pull/6473#discussion_r1642652411
## Describe Your Solution 🔧
add a configuration to control whether to skip the cancellation here for incremental collect queries, skipping by default for safety.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6482 from XorSum/features/skip-cancel-incremental.
Closes#6302
440311f07 [xorsum] reformat
edbc37868 [bkhan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
d6c99366c [xorsum] one line
9f40405c7 [xorsum] update configuration
b1526319e [xorsum] skip job group cancellation on incremental collect mode
Lead-authored-by: xorsum <xorsum@outlook.com>
Co-authored-by: bkhan <bkhan@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6302 : when the SQL is submitted by Kyuubi Rest Api (the same kyuubi path with BEELINE), the legacy job will not be cancelled, https://github.com/apache/kyuubi/issues/6302#issuecomment-2160572624 .
The reason:
Beeline session calls `cancelJobGroup` in `SparkOperation#cleanup`.
But Rest Api session doesn't call `SparkOperation#cleanup`, so the legacy job submitted by Rest Api will not be canceled.
## Describe Your Solution 🔧
The modification:
call `SparkOperation#cleanup` in `ExecuteStatement#executeStatement`'s finally clause.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6473 from XorSum/features/cancel-after-execute-stmt.
Closes#6302
16dd508e4 [xorsum] operation executeStatement cancel group
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Address comments https://github.com/apache/kyuubi/issues/4847#issuecomment-2114284381
Now, for `checkEngineConnectionAlive`, it use the client to send `TGetInfoType` to engine and cause the user session never idle for timeout.
## Describe Your Solution 🔧
We shall reuse the alive probe client.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Pass the current UT.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6468 from turboFei/engine_alive_check.
Closes#4847
e2368b206 [Wang, Fei] reuse
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Bump minikube and kubernetes version of integration test to fix failure of integration test in CI.
- minikube: v1.29.0->v1.33.1
- kubernetes: v1.26.1->v1.30.0
Docker version 25 loads images into various distributions of Kubernetes has been impossible due to what appears to be mismatching hashes of manifests/images as follows:
```
X Exiting due to GUEST_IMAGE_LOAD: save to dir: caching images: caching image "/home/runner/.minikube/cache/images/amd64/apache/kyuubi_latest": write: unable to calculate manifest: blob sha256:5bbc241b2d6dcc55b5a63c080556ad458ba1395e93af3204d12e72c9a192eb06 not found
```
Minikube and kubernetes version should be bumped to fix failure of integration test which refers to https://github.com/moby/moby/issues/47207.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6466 from cfmcgrady/bump-minikube.
Closes#6466
c4bf4b56e [Fu Chen] Bump minikube and kubernetes version of integration test
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
# 🔍 Description
## Issue References 🔗
The concurrency limit for the engine startup process is mainly used to avoid overload on the machine(or container) of the Kyuubi server, the current implementation holds startupProcessSemaphore until the session is established successfully. While for Spark on YARN cluster mode, some YARN queue resource insufficiency may block the subsequent Spark application submissions to other queues, significantly affecting the Kyuubi server's resource utilization.
## Describe Your Solution 🔧
We should immediately release the `startupProcessSemaphore` after the engine startup process exits (i.e., after the `spark-submit` process exits) as the load has already disappeared.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70%
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6463 from ic4y/master-p003.
Closes#6463
f7de68ce3 [ic4y] Improve code quality
d8b0248df [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency Limit
Authored-by: ic4y <ic4y@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
[`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues.
The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6458 from pan3793/commons-logging.
Closes#6458
114ec766a [Cheng Pan] fix
79d4121a1 [Cheng Pan] fix
6633e83ee [Cheng Pan] fix
21127ed0b [Cheng Pan] always perform install on dep check
98b13dfcf [Cheng Pan] Remove commons-logging from binary release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR removes two dependencies from the `kyuubi-rest-client` module
- `commons-collections` - has CVE Cx78f40514-81ff and is only used in one place, just rewrite to remove the dependency
- `javax.servlet-api` - only used for UT, correct the scope from `compile` to `test`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6457 from pan3793/rest-client-dep.
Closes#6457
c24af70ac [Cheng Pan] dep list
80969d0d3 [Cheng Pan] nit
950d4b730 [Cheng Pan] fix
169281903 [Cheng Pan] fix import
c068ba10f [Cheng Pan] nit
d46653d53 [Cheng Pan] fix
ca7831732 [Cheng Pan] Strip dependencies from REST client
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR follows SPARK-45508 to modify the JVM args by adding `--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED`, without this arg should not affect Spark engines as `spark-submit` will handle that by launching a sub-process and adding those JVM args.
The change benefits Spark in-process cases in UT, a little.
This PR also renames maven properties `maven.plugin.surefire.argLine` back to `extraJavaTestArgs` as it is used in both surefire and scalatest plugins, and makes it always takes effect instead of only JDK 17, then we don't need a copy when adding a new profile `java-21`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6456 from pan3793/add-open.
Closes#6456
ef7310a2e [Cheng Pan] add opens
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR makes KSHC support Spark 4.0, and also makes sure that the KSHC jar compiled against Spark 3.5 is binary compatible with Spark 4.0.
We are ready to enable CI for Spark 4.0, except for authZ module.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6453 from pan3793/spark4-ci.
Closes#6453
695e3d7f7 [Cheng Pan] Update pom.xml
2eaa0f88a [Cheng Pan] Update .github/workflows/master.yml
b1f540a34 [Cheng Pan] cross test
562839982 [Cheng Pan] fix
9f0c2e1be [Cheng Pan] fix
45f182462 [Cheng Pan] kshc
227ef5bae [Cheng Pan] fix
690a3b8b2 [Cheng Pan] Revert "fix"
87fe7678b [Cheng Pan] fix
60f55dbed [Cheng Pan] CI for Spark 4.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This is the root cause of #5957. Which is accidentally introduced in b315123a6b, thus affects 1.8.0, 1.8.1， 1.8.2, 1.9.0, 1.9.1.
`kyuubi-defaults.conf` is kind of a server side configuration file, all Kyuubi confs engine required should be passed via CLI args to sub-process.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6455 from pan3793/flink-conf-load.
Closes#5957
2972fbc98 [Cheng Pan] Flink engine should not load kyuubi-defaults.conf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
…pendencies
# 🔍 Description
## Issue References 🔗
This pull request fixes#6445
## Describe Your Solution 🔧
use `hive-pure-sasl` instead of `hive_pure_sasl` for extra name for optional distribution dependencies.
this avoid potential `WARNING: pyhive x.y.z does not provide the extra 'hive-pure-sasl'` and missing dependencies when a package depends on `pyhive[hive_pure_sasl]` and python build system choose to normalize it to `pyhive[hive-pure-saal]`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
a package depends on `pyhive[hive_pure_sasl]` may complain `WARNING: pyhive x.y.z does not provide the extra 'hive-pure-sasl'` and missing dependencies to support hive feature.
#### Behavior With This Pull Request 🎉
based on my test on our internal pypi, users use `pyhive[hive_pure_sasl]` are not affected by this change. But we should update README when we release a new version of pyhive.
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6454 from camper42/python.
Closes#6445
d82e5cae9 [camper42] [KYUUBI #6445] use normalized extra name for optional distribution dependencies
Authored-by: camper42 <camper.xlii@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6353
## Describe Your Solution 🔧
Catch exception for closing flink internal session.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6354 from wForget/KYUUBI-6353.
Closes#6353
32fc9afd9 [wforget] [KYUUBI #6353] Catch exception for closing flink internal session
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR adds Spark 4.0 cross-test for the Spark SQL engine - built against Spark 3.5, run with Spark 4.0.
As Spark 4.0 requires Java 17 or 21, the cross test uses Java 17.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6452 from pan3793/xtest-4.0.
Closes#6452
c49883756 [Cheng Pan] Add cross test for Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
