# 🔍 Description
## Issue References 🔗
This pull request fixes#6524
## Describe Your Solution 🔧
Trino engine supports insecure configuration, just as trino client supports --insecure parameter
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6525 from jiaoqingbo/6524.
Closes#6524
b414b2e05 [jiaoqingbo] update settings.md
129d40742 [jiaoqingbo] [KYUUBI #6524] Trino engine supports insecure configuration
24f374b38 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6515
## Describe Your Solution 🔧
Bumped up dropwizard metrics from 4.2.23 to 4.2.26
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6527 from avishnus/metrics.
Closes#6515
253036ca3 [avishnus] Bumping up dropwizard metrics to 4.2.26
Authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6516
## Describe Your Solution 🔧
1. Using `buildStaticChecked` instead of `build` for static method `fromUri`
2. Using `Array.empty[Object]` instead of empty argument when invoking the build method
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Throw error like when running the unit test
```
java.lang.RuntimeException
at org.apache.kyuubi.util.reflect.DynMethods$UnboundMethod.invoke(DynMethods.java:80)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.buildURI(KyuubiSparkUtil.scala:143)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtilSuite.$anonfun$new$1(KyuubiSparkUtilSuite.scala:33)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.scalatest.funsuite.AnyFunSuite.run(AnyFunSuite.scala:1564)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
```
#### Behavior With This Pull Request 🎉
Test passed
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6517 from jeanlyn/issue-6516.
Closes#6516
c6bf8b622 [jeanlyn] adjust
e50e58531 [jeanlyn] adjust
8afdc0c97 [jeanlyn] fix
a61ae7012 [jeanlyn] fix buildURI error and adding unit tests
Authored-by: jeanlyn <me@jeanlyn.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
I'd like to introduce the feature that allows users to forcibly kill an engine through API.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6008 from zhaohehuhu/dev-0123.
Closes#6008
00c208a26 [Cheng Pan] fix
8721a2d2a [Cheng Pan] log
efc7587f7 [Cheng Pan] client
cd5129db3 [Cheng Pan] fix ut
5e1b6a161 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
72d7df357 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
6d5d08710 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala
b013194d1 [zhaohehuhu] move the position of log
0cdeede7a [zhaohehuhu] restore ENGINE_SPARK_REGISTER_ATTRIBUTES
f826d0515 [zhaohehuhu] reformat
a13466e37 [zhaohehuhu] update doc and log string encoded
3a2f5970a [zhaohehuhu] refactor
ae24ea74d [zhaohehuhu] refactor UT
936a54e27 [Wang, Fei] register app mgr info
9bacc2c8b [hezhao2] fix UTs
11106d75b [Wang, Fei] comments
ba57c2c3f [hezhao2] refactor code to delete the node and then kill application
634ceb677 [hezhao2] reformat
ab31382ee [hezhao2] reformat
513bcdc57 [hezhao2] fix UT
506220654 [hezhao2] get refId by user, sharelevel and subdomain
3ad9577df [hezhao2] rename params to support multiple engines
632c56b88 [hezhao2] fix unused import
bd7bb45f0 [hezhao2] refactor
fb9b25176 [hezhao2] add default value for forceKill param
070aad06f [hezhao2] refactor
51827ecde [hezhao2] fix UT
f11e7657e [hezhao2] add an UT
8a65cf113 [hezhao2] refactor code
d6f82ff9a [hezhao2] refactor code
f3ab9c546 [hezhao2] new parameter added to decide whether to kill forcefully handle the result of killApplication
5faa5b54f [hezhao2] kill engine forcibly
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: zhaohehuhu <luoyedeyi459@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6508
## Describe Your Solution 🔧
Add the key-value pairs in optimizedConf to session conf，to make the Kyuubi.env.SPARK_HOME take effect
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
When I configure the following parameters in kyuubi-env.sh
Then submit a spark batch task through the restful interface and pass the kyuubi.engineEnv.SPARK_HOME parameter to change the default value
```curl -H "Content-Type: application/json" -X POST -d '{"batchType": "SPARK", "resource":"xxxx/spark-examples_2.12-3.3.2.jar", "name": "Spark-PI", "conf": {"spark.master":"yarn","hive.server2.proxy.user":"XXXX","kyuubi.engineEnv.SPARK_HOME":"XXXXX/cluster114/spark","kyuubi.engineEnv.HADOOP_CONF_DIR":"XXXXX/conf"}, "args": [10],"className": "org.apache.spark.examples.SparkPi"}' http://XXXXX:XXXX/api/v1/batches```
Observe the log and find that SPARK_HOME is not set and takes effect. It still uses the value in kyuubi-env.sh.
#### Behavior With This Pull Request 🎉
After this PR, the update was successful
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6510 from jiaoqingbo/6508.
Closes#6508
e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Canonicalize the words, and enrich the description for KSCH.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6512 from pan3793/kshc-doc.
Closes#6512
201c11341 [Cheng Pan] nit
1becc1ebb [Cheng Pan] nit
8d48c7c93 [Cheng Pan] fix
aea1e0386 [Cheng Pan] fix
5ba5094ab [Cheng Pan] fix
0c40de43d [Cheng Pan] fix
63dd21d11 [Cheng Pan] nit
1be266163 [Cheng Pan] Improve docs for KSHC
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
[Good First Issues](https://kyuubi.readthedocs.io/en/master/contributing/code/get_started.html) image links to the Github issues page while the image on [Contributing Documentation](https://kyuubi.readthedocs.io/en/master/contributing/doc/get_started.html) does not.
## Describe Your Solution 🔧
Added a link to https://github.com/apache/kyuubi/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Adocumentation to match the behaviour from good first issues
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6514 from artilexx/patch-1.
Closes#6514
3d54ce982 [artilexx] Add link to github issues on documentation issues image
Lead-authored-by: artilexx <arkhanlewis@gmail.com>
Co-authored-by: artilexx <97213042+artilexx@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6506
## Describe Your Solution 🔧
Set the alias `--conf` to `--hiveconf`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
```
# use --hiveconf or --hivevar to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--hiveconf kyuubi.operation.result.format=arrow \
--hiveconf kyuubi.operation.incremental.collect=true-f \
--hivevar app_name=xxx \
xxx.sql
```
#### Behavior With This Pull Request 🎉
```
# use --conf to set some configurations
kyuubi-beeline -u 'jdbc:kyuubi://kyuubi:10009/' \
--conf kyuubi.operation.result.format=arrow \
--conf kyuubi.operation.incremental.collect=true-f \
--conf app_name=xxx \
xxx.sql
```
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6511 from BruceWong96/support-beeline-conf.
Closes#6506
7a81455d8 [Bruce Wong] fix code style.
8d1fec242 [Bruce Wong] delete some spaces.
d64505dd7 [Bruce Wong] [FEATURE] kyuubi-beeline supports --conf.
Authored-by: Bruce Wong <603334301@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Hive 2.3.0 (HIVE-7224) turns it on by default, while Kyuubi BeeLine is a fork from Hive BeeLine 3.1.3, we should update the help message to reflect the change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6507 from pan3793/beeline-incr.
Closes#6507
8b1ed655a [Cheng Pan] fix
7a872eeec [Cheng Pan] fix
88e751d3a [Cheng Pan] Correct Kyuubi BeeLine help message for incremental
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Update the outdated docs to mention new features:
1. Arrow serialization
2. Sava query results into files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Docs changes, review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6504 from pan3793/docs-large-result.
Closes#6504
77043bd4d [Cheng Pan] fix
e8ba1bac1 [Cheng Pan] grammar
198d00e33 [Cheng Pan] grammar
a37f631eb [Cheng Pan] Improve docs for Solution for Large Query Results
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Default Kyuubi version in the chart is not up to date with the latest release version.
## Describe Your Solution 🔧
Upgrade default version to the latest 1.9.1 to be up to date with the release version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### The chart deploys Kyuubi 1.9.1
Install the chart:
```sh
helm install kyuubi charts/kyuubi
```
Check version:
```sh
kubectl exec kyuubi-0 -- cat /opt/kyuubi/RELEASE
Kyuubi 1.9.1 (git revision c232a4b) built for
Java 1.8.0_412
Scala 2.12
Flink 1.17.2
Spark 3.5.1
Kyuubi Hadoop 3.3.6
Hive 3.1.3
Build flags:
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6493 from dnskr/helm-default-kyuubi-version-1.9.1.
Closes#6493
e9b74067c [dnskr] [K8S][HELM] Update default Kyuubi version to 1.9.1
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
# 🔍 Description
## Issue References 🔗
as title
This pull request closes #6255
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6339 from zhaohehuhu/dev-0426.
Closes#6339
38b36f260 [Cheng Pan] fix
b2e3fcf3d [Cheng Pan] rewrite
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
It was fixed in https://github.com/apache/kyuubi-shaded/pull/44
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6500 from pan3793/rm-hms-dt-identifiler.
Closes#6500
f535e3f1b [Cheng Pan] fix
47b32e3b7 [Cheng Pan] Remove org.apache.kyuubi.shaded.hive.metastore.security.DelegationTokenIdentifier
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR rewrites some utility methods in Java, specifically,
```
Utils.isWindows
Utils.isMac
Utils.findLocalInetAddress
```
and moves them from `kyuubi-common`'s `Utils` to the `kyuubi-util`'s `JavaUtils`, so that they could be used in other modules that do not depend on `kyuubi-common`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6499 from pan3793/javautils.
Closes#6499
565936def [Cheng Pan] fix
f06a85e9f [Cheng Pan] Move some untiliy methods in Java
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Small notice change to include that pyhive was originally developed at Dropbox.
Closes#6434 from bkyryliuk/bkyryliuk-patch-1.
Closes#6434
d2ed97b42 [Bogdan] Add footnote about pyhive origin
Authored-by: Bogdan <b.kyryliuk@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6489
## Describe Your Solution 🔧
After my investigation, I found the bug and solution.
The function get_table_names returns an incorrect value when I used Superset to connect to Kyuubi for Spark SQL.
[get_table_names](https://github.com/apache/kyuubi/blob/master/python/pyhive/sqlalchemy_hive.py#L380)
The following code is used to connect to hive directly.
`return [row[0] for row in connection.execute(text(query))]`
Because The following value is returned when the Hive is connected.
show tables in default :
[('student',), ('student_scores',)]
The following code is used to connect to Kyuubi.
`return [row[1] for row in connection.execute(text(query))]`
Because The following value is returned when the Kyuubi is connected.
show tables in default :
[('default', 'employees', False), ('default', 'student', False), ('default', 'student_scores', False)]
So, for the difference in return value, I modified the code.
And I test them in Superset. The code works.
Hive
<img width="1214" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/9048b21d-053e-4b5d-be35-ba29d3bd6848">
Kyuubi
<img width="1085" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/d600dfed-1127-41ea-a0bf-ca662a5487df">
Spark SQL also works properly.
<img width="1199" alt="image" src="https://github.com/apache/kyuubi/assets/29974394/7026e39e-6d63-473d-9e43-eeab580719ea">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6490 from BruceWong96/branch-kyuubi-6489.
Closes#6489
94a52c0e5 [wenjie.wang01] add else branch.
8ab20becf [wenjie.wang01] fix bug for function get_table_names.
136c7b795 [wenjie.wang01] fix bug for function get_table_names.
Authored-by: wenjie.wang01 <wenjie.wang01@liulishuo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6469
## Describe Your Solution 🔧
Instead of initializing all RecordReaderIterator when create OrcFileIterator，we can lazily initialize the RecordReaderIterator to make sure that there is only one RecordReaderIterator which reads file current fetching by client in driver memory.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
Closes#6470 from Z1Wu/bugfix-fetch-big-resultset-lazily.
Closes#6469
83208018c [吴梓溢] update
56284e68e [吴梓溢] update
Authored-by: 吴梓溢 <wuziyi02@corp.netease.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6476 : spark historyserver -> Show incomplete applications -> kyuubi query engine ui error（java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long）.
The reason:
it's related to https://github.com/FasterXML/jackson-module-scala/issues/62
## Describe Your Solution 🔧
add JsonDeserialize(contentAs = classOf[java.lang.Long]) annotation
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6479 from felixzh2020/issues/6476.
Closes#6476
034bfe53c [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
b7b0db278 [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
a66163a5a [felixzh] [KYUUBI apache#6476] spark historyserver Show incomplete applications kyuubi query engine ui error
Authored-by: felixzh <felixzh2020@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Bumps [ws](https://github.com/websockets/ws) from 8.8.1 to 8.17.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p>
<blockquote>
<h2>8.17.1</h2>
<h1>Bug fixes</h1>
<ul>
<li>Fixed a DoS vulnerability (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>).</li>
</ul>
<p>A request with a number of headers exceeding the[<code>server.maxHeadersCount</code>][]
threshold could be used to crash a ws server.</p>
<pre lang="js"><code>const http = require('http');
const WebSocket = require('ws');
<p>const wss = new WebSocket.Server({ port: 0 }, function () {
const chars = "!#$%&'*+-.0123456789abcdefghijklmnopqrstuvwxyz^_`|~".split('');
const headers = {};
let count = 0;</p>
<p>for (let i = 0; i < chars.length; i++) {
if (count === 2000) break;</p>
<pre><code>for (let j = 0; j &lt; chars.length; j++) {
const key = chars[i] + chars[j];
headers[key] = 'x';
if (++count === 2000) break;
}
</code></pre>
<p>}</p>
<p>headers.Connection = 'Upgrade';
headers.Upgrade = 'websocket';
headers['Sec-WebSocket-Key'] = 'dGhlIHNhbXBsZSBub25jZQ==';
headers['Sec-WebSocket-Version'] = '13';</p>
<p>const request = http.request({
headers: headers,
host: '127.0.0.1',
port: wss.address().port
});</p>
<p>request.end();
});
</code></pre></p>
<p>The vulnerability was reported by <a href="https://github.com/rrlapointe">Ryan LaPointe</a> in <a href="https://redirect.github.com/websockets/ws/issues/2230">websockets/ws#2230</a>.</p>
<p>In vulnerable versions of ws, the issue can be mitigated in the following ways:</p>
<ol>
<li>Reduce the maximum allowed length of the request headers using the
[<code>--max-http-header-size=size</code>][] and/or the [<code>maxHeaderSize</code>][] options so
that no more headers than the <code>server.maxHeadersCount</code> limit can be sent.</li>
</ol>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="3c56601092"><code>3c56601</code></a> [dist] 8.17.1</li>
<li><a href="e55e5106f1"><code>e55e510</code></a> [security] Fix crash when the Upgrade header cannot be read (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>)</li>
<li><a href="6a00029edd"><code>6a00029</code></a> [test] Increase code coverage</li>
<li><a href="ddfe4a804d"><code>ddfe4a8</code></a> [perf] Reduce the amount of <code>crypto.randomFillSync()</code> calls</li>
<li><a href="b73b11828d"><code>b73b118</code></a> [dist] 8.17.0</li>
<li><a href="29694a5905"><code>29694a5</code></a> [test] Use the <code>highWaterMark</code> variable</li>
<li><a href="934c9d6b93"><code>934c9d6</code></a> [ci] Test on node 22</li>
<li><a href="1817bac06e"><code>1817bac</code></a> [ci] Do not test on node 21</li>
<li><a href="96c9b3dedd"><code>96c9b3d</code></a> [major] Flip the default value of <code>allowSynchronousEvents</code> (<a href="https://redirect.github.com/websockets/ws/issues/2221">#2221</a>)</li>
<li><a href="e5f32c7e1e"><code>e5f32c7</code></a> [fix] Emit at most one event per event loop iteration (<a href="https://redirect.github.com/websockets/ws/issues/2218">#2218</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.8.1...8.17.1">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6484 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/ws-8.17.1.
Closes#6484
58a7da865 [dependabot[bot]] ⬆️ Bump ws from 8.8.1 to 8.17.1 in /kyuubi-server/web-ui
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/pull/6473#discussion_r1642652411
## Describe Your Solution 🔧
add a configuration to control whether to skip the cancellation here for incremental collect queries, skipping by default for safety.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6482 from XorSum/features/skip-cancel-incremental.
Closes#6302
440311f07 [xorsum] reformat
edbc37868 [bkhan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala
d6c99366c [xorsum] one line
9f40405c7 [xorsum] update configuration
b1526319e [xorsum] skip job group cancellation on incremental collect mode
Lead-authored-by: xorsum <xorsum@outlook.com>
Co-authored-by: bkhan <bkhan@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6302 : when the SQL is submitted by Kyuubi Rest Api (the same kyuubi path with BEELINE), the legacy job will not be cancelled, https://github.com/apache/kyuubi/issues/6302#issuecomment-2160572624 .
The reason:
Beeline session calls `cancelJobGroup` in `SparkOperation#cleanup`.
But Rest Api session doesn't call `SparkOperation#cleanup`, so the legacy job submitted by Rest Api will not be canceled.
## Describe Your Solution 🔧
The modification:
call `SparkOperation#cleanup` in `ExecuteStatement#executeStatement`'s finally clause.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6473 from XorSum/features/cancel-after-execute-stmt.
Closes#6302
16dd508e4 [xorsum] operation executeStatement cancel group
Authored-by: xorsum <xorsum@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Address comments https://github.com/apache/kyuubi/issues/4847#issuecomment-2114284381
Now, for `checkEngineConnectionAlive`, it use the client to send `TGetInfoType` to engine and cause the user session never idle for timeout.
## Describe Your Solution 🔧
We shall reuse the alive probe client.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Pass the current UT.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6468 from turboFei/engine_alive_check.
Closes#4847
e2368b206 [Wang, Fei] reuse
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Bump minikube and kubernetes version of integration test to fix failure of integration test in CI.
- minikube: v1.29.0->v1.33.1
- kubernetes: v1.26.1->v1.30.0
Docker version 25 loads images into various distributions of Kubernetes has been impossible due to what appears to be mismatching hashes of manifests/images as follows:
```
X Exiting due to GUEST_IMAGE_LOAD: save to dir: caching images: caching image "/home/runner/.minikube/cache/images/amd64/apache/kyuubi_latest": write: unable to calculate manifest: blob sha256:5bbc241b2d6dcc55b5a63c080556ad458ba1395e93af3204d12e72c9a192eb06 not found
```
Minikube and kubernetes version should be bumped to fix failure of integration test which refers to https://github.com/moby/moby/issues/47207.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6466 from cfmcgrady/bump-minikube.
Closes#6466
c4bf4b56e [Fu Chen] Bump minikube and kubernetes version of integration test
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
# 🔍 Description
## Issue References 🔗
The concurrency limit for the engine startup process is mainly used to avoid overload on the machine(or container) of the Kyuubi server, the current implementation holds startupProcessSemaphore until the session is established successfully. While for Spark on YARN cluster mode, some YARN queue resource insufficiency may block the subsequent Spark application submissions to other queues, significantly affecting the Kyuubi server's resource utilization.
## Describe Your Solution 🔧
We should immediately release the `startupProcessSemaphore` after the engine startup process exits (i.e., after the `spark-submit` process exits) as the load has already disappeared.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70%
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6463 from ic4y/master-p003.
Closes#6463
f7de68ce3 [ic4y] Improve code quality
d8b0248df [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency Limit
Authored-by: ic4y <ic4y@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
[`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues.
The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6458 from pan3793/commons-logging.
Closes#6458
114ec766a [Cheng Pan] fix
79d4121a1 [Cheng Pan] fix
6633e83ee [Cheng Pan] fix
21127ed0b [Cheng Pan] always perform install on dep check
98b13dfcf [Cheng Pan] Remove commons-logging from binary release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR removes two dependencies from the `kyuubi-rest-client` module
- `commons-collections` - has CVE Cx78f40514-81ff and is only used in one place, just rewrite to remove the dependency
- `javax.servlet-api` - only used for UT, correct the scope from `compile` to `test`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6457 from pan3793/rest-client-dep.
Closes#6457
c24af70ac [Cheng Pan] dep list
80969d0d3 [Cheng Pan] nit
950d4b730 [Cheng Pan] fix
169281903 [Cheng Pan] fix import
c068ba10f [Cheng Pan] nit
d46653d53 [Cheng Pan] fix
ca7831732 [Cheng Pan] Strip dependencies from REST client
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR follows SPARK-45508 to modify the JVM args by adding `--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED`, without this arg should not affect Spark engines as `spark-submit` will handle that by launching a sub-process and adding those JVM args.
The change benefits Spark in-process cases in UT, a little.
This PR also renames maven properties `maven.plugin.surefire.argLine` back to `extraJavaTestArgs` as it is used in both surefire and scalatest plugins, and makes it always takes effect instead of only JDK 17, then we don't need a copy when adding a new profile `java-21`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6456 from pan3793/add-open.
Closes#6456
ef7310a2e [Cheng Pan] add opens
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR makes KSHC support Spark 4.0, and also makes sure that the KSHC jar compiled against Spark 3.5 is binary compatible with Spark 4.0.
We are ready to enable CI for Spark 4.0, except for authZ module.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6453 from pan3793/spark4-ci.
Closes#6453
695e3d7f7 [Cheng Pan] Update pom.xml
2eaa0f88a [Cheng Pan] Update .github/workflows/master.yml
b1f540a34 [Cheng Pan] cross test
562839982 [Cheng Pan] fix
9f0c2e1be [Cheng Pan] fix
45f182462 [Cheng Pan] kshc
227ef5bae [Cheng Pan] fix
690a3b8b2 [Cheng Pan] Revert "fix"
87fe7678b [Cheng Pan] fix
60f55dbed [Cheng Pan] CI for Spark 4.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This is the root cause of #5957. Which is accidentally introduced in b315123a6b, thus affects 1.8.0, 1.8.1， 1.8.2, 1.9.0, 1.9.1.
`kyuubi-defaults.conf` is kind of a server side configuration file, all Kyuubi confs engine required should be passed via CLI args to sub-process.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6455 from pan3793/flink-conf-load.
Closes#5957
2972fbc98 [Cheng Pan] Flink engine should not load kyuubi-defaults.conf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
…pendencies
# 🔍 Description
## Issue References 🔗
This pull request fixes#6445
## Describe Your Solution 🔧
use `hive-pure-sasl` instead of `hive_pure_sasl` for extra name for optional distribution dependencies.
this avoid potential `WARNING: pyhive x.y.z does not provide the extra 'hive-pure-sasl'` and missing dependencies when a package depends on `pyhive[hive_pure_sasl]` and python build system choose to normalize it to `pyhive[hive-pure-saal]`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
a package depends on `pyhive[hive_pure_sasl]` may complain `WARNING: pyhive x.y.z does not provide the extra 'hive-pure-sasl'` and missing dependencies to support hive feature.
#### Behavior With This Pull Request 🎉
based on my test on our internal pypi, users use `pyhive[hive_pure_sasl]` are not affected by this change. But we should update README when we release a new version of pyhive.
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6454 from camper42/python.
Closes#6445
d82e5cae9 [camper42] [KYUUBI #6445] use normalized extra name for optional distribution dependencies
Authored-by: camper42 <camper.xlii@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6353
## Describe Your Solution 🔧
Catch exception for closing flink internal session.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6354 from wForget/KYUUBI-6353.
Closes#6353
32fc9afd9 [wforget] [KYUUBI #6353] Catch exception for closing flink internal session
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR adds Spark 4.0 cross-test for the Spark SQL engine - built against Spark 3.5, run with Spark 4.0.
As Spark 4.0 requires Java 17 or 21, the cross test uses Java 17.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6452 from pan3793/xtest-4.0.
Closes#6452
c49883756 [Cheng Pan] Add cross test for Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Improve data lake dependency management by extracting the following Maven properties:
- `delta.artifact`
- `hudi.artifact`
- `iceberg.artifact`
- `paimon.artifact`
It often takes a while for the downstream data lakes to support the new Spark versions, extracting those properties makes it easy to override in the new profile on the Kyuubi project's `pom.xml` to workaround before data lakes jars are available.
One use case is a19bb7c18e
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6427 from pan3793/datalake-dep.
Closes#6427
74a9300e0 [Cheng Pan] Improve datalake dependency management
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6447
## Describe Your Solution 🔧
Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6448 from lsm1/branch-kyuubi-6447.
Closes#6447
467066ce5 [senmiaoliu] Use static regex Pattern instances in JavaUtils
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Kyuubi uses the Hudi Spark bundle jar in authZ module for testing, Hudi 0.15 brings Spark 3.5 and Scala 2.13 support, it also removes hacky for profile `spark-3.5`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6451 from pan3793/hudi-0.15.
Closes#6451
98d6e97c5 [Cheng Pan] fix
2d31307da [Cheng Pan] remove spark-authz-hudi-test
8896f8c3f [Cheng Pan] Enable hudi test
7e9a7c7ae [Cheng Pan] Bump Hudi 0.15.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
The `kyuubi-util-scala_2.12-<version>-tests.jar` accidentally leaked to the compile scope but should be in the test scope.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Run `build/dist` and check `dist/jars`
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6439 from pan3793/util-scala-test.
Closes#6439
0576248f5 [Cheng Pan] fix
2bf2408f5 [Cheng Pan] fix
f7151dfc6 [Cheng Pan] kyuubi-util-scala test jar leaked to compile scope
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6437
## Describe Your Solution 🔧
Use `org.apache.hadoop.fs.Path` instead of `java.nio.file.Paths` to avoid `OPERATION_RESULT_SAVE_TO_FILE_DIR` scheme unexpected change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Spark Job failed to start with error: `java.io.IOException: JuiceFS initialized failed for jfs:///` with conf `kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp`.
`hdfs://xxx:port/tmp` may encounter similar errors
#### Behavior With This Pull Request 🎉
User Can use hdfs dir as `kyuubi.operation.result.saveToFile.dir` without error.
#### Related Unit Tests
Seems no test suites added in #5591 and #5986, I'll try to build a dist and test with our internal cluster.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6444 from camper42/save-to-hdfs.
Closes#6437
990f0a728 [camper42] [Kyuubi #6437] Fix Spark engine query result save to HDFS
Authored-by: camper42 <camper.xlii@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes the issue related to incorrect casing in `kubectl` commands used for creating `serviceaccount`.
## Describe Your Solution 🔧
This PR corrects the casing for the Kubernetes `serviceaccount` creation and rolebinding command to ensure compatibility with `kubectl`. Previously, the commands used incorrect casing (`serviceAccount`), which could potentially lead to execution errors or compatibility issues with Kubernetes API expectations.
Reference: [Kubernetes Official Documentation for Creating a ServiceAccount](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_serviceaccount/)
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6440 from huangzhir/doc-fix.
Closes#6440
5c657034d [huangzhir] Fix casing in kubectl commands for serviceaccount creation
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request adds cross-version tests for Kyuubi Spark TPC-DS Connector and TPC-H Connector.
## Describe Your Solution 🔧
Add TPC-DS Connector and TPC-H Connector into GitHub Actions job `spark-connector-cross-version-test`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6441 from zhouyifan279/tcp-ds/h-cross-version.
Closes#6441
c2abc468a [zhouyifan279] Kyuubi Spark TPC-DS/H Connector cross version test
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request closes#6247
This also closes#6431
## Describe Your Solution 🔧
Add a job `spark-connector-cross-version-test` in GitHub Actions to:
1. Build KSHC package with maven opt `-Pspark-3.5`
2. Run KSHC tests with maven opt `-Pspark-3.3` and `-Pspark-3.4` and KSHC package built in step 1
3. Fix the binary-compatible issue via reflection.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6436 from zhouyifan279/kshc-cross-version-test.
Closes#6247
d3ac2ef47 [zhouyifan279] Tune the KSHC code to fix binary-compatible issues
4e14edcb5 [zhouyifan279] Fix invalid unit-tests-log name
56ca45d18 [zhouyifan279] Fix invalid unit-tests-log name
4c5ab7b9e [zhouyifan279] Update test log name
8a84e8812 [zhouyifan279] Add matrix scala
17cb67155 [zhouyifan279] [KYUUBI #6247] KSHC cross-version test
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
`master` and `branch-1.9` already added basic support for Spark 4.0.0-preivew1.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6432 from pan3793/docs-4.0.
Closes#6432
73177aab3 [Cheng Pan] Mention support for Spark 4.0 in docs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes potential `Liveness probe failed: /bin/bash: line 1: bin/kyuubi: No such file or directory`
## Describe Your Solution 🔧
livenessProbe uses absolute path based on KYUUBI_HOME
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested on our internal kyuubi deployment.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6430 from camper42/liveness.
Closes#6430
980b42cdd [camper42] livenessProbe uses absolute path based on KYUUBI_HOME
Authored-by: camper42 <camper.xlii@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6428
## Describe Your Solution 🔧
Remove axios request timeout limitation.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Ref to #6428 description.
#### Behavior With This Pull Request 🎉
Get result after a period of time.
#### Related Unit Tests
kyuubi-server/web-ui: `pnpm test` all passed.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6429 from zRains/remove-axios-timeout.
Closes#6428
e947deabe [zRain] fix: remove timeout
Authored-by: zRain <2296342883@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR fixes tests in spark engine and kyuubi server modules with Spark 4.0.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Since Spark 4.0.0-preview1 is still under voting phase, this PR does not add CI, the change was tested in https://github.com/apache/kyuubi/pull/6407 with Spark 4.0.0-preview1 RC1
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6425 from pan3793/spark-4.
Closes#6425
101986416 [Cheng Pan] Fix tests in spark engine and kyuubi server modules with Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Merge standard error to output.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6422 from wForget/redirect.
Closes#6422
277d688d5 [wforget] Merge standard error to output
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
`DeltaTest` was duplicated, while the second one should be `PaimonTest`
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review and Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6423 from pan3793/excl-tag.
Closes#6423
c94218a5d [Cheng Pan] Fix Scalatest exclusion tags
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6396
## Describe Your Solution 🔧
NPE will occur when kyuubiClientTicketCache is not specified. `ConcurrentHashMap` does not allow null key.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6420 from wForget/hotfix2.
Closes#6396
e5d5f9d33 [wforget] address comment
fe9ecd1b1 [wforget] Avoid NPE
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
