# 🔍 Description
## Issue References 🔗
This pull request fixes#6250
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass CI
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6273 from liuxiaocs7/issue-6250.
Closes#6250
c6ba1e88a [liuxiao] remove unused import
db887ef9b [liuxiao] inline method
769da013b [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala
21dbd37a7 [liuxiao] remove unused import
e869d571e [liuxiao] update for miss
7d755a879 [liuxiao] Drop support for Spark 3.1
Lead-authored-by: liuxiao <liuxiao2103@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6249.
## Describe Your Solution 🔧
According to the plan, we will no longer support Flink 1.16, and this PR will try to remove Flink 1.16.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6259 from slfan1989/drop_flink1.16_support.
Closes#6249
808acc2fb [Cheng Pan] Update docs/deployment/migration-guide.md
fc5ecf6fd [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
8d8f9ded4 [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
7a1d974b3 [Shilun Fan] [KYUUBI #6249] Drop support for Flink 1.16.
Lead-authored-by: Shilun Fan <slfan1989@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Discussion in mailing thread: https://lists.apache.org/thread/tnmz71o3rypy7qvs3899p3jkkq4xqb4r
I propose to rename the `bin/beeline` to `bin/kyuubi-beeline`, while for compatibility, we may still want to keep the alias `bin/beeline` for a while.
In a real Hadoop cluster, it’s likely to add `$HIVE_HOME/bin`, `$SPARK_HOME/bin`, `$KYUUBI_HOME/bin` to the `$PATH`, at the current state, when performing `beeline`, which one is called depends on the declaration order.
It does not matter for Spark’s `bin/beeline` because it’s a vanilla Hive BeeLine, but in Kyuubi, we have made some improvements based on vanilla Hive BeeLine, so the behavior is not exactly same as Hive’s BeeLine.
An identical name would solve this problem. And I saw some vendors[1] who shippes Kyuubi already have done the same thing.
[1] https://help.aliyun.com/zh/emr/emr-on-ecs/user-guide/connect-to-kyuubi
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manual test.
```
$ bin/beeline -u 'jdbc:kyuubi://0.0.0.0:10009/'
Warning: beeline is deprecated and will be removed in the future, please use kyuubi-beeline instead.
Connecting to jdbc:kyuubi://0.0.0.0:10009/
Connected to: Spark SQL (version 3.4.1)
Driver: Kyuubi Project Hive JDBC Client (version 1.10.0-SNAPSHOT)
Beeline version 1.10.0-SNAPSHOT by Apache Kyuubi
0: jdbc:kyuubi://0.0.0.0:10009/>
```
```
$ bin/kyuubi-beeline -u 'jdbc:kyuubi://0.0.0.0:10009/'
Connecting to jdbc:kyuubi://0.0.0.0:10009/
Connected to: Spark SQL (version 3.4.1)
Driver: Kyuubi Project Hive JDBC Client (version 1.10.0-SNAPSHOT)
Beeline version 1.10.0-SNAPSHOT by Apache Kyuubi
0: jdbc:kyuubi://0.0.0.0:10009/>
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6239 from pan3793/kyuubi-beeline.
Closes#6239
cec8f56e2 [Cheng Pan] docs
b3446baf1 [Cheng Pan] docs
46a115077 [Cheng Pan] Remove `bin/beeline` to `bin/kyuubi-beeline`
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request add a message to tell users that Flink 1.16 is deprecated.
Disscusion: https://lists.apache.org/thread/fmq7462vcfm8l3m38xh4b4jjgd4smzn7
## Describe Your Solution 🔧
Add a message to tell users that Flink 1.16 is deprecated.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6142 from zhouyifan279/dep-flink-1.16.
Closes#6142
e38a4810c [Cheng Pan] nit
d03b106cf [zhouyifan279] Deprecate Flink 1.16
Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This PR upgrades Kyuubi Shaded to 0.3.0, the major changes are:
- upgrade Thrift from 0.9.3-1 to 0.16.0
- upgrade JLine used by Zookeeper CLI from 0.9.94 to 2.14.6
- split kyuubi-relocated-thrift from kyuubi-relocated-hive-service-rpc
- introduce a light kyuubi-relocated-hive-metastore-client for getting DelegationToken
Due to THRIFT-5297 (fixed in 0.14.0) removes requestTimeout and beBackoffSlotLength, the following Kyuubi configurations are removed too.
- kyuubi.frontend.login.timeout
- kyuubi.frontend.thrift.login.timeout
- kyuubi.frontend.backoff.slot.length
- kyuubi.frontend.thrift.backoff.slot.length
Previously, Zookeeper 3.4 depends on JLine 0.9.94 while Zookeeper 3.6 and Hive Beeline depends on JLine 2.14.6, we pulls different versions of JLine into the binary distribution. Now we align them to a single JLine 2.14.6.
## Describe Your Solution 🔧
Upgrade dependencies and change code to adapt to the breaking changes, also upgrade migration guide to mention the removed configurations.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6117 from pan3793/shaded-0.3.
Closes#6117
5639c9fd8 [Cheng Pan] nit
d281cdb4b [Cheng Pan] fix
184e1b95b [Cheng Pan] fix
1628337c8 [Cheng Pan] remove unused conf
24db2d5ad [Cheng Pan] try fix
1e995bb34 [Cheng Pan] nit
1d8e9bce9 [Cheng Pan] dep list
7f33624e8 [Cheng Pan] Bump Kyuubi Shaded 0.3.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request aims to remove building support for Spark 3.1, while still keeping the engine support for Spark 3.1.
- VOTE: https://lists.apache.org/thread/670fx1qx7rm0vpvk8k8094q2d0fthw5b
- VOTE RESULT: https://lists.apache.org/thread/0zdxg5zjnc1wpxmw9mgtsxp1ywqt6qvb
The next step is to clean up code in Spark extensions to drop 3.1-related code.
## Describe Your Solution 🔧
- Remove Maven profile `spark-3.1`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.4) still works well on Spark 3.1 runtime.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6091 from pan3793/remove-spark-3.1-profile.
Closes#6091
ce2983284 [Cheng Pan] nit
5887c808b [Cheng Pan] migration guide
cf28096d3 [Cheng Pan] Log deprecation message on Spark SQL engine with 3.1
a467e618d [Cheng Pan] nit
e11c0fb31 [Cheng Pan] Remove building support for Spark 3.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request update doc for HA/Zookeeper Configuration
## Describe Your Solution 🔧
Replace conf name with the new ones and mark olds as deprecated
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6119 from kevinjmh/doc-update.
Closes#6119
49d0fffa0 [Manhua] [DOC] Update doc for HA/Zookeeper Configuration
Authored-by: Manhua <manhua@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request removes support of Derby for Kyuubi metastore.
## Describe Your Solution 🔧
Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore.
Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal.
Also, the Derby we used suffers CVE-2022-46337
> Mitigation:
>
> Users should upgrade to Java 21 and Derby 10.17.1.0.
>
> Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Some tests are modified accordingly. Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6099 from pan3793/remove-derby.
Closes#6099
7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Historically, we provided a few standalone YAML files to allow users to deploy Kyuubi on K8s, and later a Helm Chart was introduced, which is a simple and mature way for production use cases.
## Describe Your Solution 🔧
Remove the legacy standalone YAML files way and update docs to guide user to use Helm chart to deploy Kyuubi on K8s.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6080 from pan3793/k8s-cleanup.
Closes#6080
3a9fa9be3 [Cheng Pan] comment
8f681d28c [Cheng Pan] Improve instruction for deploy Kyuubi on K8s
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6010
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6011 from huangxiaopingRD/gluten-doc.
Closes#6010
88bce5865 [huangxiaoping] [KYUUBI #6010] Improve Gluten usage documentation
Authored-by: huangxiaoping <1754789345@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Similar to https://github.com/apache/kyuubi/pull/5412, this PR aims to change the default SQLite file `kyuubi_state_store.db` for Metadata store is located under `$KYUUBI_HOME` instead of `$PWD`.
## Describe Your Solution 🔧
Make `kyuubi.metadata.store.jdbc.url` supports the variables substitution: `<KYUUBI_HOME>`, and change the default value to `jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Verified locally. Make a binary distribution, and bootstrap the Kyuubi Server process from a non-KYUUBI_HOME dir, `kyuubi_state_store.db` is created at KYUUBI_HOME as expected.
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5892 from pan3793/sqlite-jdbc-url.
Closes#5892
4341f4263 [Cheng Pan] docs
7062bcdf9 [Cheng Pan] migration guide
989de0e5b [Cheng Pan] Metadata store JDBC url supports <KYUUBI_HOME> variable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5467
## Describe Your Solution 🔧
1. Add Gluten UTs.
2. Setup CI for Gluten testing
3. Write docs to guide users in setting up Kyuubi with Spark plus Gluten.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
github action ci tests: [Gluten Test CI](https://github.com/Kwafoor/incubator-kyuubi/actions/runs/7111586978)
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5800 from Kwafoor/kyuubi_5467.
Closes#5800
c6dd26f93 [wangjunbo] fix
7818ae0c5 [wangjunbo] fix Scala Test
296f08c8c [wangjunbo] remove spark-3.2 gluten test
5a704675d [wangjunbo] [KYUUBI#5467] Integrate Intel Gluten with Spark engine
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
# 🔍 Description
Infer the batch user from session or metadata, user do not need to specify the proxy user anymore.
This pr also align the behavior of BatchesResource with that of SessionsResource and OperationsResource(no proxy user parameter).
For Kyuubi Batch, Session and Operation, these resources have the user attiribute.
So we only need to check whether the authentication user has the permission to access the resource.
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5717 from turboFei/hive_server2_proxy_user.
Closes#5717
70ad7e76d [fwang12] comment
c721a751a [fwang12] ignore
da92bd5a1 [fwang12] fix ut
9a197d005 [fwang12] doc
c8ed5f9cf [fwang12] ut
cef9e329c [fwang12] do not use proxy user
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
close#5423
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes#5485 from lsm1/branch-5423.
Closes#5423
0bd3b0233 [senmiaoliu] mention in the migration guide
dddafeb9b [senmiaoliu] support multi session conf advisor
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
Both the default value of `kyuubi.zookeeper.embedded.data.log.dir` and `kyuubi.zookeeper.embedded.data.dir` are `embedded_zookeeper`. It is used based on the current dir.
If the configuration dir is a relative path , it is resolved relative to `KYUUBI_HOME`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5412 from biangjuang/master.
Closes#5412
17e502c03 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
e0aa01c82 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
bb6af93df [Cheng Pan] Update docs/deployment/migration-guide.md
53bd61bc3 [lawulu] Resolve the relative zk configuration dir based on KYUUBI_HOME
Lead-authored-by: lawulu <biangjuang@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#5381
change default metrics reporter to prometheus since Kyuubi 1.8
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5344 from zhaohehuhu/Improvement-0928.
Closes#5381
84f4c8208 [hezhao2] reset METRICS_REPORTERS for test case
b9ee5f711 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
86165a6fe [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
a3605b626 [hezhao2] set METRICS_PROMETHEUS_PORT to 0 for test cases
f1a4d2861 [hezhao2] restore version number for kyuubi.metrics.reporters in doc
dae40e1a2 [hezhao2] change default metrics reporter to prometheus
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
<img width="1080" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/cc118574-c752-449a-9b20-04d2cc59b118">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5227 from hanna-liashchuk/patch-2.
Closes#5227
baf391ba9 [Hanna Liashchuk] fix DOC style
e46dfddc3 [Cheng Pan] Update docs/deployment/engine_on_kubernetes.md
0746a350a [hanna-liashchuk] upd
233bba00b [hanna-liashchuk] Added deploy mode info
Lead-authored-by: Hanna Liashchuk <g.liashchuk@temabit.com>
Co-authored-by: hanna-liashchuk <47921651+hanna-liashchuk@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5265 from link3280/docs_jm_ha.
Closes#5265
947854a5a [Paul Lin] Improve language
f7ea8dc51 [Paul Lin] Improve language
c40b94bf1 [Paul Lin] Add notice about Jobmanager HA to Flink deployment docs
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
- Move the configuration docs to the top level of docs, which is most commonly used and referenced
- update relevant doc links
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5154 from bowenliang123/config-doc-first.
Closes#5154
b49ed3f8b [liangbowen] nit
db7f0d14d [liangbowen] update doc links
f8fd697a2 [liangbowen] move config docs to the top level
7448e4487 [liangbowen] change title of settings doc
40214ddd8 [liangbowen] move config doc in the front of deployment
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- allow using the session's user/password to connect database in the JDBC engine
- it is allowed to be applied on any share level of engines, since every kyuubi session maintains a dedicated JDBC connection in the current implementation
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5120 from bowenliang123/jdbc-user.
Closes#5120
7b8ebd137 [liangbowen] Use session's user and password to connect to database in JDBC engine
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Support SSL for trino engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3374 from hddong/support-trino-password.
Closes#3374
f39daaf78 [Cheng Pan] improve
6308c4cf7 [hongdongdong] Support SSL for trino engine
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
One of the basic ideas of Kerberos authentication is that the Client MUST provide the Server Principal Name (SPN).
[ZOOKEEPER-1467](https://issues.apache.org/jira/browse/ZOOKEEPER-1467) (fixed in 3.6.0, 3.5.7) allows to configure Zookeeper SPN at the client side.
Previously, there is no way but only `-Dzookeeper.server.principal=zookeeper/_HOSTREALM` to set ZK SPN, this PR allows the user to set it via Kyuubi configuration `kyuubi.ha.zookeeper.auth.serverPrincipal`.
In default, if `zookeeper.server.principal` is not configured, it assumes that the Zookeeper server uses `zookeeper/_HOST<SAME_REALM_WITH_CLIENT_PRINCIPAL>`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5130 from pan3793/zk-spn.
Closes#5130
6f0a1bbdc [Cheng Pan] doc
290e172aa [Cheng Pan] docs
a43b0c67a [Cheng Pan] nit
9a33446b7 [Cheng Pan] Allow set Zookeeper server principal
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5099
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5103 from lsm1/features/kyuubi_5099.
Closes#5099
84a1ecad0 [senmiaoliu] fix doc
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Updated [kyuubi on kubernetes config section](https://kyuubi.readthedocs.io/en/master/deployment/kyuubi_on_kubernetes.html#config) to state <code> Kyuubi **does** not recommend using this way on Kubernetes</code>
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5086 from mans2singh/ISSUE-5085.
Closes#5086
5faf0df2e [mans2singh] [KYUUBI # 5085] Update config section based on review comments
df9f62f36 [mans2singh] [KYUUBI # 5085] Update config section of deploy on kubernetes
Authored-by: mans2singh <mans2singh@yahoo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix bug https://github.com/apache/kyuubi/issues/5065
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5066 from ASiegeLion/master.
Closes#5065
08d1ac077 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
bf908f5af [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
9144582f9 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
f1c95e409 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
907123a93 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
f30a9fc39 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
449be44d7 [文艺攻城狮] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
987ffc7fe [文艺攻城狮] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
995386f98 [文艺攻城狮] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
ad3d11191 [liupeiyue] [KYUUBI-#5065]destroy the spark engine release the submitted temp files
Lead-authored-by: liupeiyue <liupeiyue@yy.com>
Co-authored-by: 文艺攻城狮 <945076608@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR resolves https://github.com/apache/kyuubi/issues/5010, by introducing a new configuration property that makes it possible to override the hostname or ip of the Kyuubi server's thrift frontend service that will be registered for the service discovery ensemble.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
Before applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0`, the znode becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=0.0.0.0:10009;version=1.8.0-SNAPSHOT;sequence=0000000007]
```
After applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0` and `kyuubi.frontend.advertised.host=some_kyuubi_host`, it becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=some_kyuubi_host:10009;version=1.8.0-SNAPSHOT;sequence=0000000009]
```
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
`./build/mvn clean install -pl kyuubi-common` succeeded.
Closes#5015 from touchida/issue-5010-connection-url.
Closes#5010
4598c2baf [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
bf53a8a13 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
0ddf09f0f [toshihiko.uchida] Use pattern matching instead of if-else
34c61d90d [toshihiko.uchida] Rename to kyuubi.frontend.advertised.host and apply to all frontends
7fd980baf [toshihiko.uchida] [KYUUBI #5010] Make Kyuubi server's thrift frontend connection URL configurable
Lead-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Please refer to #4997
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
1. connect to KyuubiServer with beeline
2. Confirm the Application is ACCEPTed in ResourceManager, Restart KyuubiServer
3. Confirmed that Engine was terminated shortly
```
23/06/28 10:44:59 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Current open session is 0
23/06/28 10:45:00 ERROR spark.SparkSQLEngine: Spark engine has been terminated because no incoming connection for more than 60000 ms, deregistering from engine discovery space.
23/06/28 10:45:00 WARN zookeeper.ZookeeperDiscoveryClient: This Kyuubi instance lniuhpi1616.nhnjp.ism:46588 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5002 from risyomei/feature/failfast.
Closes#5002
402d6c01f [Xieming LI] Changed runInNewThread based on comment
58f11e157 [Xieming LI] Changed runInNewThread to non-blocking
c6bb02d6a [Xieming LI] Fixed Unit Test
168d996d0 [Xieming LI] Start countdown after engine is started
48ee819f2 [Xieming LI] Fixed a typo
a8d305942 [Xieming LI] Using runInNewThread ported from Spark
21f0671df [Xieming LI] Updated document
a7d5d1082 [Xieming LI] Changed the default value to turn off this feature
437be512d [Xieming LI] Trigger CI to test agagin
42a847e84 [Xieming LI] Added Configuration for timeout, changed to ThreadPoolExecutor
639bd5239 [Xieming LI] Fail the engine fast when no incoming connection in CONNECTION mode
Authored-by: Xieming LI <risyomei@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4843
Support to submit kyuubi engine/batch to multiple kubernetes contexts and namespaces.
In this pr, the user can config the kubernetes conf for specified kubernetes context and namespace likes below.
```
kyuubi.kubernetes.<context>.master.address
kyuubi.kubernetes.<context>.<namespace>.authenticate.oauthTokenFile
```
For example:
```
kyuubi.kubernetes.28.master.address=k8s://master
kyuubi.kubernetes.28.ns1.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
kyuubi.kubernetes.28.ns2.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
for k8s context=28, namespace=ns1, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
```
for k8s context=28, namespace=ns2, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
So that, kyuubi server can build kubernetes client for each context and namespace.
### _How was this patch tested?_
Existing kubernetes integration testing.
Closes#4984 from turboFei/k8s_client_yaml.
Closes#4843
f8ffaeeb9 [fwang12] nit
d25774288 [fwang12] comments
5ae7c8433 [fwang12] save into request conf
fd6c363db [fwang12] save
ff004a529 [fwang12] procebuilder method
6b9520bfd [fwang12] save
58850387e [fwang12] save
98df67e5f [fwang12] ut
da811697c [fwang12] fix
aa568aaa4 [fwang12] save
89656f463 [fwang12] check init
a0ef6894b [fwang12] code style
00abb6568 [fwang12] default namespace
295512987 [fwang12] k8s context namespace
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#4803
The beeline has regression because of 70590f71ef (diff-993fdbefe9fe3d1c91fcedba99362a8c8d9b94793ec16cbfbc989e750367ea89)
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4965 from turboFei/revert_beeline_python_change.
Closes#4965
856d92391 [fwang12] trim for non python mdoe
f8464606b [fwang12] Revert "[KYUUBI #4619] Fix beeline with -e When there are other SQL statements before the source statement, the source statement cannot be executed normally"
e5e3c31b3 [fwang12] revert trim
bec09c254 [fwang12] migration guide
585da6fc1 [fwang12] fix'
f3fcfe97e [fwang12] save
8cb8cb9d0 [fwang12] save
e1539775a [fwang12] comments
814c970a2 [fwang12] save
b1baa773b [fwang12] save
3337ca8fa [fwang12] options
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Improve log on executing init schema DDL.
- Add `IF NOT EXISTS` on `CREATE INDEX` statement.
- Fix `kyuubi.metadata.store.jdbc.url` configuration description.
- Delete SQLite files on performing `build/mvn clean`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4953 from pan3793/sqlite-followup.
Closes#4950
a0b44f37c [Cheng Pan] [KYUUBI #4950][FOLLOWUP] Misc improvement for usage of SQLite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4881
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4886 from rhh777/jdbcengine.
Closes#4881
e0439f8e6 [haorenhui] [KYUUBI #4881] update settings.md
d667d8fe8 [haorenhui] [KYUUBI #4881] update conf/docs
cba06b4b9 [haorenhui] [KYUUBI #4881] simplify code
80be4d27c [haorenhui] [KYUUBI #4881] fix style
4f0fa3ab2 [haorenhui] [KYUUBI #4881] JDBCEngine performs initialization sql
Authored-by: haorenhui <haorenhui@kingsoft.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The upgrade section states <code>Kyuubi server supports **stop** gracefully</code>. Update to <code>Kyuubi server supports **stopping** gracefully</code>
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4864 from mans2singh/ISSUE-4863.
Closes#4864
5e853f84e [mans2singh] Updated key features section
Authored-by: mans2singh <mans2singh@yahoo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- introduce new event logger type `KAFKA`
- send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle
- use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4733 from bowenliang123/kafka-logger.
Closes#4733
b5220d234 [liangbowen] introduce kafka server event logger
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
We shall use different engine submit timeout for different resource manager.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4801 from turboFei/engine_submit_timeout.
Closes#4801
e34852a64 [fwang12] nit
ad69008e7 [fwang12] 1.7.2
db11330c5 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#4623
To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] Run CI
Closes#4625 from zwangsheng/KYUUBI_4623.
Closes#4623
a415bef7f [Cheng Pan] nit
136d0db4d [Cheng Pan] 171
b5d3c237a [Cheng Pan] re-generate conf
bf14ad870 [Cheng Pan] nit
9ee7e04f9 [Cheng Pan] nit
301162ea0 [Cheng Pan] nit
1d426922b [Cheng Pan] nit
b95d7a650 [Cheng Pan] improve
cc8d2c7f4 [zwangsheng] fix comments
d017bafdf [zwangsheng] Set resycn 0
28f9a70d9 [zwangsheng] Reorder func & slow get app info
22d9c1662 [zwangsheng] fix setting
8e0940334 [zwangsheng] fix comments
10965d3df [zwangsheng] Rename fileter function => isSparkEnginePod
b02677154 [zwangsheng] rename
78c9fdb17 [zwangsheng] fix comments
6d31f70d1 [zwangsheng] Fix IT Test
f43bba2b9 [zwangsheng] fix
17e4f55eb [zwangsheng] debug
be8da790e [zwangsheng] debug
0db45a513 [zwangsheng] retest
a93786abc [zwangsheng] Fix style
652ee837e [zwangsheng] Add Setting & Debug
4add7e4e2 [zwangsheng] improve
1f4341237 [zwangsheng] remove unused import
35acd6106 [zwangsheng] fix compile
05dfc598e [zwangsheng] [KYUUBI #4623][Improvement][K8S] Remove cached app info when out of time
4ab530e99 [zwangsheng] [KYUUBI #4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title.
With this pr, customer can execute SCALA code with `confOverlay`.
```
kyuubi.operation.language=SCALA
```
execute PYTHON code with
```
kyuubi.operation.language=PYTHON
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4639 from turboFei/init_scala.
Closes#4639
cdf828f9a [fwang12] add ut
f4f2bc883 [fwang12] doc
2fe8a1659 [fwang12] save
f840cb4d9 [fwang12] conf overlay
eb49537ea [fwang12] conf overlay
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4545 from lightning-L/kyuubi-4523.
Closes#4523
2ee634cd5 [Tianlin Liao] modify conf description; add test case
12b930456 [Tianlin Liao] refer to HIVE-13415
c8a9db6a2 [Tianlin Liao] minor fix
9942d2515 [Tianlin Liao] move logic to SessionManager.closeSession
62b990b3e [Tianlin Liao] [KYUUBI #4523] support close engine session gracefully
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Make ChatGPT model ID configurable
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4558 from bowenliang123/chatgpt-model.
Closes#4558
63f8ee30d [liangbowen] nit
3012ccaaa [liangbowen] make chatgpt model configurable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Support Chinese question
- Support proxy settings
- Support setting timeout
<img width="1228" alt="image" src="https://user-images.githubusercontent.com/3898450/225851246-8762a451-9743-4c1d-8a33-cc49a926dfec.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4548 from cxzl25/chatgpt_followup.
Closes#4548
1d5715442 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
7add6a733 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
55974f298 [sychen] fix
2d360e102 [sychen] typo
19b5d0814 [sychen] doc
bdf8e29b6 [sychen] 1.utf8;2.proxy;timeout
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce a brand new CHAT engine, it's supposed to support different backends, e.g. ChatGPT, 文心一言, etc.
This PR implements the following providers:
- ECHO, simply replies a welcome message.
- GPT: a.k.a ChatGPT, powered by OpenAI, which requires a API key for authentication. https://platform.openai.com/account/api-keys
Add the following configurations in `kyuubi-defaults.conf`
```
kyuubi.engine.chat.provider=[ECHO|GPT]
kyuubi.engine.chat.gpt.apiKey=<chat-gpt-api-key>
```
Open an ECHO beeline chat engine.
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=ECHO'
```
```
Connecting to jdbc:hive2://localhost:10009/
Connected to: Kyuubi Chat Engine (version 1.8.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Client (version 1.7.0)
Beeline version 1.7.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10009/> Hello, Kyuubi!;
+----------------------------------------+
| reply |
+----------------------------------------+
| This is ChatKyuubi, nice to meet you! |
+----------------------------------------+
1 row selected (0.397 seconds)
```
Open a ChatGPT beeline chat engine. (make sure your network can connect the open API and configure the API key)
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=GPT'
```
<img width="1109" alt="image" src="https://user-images.githubusercontent.com/26535726/225813625-a002e6e2-3b0d-4194-b061-2e215d58ba94.png">
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4544 from pan3793/chatgpt.
Closes#4544
87bdebb6d [Cheng Pan] nit
f7dee18f3 [Cheng Pan] Update docs
9beb55162 [cxzl25] chat api (#1)
af38bdc7c [Cheng Pan] update docs
9aa6d83a6 [Cheng Pan] Initial implement Kyuubi Chat Engine
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: cxzl25 <cxzl25@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
as title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4521 from turboFei/rest_docs.
Closes#3739
589237ca6 [fwang12] save
8e7d9bec6 [fwang12] final
35b67ba1e [fwang12] doc
40163311f [fwang12] save
5711d3d72 [fwang12] fix
bab07902f [fwang12] fix ut
d58c2753a [fwang12] save
989eaffd5 [fwang12] remove
2960602bc [fwang12] remove outdate docs
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The following discussion assumes using Spark cluster mode w/ `waitCompletion=false`.
In Spark on Yarn, the application is visible immediately after `spark-submit` is returned, but things are different in Spark on K8s, Driver Pod is ephemerally invisible after submitting, so NOT_FOUND is returned instead of UNKNOWN or PENDING.
To tolerate the above case, `kyuubi.engine.submit.timeout` is introduced, ApplicationManager will report UNKNOWN instead of NOT_FOUND during the Driver Pod scheduling period.
More detail in #4467
1. Remove `KubernetesApplicationOperation`'s `JpsApplicationOperation` for handle Client Deploy Mode(`YarnApplicationOperation` doesn't handle this either)
2. Add engine submit timeout for `KubernetesApplicationOperation` to return Unknown status when not found driver pod in time range.
3. GetApplicationInfo with it's submit time
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4469 from zwangsheng/4467.
Closes#4467
562b67463 [zwangsheng] [KYUUBI #4467] Fix Setting.md
362c43d1b [zwangsheng] [KYUUBI #4467] Fix Setting.md
ac69f4d81 [zwangsheng] [KYUUBI #4467] Add Config Desc
d2b9fb660 [zwangsheng] [KYUUBI #4467] save tab
eac880fcf [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
7a20b97a4 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
aa4c7716a [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
c5bd888ab [zwangsheng] [KYUUBI #4467] note it test
a86dcefba [zwangsheng] [KYUUBI #4467] Using default none
aed7f8794 [Cheng Pan] Update docs/deployment/settings.md
490df7dc0 [zwangsheng] [KYUUBI #4467] fix complie
33f3a5be8 [zwangsheng] [KYUUBI #4467] fix comments
4745790cf [zwangsheng] [KYUUBI #4467] Fix IT Test
924cfe38e [zwangsheng] [KYUUBI #4467] Fix Setting.md
5f8aeaacc [zwangsheng] [KYUUBI #4467] KubernetesApplicationOperation Wait if not fount driver pod in limit time range
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
REST frontend protocol was introduced in Kyuubi 1.4.0 https://github.com/apache/kyuubi/pull/1349, more and more users are using REST in recent days, and to simplify the planned Web UI developments, I think we should enable the REST by default for master(1.8.0-SNAPSHOT).
I do not remove the "experiment" because we don't have strong confident to ensure the API stability in this time, I would like to defer it until the community has confident to mark it as stable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4483 from pan3793/enable-rest.
Closes#4483
66fef0224 [Cheng Pan] http
91a29b453 [Cheng Pan] nit
29a4826da [Cheng Pan] Enable REST frontend protocol in default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`GetTables` operation is too slow because it queries table details info one by one, but then only a table comment is used to construct a result row, which i think could be optional.
This PR add an optional config which can control this operation. By default, `GetTables` operation queries all message. Otherwise, `GetTables` operation just return table identifiers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4444 from liaoyt/master.
Closes#4171
af5e60e36 [yeatsliao] rename config
0c9985e32 [yeatsliao] add doc
5e8687cb3 [yeatsliao] Supports ignore table comment when list all tables.
Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The key changes are
1. allow disabling metadata operation retry
2. always fail fast on "duplicated key on unique index" error
Currently, when metadata operations failed, we always do async retry, to tolerate long-time metadata store outages w/o blocking the submission request, but it can not guarantee metadata consistency eventually, e.g. when inserting data violates the unique key restriction, it will never succeed, and block any following update request for the batch job, in such cases, the client gets succeed response but the metadata can not be updated correctly.
We should distinguish between recoverable and unrecoverable errors, for unrecoverable errors, we should fail fast, but the fact is it's hard to enumerate all recoverable nor unrecoverable errors, in this PR, we just enumerate the "duplicated key" as unrecoverable errors, and provide a switch to disable async retry so that the error can propagate to client correctly.
Some configurations are renamed w/ the `async.` prefix(the original key still takes effect) because we may introduce the sync retry logic in the future.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4418 from pan3793/sync-retry.
Closes#4418
ce58ac58c [Cheng Pan] revert migration-guide.md
c2d8377a4 [Cheng Pan] simplify
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR changes the `kyuubi-defaults.conf.template` because
- remove deprecated conf `kyuubi.frontend.bind.port`
- add some properties which were frequently asked by users
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="825" alt="image" src="https://user-images.githubusercontent.com/26535726/221521262-9de7d4e0-ca68-4718-a186-fad5f71b5e5a.png">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4410 from pan3793/conf-template.
Closes#4410
aab459753 [Cheng Pan] nit
d3d8a9a28 [Cheng Pan] nit
e278fa98d [Cheng Pan] update
169db65d4 [Cheng Pan] nit
15a1f656a [Cheng Pan] Improve kyuubi-defaults.conf.template
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4376
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4405 from lightning-L/kyuubi-4376.
Closes#4376
1a01a75a8 [Tianlin Liao] rename and refactor
7324cab3d [Tianlin Liao] [KYUUBI #4376] Support to config the kyuubi service administrator with kyuubi conf
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
