apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
3,970 Commits 23 Branches 0 Tags 65 MiB
471237be92
Commit Graph

62 Commits

Author SHA1 Message Date
senmiaoliu
02e1aa3063
[KYUUBI #6387] [TRINO] Trino engine improve operation log 
trino engine increment mode support fetch log

# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

1. Trino engine supports outputting final status information, and we can also support outputting running information.
2. In increment mode, the operation status is set to FINISHED quickly, which prevents the client from fetching detailed operation logs. The client is unaware of the exact execution status of the operation. It should be possible to explicitly check if the ResultSet has more data (hasNext), and delay setting the status to output more operation information.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6387 from lsm1/branch-trino-print-info.

Closes #6387

b9b733a2f [senmiaoliu] trino engine output running info trino engine increment mode support fetch log

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
 2024-05-15 16:38:54 +08:00
Wang, Fei
3439ea03f2 [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile 
# 🔍 Description
## Issue References 🔗

This pull request fixes #
I found that, with saveToFile enabled with the default min size threshold, even I run a simple `set` command, It also save the result to file.
<img width="1718" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/5bcc0da1-201a-453a-8568-d1bfadd7adef">

I think we need to skip this kind of queries.

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6377 from turboFei/check_is_DQL.

Closes #6377

da9c2a921 [Wang, Fei] ut
04e20db5f [Wang, Fei] conf
8f20ed84b [Wang, Fei] refine the check
f558dcca5 [Wang, Fei] ut
c81340333 [Wang, Fei] DQL

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
 2024-05-09 08:43:07 -07:00
wforget
ef28a615a6 [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6338

## Describe Your Solution 🔧

Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests
KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size")

---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6340 from wForget/KYUUBI-6338.

Closes #6338

acd73a16b [wforget] fix
5e64ed3a5 [wforget] comments
48b8ec6cd [wforget] fix style and address comments
ca653492c [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf

Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
 2024-04-29 14:06:53 +08:00
Wang, Fei
90491fc07e [KYUUBI #6321] Support to get Spark Kubernetes app URL 
# 🔍 Description
## Issue References 🔗

Now for kubernetes application, there is no runtime AppURL return in the application report.

In fact, we can get the driver svc and get the spark-ui port, then build the Spark UI URL same with that of spark.

074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L67)

074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L96-L102)

Service Selector example:
```
  selector:
    kyuubi-unique-tag: bf5fa281-0b2f-4096-aa0a-13a463c147a6
    spark-app-selector: spark-bf074093a7994954a89101eb2831bd1e
```
## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6318 from turboFei/k8s_spark_ui.

Closes #6321

f77999ef0 [Wang, Fei] with label selector does not work
4e8d5767b [Wang, Fei] typo
3f1eb545c [Wang, Fei] comments
6f0d01d94 [Wang, Fei] ut
9edfde034 [Wang, Fei] ut
65d5e8175 [Wang, Fei] using pattern
9b21d30df [Wang, Fei] Support to get kubernetes app url

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
 2024-04-19 09:32:46 -07:00
senmiaoliu
11343cd6ea
[KYUUBI #6299] Support disabling Web UI 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6299

## Describe Your Solution 🔧

when disabling web ui, return 404 page

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6311 from lsm1/branch-support-disable-webui.

Closes #6299

aa96c2737 [senmiaoliu] remove enable.html
998504710 [senmiaoliu] fix style
a2622cbbc [senmiaoliu] disable web ui

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-04-16 15:25:35 +08:00
senmiaoliu
9aa5269665
[KYUUBI #6253] Support running JDBC engine on YARN AM 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6253

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

<img width="1251" alt="image" src="https://github.com/apache/kyuubi/assets/18713676/b654a300-8c79-4461-9fba-4ad1c913accc">

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6275 from lsm1/branch-jdbc-engine-on-yarn.

Closes #6253

5ed4af041 [senmiaoliu] fix style
86e032688 [senmiaoliu] fix style
b3e114445 [senmiaoliu] fix style
bf33de300 [senmiaoliu] fix style
c38404918 [senmiaoliu] jdbc engine on yarn

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
 2024-04-15 17:39:17 +08:00
wforget
a4c3c0b759 [KYUUBI #6216] Support to deny some client ips to make connection 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6216

## Describe Your Solution 🔧

Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

SessionLimiterSuite.test("test session limiter with ip deny list")
---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6217 from wForget/KYUUBI-6216.

Closes #6216

e0a058c1d [wforget] update ctl
f66426eda [wforget] typo
5c5308ad8 [wforget] typo
959242e9b [wforget] Support to deny some client ips to make connection

Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
 2024-04-07 16:32:00 +08:00
senmiaoliu
a937ae760e
[KYUUBI #6243] Add missing Impala alias for JDBC engine 
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6238 from lsm1/branch-conf-add-Impala.

Closes #6243

de3093d6a [senmiaoliu] add Impala conf

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-04-03 17:30:42 +08:00
senmiaoliu
03c3b4d69c
[KYUUBI #5374] JDBC Engine supports ClickHouse 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5374

## Describe Your Solution 🔧

JDBC Engine supports ClickHouse

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6225 from lsm1/branch-support-clickhouse.

Closes #5374

0ce4f6f0b [senmiaoliu] fix style
f6ab022b6 [senmiaoliu] use ck jdbc http jar
dee6a6bdc [senmiaoliu] add it test
aed6b33a9 [senmiaoliu] init clickhouse engine

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-04-02 20:00:01 +08:00
zhouyifan279
a196ace284
[KYUUBI #6199] Support to run HiveSQLEngine on kerberized YARN 
# 🔍 Description
## Issue References 🔗

This pull request implement a feature -  Run HiveSQLEngine on kerberized YARN

## Describe Your Solution 🔧
Introduced two configs:
- kyuubi.engine.principal
- kyuubi.engine.keytab

When do submit to a kerberized YARN, submitter uploads `kyuubi.engine.keytab` to application's staging dir.
YARN NodeManager downloads keytab to AM's working directory. AM logins to Kerberos using the principal and keytab

**Note**
I've tried to run HiveSQLEngine with only DelegationTokens but failed.

Take SQL `SELECT * FROM a` as an example:
Hive handles this simple TableScan SQL by reading directly from table's hdfs file.
When Hive invokes `FileInputFormat.getSplits` during reading, `java.io.IOException: Delegation Token can be issued only with kerberos or web authentication` will be thrown.
The simplified stacktrace from IDEA is as below:
```
getDelegationToken:734, DFSClient (org.apache.hadoop.hdfs)
getDelegationToken:2072, DistributedFileSystem (org.apache.hadoop.hdfs)
collectDelegationTokens:108, DelegationTokenIssuer (org.apache.hadoop.security.token)
addDelegationTokens:83, DelegationTokenIssuer (org.apache.hadoop.security.token)
obtainTokensForNamenodesInternal:143, TokenCache (org.apache.hadoop.mapreduce.security)
obtainTokensForNamenodesInternal:102, TokenCache (org.apache.hadoop.mapreduce.security)
obtainTokensForNamenodes:81, TokenCache (org.apache.hadoop.mapreduce.security)
listStatus:221, FileInputFormat (org.apache.hadoop.mapred)
getSplits:332, FileInputFormat (org.apache.hadoop.mapred)
getNextSplits:372, FetchOperator (org.apache.hadoop.hive.ql.exec)
getRecordReader:304, FetchOperator (org.apache.hadoop.hive.ql.exec)
getNextRow:459, FetchOperator (org.apache.hadoop.hive.ql.exec)
pushRow:428, FetchOperator (org.apache.hadoop.hive.ql.exec)
fetch:147, FetchTask (org.apache.hadoop.hive.ql.exec)
getResults:2208, Driver (org.apache.hadoop.hive.ql)
getNextRowSet:494, SQLOperation (org.apache.hive.service.cli.operation)
getNextRowSetInternal:105, HiveOperation (org.apache.kyuubi.engine.hive.operation)
```

Theoretically, it can be solved by add AM DelegationTokens into
 `org.apache.hadoop.hive.ql.exec.FetchOperator.job.credentials`.
But actually, it is impossible without modifying Hive's source code.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
HiveSQLEngine can not run on a kerberized YARN

#### Behavior With This Pull Request 🎉
HiveSQLEngine can run on a kerberized YARN

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6199 from zhouyifan279/kerberized-hive-engine-on-yarn.

Closes #6199

383d1cdcb [zhouyifan279] Fix tests
458493a91 [zhouyifan279] Warn if run Hive on YARN without principal and keytab
118afe280 [zhouyifan279] Warn if run Hive on YARN without principal and keytab
41fed0c44 [zhouyifan279] Ignore Principal&Keytab when hadoop security is no enabled.
9e2d86237 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/hive/HiveProcessBuilder.scala
5ae0a3eac [zhouyifan279] Remove redundant checks
5d3013aaf [zhouyifan279] Use principal & keytab in Local mode
5733dfdcb [zhouyifan279] Use principal & keytab in Local mode
85ce9bb7a [zhouyifan279] Use principal & keytab in Local mode
061223dbe [zhouyifan279] Resolve comments
e706936e7 [zhouyifan279] Resolve comments
f84c7bccc [zhouyifan279] Support run HiveSQLEngine on kerberized YARN
4d262c847 [zhouyifan279] Support run HiveSQLEngine on kerberized YARN

Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-03-22 16:18:43 +08:00
senmiaoliu
67f099a4cd
[KYUUBI #6201] Distinguish the collect mode of different engines 
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Currently, Spark, JDBC, and TRINO engines support incremental collection, but they share the same configuration option `kyuubi.operation.incremental.collect`. Sometimes, it's necessary to enable incremental collection only for specific engines, which requires distinguishing between them.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6201 from lsm1/branch-distinguish-engine-collect-mode.

Closes #6201

3c43d2cac [senmiaoliu] fix conf version
aa53231aa [senmiaoliu] distinguish kyuubi.operation.incremental.collect for engine type

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-03-20 20:27:01 +08:00
minyk
7413ba584a
[KYUUBI #6165] Add some HttpClient options for EngineUIProxyServlet 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6165

## Describe Your Solution 🔧

Add HttpClient options for engine ui proxy servlet.
- idleTimeout
- maxConnections
- maxThreads
- requestBufferSize
- responseBufferSize
- timeout

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
Engine UI proxy requests with large headers( > 4k) are failed with 500 http status.

#### Behavior With This Pull Request 🎉
Increase the value of `kyuubi.frontend.rest.proxy.jetty.client.requestBufferSize` properly(ex: 5k), request with large header has succeeded with 200 status.

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6192 from minyk/kyuubi-6165.

Closes #6165

6ef26aa43 [minyk] fix kyuubi version of configs
1812a96eb [minyk] apply spotless
3527cada9 [minyk] set initial parameters to the httpclient of proxy servletholder.

Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-03-19 10:45:57 +08:00
Cheng Pan
182f4349a7
[KYUUBI #6117] Bump Kyuubi Shaded 0.3.0 
# 🔍 Description
## Issue References 🔗

This PR upgrades Kyuubi Shaded to 0.3.0, the major changes are:

- upgrade Thrift from 0.9.3-1 to 0.16.0
- upgrade JLine used by Zookeeper CLI from 0.9.94 to 2.14.6
- split kyuubi-relocated-thrift from kyuubi-relocated-hive-service-rpc
- introduce a light kyuubi-relocated-hive-metastore-client for getting DelegationToken

Due to THRIFT-5297 (fixed in 0.14.0) removes requestTimeout and beBackoffSlotLength, the following Kyuubi configurations are removed too.

- kyuubi.frontend.login.timeout
- kyuubi.frontend.thrift.login.timeout
- kyuubi.frontend.backoff.slot.length
- kyuubi.frontend.thrift.backoff.slot.length

Previously, Zookeeper 3.4 depends on JLine 0.9.94 while Zookeeper 3.6 and Hive Beeline depends on JLine 2.14.6, we pulls different versions of JLine into the binary distribution. Now we align them to a single JLine 2.14.6.

## Describe Your Solution 🔧

Upgrade dependencies and change code to adapt to the breaking changes, also upgrade migration guide to mention the removed configurations.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6117 from pan3793/shaded-0.3.

Closes #6117

5639c9fd8 [Cheng Pan] nit
d281cdb4b [Cheng Pan] fix
184e1b95b [Cheng Pan] fix
1628337c8 [Cheng Pan] remove unused conf
24db2d5ad [Cheng Pan] try fix
1e995bb34 [Cheng Pan] nit
1d8e9bce9 [Cheng Pan] dep list
7f33624e8 [Cheng Pan] Bump Kyuubi Shaded 0.3.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-03-08 10:19:30 +08:00
Manhua
97d02a492d [KYUUBI #6119] [DOC] Update doc for HA/Zookeeper Configuration 
# 🔍 Description
## Issue References 🔗

This pull request update doc for HA/Zookeeper Configuration

## Describe Your Solution 🔧

Replace conf name with the new ones and mark olds as deprecated

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6119 from kevinjmh/doc-update.

Closes #6119

49d0fffa0 [Manhua] [DOC] Update doc for HA/Zookeeper Configuration

Authored-by: Manhua <manhua@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-03-04 05:38:52 +00:00
Cheng Pan
d007d7af10
[KYUUBI #6099] Remove support of Derby for Kyuubi metastore 
# 🔍 Description
## Issue References 🔗

This pull request removes support of Derby for Kyuubi metastore.

## Describe Your Solution 🔧

Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore.

Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal.

Also, the Derby we used suffers CVE-2022-46337

> Mitigation:
>
> Users should upgrade to Java 21 and Derby 10.17.1.0.
>
> Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Some tests are modified accordingly. Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6099 from pan3793/remove-derby.

Closes #6099

7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-02-27 19:15:41 +08:00
Cheng Pan
4390cc6fa4
[KYUUBI #6058] Make Jetty server stop timeout configurable 
# 🔍 Description
## Issue References 🔗

This pull request aims to make the Jetty server stop timeout configurable.

The RESTful frontend protocol has been enabled by default since 1.8.0, while a side-effect is observed, the process shutdown is always blocked by the Jetty server until reaches the default 30s timeout.

## Describe Your Solution 🔧

Two configurations are introduced in this PR
```
kyuubi.frontend.trino.jetty.stopTimeout=5s
kyuubi.frontend.rest.jetty.stopTimeout=5s
```
while the 5s is chosen as the default value according to https://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/jetty-server/src/main/config/etc/jetty.xml?h=jetty-9.3.x#n124

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

1. perform `bin/kyuubi run`
2. press `ctrl + c` to shutdown the Kyuubi process

#### Behavior Without This Pull Request ⚰️

takes ~30s to complete the shutdown progress.

#### Behavior With This Pull Request 🎉

takes ~5s to complete the shutdown progress.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6058 from pan3793/jetty-timeout.

Closes #6058

47d15f9aa [Cheng Pan] fix
cbac2b13e [Cheng Pan] Make Jetty server stop timeout configurable

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-02-17 00:38:44 +08:00
Fei Wang
8c3f471ae3 [KYUUBI #5952] Disconnect connections without running operations after engine maxlife time graceful period 
# 🔍 Description
## Issue References 🔗

We found that, some kyuubi connections(maybe managed by jdbc connection pool likes hikari) always keep alive, and the engine can not be terminated after exceeds the max life time.

So, In this pr, I introduce a graceful period after spark engine max life time, after the graceful period, the connections without running operations will be disconnected forcibly.

Close #5952

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6040 from turboFei/close_on_nooperation_rebase.

Closes #5952

0b05ddc0c [Fei Wang] comments
4329a85cf [Fei Wang] async stop
b39c9b3a0 [Fei Wang] use short sleep
7c123f612 [Fei Wang] save
45ad3489d [Fei Wang]  check no running operation

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2024-02-06 18:16:46 -08:00
Tigran Manasyan
35a910a43e
[KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5674

## Describe Your Solution 🔧

Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests
`org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite`

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6027 from tigrulya-exe/feature/5674-postgresql-metadata-backend.

Closes #5674

111658002 [Tigran Manasyan] Fix metastore db type option comment
44a22bfcf [Tigran Manasyan] Fix dependency list and metastore db type option comment
b638c8942 [Tigran Manasyan] Add PostgreSQL as backend database for kyuubi metadata

Authored-by: Tigran Manasyan <t.manasyan@arenadata.io>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-30 13:15:35 +08:00
Cheng Pan
3f993f405a
[KYUUBI #6003] Allow disabling user impersonation on launching engine 
# 🔍 Description
## Issue References 🔗

HiveServer2 has a configuration `hive.server2.enable.doAs` to control the execution user between the session user and the server user, Kyuubi's CONNECTION and USER share levels always perform like doAs enabled do. In CDH 5/6, this is disabled by default, users who want to migrate from CDH to Kyuubi may encounter permission issues with the current implementation.

## Describe Your Solution 🔧

This pull request introduces a new configuration `kyuubi.engine.doAs.enabled` to allow enable/disable user impersonation on launching engine. For security purpose, it's not allowed to be overridden by session conf.

The change in this PR has certain limitations:

- only supports Spark engine
- only supports interactive mode, specifically, it does not take effect on Spark batch mode now.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

The first step is passing all existing UTs when `kyuubi.engine.doAs.enabled=true`.

Tested on internal Kerberized-environment, when `kyuubi.engine.share.level=CONNECTION` and `kyuubi.engine.doAs.enabled=false`, use user 'spark' to launch engine, and the engine submitted without `--proxy-user spark`, thus engine launched by server user `hive`, then run `select session_user(), current_user()` and returns

```
+-----------------+-----------------+
| session_user()  | current_user()  |
+-----------------+-----------------+
| spark           | hive            |
+-----------------+-----------------+
```

And I checked the `spark.app.name` and registered path on Zookeeper also expected.
```
+-----------------+--------------------------------------------------------------------------+
|       key       |                       value                                              |
+-----------------+--------------------------------------------------------------------------+
| spark.app.name  | kyuubi_USER_SPARK_SQL_spark_default_51a416e5-6023-4bac-a964-cd9605f17c61 |
+-----------------+--------------------------------------------------------------------------+
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6003 from pan3793/doas.

Closes #6003

c4002fef5 [Cheng Pan] grammar
add20fd57 [Cheng Pan] nit
8711c2265 [Cheng Pan] address comment
033a32252 [Cheng Pan] 1.9.0
9273b9426 [Cheng Pan] fix
a1563e1ca [Cheng Pan] HadoopCredentialsManager
e982e2364 [Cheng Pan] Allow disable user impersonation on launching engine

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-29 21:30:42 +08:00
Cheng Pan
d474768d97
[KYUUBI #6018] Speed up GetTables operation for Spark session catalog 
# 🔍 Description
## Issue References 🔗

This pull request aims to speed up the GetTables operation for the Spark session catalog.
As reported in https://github.com/apache/kyuubi/discussions/4956, https://github.com/apache/kyuubi/discussions/5949, the GetTables operation is quite slow in some cases, and in https://github.com/apache/kyuubi/pull/4444, `kyuubi.operation.getTables.ignoreTableProperties` was introduced to speed up the V2 catalog, but not covers session catalog.

## Describe Your Solution 🔧

Extend the scope of `kyuubi.operation.getTables.ignoreTableProperties` to cover the GetTables operation for the Spark session catalog.

Currently, the basic step of GetTables in the Spark engine is
```
val catalog: String = getCatalog(spark, catalogName)
val databases: Seq[String] = sessionCatalog.listDatabases(schemaPattern)
val identifiers: Seq[TableIdentifier] = catalog.listTables(db, tablePattern, includeLocalTempViews = false)
val tableObjects: Seq[CatalogTable] = catalog.getTablesByName(identifiers)
```
then filter `tableObjects` with `tableTypes: Set[String]`.

The cost of `catalog.getTablesByName(identifiers)` is quite high when the table number is large, e.g. dozen thousand.

For some cases, listing tables only for table name display, it is worth speeding up the operation while ignoring some properties(e.g. table comments) and query criteria(specifically in this case, when `kyuubi.operation.getTables.ignoreTableProperties=true`, criteria `tableTypes` will be ignored, and all tables and views will be treated as TABLE to return.)

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6018 from pan3793/fast-get-table.

Closes #6018

058001c6f [Cheng Pan] fix
405b12484 [Cheng Pan] fix
615b7470f [Cheng Pan] Speed up GetTables operation

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-29 14:21:09 +08:00
senmiaoliu
777b784439
[KYUUBI #5968] Support set authentication user for Trino engine 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5968

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5970 from lsm1/branch-kyuubi-5968.

Closes #5968

580eb45d2 [senmiaoliu] fix style
a36380cde [senmiaoliu] add auth user conf for trino engine

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-21 14:26:12 +08:00
Cheng Pan
3b2e674364
[KYUUBI #5990] Always take the first declared SASL/PLAIN auth type 
# 🔍 Description
## Issue References 🔗

https://github.com/apache/kyuubi/pull/5185 changed the type of `kyuubi.authentication` from `Seq`(ordered) to `Set`(unordered), which break the assumption of
```
Note that: for SASL authentication, KERBEROS and PLAIN auth types are supported
at the same time, and only the first specified PLAIN auth type is valid.
```

## Describe Your Solution 🔧

Restore the type to Seq

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

UT is updated

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5990 from pan3793/auth-plain.

Closes #5990

acae25f68 [Cheng Pan] fix doc
cef7dba90 [Cheng Pan] fix
87b370ffe [Cheng Pan] Always take the first declared SASL/PLAIN auth type

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-18 10:03:28 +08:00
yikaifei
518e918fa7
[KYUUBI #5867][FOLLOWUP] Remove unused config 
# 🔍 Description
## Issue References 🔗

This pull request aims to remove unused config as #5867 followup.

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5976 from Yikf/remove-INTERVAL.

Closes #5867

e4183ffca [yikaifei] [KYUUBI #5867][FOLLOWUP] Remove unused config

Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2024-01-15 17:38:18 +08:00
Cheng Pan
5d59cf1d6a
[KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo 
# 🔍 Description
## Issue References 🔗

This pull request is a follow-up of #5867, to fix generated configuration docs and typo to recover CI.

## Describe Your Solution 🔧

Regenerated docs by performing `dev/gen/gen_all_config_docs.sh`
Address comment https://github.com/apache/kyuubi/pull/5868#discussion_r1438080698

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5932 from pan3793/fix-docs.

Closes #5867

93b8f97b0 [Cheng Pan] docs
0ad0d29f3 [Cheng Pan] typo

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-29 21:51:07 +08:00
yikaifei
679aca5a6c
[KYUUBI #5867] HiveEngine support run on YARN mode 
# 🔍 Description
## Issue References 🔗

This PR aims to support hive engine run on yarn mode, close https://github.com/apache/kyuubi/issues/5867

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5868 from Yikf/hive-on-yarn.

Closes #5867

44f7287f5 [yikaifei] fix
3c17d2c4a [yikaifei] fix test
5474ebfba [yikaifei] parse classpath
6b97c4213 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala
34a67b452 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala
5e5045e66 [yikaifei] fix app type
d1eb5aea7 [yikaifei] fix
d89d09cfe [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala
1fa18ba1b [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala
1b0b77f4d [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala
2ed1d4492 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala
98ff19ce6 [yikaifei] HiveEngine support run on YARN mode

Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-29 18:50:12 +08:00
Kent Yao
c18dd61952
[KYUUBI #5916] Revert #5916 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5916 as a revert

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5928 from yaooqinn/revert.

Closes #5916

5c43c0286 [Kent Yao] [KYUUBI #5916] Revert #5916

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-29 16:17:50 +08:00
Bowen Liang
b1e726dcf5 [KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution 
# 🔍 Description
## Issue References 🔗

As decribed.

## Describe Your Solution 🔧

Drop the unused config of KyuubiConf, which are never used and linked in source code and test code:
- FRONTEND_THRIFT_HTTP_ALLOW_USER_SUBSTITUTION: `kyuubi.frontend.thrift.http.allow.user.substitution`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5912 from bowenliang123/unused-configs.

Closes #5912

b938058b8 [Bowen Liang] update doc
9fd441dd0 [Bowen Liang] fix
9b0b31ee1 [Bowen Liang] cleanup unused config entries in KyuubiConf

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
 2023-12-28 15:52:52 +08:00
Bowen Liang
0f8edb2b2f [KYUUBI #5916] Use "(none)" placeholder for default value of un-preset configs on confing doc page 
# 🔍 Description
## Issue References 🔗

As descirbed.

## Describe Your Solution 🔧

- Replace the placeholder for default value of un-preset configs on doc page, changed from `<undefined>` to `(none)`
- The `(none)` placeholder style is also used in the docs of
    - Spark : https://spark.apache.org/docs/latest/configuration.html
    - Flink : https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
    - Trino (as `NONE`): https://trino.io/docs/current/admin/properties.html
- Only affects the config doc page, no functional changes.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
No functional changes.

Config doc page:
![image](https://github.com/apache/kyuubi/assets/1935105/46c40f03-d29e-4237-a0b9-a8447f032769)

#### Behavior With This Pull Request 🎉
No functional changes.

Config doc page:
![image](https://github.com/apache/kyuubi/assets/1935105/01e3a079-2ea3-405d-8346-ea4e75606ee7)

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5916 from bowenliang123/none-ph.

Closes #5916

3b141dd61 [Bowen Liang] update test
04cf8cd2a [Bowen Liang] Use "(none)" placeholder for default value of unset configs on config page

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
 2023-12-25 15:30:29 +08:00
Bowen Liang
5a751027f5 [KYUUBI #5905] [DOC] Workaround for long inlined code sample in config page 
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Workaround to fix the display issue of long inlined code sample on config page.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
![image](https://github.com/apache/kyuubi/assets/1935105/7910808d-7235-4d21-bdc5-84260af53e80)

#### Behavior With This Pull Request 🎉
![image](https://github.com/apache/kyuubi/assets/1935105/877f45a5-c49b-45fc-90a1-3c23eb2aa508)

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5905 from bowenliang123/doc-block.

Closes #5905

b4d28ed85 [Bowen Liang] update
7aa995b38 [Bowen Liang] update
ef947ea15 [Bowen Liang] update
7b3263bee [Bowen Liang] update
aefd206ae [Bowen Liang] update
f02ec3acf [Bowen Liang] update
d3302d55c [Bowen Liang] fix display by putting url demo in code block

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-22 17:27:40 +08:00
Fei Wang
fd397d6277 [KYUUBI #5900] Support to deregister the engine on open failure 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5900

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5901 from turboFei/delete_engine.

Closes #5900

57bcf1f24 [Fei Wang] docs
557714eed [Fei Wang] enum
bc479e6e0 [Fei Wang] save
2b2a4952c [Fei Wang] refine
f0ace3574 [Fei Wang] reset after close client
cf07877fe [Fei Wang] re-version
d93dd6add [Fei Wang] refactor

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2023-12-21 21:27:45 -08:00
Fei Wang
58db6ca225 [KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook 
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5898 from turboFei/notebook_mode.

Closes #5877

7f1c607b9 [Fei Wang] PySpark
644d036bc [Fei Wang] docs
7c68b7742 [Fei Wang] add option to support notebook well

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2023-12-21 21:26:35 -08:00
Cheng Pan
2aa330c850 [KYUUBI #5892] Metadata store JDBC url supports <KYUUBI_HOME> variable 
# 🔍 Description
## Issue References 🔗

Similar to https://github.com/apache/kyuubi/pull/5412, this PR aims to change the default SQLite file `kyuubi_state_store.db` for Metadata store is located under `$KYUUBI_HOME` instead of `$PWD`.

## Describe Your Solution 🔧

Make `kyuubi.metadata.store.jdbc.url` supports the variables substitution: `<KYUUBI_HOME>`, and change the default value to `jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Verified locally. Make a binary distribution, and bootstrap the Kyuubi Server process from a non-KYUUBI_HOME dir, `kyuubi_state_store.db` is created at KYUUBI_HOME as expected.

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5892 from pan3793/sqlite-jdbc-url.

Closes #5892

4341f4263 [Cheng Pan] docs
7062bcdf9 [Cheng Pan] migration guide
989de0e5b [Cheng Pan] Metadata store JDBC url supports <KYUUBI_HOME> variable

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2023-12-21 14:19:24 -08:00
Bowen Liang
4be20e1ca7 [KYUUBI #5404] JDBC Engine supports StarRocks 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5404

## Describe Your Solution 🔧

- Introduce StarRocks support in JDBC Engine
- Adding dialects and tests for StarRocks
- Tested with StarRocks 3.x official open-sourced version docker image, with a mini cluster with FE and BE setup.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉
Supported StarRocks in JDBC engine.

#### Related Unit Tests
Add tests of `StarRocksOperationSuite`, `StarRocksOperationWithEngineSuite`, `StarRocksSessionSuite` and `StarRocksStatementSuite`.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5882 from bowenliang123/jdbc-sr.

Closes #5404

54927d341 [Bowen Liang] update
5b1bbf71f [Bowen Liang] update doc
f01da74ea [Bowen Liang] update StarRocksStatementSuite
1018bc95a [Bowen Liang] MySQL8ConnectionProvider
59cba957f [Bowen Liang] simplify StarRocksDialect
4bf55bba7 [Bowen Liang] simplify StarRocksTRowSetGenerator
c85722481 [Bowen Liang] jdbc starrocks

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
 2023-12-21 16:19:36 +08:00
wforget
7b7e17b1ad
[KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5244

## Describe Your Solution 🔧

As discussed in https://github.com/apache/kyuubi/pull/5888#discussion_r1433511355, we need to correct the ENGINE_ALIVE_MAX_FAILURES  conf version.

## Types of changes 🔖

- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5889 from wForget/KYUUBI-5244-FOLLOWUP.

Closes #5244

a7854496e [wforget] [KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version

Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-21 16:03:45 +08:00
senmiaoliu
4c029f991f [KYUUBI #5377] Spark engine query result save to file 
### _Why are the changes needed?_

close #5377

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5591 from lsm1/branch-kyuubi-5377.

Closes #5377

9d1a18c15 [senmiaoliu] ignore empty file
3c70a1e9e [LSM] fix doc
73d3c3abf [senmiaoliu] fix style and add some comment
80e1f0d70 [senmiaoliu] Close orc fetchOrcStatement and remove result save file when ExecuteStatement close
42634a1d0 [senmiaoliu] fix style
979125d9b [senmiaoliu] fix style
1dc07a553 [senmiaoliu] spark engine save into hdfs file

Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: LSM <senmiaoliu@trip.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
 2023-12-13 16:03:11 +08:00
hezhao2
8ab4763ce4 [KYUUBI #5828] [CHAT] Kyuubi chat engine supports ernie bot(文心一言) 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5386

## Describe Your Solution 🔧

add a new backend(ernie bot) for the Chat engine.
<img width="1672" alt="Screenshot 2023-12-07 at 16 20 56" src="https://github.com/apache/kyuubi/assets/32693629/9850916c-3c4a-433c-8278-3068e7b37314">

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5828 from zhaohehuhu/dev-1207.

Closes #5828

c7314d4a8 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/ernie/service/ErnieBotService.scala
0c4e01007 [hezhao2] update doc
78e51b3d1 [hezhao2] add ernie into doc
2f4a63845 [hezhao2] refactor
832bc0453 [hezhao2] delete enum model
67214c575 [hezhao2] get rid of some java code
567772679 [hezhao2] java bean to scale case class
4d5c5940d [hezhao2] refactor some params
7c44eb83f [hezhao2] refactor some params
56b9ad13a [hezhao2] refactor
a8e3d6cf6 [hezhao2] refactor
7376d800d [hezhao2] Kyuubi chat engine supports ernie bot
4b72a09c0 [hezhao2] Kyuubi chat engine supports ernie bot

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-11 10:18:16 +08:00
yeatsliao
27ad1024b5
[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5795

## Describe Your Solution 🔧

Create a single daemon thread to traverse cache map periodically, which will evict expired cache and trigger a pod clean up operation.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5806 from liaoyt/master.

Closes #5795

75c2b68cc [yeatsliao] cleanup driver pod periodically

Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-07 13:55:16 +08:00
kandy01.wang
f3f643a309
[KYUUBI #5756] Introduce specified initialized SQL to every engine 
# 🔍 Description
## Issue References 🔗

This pull request fixes #5756

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5821 from hadoopkandy/KYUUBI-5756.

Closes #5756

046fe2a58 [kandy01.wang] [KYUUBI #5756] Introduce specified initialized SQL to every engine

Authored-by: kandy01.wang <kandy01.wang@vipshop.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-12-07 11:10:57 +08:00
wangjunbo
5481bf54f2 [KYUUBI #5464] JDBC Engine supports MySQL 
### _Why are the changes needed?_

To close https://github.com/apache/kyuubi/issues/5464.
To support JDBC engine use MySQL Dialect (kyuubi.engine.jdbc.type=mysql).

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5588 from Kwafoor/kyuubi_5464.

Closes #5464

1019a6118 [wangjunbo] [KYUUBI #5464]rename function name `getProviderClass` to `getDriverClass`
9901bbad4 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
b33d79ed2 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
86e6ee2b3 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
d76cb3275 [wangjunbo] [KYUUBI #5464]update the docs
4a1acffd1 [wangjunbo] [KYUUBI #5464]update the docs
1aff55ecd [wangjunbo] [KYUUBI #5464]update the docs of kyuubi.engine.type
84202ea0c [wangjunbo] [KYUUBI #5464] update the docs of kyuubi.engine.type
e3c1e94db [wangjunbo] [KYUUBI #5464] fix check
cdf820da0 [wangjunbo] [KYUUBI #5464] fix check
ff0f30ad8 [wangjunbo] [KYUUBI #5464] fix check
01321dc44 [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL
756f5303c [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL

Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-24 21:17:17 +08:00
Yuwei Zhan
931b5026e7 [KYUUBI #5737] [DOC] Correct JDBC URL demo in Flink docs 
# 🔍 Description
flink setting need additional prefix flink. to override settings
## Issue References 🔗

This pull request fixes #5737

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5745 from dik111/update-doc.

Closes #5737

27ac6c1db [yuwei zhan] Update AllKyuubiConfiguration about flink jdbc url
a1b1b2cba [yuwei zhan] Update settings.md

Lead-authored-by: Yuwei Zhan <195812208@qq.com>
Co-authored-by: yuwei zhan <195812208@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-21 21:56:07 +08:00
yeatsliao
dc03687248 [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy 
# 🔍 Description

## Describe Your Solution 🔧

A new feature introduced from #5714 supports kyuubi to clean up spark driver pods automatically, but all pod would be clean up without considering app's terminated state.
This PR make user can chose which pods should be delete by setting up a cleanup strategy.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Test locally.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5728 from liaoyt/master.

Closes #5731

d2cc8cb55 [yeatsliao] regenerate docs
4caf8b152 [yeatsliao] rename conf 'KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION' to 'KUBERNETES_SPARK_CLEANUP_TERMINATED_DRIVER_POD'
4d970fa14 [yeatsliao] [K8S] Support to cleanup the spark driver pod with specified clean up strategy

Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-20 16:59:57 +08:00
fwang12
765cdaa255 [KYUUBI #5711] [K8S] Support to build application state and error from pod container 
# 🔍 Description

This pr is to support to build the application state from pod container instead of pod.

The use case is that:
For spark on k8s, the pod contains two containers.
- one is the spark driver main container
- the another one is a sidecar container

If the spark driver main container completed but the sidecar container not, the batch application state is always RUNNING as the pod state is `NotReady`.

It is not expected.

So in this pr, I want to introduce a new option config to support to build application state from the pod container.

## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5711 from turboFei/k8s_pod_state.

Closes #5711

9e0575b5f [fwang12] save
eaec5d2bb [fwang12] comment
61ae5f723 [fwang12] comment
1eac84f2d [fwang12] save
bca19be3a [fwang12] nit
38115b752 [fwang12] [K8S] Support to build application state and error from pod container

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2023-11-17 14:45:16 +08:00
fwang12
88fae49d7f [KYUUBI #5714] [K8S] Support to cleanup the spark driver pod after application terminates for retain period 
# 🔍 Description

## Describe Your Solution 🔧

As title, support to cleanup the application pod after application terminates for retain period

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Test locally.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5714 from turboFei/kill_k8s_pod.

Closes #5714

1e0787868 [fwang12] doc
0c9ff1a44 [fwang12] cleanup pod
ab95d4c03 [fwang12] save

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2023-11-16 19:33:00 +08:00
senmiaoliu
e498bdba00
[KYUUBI #5582] JDBC Engine supports configurable default fetchSize 
### _Why are the changes needed?_

close #5582

JDBC Engine supports configurable default fetchSize

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5614 from lsm1/branch-kyuubi-5582.

Closes #5582

dd58a6d51 [senmiaoliu] fix style
d0eb7c5dc [senmiaoliu] set jdbc fetch size in operation manager
4eee26b7b [senmiaoliu] rename conf key
4d8b74796 [senmiaoliu] add conf for JDBC engine default fetch size

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
 2023-11-15 12:15:05 +08:00
mrtisttt
50cd91b59e [KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user 
### _Why are the changes needed?_

We need a configuration to be an alternative to `hive.server2.proxy.user`. `hive.server2.proxy.user` is derived from the hive server, and it's perfect for keeping the behavior the same as the hive. But hive thrift protocol is only one of the FrontendServices. There are also other FrontendServices not necessary to follow `hive.server2.proxy.user`. So we need `kyuubi.proxy.user`.

In this commit, support for using the `kyuubi.proxy.user` configuration has been implemented in scenarios where the original `hive.server2.proxy.user` configuration was used.In the future, this feature will continue to be improved according to other scenarios where it is used.

Close #5683

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5653 from mrtisttt/add-kyuubi-proxy-user.

Closes #5683

6d60e7632 [mrtisttt] Rollback unnecessary modifications
7298e4137 [mrtisttt] fix doc violations
19f6f4084 [mrtisttt] For API side, shorten the name of kyuubi.session.proxy.user and change it to proxyUser.
0fbcc704d [mrtisttt] Change kyuubi.proxy.user to kyuubi.session.proxy.user and improve documentation
c2a810ce2 [mrtisttt] fix format violations
a6413e8aa [mrtisttt] Compatible with scenarios where hive.server2.proxy.user is used
b687c21b9 [mrtisttt] Add kyuubi.proxy.user as an alternative to hive.server2.proxy.user

Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-13 20:08:33 +08:00
hezhao2
0ce697eefa [KYUUBI #5654] Introduce new config to forcibly rewrite pod name 
### _Why are the changes needed?_
close #5654
This pr is to introduce a config for rewriting pod name, which can help users forcibly rewrite driver or executor pod name prefix.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5672 from zhaohehuhu/dev-1111.

Closes #5654

6a450e444 [hezhao2] introduce new config to force rewrite pod name

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-13 10:21:05 +08:00
Cheng Pan
9bf234473a [KYUUBI #5568] Check administrator only when security is enabled 
### _Why are the changes needed?_

This PR aims to improve out-of-box UX.

Currently, administrators' checking is enforced even when security is disabled, and we are going to deliver the preview of Web UI in 1.8, it may confuse users who use the default configuration to launch Kyuubi sever and explore Kyuubi WebUI (because the user "anonymous" can not visit all users' sessions/engines without being configured as an administrator).

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5568 from pan3793/admin.

Closes #5568

2eccf659f [Cheng Pan] code dedup and fix test
76f4fd4a3 [Cheng Pan] fix test
d7ca94650 [Cheng Pan] Check administrator only when security is enabled

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-11-02 10:09:01 +08:00
senmiaoliu
c24e984a3e
[KYUUBI #5423] Support chaining SessionConfAdvisors 
### _Why are the changes needed?_

close #5423

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5485 from lsm1/branch-5423.

Closes #5423

0bd3b0233 [senmiaoliu] mention in the migration guide
dddafeb9b [senmiaoliu] support multi session conf advisor

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
 2023-10-30 09:48:25 +08:00
lawulu
30a127a1ba [KYUUBI #5412] Resolve the relative zk configuration dir based on KYUUBI_HOME 
### _Why are the changes needed?_
Both   the default  value of `kyuubi.zookeeper.embedded.data.log.dir` and `kyuubi.zookeeper.embedded.data.dir` are `embedded_zookeeper`. It is used based on the current dir.
If the configuration dir is a relative path , it is resolved relative to `KYUUBI_HOME`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5412 from biangjuang/master.

Closes #5412

17e502c03 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
e0aa01c82 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
bb6af93df [Cheng Pan] Update docs/deployment/migration-guide.md
53bd61bc3 [lawulu] Resolve the relative zk configuration dir based on KYUUBI_HOME

Lead-authored-by: lawulu <biangjuang@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-10-27 10:48:22 +08:00
liupeiyue
9a409956a1 [KYUUBI #5282] Support configure Trino session conf in kyuubi-default.conf 
### _Why are the changes needed?_

Refer to spark and flink settings conf,  support configure Trino session conf in kyuubi-default.conf
issue : https://github.com/apache/kyuubi/issues/5282

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5283 from ASiegeLion/kyuubi-master-trino.

Closes #5282

87a3f57b4 [Cheng Pan] Apply suggestions from code review
effdd79f4 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
399a200f7 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
7462b32c2 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml--Update documentation
5295f5f94 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml

Lead-authored-by: liupeiyue <liupeiyue@yy.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2023-10-24 16:19:44 +08:00