Commit Graph

709 Commits

Author SHA1 Message Date
Cheng Pan
8f59355eb6
Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table"
This reverts commit 97a0ba461c.
2023-12-24 01:53:05 +08:00
Cheng Pan
97a0ba461c [KYUUBI #5908] [DOCS] Remove workaround for malformed table
# 🔍 Description
## Issue References 🔗

The `custom.css` was used to address the `sphinx-book-theme` bugs which generated malformed tables, see #1889 #2764 for details.

With the recent upgrading of `sphinx-book-theme 1.1.0`, it's unnecessary.

## Describe Your Solution 🔧

Remove useless `custom.css`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

<img width="1376" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1ef116f8-6be1-49ee-8f6e-3b844eae284a">

#### Behavior With This Pull Request 🎉

<img width="1376" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/b12e461b-6294-4a4e-b034-fbd093b10ca4">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5908 from pan3793/remove-custom-css.

Closes #5908

a958b774c [Cheng Pan] nit
a3f592645 [Cheng Pan] [DOCS] Remove workaround for malformed table

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-12-23 00:10:13 +08:00
Bowen Liang
5a751027f5 [KYUUBI #5905] [DOC] Workaround for long inlined code sample in config page
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Workaround to fix the display issue of long inlined code sample on config page.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
![image](https://github.com/apache/kyuubi/assets/1935105/7910808d-7235-4d21-bdc5-84260af53e80)

#### Behavior With This Pull Request 🎉
![image](https://github.com/apache/kyuubi/assets/1935105/877f45a5-c49b-45fc-90a1-3c23eb2aa508)

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5905 from bowenliang123/doc-block.

Closes #5905

b4d28ed85 [Bowen Liang] update
7aa995b38 [Bowen Liang] update
ef947ea15 [Bowen Liang] update
7b3263bee [Bowen Liang] update
aefd206ae [Bowen Liang] update
f02ec3acf [Bowen Liang] update
d3302d55c [Bowen Liang] fix display by putting url demo in code block

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-22 17:27:40 +08:00
Fei Wang
fd397d6277 [KYUUBI #5900] Support to deregister the engine on open failure
# 🔍 Description
## Issue References 🔗

This pull request fixes #5900

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5901 from turboFei/delete_engine.

Closes #5900

57bcf1f24 [Fei Wang] docs
557714eed [Fei Wang] enum
bc479e6e0 [Fei Wang] save
2b2a4952c [Fei Wang] refine
f0ace3574 [Fei Wang] reset after close client
cf07877fe [Fei Wang] re-version
d93dd6add [Fei Wang] refactor

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2023-12-21 21:27:45 -08:00
Fei Wang
58db6ca225 [KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5898 from turboFei/notebook_mode.

Closes #5877

7f1c607b9 [Fei Wang] PySpark
644d036bc [Fei Wang] docs
7c68b7742 [Fei Wang] add option to support notebook well

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2023-12-21 21:26:35 -08:00
Bowen Liang
0640a00e3c [KYUUBI #5893] [DOC] Update logos in layer picture on welcome page
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- update the logos of DBeaver, Mesos and DeltaLake with white background
- add StarRocks logo to the engine layer
- add Apache Paimon(Incubating) logo to the storage layer
- update JDBC and Python logo on driver layer
- add REST API text on driver layer
- remove outdated SQuirrelSQLClient logo
- fix typo in `etc,` into `etc.,`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

![image](https://github.com/apache/kyuubi/assets/1935105/6b6c0cf3-11a4-447a-b593-0aa561e4c149)

![image](https://github.com/apache/kyuubi/assets/1935105/f2792695-7fdb-4374-a4f3-4b858dbd93e8)

#### Behavior With This Pull Request 🎉

![image](https://github.com/apache/kyuubi/assets/1935105/c11aef1e-69eb-4ebb-a72f-a82c910af708)

<img width="750" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/6d6b058d-b295-478f-a561-0581107c8206">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5893 from bowenliang123/layer-logo.

Closes #5893

8b13ec65f [Bowen Liang] mentioning paimon
ea4686d71 [liangbowen] update
f1a9c1b1c [Bowen Liang] update
87c8b94e5 [Bowen Liang] update logos in layer

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-22 10:47:00 +08:00
Bowen Liang
feddca4523 [KYUUBI #5902] Bump doc build dependencies
# 🔍 Description
## Issue References 🔗

As descirbed.

## Describe Your Solution 🔧

- Bump pip dependencies' version of doc building tools
    - sphinx 4.5.0 -> 7.2.6
    - sphinx-book-theme 0.3.3-> 1.1.0
    - markdown 3.4.1-> 3.5.1
- Upgraded `sphinx-book-theme` brings
    - dark mode support (light/dark/system)
    - new search bar style
    - less text hits shown in searched results

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
![image](https://github.com/apache/kyuubi/assets/1935105/561df239-ba51-4044-a9f4-25d213f2e82f)

![image](https://github.com/apache/kyuubi/assets/1935105/6dacbb12-c0d8-4a90-aa33-88e326bd3590)

#### Behavior With This Pull Request 🎉
![image](https://github.com/apache/kyuubi/assets/1935105/395629f3-1f25-4c31-bdbe-717658a85022)

![image](https://github.com/apache/kyuubi/assets/1935105/9f43ae96-f410-4b30-a7cf-0ee2143ef0b7)

![image](https://github.com/apache/kyuubi/assets/1935105/5d14ca7a-92e2-4215-83ef-1fd056303094)

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [ ] Milestone correctly set?
- [x] Test coverage is ok
- [ ] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5902 from bowenliang123/docdep-bump.

Closes #5902

db9d8671a [Bowen Liang] bump doc build dependencies

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2023-12-21 18:37:43 -08:00
Cheng Pan
2aa330c850 [KYUUBI #5892] Metadata store JDBC url supports <KYUUBI_HOME> variable
# 🔍 Description
## Issue References 🔗

Similar to https://github.com/apache/kyuubi/pull/5412, this PR aims to change the default SQLite file `kyuubi_state_store.db` for Metadata store is located under `$KYUUBI_HOME` instead of `$PWD`.

## Describe Your Solution 🔧

Make `kyuubi.metadata.store.jdbc.url` supports the variables substitution: `<KYUUBI_HOME>`, and change the default value to `jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Verified locally. Make a binary distribution, and bootstrap the Kyuubi Server process from a non-KYUUBI_HOME dir, `kyuubi_state_store.db` is created at KYUUBI_HOME as expected.

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5892 from pan3793/sqlite-jdbc-url.

Closes #5892

4341f4263 [Cheng Pan] docs
7062bcdf9 [Cheng Pan] migration guide
989de0e5b [Cheng Pan] Metadata store JDBC url supports <KYUUBI_HOME> variable

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2023-12-21 14:19:24 -08:00
Bowen Liang
4be20e1ca7 [KYUUBI #5404] JDBC Engine supports StarRocks
# 🔍 Description
## Issue References 🔗

This pull request fixes #5404

## Describe Your Solution 🔧

- Introduce StarRocks support in JDBC Engine
- Adding dialects and tests for StarRocks
- Tested with StarRocks 3.x official open-sourced version docker image, with a mini cluster with FE and BE setup.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉
Supported StarRocks in JDBC engine.

#### Related Unit Tests
Add tests of `StarRocksOperationSuite`, `StarRocksOperationWithEngineSuite`, `StarRocksSessionSuite` and `StarRocksStatementSuite`.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5882 from bowenliang123/jdbc-sr.

Closes #5404

54927d341 [Bowen Liang] update
5b1bbf71f [Bowen Liang] update doc
f01da74ea [Bowen Liang] update StarRocksStatementSuite
1018bc95a [Bowen Liang] MySQL8ConnectionProvider
59cba957f [Bowen Liang] simplify StarRocksDialect
4bf55bba7 [Bowen Liang] simplify StarRocksTRowSetGenerator
c85722481 [Bowen Liang] jdbc starrocks

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-21 16:19:36 +08:00
wforget
7b7e17b1ad
[KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version
# 🔍 Description
## Issue References 🔗

This pull request fixes #5244

## Describe Your Solution 🔧

As discussed in https://github.com/apache/kyuubi/pull/5888#discussion_r1433511355, we need to correct the ENGINE_ALIVE_MAX_FAILURES  conf version.

## Types of changes 🔖

- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5889 from wForget/KYUUBI-5244-FOLLOWUP.

Closes #5244

a7854496e [wforget] [KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version

Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-21 16:03:45 +08:00
senmiaoliu
4c029f991f [KYUUBI #5377] Spark engine query result save to file
### _Why are the changes needed?_

close #5377

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5591 from lsm1/branch-kyuubi-5377.

Closes #5377

9d1a18c15 [senmiaoliu] ignore empty file
3c70a1e9e [LSM] fix doc
73d3c3abf [senmiaoliu] fix style and add some comment
80e1f0d70 [senmiaoliu] Close orc fetchOrcStatement and remove result save file when ExecuteStatement close
42634a1d0 [senmiaoliu] fix style
979125d9b [senmiaoliu] fix style
1dc07a553 [senmiaoliu] spark engine save into hdfs file

Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: LSM <senmiaoliu@trip.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-12-13 16:03:11 +08:00
hezhao2
8ab4763ce4 [KYUUBI #5828] [CHAT] Kyuubi chat engine supports ernie bot(文心一言)
# 🔍 Description
## Issue References 🔗

This pull request fixes #5386

## Describe Your Solution 🔧

add a new backend(ernie bot) for the Chat engine.
<img width="1672" alt="Screenshot 2023-12-07 at 16 20 56" src="https://github.com/apache/kyuubi/assets/32693629/9850916c-3c4a-433c-8278-3068e7b37314">

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5828 from zhaohehuhu/dev-1207.

Closes #5828

c7314d4a8 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/ernie/service/ErnieBotService.scala
0c4e01007 [hezhao2] update doc
78e51b3d1 [hezhao2] add ernie into doc
2f4a63845 [hezhao2] refactor
832bc0453 [hezhao2] delete enum model
67214c575 [hezhao2] get rid of some java code
567772679 [hezhao2] java bean to scale case class
4d5c5940d [hezhao2] refactor some params
7c44eb83f [hezhao2] refactor some params
56b9ad13a [hezhao2] refactor
a8e3d6cf6 [hezhao2] refactor
7376d800d [hezhao2] Kyuubi chat engine supports ernie bot
4b72a09c0 [hezhao2] Kyuubi chat engine supports ernie bot

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-11 10:18:16 +08:00
Cheng Pan
b594105f14
[KYUUBI #5830] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
# 🔍 Description
## Issue References 🔗

The CDH Hive 2.1.1-cdh6.3.2 is the latest free version of CDH, which is adopted widely, it would be great if the Kyuubi Hive engine could support this version.

## Describe Your Solution 🔧

Actually, the Kyuubi Hive engine could work with CDH Hive 2.1.1-cdh6.3.2, this pull request just adds an integration test and fixes one test case.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5830 from pan3793/hive-211-cdh6.

Closes #5830

afa80bd82 [Cheng Pan] nit
89c9ae96e [Cheng Pan] nit
e402e49cd [Cheng Pan] nit
3c126a2a6 [Cheng Pan] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-08 09:57:08 +08:00
yeatsliao
27ad1024b5
[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically
# 🔍 Description
## Issue References 🔗

This pull request fixes #5795

## Describe Your Solution 🔧

Create a single daemon thread to traverse cache map periodically, which will evict expired cache and trigger a pod clean up operation.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5806 from liaoyt/master.

Closes #5795

75c2b68cc [yeatsliao] cleanup driver pod periodically

Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-07 13:55:16 +08:00
kandy01.wang
f3f643a309
[KYUUBI #5756] Introduce specified initialized SQL to every engine
# 🔍 Description
## Issue References 🔗

This pull request fixes #5756

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5821 from hadoopkandy/KYUUBI-5756.

Closes #5756

046fe2a58 [kandy01.wang] [KYUUBI #5756] Introduce specified initialized SQL to every engine

Authored-by: kandy01.wang <kandy01.wang@vipshop.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-07 11:10:57 +08:00
wangjunbo
5b6a729fa8
[KYUUBI #5800] [KYUUBI#5467] Integrate Intel Gluten with Spark engine
# 🔍 Description
## Issue References 🔗

This pull request fixes #5467

## Describe Your Solution 🔧

1. Add Gluten UTs.
2. Setup CI for Gluten testing
3. Write docs to guide users in setting up Kyuubi with Spark plus Gluten.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

github action ci tests: [Gluten Test CI](https://github.com/Kwafoor/incubator-kyuubi/actions/runs/7111586978)

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5800 from Kwafoor/kyuubi_5467.

Closes #5800

c6dd26f93 [wangjunbo] fix
7818ae0c5 [wangjunbo] fix Scala Test
296f08c8c [wangjunbo] remove spark-3.2 gluten test
5a704675d [wangjunbo] [KYUUBI#5467] Integrate Intel Gluten with Spark engine

Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-12-07 10:47:00 +08:00
Cheng Pan
52d25c7cf1
[KYUUBI #5819] [DOCS] Fix rst syntax of missing blank line after code-block
# 🔍 Description
## Issue References 🔗

This pull request fixes a rst syntax of missing blank line after code-block

## Describe Your Solution 🔧

Add a blank line.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
<img width="1316" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/a35725aa-03be-4402-a482-6cfa74430293">

#### Behavior With This Pull Request 🎉
<img width="1337" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/acc643c0-9b91-49c6-97ae-44cc83874ea5">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5819 from pan3793/hudi-doc.

Closes #5819

45149c5ac [Cheng Pan] fix
3236c9692 [Cheng Pan] [DOCS] Fix rst syntax in Hudi connector page

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-05 21:56:24 +08:00
liangbowen
3ff4593b09 [KYUUBI #5802] [DOC] Add scala-2.13 as supported Scala profile to the docs of building from source
# 🔍 Description
## Issue References 🔗

As described.

## Describe Your Solution 🔧

Add docs for spark-3.4 and spark-3.5 as supported Spark profiles.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
No related docs.

#### Behavior With This Pull Request 🎉
<img width="766" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/a217b939-f97a-4176-b320-a047eb142609">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5802 from bowenliang123/docs-scala-213.

Closes #5802

d73420713 [liangbowen] add scala-2.13 profile to the docs of building from code

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: wforget <643348094@qq.com>
2023-12-05 11:37:20 +08:00
zml1206
762ccd8295 [KYUUBI #5786] Disable spark script transformation
# 🔍 Description
## Issue References 🔗

This pull request fixes #5786.

## Describe Your Solution 🔧

Add spark check rule.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests
org.apache.kyuubi.plugin.spark.authz.rule.AuthzUnsupportedOperationsCheckSuite.test("disable script transformation")

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5788 from zml1206/KYUUBI-5786.

Closes #5786

06c0098be [zml1206] fix
e2c3fee22 [zml1206] fix
37744f4c3 [zml1206] move to spark extentions
deb09fb30 [zml1206] add configuration
cfea4845a [zml1206] Disable spark script transformation in Authz

Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: wforget <643348094@qq.com>
2023-12-05 11:16:30 +08:00
liangbowen
2a39d697c0 [KYUUBI #5801] [DOC] Add spark-3.4 and spark-3.5 as supported Spark profiles to the docs of building from source
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Add docs for spark-3.4 and spark-3.5 as supported Spark profiles.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

<img width="774" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/29b0690d-574f-4b3c-ac70-50d5fb67a75e">

#### Behavior With This Pull Request 🎉

<img width="766" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/58af356b-0f9c-4fe9-8f2b-8769ccebcc3a">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5801 from bowenliang123/doc-spark-profiles.

Closes #5801

35254df09 [liangbowen] style
2b55824fd [liangbowen] add spark-3.4 and spark-3.5 to the docs of building from code

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-04 08:46:17 +08:00
hezhao2
3677f3a26e
[KYUUBI #5762] [DOC] Introduce Kyuubi Web UI Engine Page
# 🔍 Description
## Issue References 🔗

This pull request fixes #5572

## Describe Your Solution 🔧

Introduce Kyuubi Web UI Engine Page as the screenshot shown.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

<img width="835" alt="Screenshot 2023-11-27 at 10 39 55" src="https://github.com/apache/kyuubi/assets/32693629/363426e8-9808-4fd7-a711-2f0f80164147">

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5762 from zhaohehuhu/dev-1123.

Closes #5762

4a85af03e [hezhao2] compress picture
5aed43e63 [hezhao2] refactor
16d2d2f83 [hezhao2] reformat
ad9c30bf1 [hezhao2] Introduce Kyuubi Web UI Engine Page

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Kent Yao <yao@apache.org>
2023-12-01 18:00:13 +08:00
wangjunbo
5481bf54f2 [KYUUBI #5464] JDBC Engine supports MySQL
### _Why are the changes needed?_

To close https://github.com/apache/kyuubi/issues/5464.
To support JDBC engine use MySQL Dialect (kyuubi.engine.jdbc.type=mysql).

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5588 from Kwafoor/kyuubi_5464.

Closes #5464

1019a6118 [wangjunbo] [KYUUBI #5464]rename function name `getProviderClass` to `getDriverClass`
9901bbad4 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
b33d79ed2 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
86e6ee2b3 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity
d76cb3275 [wangjunbo] [KYUUBI #5464]update the docs
4a1acffd1 [wangjunbo] [KYUUBI #5464]update the docs
1aff55ecd [wangjunbo] [KYUUBI #5464]update the docs of kyuubi.engine.type
84202ea0c [wangjunbo] [KYUUBI #5464] update the docs of kyuubi.engine.type
e3c1e94db [wangjunbo] [KYUUBI #5464] fix check
cdf820da0 [wangjunbo] [KYUUBI #5464] fix check
ff0f30ad8 [wangjunbo] [KYUUBI #5464] fix check
01321dc44 [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL
756f5303c [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL

Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-24 21:17:17 +08:00
Yuwei Zhan
931b5026e7 [KYUUBI #5737] [DOC] Correct JDBC URL demo in Flink docs
# 🔍 Description
flink setting need additional prefix flink. to override settings
## Issue References 🔗

This pull request fixes #5737

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5745 from dik111/update-doc.

Closes #5737

27ac6c1db [yuwei zhan] Update AllKyuubiConfiguration about flink jdbc url
a1b1b2cba [yuwei zhan] Update settings.md

Lead-authored-by: Yuwei Zhan <195812208@qq.com>
Co-authored-by: yuwei zhan <195812208@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-21 21:56:07 +08:00
yeatsliao
dc03687248 [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy
# 🔍 Description

## Describe Your Solution 🔧

A new feature introduced from #5714 supports kyuubi to clean up spark driver pods automatically, but all pod would be clean up without considering app's terminated state.
This PR make user can chose which pods should be delete by setting up a cleanup strategy.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Test locally.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5728 from liaoyt/master.

Closes #5731

d2cc8cb55 [yeatsliao] regenerate docs
4caf8b152 [yeatsliao] rename conf 'KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION' to 'KUBERNETES_SPARK_CLEANUP_TERMINATED_DRIVER_POD'
4d970fa14 [yeatsliao] [K8S] Support to cleanup the spark driver pod with specified clean up strategy

Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-20 16:59:57 +08:00
fwang12
3478fc9dfb [KYUUBI #5717] Infer the proxy user automatically for delete batch operation
# 🔍 Description
Infer the batch user from session or metadata, user do not need to specify the proxy user anymore.

This pr also align the behavior of BatchesResource with that of SessionsResource and OperationsResource(no proxy user parameter).

For Kyuubi Batch, Session and Operation, these resources have the user attiribute.

So we only need to check whether the authentication user has the permission to access the resource.

## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5717 from turboFei/hive_server2_proxy_user.

Closes #5717

70ad7e76d [fwang12] comment
c721a751a [fwang12] ignore
da92bd5a1 [fwang12] fix ut
9a197d005 [fwang12] doc
c8ed5f9cf [fwang12] ut
cef9e329c [fwang12] do not use proxy user

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-11-17 20:52:40 +08:00
fwang12
765cdaa255 [KYUUBI #5711] [K8S] Support to build application state and error from pod container
# 🔍 Description

This pr is to support to build the application state from pod container instead of pod.

The use case is that:
For spark on k8s, the pod contains two containers.
- one is the spark driver main container
- the another one is a sidecar container

If the spark driver main container completed but the sidecar container not, the batch application state is always RUNNING as the pod state is `NotReady`.

It is not expected.

So in this pr, I want to introduce a new option config to support to build application state from the pod container.

## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5711 from turboFei/k8s_pod_state.

Closes #5711

9e0575b5f [fwang12] save
eaec5d2bb [fwang12] comment
61ae5f723 [fwang12] comment
1eac84f2d [fwang12] save
bca19be3a [fwang12] nit
38115b752 [fwang12] [K8S] Support to build application state and error from pod container

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-11-17 14:45:16 +08:00
fwang12
88fae49d7f [KYUUBI #5714] [K8S] Support to cleanup the spark driver pod after application terminates for retain period
# 🔍 Description

## Describe Your Solution 🔧

As title, support to cleanup the application pod after application terminates for retain period

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Test locally.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5714 from turboFei/kill_k8s_pod.

Closes #5714

1e0787868 [fwang12] doc
0c9ff1a44 [fwang12] cleanup pod
ab95d4c03 [fwang12] save

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-11-16 19:33:00 +08:00
Kent Yao
10d0aecde1
[KYUUBI #5701] Enhance the PR Template to make PR desc informative
# 🔍 Description

## Issue References 🔗

This pull request fixes #5701

## Describe Your Solution 🔧

This PR updates the PR template to ensure that PR authors provide informative PR descriptions for reviewers to speed up the review process.

It's also suitable for improving our code quality.

Besides, `docs/contributing/code/style.rst` is also changed to add a link to the doc style so that we can keep the PR template neat.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

See the screenshot below.
![image](https://github.com/apache/kyuubi/assets/8326978/94a71e97-a573-47d9-8f44-f84b0f38db72)

#### Behavior With This Pull Request 🎉

It's able to check it from the description of this PR directly.

#### Related Unit Tests

No

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5702 from yaooqinn/pr.

Closes #5701

19bdea9a4 [Kent Yao] not
d7d9d8cfa [Kent Yao] Enhance PR Template to make PR desc informative
be026e4a6 [Kent Yao] Enhance PR Template to make PR desc informative

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2023-11-15 18:36:47 +08:00
senmiaoliu
e498bdba00
[KYUUBI #5582] JDBC Engine supports configurable default fetchSize
### _Why are the changes needed?_

close #5582

JDBC Engine supports configurable default fetchSize

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5614 from lsm1/branch-kyuubi-5582.

Closes #5582

dd58a6d51 [senmiaoliu] fix style
d0eb7c5dc [senmiaoliu] set jdbc fetch size in operation manager
4eee26b7b [senmiaoliu] rename conf key
4d8b74796 [senmiaoliu] add conf for JDBC engine default fetch size

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-11-15 12:15:05 +08:00
mrtisttt
50cd91b59e [KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user
### _Why are the changes needed?_

We need a configuration to be an alternative to `hive.server2.proxy.user`. `hive.server2.proxy.user` is derived from the hive server, and it's perfect for keeping the behavior the same as the hive. But hive thrift protocol is only one of the FrontendServices. There are also other FrontendServices not necessary to follow `hive.server2.proxy.user`. So we need `kyuubi.proxy.user`.

In this commit, support for using the `kyuubi.proxy.user` configuration has been implemented in scenarios where the original `hive.server2.proxy.user` configuration was used.In the future, this feature will continue to be improved according to other scenarios where it is used.

Close #5683

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5653 from mrtisttt/add-kyuubi-proxy-user.

Closes #5683

6d60e7632 [mrtisttt] Rollback unnecessary modifications
7298e4137 [mrtisttt] fix doc violations
19f6f4084 [mrtisttt] For API side, shorten the name of kyuubi.session.proxy.user and change it to proxyUser.
0fbcc704d [mrtisttt] Change kyuubi.proxy.user to kyuubi.session.proxy.user and improve documentation
c2a810ce2 [mrtisttt] fix format violations
a6413e8aa [mrtisttt] Compatible with scenarios where hive.server2.proxy.user is used
b687c21b9 [mrtisttt] Add kyuubi.proxy.user as an alternative to hive.server2.proxy.user

Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 20:08:33 +08:00
hezhao2
0ce697eefa [KYUUBI #5654] Introduce new config to forcibly rewrite pod name
### _Why are the changes needed?_
close #5654
This pr is to introduce a config for rewriting pod name, which can help users forcibly rewrite driver or executor pod name prefix.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5672 from zhaohehuhu/dev-1111.

Closes #5654

6a450e444 [hezhao2] introduce new config to force rewrite pod name

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 10:21:05 +08:00
GianlucaPrincipini
4be584cabe [KYUUBI #5599] Correct the parameter name of fetching op log in RESTful API
### _Why are the changes needed?_

It corrects the fetching op log Rest API documentation to match request parameter case in code, solving issue #5599

### _How was this patch tested?_
No tests needed.

### _Was this patch authored or co-authored using generative AI tooling?_

No AI tool has been used for this patch.

Closes #5673 from GianlucaPrincipini/fix-fetching-op-log-param-doc.

Closes #5599

f134af050 [GianlucaPrincipini] Change maxRows "param" to "maxrows" in documentation

Authored-by: GianlucaPrincipini <gianluca.principini@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 10:10:07 +08:00
fwang12
c149809fb9 [KYUUBI #5564] Support to config ticket cache location in JDBC URL
### _Why are the changes needed?_

Support to config customized ticket cache location in kyuubi connection URL.

Before, we can only set it via environment variable `KRB5CCNAME`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5564 from turboFei/ticket_cache.

Closes #5564

140ae6ec2 [fwang12] since 1.8.0
caaf33a44 [fwang12] check is blank
0fae9f9c5 [fwang12] doc
05b9e9df8 [fwang12] ticket cache

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-03 00:35:27 +08:00
Cheng Pan
8980e0752d
Revert "[KYUUBI #4994][REST] Support listing all engines"
This reverts commit 16ae852891.
2023-11-03 00:25:18 +08:00
Cheng Pan
9bf234473a [KYUUBI #5568] Check administrator only when security is enabled
### _Why are the changes needed?_

This PR aims to improve out-of-box UX.

Currently, administrators' checking is enforced even when security is disabled, and we are going to deliver the preview of Web UI in 1.8, it may confuse users who use the default configuration to launch Kyuubi sever and explore Kyuubi WebUI (because the user "anonymous" can not visit all users' sessions/engines without being configured as an administrator).

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5568 from pan3793/admin.

Closes #5568

2eccf659f [Cheng Pan] code dedup and fix test
76f4fd4a3 [Cheng Pan] fix test
d7ca94650 [Cheng Pan] Check administrator only when security is enabled

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-02 10:09:01 +08:00
zhouyifan279
d9a26c3b09 [KYUUBI #5480] Support setting kyuubi hive jdbc client protocol version
### _Why are the changes needed?_
When using Kyuubi Hive JDBC to Hive Server2, TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10 is used and can not be changed.

When we connected to Hive Server2 with version lower than 2.2.0, we got the following error:
```
org.apache.kyuubi.shade.org.apache.thrift.TApplicationException:
Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, configuration:{kyuubi.client.version=1.7.3, set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, kyuubi.client.ipAddress=172.16.19.113, use:database=default})
```

In this PR, we introduced a session conf `clientProtocolVersion`.
By adding `clientProtocolVersion=8` to jdbc url, the error got fixed.

Changes of `kyuubi_jdbc.rst`

<img width="867" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/8f98edf9-15c4-4d1b-9299-83b24136352b">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Manual tests against Hive Server2 version 2.1.1-cdh6.3.0

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No.

Closes #5480 from zhouyifan279/protocol-version.

Closes #5480

0ee7b1f68 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
61b8038d3 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
ed4c29f2b [Cheng Pan] Update docs/deployment/migration-guide.md
dc16a05fc [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
1a6bfd88c [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
cb00edc37 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
c99fc4802 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version

Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-30 23:51:54 +08:00
senmiaoliu
c24e984a3e
[KYUUBI #5423] Support chaining SessionConfAdvisors
### _Why are the changes needed?_

close #5423

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5485 from lsm1/branch-5423.

Closes #5423

0bd3b0233 [senmiaoliu] mention in the migration guide
dddafeb9b [senmiaoliu] support multi session conf advisor

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-10-30 09:48:25 +08:00
lawulu
30a127a1ba [KYUUBI #5412] Resolve the relative zk configuration dir based on KYUUBI_HOME
### _Why are the changes needed?_
Both   the default  value of `kyuubi.zookeeper.embedded.data.log.dir` and `kyuubi.zookeeper.embedded.data.dir` are `embedded_zookeeper`. It is used based on the current dir.
If the configuration dir is a relative path , it is resolved relative to `KYUUBI_HOME`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5412 from biangjuang/master.

Closes #5412

17e502c03 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
e0aa01c82 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
bb6af93df [Cheng Pan] Update docs/deployment/migration-guide.md
53bd61bc3 [lawulu] Resolve the relative zk configuration dir based on KYUUBI_HOME

Lead-authored-by: lawulu <biangjuang@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-27 10:48:22 +08:00
Cheng Pan
cb93d6fd1a [KYUUBI #5523] [DOC] Update the Kyuubi supported components version
### _Why are the changes needed?_

Kyuubi 1.8 added the support for JDK 17 and Flink 1.18.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

<img width="1139" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/2316fce3-20fc-49dc-a967-a40f9ea8f706">

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5523 from pan3793/doc-ver.

Closes #5523

d9024ccc7 [Cheng Pan] Update the Kyuubi supported components version

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-25 22:12:08 +08:00
liupeiyue
9a409956a1 [KYUUBI #5282] Support configure Trino session conf in kyuubi-default.conf
### _Why are the changes needed?_

Refer to spark and flink settings conf,  support configure Trino session conf in kyuubi-default.conf
issue : https://github.com/apache/kyuubi/issues/5282

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5283 from ASiegeLion/kyuubi-master-trino.

Closes #5282

87a3f57b4 [Cheng Pan] Apply suggestions from code review
effdd79f4 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
399a200f7 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
7462b32c2 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml--Update documentation
5295f5f94 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml

Lead-authored-by: liupeiyue <liupeiyue@yy.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-24 16:19:44 +08:00
yangming
289549f7f3 [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine
### _Why are the changes needed?_

As https://github.com/apache/kyuubi/pull/4000 adds the Apache Phoenix support for the JDBC engine without upgrading the configuration's description.

This PR aims to update the outdated configuration description to reflect the capabilities of the Kyuubi JDBC engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5294 from ymZhao1001/doc0915.

Closes #5294

8d61c36f9 [yangming] update jdbc doc
d082b4d34 [yangming] run dev/reformat and dev/gen/gen_all_config_docs.sh
5b814d86b [zhaoyangming] add jdbc Phoenix dialect

Lead-authored-by: yangming <261635393@qq.com>
Co-authored-by: zhaoyangming <zhaoyangming@deepexi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-24 16:16:29 +08:00
Kent Yao
1224b0d2c9
[KYUUBI #5500] Add Kyuubi Code Program to Doc
### _Why are the changes needed?_

Add Kyuubi Code Program to Doc for long-term management and sponsorship.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

no

Closes #5500 from yaooqinn/program.

Closes #5500

14f144747 [Kent Yao] Add Kyuubi Code Program to Doc

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2023-10-23 17:39:37 +08:00
yikaifei
59c25b9851
[KYUUBI #5427] [AUTHZ] Shade spark authz plugin
### _Why are the changes needed?_

This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5427 from Yikf/shade-authz.

Closes #5427

d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin

Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
2023-10-20 20:10:34 +08:00
senmiaoliu
16752164a1
[KYUUBI #5408] MetadataManager tries MySQL 8 driver class first
### _Why are the changes needed?_

close #5408

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5433 from lsm1/branch-kyuubi-5408.

Closes #5408

8c6cc223d [senmiaoliu] fix style
20415dc44 [senmiaoliu] use com.mysql.cj.jdbc.Driver first

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-10-18 12:55:00 +08:00
zwangsheng
b24d94e74f
[KYUUBI #5328] Batch supports priority scheduling
### _Why are the changes needed?_

Follow #5329 and close #5328:

1. Add new config `kyuubi.metadata.store.jdbc.priority.enabled` to control whether enable priority scheduling, due to users may experience performance issues when using MySQL5.7 as metastore backend and enabling kyuubi batch v2 priority feature.
2. When priority scheduling is enabled, `KyuubiBatchService` picks metadata job with `ORDER BY priority DESC, create_time ASC`.
3. Insert metadata with priority field, default priority value is `10`.
4. Add new config `kyuubi.batch.priority` for each batch priority.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5352 from zwangsheng/KYUUBI#5328.

Closes #5328

687ed1ed6 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
58621b557 [zwangsheng] fix comments
1bf81e75c [zwangsheng] fix style
7ed2551b3 [zwangsheng] update default priority desc & improve UT
21ceccb01 [zwangsheng] fix doc
27fc5e825 [zwangsheng] enrich desc
c0bbc0dfd [zwangsheng] fix style
6b8d0f091 [zwangsheng] fix comment
67eb2524d [zwangsheng] fix comment
e1705c34d [zwangsheng] Add config to control whether pick order by priority or not
129a46729 [zwangsheng] Add unit test for pickBatchForSubmitting
fcaf85d92 [zwangsheng] Fix unit test
f7ca2219e [zwangsheng] Fix unit test
8d4b276ff [wangsheng] fix code style
4c6b99090 [wangsheng] fix comments
654ad843a [zwangsheng] [KYUUBI #5328][V2] Kyuubi Server Pick Metadata job with priority

Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: wangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-16 22:43:02 +08:00
hezhao2
4bb67bdf3b
[KYUUBI #5381] Change the default metrics reporter to Prometheus
### _Why are the changes needed?_

Close #5381

change default metrics reporter to prometheus since Kyuubi 1.8

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5344 from zhaohehuhu/Improvement-0928.

Closes #5381

84f4c8208 [hezhao2] reset METRICS_REPORTERS for test case
b9ee5f711 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
86165a6fe [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
a3605b626 [hezhao2] set METRICS_PROMETHEUS_PORT to 0 for test cases
f1a4d2861 [hezhao2] restore version number for kyuubi.metrics.reporters in doc
dae40e1a2 [hezhao2] change default metrics reporter to prometheus

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-16 11:46:56 +08:00
Bowen Liang
74e52f6924 [KYUUBI #5409] [DOCS] Update config docs regeneratoin hints in developer guide
### _Why are the changes needed?_

- Update the regeneration hits for `setting.md` in `develop.md`, as the config doc`settings.md` is checked and able to regenearted by `dev/gen/gen_all_config_docs.sh` script.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5409 from bowenliang123/regen-settings.

Closes #5409

985f39e51 [Bowen Liang] update docs

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-10-12 15:11:18 +08:00
ITzhangqiang
e51095edaa
[KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
### _Why are the changes needed?_

The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.

This PR to fix the performance issue on our side.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No.

Closes #5400 from ITzhangqiang/KYUUBI_5365.

Closes #5365

dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations

Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-11 21:41:22 +08:00
Angerszhuuuu
79b147ad3a [KYUUBI #5362] Remove Spark 3.0 support for Authz
### _Why are the changes needed?_
To close #5362 .

Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5363 from AngersZhuuuu/KYUUBI-5362.

Closes #5362

d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-11 10:18:50 +08:00
mrtisttt
42f9c7c921 [KYUUBI #5347] [DOCS] Fix java code errors in quick_start_with_jdbc
### _Why are the changes needed?_

There were obvious errors in the case code, so these cases were fixed.

### _How was this patch tested?_

Just fix case code, no test need. Of course, I have run the case code and have confirmed that there is no problem.

Closes #5347 from mrtisttt/fix-hive-jdbc-example-code.

Closes #5347

ec6ecf758 [mrtisttt] [DOCS] Fix java code errors in quick_start_with_jdbc

Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-10-06 11:18:28 +08:00