# 🔍 Description
## Issue References 🔗
This pull request aims to bump labeler action to v5.
## Describe Your Solution 🔧
The full release notes: https://github.com/actions/labeler/releases/tag/v5.0.0
Version 5 of this action updated the [runtime to Node.js 20](https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#runs-for-javascript-actions). All scripts are now run with Node.js 20 instead of Node.js 16 and are affected by any breaking changes between Node.js 16 and 20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5874 from panbingkun/github_labeler_v5.
Closes#5874
85ce1f687 [panbingkun] Bump labeler action to v5
Authored-by: panbingkun <pbk1982@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5856
## Describe Your Solution 🔧
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5859 from zml1206/KYUUBI-5856.
Closes#5856
872fd06d2 [zml1206] Revert changes in SparkProcessBuilderSuite
bc4996f90 [zml1206] Bump spark 3.4.2
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Sometimes developers may directly change `dev/dependencyList`, but CI will not check it, which will cause the subsequent upgraded versions of PR execution CI to fail.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5857 from cxzl25/deps_check.
Closes#5857
f2e1a1d16 [sychen] check
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Currently, we always enable Jacoco, which would add additional building time and produce lots of HTML files, also increase the IDEA indexing burden.
<img width="1065" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/6276f309-8ba4-4563-96d6-91cb53ce20a5">
## Describe Your Solution 🔧
Add a new maven profile `codecov`, only enable it in CI with codecov enabled.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
See CI result.
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5837 from pan3793/codecov.
Closes#5837
97d0b22de [Cheng Pan] fi
c60367dd1 [Cheng Pan] fix
a0e8910bc [Cheng Pan] Only enable Jacoco with explicit profile
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The program ended on 2023/12/08 at 18:00 CST.
## Describe Your Solution 🔧
Remove the template, we can recover it from git history if when we decide to start the next program.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5835 from pan3793/remove-task-template.
Closes#5835
af3d17ead [Cheng Pan] Remove issue template for 2023 Kyuubi Code Contribution Task
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
As described.
This pull request fixes #
## Describe Your Solution 🔧
- `labeler` Github Action v5 release note: https://github.com/actions/labeler/releases/tag/v5.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5840 from bowenliang123/actions-labeler-5.
Closes#5840
6d067bcec [Bowen Liang] Bump actions/labeler from v4 to v5
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- `setup-node` Github Action v4 release note: https://github.com/actions/setup-node/releases/tag/v4.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5842 from bowenliang123/actions-node-4.
Closes#5842
8610525a8 [Bowen Liang] Bump actions/setup-node from v3 to v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- `stale` Github Action v9 release note: https://github.com/actions/stale/releases/tag/v9.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5841 from bowenliang123/actions-stale-9.
Closes#5841
e1b8ff0d1 [Bowen Liang] Bump actions/stale from v7 to v9
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- `setup-java` Github Action v4 release note: https://github.com/actions/setup-java/releases/tag/v4.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5839 from bowenliang123/actions-java-4.
Closes#5839
af76bf44c [Bowen Liang] Bump actions/setup-java from v3 to v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- `setup-python` Github Action v5 release note: https://github.com/actions/setup-python/releases/tag/v5.0.0. Update node version runtime for the actions from node16 to node20.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5838 from bowenliang123/actions-python-5.
Closes#5838
f1703d690 [Bowen Liang] Bump actions/setup-python from v4 to v5
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
The CDH Hive 2.1.1-cdh6.3.2 is the latest free version of CDH, which is adopted widely, it would be great if the Kyuubi Hive engine could support this version.
## Describe Your Solution 🔧
Actually, the Kyuubi Hive engine could work with CDH Hive 2.1.1-cdh6.3.2, this pull request just adds an integration test and fixes one test case.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5830 from pan3793/hive-211-cdh6.
Closes#5830
afa80bd82 [Cheng Pan] nit
89c9ae96e [Cheng Pan] nit
e402e49cd [Cheng Pan] nit
3c126a2a6 [Cheng Pan] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5467
## Describe Your Solution 🔧
1. Add Gluten UTs.
2. Setup CI for Gluten testing
3. Write docs to guide users in setting up Kyuubi with Spark plus Gluten.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
github action ci tests: [Gluten Test CI](https://github.com/Kwafoor/incubator-kyuubi/actions/runs/7111586978)
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5800 from Kwafoor/kyuubi_5467.
Closes#5800
c6dd26f93 [wangjunbo] fix
7818ae0c5 [wangjunbo] fix Scala Test
296f08c8c [wangjunbo] remove spark-3.2 gluten test
5a704675d [wangjunbo] [KYUUBI#5467] Integrate Intel Gluten with Spark engine
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
- Bump docker/setup-buildx-action from v2 to v3
- docker/setup-buildx-action v3.0.0 release note: https://github.com/docker/setup-buildx-action/releases/tag/v3.0.0
- Bump docker/build-push-action from v3 to v5
- docker/buld-push action v5.1.0 release note: https://github.com/docker/build-push-action/releases/tag/v5.1.0
- Bump docker/login-action from v2 to v3
- v3.0.0 release note: https://github.com/docker/login-action/releases/tag/v3.0.0
- Bump setup-qemu-action from v2 to v3
- v3.0.0 release note: https://github.com/docker/setup-qemu-action/releases/tag/v3.0.0
- All upgraded actions now use NodeJS 20 as default runtime
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
No behavior changes.
#### Behavior With This Pull Request 🎉
No behavior changes.
#### Related Unit Tests
No behavior changes.
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5779 from bowenliang123/docker-push-v5.
Closes#5779
ceed9e0ec [Bowen Liang] bump docker/setup-qemu-action to v3
810bc18a6 [Bowen Liang] bump
ad36ebc2c [Bowen Liang] Bump docker/build-push-action from v3 to v5
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
As described.
## Describe Your Solution 🔧
Bump actions/checkout from v3 to v4
- v4.1.1 release notes: https://github.com/actions/checkout/releases/tag/v4.1.1
- default Nodejs runtime for the action bumped from 16 to 20
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
No behavior changes.
#### Behavior With This Pull Request 🎉
No behavior changes.
#### Related Unit Tests
CI workflows.
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes#5774 from bowenliang123/checkout-v4.
Closes#5774
c31edbffb [liangbowen] Bump actions/checkout from v3 to v4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Try to make the Hive engine compatible with Hive 2.3.
It is not easy to align all dependencies with Hive 2.3, thus this PR just added a new integration test to make sure that `kyuubi hive engine` compiled against Hive 3.1.3 could work with Hive 2.3.9 runtime, instead of introducing new a profile to make it pass building with Hive 2.3.9
There are potential class conflict issues, because the Apache Hive 2.3.9 and 3.1.3 official binary artifacts ship Scala 2.11 jars but Kyuubi uses Scala 2.12.
Also tested with `apache-hive-2.1.1-cdh6.3.2-bin.tar.gz` locally, which can successfully bootstrap and run queries.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5280 from pan3793/hive-2.3.
Closes#5280
8ca02a351 [Cheng Pan] corss verify hive engine on hive 2.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5701
## Describe Your Solution 🔧
This PR updates the PR template to ensure that PR authors provide informative PR descriptions for reviewers to speed up the review process.
It's also suitable for improving our code quality.
Besides, `docs/contributing/code/style.rst` is also changed to add a link to the doc style so that we can keep the PR template neat.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
See the screenshot below.
#### Behavior With This Pull Request 🎉
It's able to check it from the description of this PR directly.
#### Related Unit Tests
No
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5702 from yaooqinn/pr.
Closes#5701
19bdea9a4 [Kent Yao] not
d7d9d8cfa [Kent Yao] Enhance PR Template to make PR desc informative
be026e4a6 [Kent Yao] Enhance PR Template to make PR desc informative
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Since the kyuubi official website(latest branch) had been changed from `https://kyuubi.readthedocs.io/en/latest` to `https://kyuubi.readthedocs.io/en/master`, so we need to update the old and incorrect url to the new one.
Close#5697
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Just fix incorrect url reference in code, it's no need to run test.
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5694 from mrtisttt/fix-incorrect-url-reference.
Closes#5697
7b7621cb4 [mrtisttt] Fix incorrect url reference in code or doc.
Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Add the link of [2023 Kyuubi Code Contribution Program](https://github.com/apache/kyuubi/issues/5357)
- Fix typos
- Add task level dropdown
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Reviewers can preview it at https://github.com/pan3793/test/issues/new/choose
<img width="1004" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/29bd9dab-528c-4982-947d-a406db412a43">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5602 from pan3793/template.
Closes#5602
c71c4b915 [Cheng Pan] typo
1f29eaa55 [Cheng Pan] fix
fba5cd249 [Cheng Pan] nit
a1485fccd [Cheng Pan] Improve issue template for 2023 Kyuubi Code Contribution Program
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Update codecov token to fix stale code coverage action
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
no
Closes#5501 from yaooqinn/token.
Closes#5501
1f4053231 [Kent Yao] java-17
fcd8391e7 [Kent Yao] java-17
5ccead77d [Kent Yao] upgrade jacoco
5cb4a8374 [Kent Yao] Revert "rm"
45d7a2134 [Kent Yao] rm
0a3dd2f42 [Kent Yao] Update codecov token
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Resolve: #5405 Support the Flink 1.18
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5465 from YesOrNo828/flink-1.18.
Closes#5405
a0010ca14 [Xianxun Ye] [KYUUBI #5405] [FLINK] Remove flink1.18 rc repo
2a4ae365c [Xianxun Ye] Update .github/workflows/master.yml
d4d458dc7 [Xianxun Ye] [KYUUBI #5405] [FLINK] Update the flink1.18-rc3 repo
99172e3da [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage
4c0cf887b [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage
c74f5c31b [Xianxun Ye] [KYUUBI #5405] [FLINK] fixed Pan's comments.
1933ebadd [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18
Authored-by: Xianxun Ye <yesorno828423@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Improve Kyuubi On Kubernetes IT
Done:
1. Copy spark submit engine log in kyuubi pod to local and upload when test failed.
2. pre install spark image into minikube to avoid image pull error
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5437 from zwangsheng/KYUUBI#5435.
Closes#5435
0cbbafce7 [zwangsheng] add comment
1f1336c59 [zwangsheng] ready
e1c10a6ea [zwangsheng] debug
32759015c [zwangsheng] debug
8e2f1eaf1 [zwangsheng] debug
80eaae30a [zwangsheng] [KYUUBI #5435][NOT_MERGE][TEST] Improve Kyuubi On Kubernetes IT
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Check style with profile of Spark 3.4 and 3.5 in the style workflow
- Isolated scalastyle check for Spark 3.1 profile, as Iceberg 1.3.1 does not support Spark 3.5
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5444 from bowenliang123/spotless-spark34.
Closes#5444
ca98c52e6 [Bowen Liang] update
9550a4a0a [Bowen Liang] enable spark-3.1 profile for spotless style checks
dae35290b [Bowen Liang] revert order
3e488cf48 [Bowen Liang] Check with Spark 3.1 profile separately
e879f09f8 [Bowen Liang] revert the order of spark versions in profiles
16456a7fd [Bowen Liang] check style with profile of Spark 3.4 and 3.5
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Apache Paimon is an incubating Apache project of data lake platform for high-speed data ingestion, changelog tracking and efficient real-time analytics.
- Initial support for Paimon tables in Authz plugin
- Create Table Command: https://paimon.apache.org/docs/master/engines/spark3/#create-table
- Paimon `0.5.0-incubating` supports Spark 3.1/3.2/3.3/3.4.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5462 from bowenliang123/authz-paimon-createtable.
Closes#5462
8393f6ecc [liangbowen] remove spark-authz-paimon-test profile, use 3.4 as spark binary version for Paimon
a4e01a2a8 [liangbowen] update workflow inclusions
f372882aa [liangbowen] update workflow exclusions
7785dec52 [liangbowen] update workflow include
64ee20106 [liangbowen] update
73edea38f [liangbowen] add spark-authz-paimon-test profile and excluded for Spark 3.5
de58605bb [liangbowen] nit
81a4a4370 [liangbowen] nit
eb2de8359 [Bowen Liang] exclude paimon tests for spark-3.5
4f58d7727 [Bowen Liang] fix typo
cf360e6da [Bowen Liang] drop database
af8ffda14 [Bowen Liang] drop database
dc3ee9462 [Bowen Liang] support Paimon create table
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5284.
Hudi also is a common used data format, since kyuubi already support iceberg and delta, we should also support hudi.
In this pr we support hoodie sql about ALTER COMMAND in authz
In this PR we use default Hudi version 0.14.0.
We support from spark 3.1 to spark 3.4, since Hudi don't support spark 3.5 yet
- [x] spark 3.1
- [x] spark 3.2
- [x] spark 3.3
- [x] spark 3.4
- [ ] spark 3.5
Also since Hudi only supports Scala 2.12, I also made Hudi as a separate profile to avoid importing Hudi when enable Scala 2.13
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5287 from AngersZhuuuu/KYUUBI-5284.
Closes#5284
f171e11af [Angerszhuuuu] Update pom.xml
3f57a3dc5 [Angerszhuuuu] follow comment
f6c764028 [Angerszhuuuu] follow comment
51797e25c [Angerszhuuuu] trigger
b3c059af9 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
3510e7601 [liangbowen] remove scope in dependencyManagement
14ea0d498 [liangbowen] change to use `spark.binary.version` for hudi dependency by default
354260eb0 [liangbowen] remove the abbreviation tite
658bddbab [liangbowen] remove clarification and use ALTERTABLE_PROPERTIES for opType
150edcd40 [Angerszhuuuu] update
30c417b19 [Angerszhuuuu] trigger
56e5cb17b [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
fe9b75270 [Angerszhuuuu] update
888943831 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
db749a277 [Angerszhuuuu] update
9b09e78c2 [Angerszhuuuu] Update HudiCommands.scala
87de62e52 [Angerszhuuuu] follow comment
2d551d112 [Angerszhuuuu] Update master.yml
89082e06b [Angerszhuuuu] Update master.yml
7c7846378 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
d32ca9839 [Angerszhuuuu] Update master.yml
ec43e2a7b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
b3611fd3e [Angerszhuuuu] update
2a0dfa74f [Angerszhuuuu] Update AuthZUtils.scala
45ee9e251 [Angerszhuuuu] update
0560a5e14 [Angerszhuuuu] Update pom.xml
97c50f622 [Angerszhuuuu] update
f57ee0093 [Angerszhuuuu] Update table_command_spec.json
fb72197e6 [Angerszhuuuu] update
2154cf928 [Angerszhuuuu] trigger
44469359f [Angerszhuuuu] trigger
b0e768cb8 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
83795ed63 [Angerszhuuuu] Update pom.xml
eed190f92 [Angerszhuuuu] update
361660145 [Angerszhuuuu] update
1ed1f3ab6 [Angerszhuuuu] Update
7ee3c7dd5 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
ee0916f63 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
010260fa4 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
c11d02def [Angerszhuuuu] update
b84f91f65 [Angerszhuuuu] update
42fbb0ffa [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
c1346adb1 [Angerszhuuuu] update
2ec63ae94 [Angerszhuuuu] Update pom.xml
39bce7468 [Angerszhuuuu] update
c70b0ea2f [Angerszhuuuu] Update pom.xml
e1d85ff77 [Angerszhuuuu] Update pom.xml
59012ac25 [Angerszhuuuu] Update pom.xml
a46de65b5 [Angerszhuuuu] Update HoodieTest.java
b8173b893 [Angerszhuuuu] update
055713329 [Angerszhuuuu] Update table_command_spec.json
d7b21e820 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
0a93ff794 [Angerszhuuuu] [KYUUBI #5284] Kyuubi authz support Hoodie Alter Table Command
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#5362 .
Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5363 from AngersZhuuuu/KYUUBI-5362.
Closes#5362
d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar in the following order:
1. `SPARK_SCALA_VERSION` system env
2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5196 from bowenliang123/scala213-test.
Closes#5196
97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
It is basically copied from `kyuubi-extension-spark-3-4`.
### _How was this patch tested?_
Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5336 from wForget/dev_spark_3_5.
Closes#5336
7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Prepare for 2023 Kyuubi Code Contribution Program
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1002" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/b6f6ea69-75fa-41b6-b58a-b76e3d87f316">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5356 from pan3793/task.
Closes#5356
6bc5add3d [Cheng Pan] nit
9a68250be [Cheng Pan] nit
44df6ed37 [Cheng Pan] am1
0fa8c9fff [Cheng Pan] nit
498ef8a29 [Cheng Pan] [INFRA] GitHub Issue template for 2023 Kyuubi Code Contribution Program
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Spark 3.5.0 released in few days ago, this PR aims to add cross-version verification to ensure that Kyuubi engine built against Spark 3.4.1 works on Spark 3.5.0 runtime.
https://spark.apache.org/releases/spark-release-3-5-0.html
For users who want to try Kyuubi with Spark 3.5.0, please compile master/branch-1.8 via
```
./build/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided
```
And then follow the [Quick Start](https://kyuubi.readthedocs.io/en/master/quick_start/quick_start.html) to setup Kyuubi and Spark properly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```log
Connected to: Spark SQL (version 3.5.0)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0-SNAPSHOT)
Beeline version 1.8.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://0.0.0.0:10009/default> select version();
+-------------------------------------------------+
| version() |
+-------------------------------------------------+
| 3.5.0 ce5ddad990373636e94071e7cef2f31021add07b |
+-------------------------------------------------+
1 row selected (0.39 seconds)
0: jdbc:hive2://0.0.0.0:10009/default> select kyuubi_version();
+-------------------+
| kyuubi_version() |
+-------------------+
| 1.8.0-SNAPSHOT |
+-------------------+
1 row selected (0.16 seconds)
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5291 from pan3793/cross-3.5.
Closes#5291
24107a414 [Cheng Pan] Update .github/workflows/master.yml
1554e7131 [Cheng Pan] Verify Spark engine on Spark 3.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
branch-1.8 has been cut, we should update the GitHub workflow to daily publish snapshot jars on all active branches.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5245 from pan3793/daily-snapshot-1.8.
Closes#5245
df3979743 [Cheng Pan] [INFRA] Daily deploy snapshot version on branch-1.8
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`branch-1.5` and `branch-1.6` don't active for a while, the currently active maintained branches are `master` and `branch-1.7`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5235 from pan3793/snapshot.
Closes#5235
d63e1a570 [Cheng Pan] [INFRA] Daily deploy snapshot version on master and branch-1.7
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi fully supports Spark 3.4 now, it's time to move forward.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5202 from pan3793/default-3.4.
Closes#5202
a0efccdbf [Cheng Pan] nit
30456dbb9 [Cheng Pan] nit
1cc83c871 [Cheng Pan] enable lineage test
d8ca7c7d8 [Cheng Pan] Switch to Spark 3.4 by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Make Spark TPC-H connector compilable on Scala 2.13
- adding spark extensions to scala 2.13 compilation CI test
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5195 from bowenliang123/scala213-conn.
Closes#5195
ad2558a16 [liangbowen] update
c50d29111 [liangbowen] update
d38eb3b6c [liangbowen] adapt spark connector plugin compilable on Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adding a basic compilation CI test on Scala 2.13 , skipping test runs
- separate versions of `KyuubiSparkILoop` for Scala 2.12 and 2.13, adapting the changes of Scala interpreter packages
- rename `export` variable
```
[Error] /Users/bw/dev/kyuubi/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala:563: Wrap `export` in backticks to use it as an identifier, it will become a keyword in Scala 3.
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5188 from bowenliang123/sparksql-213.
Closes#5188
04f192064 [liangbowen] update
9e764271b [liangbowen] add ci for compilation with server module
a465375bd [liangbowen] update
5c3f24fdf [liangbowen] update
4b6a6e339 [liangbowen] use Iterable for the row in MySQLTextResultSetRowPacket
f09d61d26 [liangbowen] use ListMap.newBuilder
6b5480872 [liangbowen] Use Iterable for collections
1abfa29d8 [liangbowen] 2.12's KyuubiSparkILoop
b1c9da591 [liangbowen] rename export variable
b6a6e077b [liangbowen] remove original KyuubiSparkILoop
15438b503 [liangbowen] move back scala 2.12's KyuubiSparkILoop
dd3244351 [liangbowen] adapt spark sql module to 2.13
62d3abbf0 [liangbowen] adapt server module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
In minor cases, Spark Stage hangs forever when spark.sql.finalWriteStage.eagerlyKillExecutors.enabled is true.
The bug occurs if two conditions are met in the same time:
1. All executors are either removed because of idle time out or killed by FinalStageResourceManager.
Target executor num in YarnAllocator will be set to 0 and no more executor will be launched.
2. Target executor num in ExecutorAllocationManager equals to the executor num needed by final stage.
Then ExecutorAllocationManager will not sync target executor num to YarnAllocator.
### _How was this patch tested?_
- [x] Add a new test suite `FinalStageResourceManagerSuite`
Closes#5141 from zhouyifan279/adjust-executors.
Closes#5136
c4403eefa [zhouyifan279] assert adjustedTargetExecutors == 1
ea8f24733 [zhouyifan279] Add comment
5f3ca1d9c [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
12687eee7 [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
9dcbc780d [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5162
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5163 from lsm1/features/kyuubi-5162.
Closes#5162
c5bba8c20 [senmiaoliu] fix
af5e816fb [Kent Yao] Update docs/contributing/code/developer.md
ceb44358d [senmiaoliu] Add statements about contributions co-authored with AIGC in developer guide page
Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It was planned but actually delayed, remove this dummy module to save CI and avoid confusing users and release managers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5113 from pan3793/remove-kudu.
Closes#5113
ff8fd2e6a [Cheng Pan] Remove Spark Kudu connector
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The main change is copy code from `kyuubi-extension-spark-common_2.12` and `kyuubi-extension-spark-3-3`. The reason copy `kyuubi-extension-spark-common_2.12` is that, Spark3.4 v1 writes does not have ctas...
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5018 from ulysses-you/extension.
Closes#5018
1ca4951e1 [ulysses-you] fix
fb20bb1f0 [ulysses-you] fix
338a3120a [ulysses-you] rebase
241a436fd [ulysses-you] fix
70618f368 [ulysses-you] fix
5f5e2b7c8 [ulysses-you] fix
301ae5c04 [ulysses-you] fix
bb9416811 [ulysses-you] fix
a38595d0a [ulysses-you] copy
9dec27d6c [ulysses-you] copy
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
The run test link in the template is broken.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4976 from mans2singh/ISSUE-4975.
Closes#4975
593172786 [Cheng Pan] Update .github/PULL_REQUEST_TEMPLATE
83103e26f [Cheng Pan] Update .github/PULL_REQUEST_TEMPLATE
ddec082d4 [mans2singh] ISSUE-4975 - Reverted unrelated change
37342792d [mans2singh] ISSUE-4975 - Fixed run tests link
Lead-authored-by: mans2singh <mans2singh@yahoo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- fetch versions for NodeJS and Pnpm from Maven properties
- enabled npm dependency caching for global packages and web-ui module, as the step above takes extra time in builds.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4948 from bowenliang123/pnpm-version.
Closes#4948
80b0f6934 [liangbowen] Reuse versions of NodeJS and Pnpm from maven properties
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Authz plugin's tests now pass with Spark3.4.
` mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -am -DskipTests
&& mvn test -pl :kyuubi-spark-authz_2.12 -Pspark-3.4`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4909 from bowenliang123/authz-enabletest-3.4.
Closes#4909
b0bdc106e [liangbowen] enable authz plugin tests with Spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After previous efforts, we are close to the goal. This is the latest step to to allow Kyuubi run on JDK 17 by adding `--add-open` in bootstrap scripts and test the plugin's parameter.
This PR also enables JDK 17 test in CI, but only for Kyuubi server and Spark(3.3+) engine moudles, because
- Spark supports JDK 17 since 3.3, see SPARK-33772
- Flink does not support JDK 17 until the latest 1.17
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- Run Kyuubi server on JDK 17
- Run simple Spark (3.3.2) query on JDK 17
- Run `kyuubi-zk-cli` on JDK 17
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4849 from pan3793/jdk-17.
Closes#4849
165e3558c [Cheng Pan] Enable JDK 17 in CI
f5778e7bd [Cheng Pan] Open modules in bootstrap scripts
76b507ffd [Cheng Pan] Open modules in UT
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17.
There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical.
With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested
zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).
We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4852 from pan3793/shaded-zk.
Closes#4852
d960cc945 [Cheng Pan] remove staging repo
1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
add `restore-keys` to restore the most recently maven binary
[github docs](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key)
> restore-keys allows you to specify a list of alternate restore keys to use when there is a cache miss on key. You can create multiple restore keys ordered from the most specific to least specific. The cache action searches the restore-keys in sequential order. When a key doesn't match directly, the action searches for keys prefixed with the restore key. If there are multiple partial matches for a restore key, the action returns the most recently created cache.
logs from [github action](https://github.com/apache/kyuubi/actions/runs/5020359871/jobs/9001759790?pr=4856) (after we update the file `pom.xml` hash)
```
Prepare all required actions
Getting action download info
Run ./.github/actions/setup-maven
Run actions/cachev3
Received 0 of 8173297 (0.0%), 0.0 MBs/sec
Received 8173297 of 8173297 (100.0%), 6.5 MBs/sec
Cache Size: ~8 MB (8173297 B)
/usr/bin/tar -xf /home/runner/work/_temp/0e0ca716-97a4-4d83-af3b-9703003cdd33/cache.tzst -P -C /home/runner/work/kyuubi/kyuubi --use-compress-program unzstd
Cache restored successfully
Cache restored from key: setup-maven-e425be8ee60f158fc4f40b3c6b1a51cb2896e5c97dbbe7c61a96d91380342b89
Run build/mvn -v
Using `mvn` from path: /usr/bin/mvn
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /usr/share/apache-maven-3.8.8
Java version: 1.8.0_362, vendor: Temurin, runtime: /usr/lib/jvm/temurin-8-jdk-amd64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-1037-azure", arch: "amd64", family: "unix"
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4856 from cfmcgrady/ga-cache.
Closes#4856
779e9030e [Fu Chen] Update .github/actions/setup-maven/action.yaml
d663b1a42 [Fu Chen] Revert "test cache restore"
3d3f16013 [Fu Chen] test cache restore
d2d18b613 [Fu Chen] add restore-keys
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- nodejs v18 is the current active LTS version of nodejs since October 2022, while v16 is in maintenance mode and no longer in the active status
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#4784 from bowenliang123/nodejs18.
Closes#4784
483c3ac56 [liangbowen] use nodejs v18 in ci builds and maven builds
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- v8 is now the stable version of pnpm
- lockfile v6 is used by default since pnpm v8.0, which improves the readability of the lockfile by removing hashes from package IDs and more resistant to git merge conflicts (https://github.com/pnpm/pnpm/releases/tag/v8.0.0)
- pnpm 8 supports nodejs 16 and 18 (https://pnpm.io/installation#compatibility)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4783 from bowenliang123/pnpm8.
Closes#4783
8b3db3820 [liangbowen] use pnpm v8 and update lock file to v6
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Guidelines for Document and Code Contributions.
- code style
- documentation style
- quick starts
- sphinx-copybutton for copy button in code blocks
- add kind:minor,help wanted,good first issue to doc issue template
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1325" alt="image" src="https://user-images.githubusercontent.com/8326978/234193368-473e2a1d-2deb-4d91-901a-19cea58a489b.png">
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4768 from yaooqinn/guide.
Closes#4768
3a0bd395b [Kent Yao] Update docs/contributing/doc/build.rst
8fe4f6d14 [Kent Yao] Update docs/contributing/doc/build.rst
a753bdde5 [Kent Yao] Guidelines for Document and Code contributions
f8393fe8d [Kent Yao] Guidelines for Document and Code contributions
8f2d3ce82 [Kent Yao] Guidelines for Document and Code contributions
4f3a2128b [Kent Yao] Guidelines for Document and Code contributions
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Add CI for Spark 3.4
- Kyuubi Spark TPC-DS/H connectors support Spark 3.4
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4741 from pan3793/spark-3.4.
Closes#4741
84a2d6ad7 [Cheng Pan] log
b9b2ec1fb [Cheng Pan] Add spark-3.4 profile
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
