# 🔍 Description
## Issue References 🔗
Historically, we provided a few standalone YAML files to allow users to deploy Kyuubi on K8s, and later a Helm Chart was introduced, which is a simple and mature way for production use cases.
## Describe Your Solution 🔧
Remove the legacy standalone YAML files way and update docs to guide user to use Helm chart to deploy Kyuubi on K8s.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6080 from pan3793/k8s-cleanup.
Closes#6080
3a9fa9be3 [Cheng Pan] comment
8f681d28c [Cheng Pan] Improve instruction for deploy Kyuubi on K8s
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
It's RTB.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6075 from pan3793/kt.
Closes#6075
0a8f65785 [Cheng Pan] Update known_translations
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Currently, we use a rather primitive way to manually write release notes from scratch, and some of the mechanical and repetitive work can be simplified by the scripts.
## Describe Your Solution 🔧
Adds a script to simplify the process of creating release notes.
Note: it just simplifies some processes, the release manager still needs to tune the outputs by hand.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
```
RELEASE_TAG=v1.8.1 PREVIOUS_RELEASE_TAG=v1.8.0 build/release/pre_gen_release_notes.py
```
```
$ head build/release/commits-v1.8.1.txt
[KYUUBI #5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central
[KYUUBI #6058] Make Jetty server stop timeout configurable
[KYUUBI #5952][1.8] Disconnect connections without running operations after engine maxlife time graceful period
[KYUUBI #6048] Assign serviceNode and add volatile for variables
[KYUUBI #5991] Error on reading Atlas properties composed of multi values
[KYUUBI #6045] [REST] Sync the AdminRestApi with the AdminResource Apis
[KYUUBI #6047] [CI] Free up disk space
[KYUUBI #6036] JDBC driver conditional sets fetchSize on opening session
[KYUUBI #6028] Exited spark-submit process should not block batch submit queue
[KYUUBI #6018] Speed up GetTables operation for Spark session catalog
```
```
$ head build/release/contributors-v1.8.1.txt
* Shaoyun Chen -- [KYUUBI #5857][KYUUBI #5720][KYUUBI #5785][KYUUBI #5617]
* Chao Chen -- [KYUUBI #5750]
* Flyangz -- [KYUUBI #5832]
* Pengqi Li -- [KYUUBI #5713]
* Bowen Liang -- [KYUUBI #5730][KYUUBI #5802][KYUUBI #5767][KYUUBI #5831][KYUUBI #5801][KYUUBI #5754][KYUUBI #5626][KYUUBI #5811][KYUUBI #5853][KYUUBI #5765]
* Paul Lin -- [KYUUBI #5799][KYUUBI #5814]
* Senmiao Liu -- [KYUUBI #5969][KYUUBI #5244]
* Xiao Liu -- [KYUUBI #5962]
* Peiyue Liu -- [KYUUBI #5331]
* Junjie Ma -- [KYUUBI #5789]
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6074 from pan3793/release-script.
Closes#6074
3d5ec20a4 [Cheng Pan] credits
176527995 [Cheng Pan] Add a script to simplify the process of creating release notes
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The release process will fail with the following errors if `SVN_STAGING_DIR`(`${KYUUBI_DIR}/work/svn-dev"`) exist.
```
+ upload_svn_staging
+ svn checkout --depth=empty https://dist.apache.org/repos/dist/dev/kyuubi /home/ubuntu/apache-kyuubi/work/svn-dev
svn: warning: cannot set LC_CTYPE locale
svn: warning: environment variable LC_CTYPE is UTF-8
svn: warning: please check that your locale name is correct
svn: E155016: The working copy database at '/home/ubuntu/apache-kyuubi/work/svn-dev' is missing.
```
## Describe Your Solution 🔧
Delete `SVN_STAGING_DIR` before checking out
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually test.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6061 from pan3793/release-rm.
Closes#6061
3a9d51e89 [Cheng Pan] Delete svn staging dir before checking out
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request closes#5981
## Describe Your Solution 🔧
As title
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
N/A, will test in v1.8.1 release process.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6060 from pan3793/hive-scala213.
Closes#5981
506ab4883 [Cheng Pan] [KYUUBI #5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5941
## Describe Your Solution 🔧
Originally aims to support Spark On Kubernetes Shuffle Data Clean, limitations became more and more apparent over time, so let's drop this.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#5942 from zwangsheng/KYUUBI#5941.
Closes#5941
23bf14f37 [Cheng Pan] Update docs/tools/spark_block_cleaner.md
1c3350186 [zwangsheng] fix comment
0bdbb1104 [zwangsheng] nit
0a5aa2bfa [zwangsheng] fix comments
Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR proposes to add a contributor name translation mapping, it's for the contributor's contributions calculation.
Note, that this PR only contains the mapping, but not the scripts (I still need to polish them).
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5637 from pan3793/trans.
Closes#5637
727923bb9 [sychen] sort
8a65304fd [sychen] add
166898e23 [Cheng Pan] fix
472d3f0bd [Cheng Pan] nit
5a24e6907 [Cheng Pan] ni
2efe704a4 [Cheng Pan] [INFRA] Add known contributor translation
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Minor, just rename a folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5598 from pan3793/db-scripts.
Closes#5561
17cdd661a [Cheng Pan] [KYUUBI #5561][FOLLOWUP] Rename folder to db-scripts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5561
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes#5589 from lsm1/branch-kyuubi-5561.
Closes#5561
831f06ae5 [senmiaoliu] just copy server resource sql
64b90b144 [senmiaoliu] copy metadata sql to dist
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As we already added support for Spark 3.5, we should publish it to Maven Central in 1.8.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5544 from pan3793/sext-3.5.
Closes#5544
5d835f511 [Cheng Pan] Release contains Spark 3.5 extension
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title, it fixes the command issue identified during the preparation of 1.8.0-rc0.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5536 from pan3793/5483-followup.
Closes#5483
b508b201b [Cheng Pan] [KYUUBI #5483] Fix deploy command for Scala 2.13 TPC-DS/H connectors
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark TPC-H/DS Connectors are mostly used for testing, we'd better provide both Scala 2.12 and 2.13 for next release
Close#5483
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5521 from pan3793/5483.
Closes#5483
49cd2a1ec [Cheng Pan] Update build/release/release.sh
13f3d8bb1 [Cheng Pan] Update build/release/release.sh
4a30b0829 [Cheng Pan] nit
ed388f995 [Cheng Pan] [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It is basically copied from `kyuubi-extension-spark-3-4`.
### _How was this patch tested?_
Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5336 from wForget/dev_spark_3_5.
Closes#5336
7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
- [x] Manually review.
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5277 from pan3793/spark-3.4-release.
Closes#5202
ce513b333 [Cheng Pan] [KYUUBI #5202][FOLLOWUP] Update upload_nexus_staging
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Avoid releasing new version with incorrect java version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5308 from wForget/release_check_java_version.
Closes#5308
8bf3c13ba [Cheng Pan] Update build/release/release.sh
90ddd145d [wforget] comment
b6ab1a480 [wforget] Check java version for release script
Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
followup : #4187
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5032 from cxzl25/PR_4187_followup.
Closes#4187
f0b6c89df [sychen] cmd revision_time
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The main change is copy code from `kyuubi-extension-spark-common_2.12` and `kyuubi-extension-spark-3-3`. The reason copy `kyuubi-extension-spark-common_2.12` is that, Spark3.4 v1 writes does not have ctas...
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5018 from ulysses-you/extension.
Closes#5018
1ca4951e1 [ulysses-you] fix
fb20bb1f0 [ulysses-you] fix
338a3120a [ulysses-you] rebase
241a436fd [ulysses-you] fix
70618f368 [ulysses-you] fix
5f5e2b7c8 [ulysses-you] fix
301ae5c04 [ulysses-you] fix
bb9416811 [ulysses-you] fix
a38595d0a [ulysses-you] copy
9dec27d6c [ulysses-you] copy
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
To speed up the making distribution.
Refer https://github.com/apache/spark/pull/41141
This PR add more skip properties when making distribution:
- -Dmaven.javadoc.skip=true to skip generating javadoc
- -Dmaven.scaladoc.skip=true to skip generating scaladoc. Please see: https://davidb.github.io/scala-maven-plugin/doc-jar-mojo.html#skip
- -Dmaven.source.skip to skip generating sources.jar
### _How was this patch tested?_
Manual test:
<img width="267" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/af4e7bb5-10f1-4fa0-b212-b779b77c9ff3">
```
./build/dist --spark-provided --hive-provided --flink-provided
```
Before this pr:
```
[INFO] Total time: 09:20 min
```
After this pr:
```
[INFO] Total time: 06:46 min
```
Closes#4880 from turboFei/speedup_dist.
Closes#4880
47f5ef2ac [fwang12] [BUILD] Skip compiling useless files when making distribution
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add args `--no-mac-metadata --no-xattrs --no-fflags` to `tar` on macOS in `build/dist` to exclude macOS-specific extended metadata.
The binary tarball created on macOS includes extended macOS-specific metadata and xattrs, which causes warnings when unarchiving it on Linux.
Step to reproduce
1. create tarball on macOS (13.3.1)
```
➜ apache-kyuubi git:(master) tar --version
bsdtar 3.5.3 - libarchive 3.5.3 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.8
```
```
➜ apache-kyuubi git:(master) build/dist --tgz
```
2. unarchive the binary tarball on Linux (CentOS-7)
```
➜ ~ tar --version
tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.
```
```
➜ ~ tar -xzf apache-kyuubi-1.8.0-SNAPSHOT-bin.tgz
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
```
### _How was this patch tested?_
- [x] Manual tests
Create binary tarball on macOS then unarchive on Linux, warnings disappear after this change.
Closes#4828 from pan3793/dist.
Closes#4828
7bc49d847 [Cheng Pan] [BUILD] Exclude macOS tar extended metadata in build/dist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- bump Maven from 3.8.7 to 3.9.1, 3.9.1 fixed the performance issue [MNG-7677](https://issues.apache.org/jira/browse/MNG-7677) in 3.9.0, release notes: https://maven.apache.org/docs/3.9.1/release-notes.html
- Mvnd from 0.9.0 to 1.0-m6 (with embedded maven 3.9.1), release notes: https://github.com/apache/maven-mvnd/releases/tag/1.0-m6
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4647 from bowenliang123/maven-3.9.1.
Closes#4647
f803394df [liangbowen] remove property
efd199f7a [liangbowen] fix
87e18d70a [Bowen Liang] Update build/mvnd
10f4a25ff [liangbowen] bump Maven from 3.8.7 to 3.9.1, and Mvnd from 0.9.0 to 1.0-m6 (with embedded maven 3.9.1)
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Introduce a brand new CHAT engine, it's supposed to support different backends, e.g. ChatGPT, 文心一言, etc.
This PR implements the following providers:
- ECHO, simply replies a welcome message.
- GPT: a.k.a ChatGPT, powered by OpenAI, which requires a API key for authentication. https://platform.openai.com/account/api-keys
Add the following configurations in `kyuubi-defaults.conf`
```
kyuubi.engine.chat.provider=[ECHO|GPT]
kyuubi.engine.chat.gpt.apiKey=<chat-gpt-api-key>
```
Open an ECHO beeline chat engine.
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=ECHO'
```
```
Connecting to jdbc:hive2://localhost:10009/
Connected to: Kyuubi Chat Engine (version 1.8.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Client (version 1.7.0)
Beeline version 1.7.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10009/> Hello, Kyuubi!;
+----------------------------------------+
| reply |
+----------------------------------------+
| This is ChatKyuubi, nice to meet you! |
+----------------------------------------+
1 row selected (0.397 seconds)
```
Open a ChatGPT beeline chat engine. (make sure your network can connect the open API and configure the API key)
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=GPT'
```
<img width="1109" alt="image" src="https://user-images.githubusercontent.com/26535726/225813625-a002e6e2-3b0d-4194-b061-2e215d58ba94.png">
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4544 from pan3793/chatgpt.
Closes#4544
87bdebb6d [Cheng Pan] nit
f7dee18f3 [Cheng Pan] Update docs
9beb55162 [cxzl25] chat api (#1)
af38bdc7c [Cheng Pan] update docs
9aa6d83a6 [Cheng Pan] Initial implement Kyuubi Chat Engine
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: cxzl25 <cxzl25@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- to fix mvnd failure in `Install` step of `Sytle Check` job in (https://github.com/apache/kyuubi/actions/runs/4403008223/jobs/7711886893#step:7:1012)
- by lowering `mvnd.minThreads` to reduce memory pressure in concurrent builds
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4502 from bowenliang123/sytle-mvn.
Closes#4502
a3ddb6270 [liangbowen] lower `mvnd.minThreads`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- concurrent execution and smart builder is shipped with `mvnd`, but the CI log shows they are not activated
- increase maximum degree of concurrency and utilize `SmartBuilder` feature in `mvnd`. `-Dmvnd.minThreads` is set as fallback and it will be ignored if `--threads` or `-Dmvnd.threads` is used.
#### Before:
(https://github.com/apache/kyuubi/actions/runs/4276652363/jobs/7444896450#step:6:64)
```
[INFO] Build maximum degree of concurrency is 1
```
#### After:
(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:64)
```
[INFO] Using the SmartBuilder implementation with a thread count of 4
[INFO] Build maximum degree of concurrency is 4
```
(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:553)
```
[INFO] Segment walltime 13 s, segment projects service time 36 s, effective/maximum degree of concurrency 2.63/4
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4420 from bowenliang123/mvnd-smart.
Closes#4274
64f0cc53e [liangbowen] update
391da26d3 [liangbowen] quite in dep and style jobs
c776356a5 [liangbowen] use -Dmvnd.minThreads
d10bfb70c [liangbowen] use -Dmvnd.minThreads
f793d2b6b [liangbowen] warn
bbb929fe3 [liangbowen] quiet
a5e6d4914 [liangbowen] logging warn level
3cfbaad45 [liangbowen] enable quite option and cancel redirect stderr
09442ff69 [liangbowen] update
ca5f855fa [liangbowen] try to redirect stderr
0ebdd18d4 [liangbowen] set completion and print MAVEN_CLI_OPTS
d3b2c9656 [liangbowen] revert explicitly setting `mvnd.noBuffering`
e2eca0140 [liangbowen] revert explicitly setting `mvnd.noBuffering`
dba25e327 [liangbowen] explicitly set `mvnd.noBuffering` to false prevent displaying events continuously
b7583f936 [liangbowen] shell
d60ebd047 [liangbowen] change mvnd cache key
2cc576ef8 [liangbowen] dep
bbd6414c9 [liangbowen] move multithread setting to mvnd's MAVEN_CLI_OPTS
347b58c8c [liangbowen] increase concurrency for spotless check
15c5519e4 [liangbowen] increase concurrency
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
`bin/docker-image-tool.sh` should include `web-ui` when build image
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
build/dist --web-ui
cd dist
bin/docker-image-tool.sh -t test build
cat > conf/kyuubi-defaults.conf << EOF
kyuubi.frontend.protocols REST
EOF
docker run --rm -v $PWD/conf:/opt/kyuubi/conf -p10099:10099 kyuubi:test
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4413 from pan3793/web-ui-package.
Closes#4413
787f89515 [Cheng Pan] [BUILD] Build docker image should include web UI
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
```
Usage:
+----------------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--web-ui] [--flink-provided] [--hive-provided] |
| [--spark-provided] [--mvn <maven_executable>] <maven build options> |
+----------------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
web-ui: - whether to include web ui
flink-provided: - whether to make a package without Flink binary
hive-provided: - whether to make a package without Hive binary
spark-provided: - whether to make a package without Spark binary
mvn: - external maven executable location
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Create binary artifacts using `build/dist --tgz --web-ui` and run, then open `http://0.0.0.0:10099/ui`
<img width="1401" alt="image" src="https://user-images.githubusercontent.com/26535726/220753103-ce801f12-f394-4ece-92a4-1902e93c62c7.png">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4397 from pan3793/webui-build.
Closes#4397
97901d63e [Cheng Pan] doc
37d5e2ad3 [Cheng Pan] mirror-cdn
c5751dd5b [Cheng Pan] remove unused dep
d308defb7 [Cheng Pan] nit
9abca4705 [Cheng Pan] nit
c1d184afd [Cheng Pan] nit
7091d5bf5 [Cheng Pan] regex
f0ac16b3c [Cheng Pan] [BUILD] `build/dist` support --web-ui
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The release-related shell scripts should have execution permission.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4377 from pan3793/x.
Closes#4377
fb727adc [Cheng Pan] Grant execute permission to release scripts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The module is missed because it's only included when `spark-3.3` is activated.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4374 from pan3793/release.
Closes#4374
ffc0dc22 [Cheng Pan] fix
d21738f1 [Cheng Pan] Release uploading should include kyuubi-spark-connector-hive
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The uploading destination is the svn repo location, not the local temp folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4373 from pan3793/minor.
Closes#4373
96aa1d4c9 [Cheng Pan] Fix echo messeage in release script
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
eclipse-temurin is the successor for openjdk, see https://github.com/apache/spark/pull/37705
"The core change is: the OS of base image changes debian-bullseye to ubuntu-focal (based on debian bullseye)."
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4288 from pan3793/image.
Closes#4288
b7b619018 [Cheng Pan] Use eclipse-temurin:8-jdk-focal as default base image
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`build/mvn` should download the maven when the local `mvn` version does not match, this corrects the change in #4261
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4291 from pan3793/mvn-version.
Closes#4247
42717a73c [Cheng Pan] [KYUUBI #4247][FOLLOWUP] Fix build/mvn version comparison
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As mentioned in https://github.com/apache/kyuubi/issues/4139, we are not able to use Travis for AMR testing in the future.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4287 from pan3793/rm-travis.
Closes#4287
efe2a4854 [Cheng Pan] nit
c72a8b00c [Cheng Pan] Remove travis
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As mentioned in #4247, it seems to be better that using the same maven version with master. Thus, modify the version check logic in `build/mvn` from equal or greater to equal.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4261 from edddddy/mvn_ver_cp.
Closes#4247
11f19eca1 [1456173400] change the if condition to equal
Lead-authored-by: edddddy <rmrfall@qq.com>
Co-authored-by: 1456173400 <1456173400@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Skip Maven dependency snapshot update fetching to building `dependencyList` faster without waiting for fetching daily snapshots checking (only snapshots of submodules used).
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4191 from bowenliang123/dependency-resolve.
Closes#4191
0bf36815 [liangbowen] apply `--no-snapshot-updates` option in build_classpath
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- commit date is more readable and helpful to determine earlier/later version, especially for snapshot branch
- adding revision commit date from git to `kyuubi-version-info.properties`
- Print revision date on server startup
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4187 from bowenliang123/print-revision-date.
Closes#4187
b0b0a6c8 [liangbowen] print revision date on server startup
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
After changes in #4034, we should modify `./build/dist` to build kyuubi binary with charts
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4143 from zwangsheng/helm/build_dist_with_charts.
Closes#4034
78edac4e [zwangsheng] [BUILD] Should build binary with charts dir
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#2062
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4078 from xiaoyuandajian/fix-#2062.
Closes#4078
a5a01945 [Cheng Pan] Update .github/workflows/style.yml
3c08e692 [袁福元] fix the shellcheck found error file
a33cbb22 [袁福元] fix the shellcheck found error file
da195782 [袁福元] fix the shellcheck found error file
b24e6adf [袁福元] fix the shellcheck found error file
90ef0712 [袁福元] fix the shellcheck found error file
42acb31f [袁福元] fixup! test1
31c1783d [袁福元] fixup! test
c40b5f38 [袁福元] Revert "test"
d97801fa [袁福元] Merge remote-tracking branch 'origin/fix-#2062' into fix-#2062
39a9ff31 [袁福元] test
fc7198da [xiaoyuandajian] Update style.yml
62e80726 [袁福元] test
0abdef36 [xiaoyuandajian] Delete main.yml
1e252e2a [xiaoyuandajian] Create main.yml
c19728ec [xiaoyuandajian] Delete main.yml
5db61e5a [xiaoyuandajian] Delete main1.yml
9e27a363 [xiaoyuandajian] Create main1.yml
69540652 [xiaoyuandajian] Create main.yml
Lead-authored-by: 袁福元 <yuanfuyuan@mafengwo.com>
Co-authored-by: xiaoyuandajian <51512358+xiaoyuandajian@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changelogs are exported from the git command, which is quite duplicated and does not help much, after removing, users still can get them from GitHub or git history easily.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4094 from pan3793/changelog.
Closes#4094
2d0bfe07 [Cheng Pan] [INFRA] Remove changelogs and generate_changelog.sh
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is part of https://github.com/apache/kyuubi/issues/4020, and should finalize the update for repo https://github.com/apache/kyuubi
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4089 from pan3793/graduate.
Closes#4089
60ece0c1 [Cheng Pan] [INFRA] Update release scripts and templates because of graduation
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- DISCLAIMER file on top-level folder of project was already removed after graduation in (https://github.com/apache/kyuubi/issues/4020)
- fix build/dist for `cp` non-existed DISCLAIMER file
- remove DISCLAIMER file reference in dev/dist script, doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4086 from bowenliang123/desclaimer-ref.
Closes#4086
18be7489 [liangbowen] remove DISCLAIMER file reference in dev/dist script, doc, docker , labeler.yml
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
#3847
Windows users need to generate `kyuubi-version-info.properties` to debug.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3903 from cxzl25/win_gen_version_info.
Closes#3903
089cf9eb [sychen] support windows generate kyuubi-version-info.properties
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3816 from cxzl25/release_typo.
Closes#3816
614d60fe [sychen] typo
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3441;
This pr aims to two points as follow:
- Bump Spark 3.3.0 to Spark 3.3.1 in spark-3.3 profile
- Change default Spark version to 3.3.1
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3507 from Yikf/pr/3471.
Closes#3441
7449280b [yikf] Change spark 3.3.0 as default version
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
