<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1104 from pan3793/release.
Closes#1104
f73706e1 [Cheng Pan] Fix release script finalize_svn
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 9e8b974c23)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1030 from pan3793/pkg.
Closes#1024
4fad69f9 [Cheng Pan] [KYUUBI #1024][FOLLOWUP] Fix source release folder name
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 057bac1913)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Close#1025
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1027 from pan3793/sha.
Closes#1027
3e384b80 [Cheng Pan] [BUILD] Use relative path to generate sha512 for release tarballs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 05fc1c0ba8)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Close#1024
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1026 from pan3793/tarball.
Closes#1026
bc515aab [Cheng Pan] [BUILD] Rename tarball to apache-kyuubi
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit fdd51d55f8)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
The building will fail if the project name contains `kyuubi-`.
For example: for our jenkins job, its name is `kyuubi-binary-build`, and the TMP dir won't be created because all the jar files are filtered because their absolute path contains `kyuubi-`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#997 from turboFei/fix_collect_license.
Closes#997
eef47b79 [fwang12] follow up
45fe7834 [fwang12] replace tab with 2 spaces
f6589403 [fwang12] refactor
c0cddd96 [fwang12] fix build error
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 7141f2f13e)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix dist bug
`$SPARK_EXTENSION_VERSION_` is empty
`${SPARK_EXTENSION_VERSION}_` is `3-1_`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#992 from ulysses-you/fix-SPARK_EXTENSION_VERSION_.
Closes#992
4379093e [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit c59e54a675)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support build, run unit tests, and create binary tarball from source release.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
The verify steps:
1. create a source release tarball
```
RELEASE_VERSION=1.4.0-SNAPSHOT build/release/create-package.sh source
```
2. untar tarball
```
cd work/release
tar -xzf kyuubi-1.4.0-SNAPSHOT-bin.tgz
cd kyuubi-1.4.0-SNAPSHOT-bin
```
3. compile and run unit test
```
build/mvn clean install
```
4. create binary release tarball
```
build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1
```
Closes#976 from pan3793/source-build.
Closes#976
354968ae [Cheng Pan] Compress imgs
9c47b64f [Cheng Pan] Restore docs/imgs in source release
5d86ecc7 [Cheng Pan] Clean up imgs
a3399279 [Cheng Pan] Support create binary from source release
bb279863 [Cheng Pan] [BUILD] Fix source release ut failures
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit d58a092245)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#971 from pan3793/disclaimer.
Closes#971
c0fe3821 [Cheng Pan] [LICENSE] Add DISCLAIMER to binary release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 2f82f33e9d)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- Add release script
- Add release guide
- Add license header in some files
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
The release script has not been tested yet.
Closes#874 from pan3793/asf-pub.
Closes#874
5e798a96 [Cheng Pan] LICENSE header
825f2a84 [Cheng Pan] Fix rat
aecc3e6b [Cheng Pan] release doc
a55503c5 [Cheng Pan] Kyuubi Release Guide
2f1a25a2 [Cheng Pan] Release Guide
c5a40c24 [Cheng Pan] Release script
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
```
export RELEASE_VERSION=1.3.0-SNAPSHOT # keep same version with pom.xml
export SKIP_GPG=false # optional, default false
build/release/create-package.sh source
build/release/create-package.sh binary
```
```
➜ apache-kyuubi git:(asf-pub) ls -lh work/release
total 161256
-rw-r--r-- 1 chengpan staff 60M Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz
-rw-r--r-- 1 chengpan staff 488B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.asc
-rw-r--r-- 1 chengpan staff 212B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.sha512
-rw-r--r-- 1 chengpan staff 18M Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz
-rw-r--r-- 1 chengpan staff 488B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.asc
-rw-r--r-- 1 chengpan staff 215B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.sha512
```
Planning to add release script and release doc in following PRs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#889 from pan3793/asf-package.
Closes#889
0aef5d77 [Cheng Pan] Update docs
4b0be7da [Cheng Pan] [BUILD] Package release tarball
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#888 from pan3793/scala.
Closes#888
571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#887 from pan3793/publish-snapshot.
Closes#887
5b2044a1 [Cheng Pan] Pin ubuntu-20.04 in GA
98945a12 [Cheng Pan] Remove Kyuubi local cache before deploy
f29cb6e4 [Cheng Pan] [GA] Daily publish snapshot
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
`kyuubi-extension-spark_3.1` is conflict with the scala naming convention, e.g. `spark-core_2.12`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#882 from pan3793/rename-spark-3-1.
Closes#882
d3297409 [Cheng Pan] Rename kyuubi-extension-spark_3.1 to kyuubi-extension-spark-3-1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark-Block-Cleaner log files are not governed by itself.
Add log4j-defaults.properties to help input both console and file.
And use rollingFileApeender to split big logfile to small logFiles.
Create a log file under /logs/spark-block-cleaner-log/ so that you can manage the log yourself with the ability to clear the folder leading to `spark-`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#831 from zwangsheng/feature/block-cleaner.
Closes#828
83dc639e [ulysses] nit
5458f84b [Binjie Yang] make init log4j static
f0d65dd7 [Binjie Yang] import format
48514496 [Binjie Yang] optimize properties
52713f1e [Binjie Yang] assign log4j.properties
a8723f1a [Binjie Yang] fix
4d44a033 [Binjie Yang] use rollingfileappender
Lead-authored-by: Binjie Yang <2213335496@qq.com>
Co-authored-by: ulysses <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Revert "Set protected_branches explicitly in a nonstrict way (#793)"
This reverts commit b16533a972.
* Revert "Revert "Set protected_branches explicitly in a nonstrict way (#793)""
This reverts commit 57f5f6cf9c010ed177bdf0cd446902feb68c14dd.
* adress comments
### _Why are the changes needed?_
Add Kyuubi tools cache-file-cleaner on kubernetes.
This tools help to start daemonSet on Kubernetes to clean shuffle file and spark cache file.
By config some envs, cache-file-cleaner will help to check some old file(which depending on configuration) and delete them for free dist.
And after first cleaning, if the free capacity lower than threshold, cache file cleaner will start deep clean to help free dist.
With the cache-file-cleaner, add Dockerfile, entrypoint and daemonSet.yaml to help user to use this tool.
Using threadPool to help delete shuffle files.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#768 from zwangsheng/shuffle-cleaner.
Closes#768
3e3d5c65 [Binjie Yang] tini
2f7eab6e [Binjie Yang] dist
d19b9d0c [Binjie Yang] dist
3c8b7828 [Binjie Yang] dist
21bf5b4d [Binjie Yang] big change
4d803504 [Binjie Yang] rename to spark-block-cleaner & fix bug
ba6515d0 [Binjie Yang] do some change
1084feaa [Binjie Yang] delete empty dir
ada682da [Binjie Yang] big change
aabbf9ba [Binjie Yang] bugfix
a504448a [Binjie Yang] big change
0e0d5811 [Binjie Yang] modify
262fe314 [Binjie Yang] pom
ad9c533c [Binjie Yang] dist
0364f6e9 [Binjie Yang] pom
6fbb5cae [Binjie Yang] dist
797bcc39 [Binjie Yang] shuffle cleaner
Authored-by: Binjie Yang <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#756 from pan3793/codecov.
Closes#756
0fc8a6dc [Cheng Pan] nit
52fc82d5 [Cheng Pan] MAVEN_OPTS
20bac836 [Cheng Pan] try recover codecov
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Enable mvn cli options `--batch-mode` `--no-transfer-progress` in CI mode to suppress noising logs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#742 from pan3793/mvn.
Closes#742
79d8901b [Cheng Pan] --no-transfer-progress
218852a2 [Cheng Pan] fix syntax
309e45bf [Cheng Pan] MAVEN_CLI_OPTS
44907b65 [Cheng Pan] [BUILD] Enhance build/mvn in CI mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Fix bug.
### _How was this patch tested?_
Manual test with follow two cmd:
```
./build/dist --name 1.2.0-test --tgz -Pspark-3.1 -Pspark-hadoop-2.7 -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1
./build/dist --name 1.2.0-test --tgz -Pspark-3.1 -Pspark-hadoop-2.7 -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1 -Pkyuubi-extension-spark_3.1
```
Closes#639 from ulysses-you/fix-extension-package.
Closes#639
9852413 [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make Kyuubi jars clear.
### _How was this patch tested?_
Manual test. After this PR, the packaged dir looks like:
Closes#634 from ulysses-you/extension-dir.
Closes#634
2c5009a [ulysses-you] comment
35063ea [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into extension-dir
f2b4fff [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#628 from zhang1002/dev.
Closes#628
79b3d5a [张宇翔] ## solve the folder naming problem For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
Lead-authored-by: 张宇翔 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhangyuxiang03@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/401)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Current kyuubi server dist jars
```
➜ kyuubi git:(KYUUBI-398-followup) ll dist/jars
total 122064
-rw-r--r-- 1 chengpan staff 346K Mar 5 01:00 commons-codec-1.15.jar
-rw-r--r-- 1 chengpan staff 511K Mar 5 01:00 commons-lang3-3.10.jar
-rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-client-2.12.0.jar
-rw-r--r-- 1 chengpan staff 197K Mar 5 01:00 curator-framework-2.12.0.jar
-rw-r--r-- 1 chengpan staff 277K Mar 5 01:00 curator-recipes-2.12.0.jar
-rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-test-2.12.0.jar
-rw-r--r-- 1 chengpan staff 4.5K Mar 5 01:00 failureaccess-1.0.1.jar
-rw-r--r-- 1 chengpan staff 2.7M Mar 5 01:00 guava-30.1-jre.jar
-rw-r--r-- 1 chengpan staff 17M Mar 5 01:00 hadoop-client-api-3.2.2.jar
-rw-r--r-- 1 chengpan staff 22M Mar 5 01:00 hadoop-client-runtime-3.2.2.jar
-rw-r--r-- 1 chengpan staff 1.5M Mar 5 01:00 hive-service-rpc-2.3.7.jar
-rw-r--r-- 1 chengpan staff 1.4M Mar 5 01:00 htrace-core4-4.1.0-incubating.jar
-rw-r--r-- 1 chengpan staff 697K Mar 5 01:00 javassist-3.18.1-GA.jar
-rw-r--r-- 1 chengpan staff 16K Mar 5 01:00 jcl-over-slf4j-1.7.30.jar
-rw-r--r-- 1 chengpan staff 325K Mar 5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 34K Mar 5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 81K Mar 5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 306K Mar 5 01:00 libfb303-0.9.3.jar
-rw-r--r-- 1 chengpan staff 229K Mar 5 01:00 libthrift-0.9.3.jar
-rw-r--r-- 1 chengpan staff 478K Mar 5 01:00 log4j-1.2.17.jar
-rw-r--r-- 1 chengpan staff 5.0M Mar 5 01:00 scala-library-2.12.10.jar
-rw-r--r-- 1 chengpan staff 41K Mar 5 01:00 slf4j-api-1.7.30.jar
-rw-r--r-- 1 chengpan staff 12K Mar 5 01:00 slf4j-log4j12-1.7.30.jar
-rw-r--r-- 1 chengpan staff 890K Mar 5 01:00 zookeeper-3.4.14.jar
```
Closes#401 from pan3793/KYUUBI-398-followup.
c96586b [Cheng Pan] reorder deps
207f0b1 [Cheng Pan] restore docs
0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/399)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#398
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
manual test release workflow
```
➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```
Closes#399 from pan3793/KYUUBI-398.
a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
fixes#297
Squashed commit of the following:
commit 26df4c44a57f92dd16efe69898e9e969c22093fa
Author: fwang12 <fwang12@ebay.com>
Date: Mon Jan 18 13:09:13 2021 +0800
remove unused properties in build info
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix#167
* fix ut
* add ut
* fix ut
* add doc
* fix https://github.com/apache/spark/pull/24120
* logger
* typo
* comment/dist jar
* fix ut
* typo
