<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
`kyuubi-extension-spark_3.1` is conflict with the scala naming convention, e.g. `spark-core_2.12`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#882 from pan3793/rename-spark-3-1.
Closes#882
d3297409 [Cheng Pan] Rename kyuubi-extension-spark_3.1 to kyuubi-extension-spark-3-1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark-Block-Cleaner log files are not governed by itself.
Add log4j-defaults.properties to help input both console and file.
And use rollingFileApeender to split big logfile to small logFiles.
Create a log file under /logs/spark-block-cleaner-log/ so that you can manage the log yourself with the ability to clear the folder leading to `spark-`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#831 from zwangsheng/feature/block-cleaner.
Closes#828
83dc639e [ulysses] nit
5458f84b [Binjie Yang] make init log4j static
f0d65dd7 [Binjie Yang] import format
48514496 [Binjie Yang] optimize properties
52713f1e [Binjie Yang] assign log4j.properties
a8723f1a [Binjie Yang] fix
4d44a033 [Binjie Yang] use rollingfileappender
Lead-authored-by: Binjie Yang <2213335496@qq.com>
Co-authored-by: ulysses <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Run kyuubi on kubernetes - make basic dockerfile and example
* Revert "Set protected_branches explicitly in a nonstrict way (#793)"
This reverts commit b16533a972.
* Revert "Revert "Set protected_branches explicitly in a nonstrict way (#793)""
This reverts commit 57f5f6cf9c010ed177bdf0cd446902feb68c14dd.
* adress comments
### _Why are the changes needed?_
Add Kyuubi tools cache-file-cleaner on kubernetes.
This tools help to start daemonSet on Kubernetes to clean shuffle file and spark cache file.
By config some envs, cache-file-cleaner will help to check some old file(which depending on configuration) and delete them for free dist.
And after first cleaning, if the free capacity lower than threshold, cache file cleaner will start deep clean to help free dist.
With the cache-file-cleaner, add Dockerfile, entrypoint and daemonSet.yaml to help user to use this tool.
Using threadPool to help delete shuffle files.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#768 from zwangsheng/shuffle-cleaner.
Closes#768
3e3d5c65 [Binjie Yang] tini
2f7eab6e [Binjie Yang] dist
d19b9d0c [Binjie Yang] dist
3c8b7828 [Binjie Yang] dist
21bf5b4d [Binjie Yang] big change
4d803504 [Binjie Yang] rename to spark-block-cleaner & fix bug
ba6515d0 [Binjie Yang] do some change
1084feaa [Binjie Yang] delete empty dir
ada682da [Binjie Yang] big change
aabbf9ba [Binjie Yang] bugfix
a504448a [Binjie Yang] big change
0e0d5811 [Binjie Yang] modify
262fe314 [Binjie Yang] pom
ad9c533c [Binjie Yang] dist
0364f6e9 [Binjie Yang] pom
6fbb5cae [Binjie Yang] dist
797bcc39 [Binjie Yang] shuffle cleaner
Authored-by: Binjie Yang <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#756 from pan3793/codecov.
Closes#756
0fc8a6dc [Cheng Pan] nit
52fc82d5 [Cheng Pan] MAVEN_OPTS
20bac836 [Cheng Pan] try recover codecov
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Enable mvn cli options `--batch-mode` `--no-transfer-progress` in CI mode to suppress noising logs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#742 from pan3793/mvn.
Closes#742
79d8901b [Cheng Pan] --no-transfer-progress
218852a2 [Cheng Pan] fix syntax
309e45bf [Cheng Pan] MAVEN_CLI_OPTS
44907b65 [Cheng Pan] [BUILD] Enhance build/mvn in CI mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Fix bug.
### _How was this patch tested?_
Manual test with follow two cmd:
```
./build/dist --name 1.2.0-test --tgz -Pspark-3.1 -Pspark-hadoop-2.7 -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1
./build/dist --name 1.2.0-test --tgz -Pspark-3.1 -Pspark-hadoop-2.7 -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1 -Pkyuubi-extension-spark_3.1
```
Closes#639 from ulysses-you/fix-extension-package.
Closes#639
9852413 [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make Kyuubi jars clear.
### _How was this patch tested?_
Manual test. After this PR, the packaged dir looks like:
Closes#634 from ulysses-you/extension-dir.
Closes#634
2c5009a [ulysses-you] comment
35063ea [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into extension-dir
f2b4fff [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#628 from zhang1002/dev.
Closes#628
79b3d5a [张宇翔] ## solve the folder naming problem For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
Lead-authored-by: 张宇翔 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhangyuxiang03@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/401)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Current kyuubi server dist jars
```
➜ kyuubi git:(KYUUBI-398-followup) ll dist/jars
total 122064
-rw-r--r-- 1 chengpan staff 346K Mar 5 01:00 commons-codec-1.15.jar
-rw-r--r-- 1 chengpan staff 511K Mar 5 01:00 commons-lang3-3.10.jar
-rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-client-2.12.0.jar
-rw-r--r-- 1 chengpan staff 197K Mar 5 01:00 curator-framework-2.12.0.jar
-rw-r--r-- 1 chengpan staff 277K Mar 5 01:00 curator-recipes-2.12.0.jar
-rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-test-2.12.0.jar
-rw-r--r-- 1 chengpan staff 4.5K Mar 5 01:00 failureaccess-1.0.1.jar
-rw-r--r-- 1 chengpan staff 2.7M Mar 5 01:00 guava-30.1-jre.jar
-rw-r--r-- 1 chengpan staff 17M Mar 5 01:00 hadoop-client-api-3.2.2.jar
-rw-r--r-- 1 chengpan staff 22M Mar 5 01:00 hadoop-client-runtime-3.2.2.jar
-rw-r--r-- 1 chengpan staff 1.5M Mar 5 01:00 hive-service-rpc-2.3.7.jar
-rw-r--r-- 1 chengpan staff 1.4M Mar 5 01:00 htrace-core4-4.1.0-incubating.jar
-rw-r--r-- 1 chengpan staff 697K Mar 5 01:00 javassist-3.18.1-GA.jar
-rw-r--r-- 1 chengpan staff 16K Mar 5 01:00 jcl-over-slf4j-1.7.30.jar
-rw-r--r-- 1 chengpan staff 325K Mar 5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 34K Mar 5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 81K Mar 5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar
-rw-r--r-- 1 chengpan staff 306K Mar 5 01:00 libfb303-0.9.3.jar
-rw-r--r-- 1 chengpan staff 229K Mar 5 01:00 libthrift-0.9.3.jar
-rw-r--r-- 1 chengpan staff 478K Mar 5 01:00 log4j-1.2.17.jar
-rw-r--r-- 1 chengpan staff 5.0M Mar 5 01:00 scala-library-2.12.10.jar
-rw-r--r-- 1 chengpan staff 41K Mar 5 01:00 slf4j-api-1.7.30.jar
-rw-r--r-- 1 chengpan staff 12K Mar 5 01:00 slf4j-log4j12-1.7.30.jar
-rw-r--r-- 1 chengpan staff 890K Mar 5 01:00 zookeeper-3.4.14.jar
```
Closes#401 from pan3793/KYUUBI-398-followup.
c96586b [Cheng Pan] reorder deps
207f0b1 [Cheng Pan] restore docs
0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/399)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#398
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
manual test release workflow
```
➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```
Closes#399 from pan3793/KYUUBI-398.
a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
fixes#297
Squashed commit of the following:
commit 26df4c44a57f92dd16efe69898e9e969c22093fa
Author: fwang12 <fwang12@ebay.com>
Date: Mon Jan 18 13:09:13 2021 +0800
remove unused properties in build info
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix#167
* fix ut
* add ut
* fix ut
* add doc
* fix https://github.com/apache/spark/pull/24120
* logger
* typo
* comment/dist jar
* fix ut
* typo
