6afacdcdb8
757 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ulysses-you
|
6afacdcdb8 | release v1.1.0 | ||
|
MyLanPangzi
|
46725901c6 |
[KYUUBI #413] [MINOR]: remove unused class import
 [](https://github.com/yaooqinn/kyuubi/pull/413)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
### _Why are the changes needed?_
remove unused class import
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes #413 from MyLanPangzi/master.
dc6f65a [hiscat] [MINOR]: remove unused class import
Lead-authored-by: MyLanPangzi <1251723871@qq.com>
Co-authored-by: hiscat <46845236+MyLanPangzi@users.noreply.github.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
(cherry picked from commit
|
||
|
ulysses-you
|
5e551bdd71 |
[KYUUBI #408] Fix engine log does not be overwrite
 [](https://github.com/yaooqinn/kyuubi/pull/408)       [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Bug fix, otherwise the engine log will always increase size with append mode.
### _How was this patch tested?_
Add new test.
Closes #408 from ulysses-you/fix-engine-log-overwrite.
7e4fd1d [ulysses-you] simply code
b8d92ee [ulysses-you] config
821891c [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
(cherry picked from commit
|
||
|
ulysses-you
|
7107666f1b |
[KYUUBI #407] Auto update release note
 [](https://github.com/yaooqinn/kyuubi/pull/407)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
Closes #407 from ulysses-you/auto-update-release-note.
4dc09ae [ulysses-you] Release Changelog Builder
cfbe23c [ulysses-you] remove
8fcb56c [ulysses-you] changelog
28e8618 [ulysses-you] release
e953874 [ulysses-you] config
d792eea [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
(cherry picked from commit
|
||
|
Kent Yao
|
48e359b4a7
|
[HOTFIX] User hadoop2.7 as default for spark built in | ||
|
Kent Yao
|
49094f7768
|
[KYUUBI #406] Create GitHub Release only once
 [](https://github.com/yaooqinn/kyuubi/pull/406)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes #406 from yaooqinn/release.
2fc8754 [Kent Yao] Create GitHub Release only once
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit
|
||
|
ulysses-you
|
03b7112eee |
[KYUUBI #403] FIx flaky test in ConnectionLevelSparkEngineSuite
 [](https://github.com/yaooqinn/kyuubi/pull/403)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ `ignore` -> `test` Closes #403 from ulysses-you/flaky-test2. d8aa6c7 [ulysses-you] empty fd38e9c [ulysses-you] empty 2e5a7ee [ulysses-you] empty 444a526 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
9cb507d3d1 |
[KYUUBI #395] [WIP] FIx flaky test in ConnectionLevelSparkEngineSuite
 [](https://github.com/yaooqinn/kyuubi/pull/395)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ Closes #395 from ulysses-you/flaky-test. e8bdd4d [ulysses-you] empty 08febfc [ulysses-you] empty c3805d1 [ulysses-you] empty 702ac2c [ulysses-you] empty a544ef0 [ulysses-you] empty 0bd8878 [ulysses-you] empty dbd53e4 [ulysses-you] fix f4be895 [ulysses-you] empty 195933a [ulysses-you] empty cb69e0d [ulysses-you] wip Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
352b4eb99e |
[KYUUBI #400] Implement a simpler toHiveString to convert spark row
 [](https://github.com/yaooqinn/kyuubi/pull/400)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Hive compatibility is essential! This PR tries to restore the previous behavior. 1. Not use Spark's internal class HiveResult to convert spark catalyst type-values to JDBC results, the interval APIs are not APIs and changed unexpectedly. It involves a lot of overhead. (For this reason, pan3793 and I propose a Casting way to solve it.) 2. Unfortunately, the casting way breaks things like #397. So this is a reverting PR. 3. I checked the spark `HiveResult` carefully, I noticed that the `nested` logic is not needed in Kyuubi as we already matched primitives ahead. 4. Also the date time formatter used in `HiveResult` is excessively packaged with massively complicated logic inside spark. It's not clear and simple enough to call. 5. (tiny breaking change)the trailing spaces of a timestamp in milliseconds may not be trimmed. But this is a correctness issue and acceptable. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #400 from yaooqinn/397. 388c86d [Kent Yao] nit bbf458b [Kent Yao] time stamp formatter e5a7436 [Kent Yao] more tests 83f3142 [Kent Yao] timezone 38f5a76 [Kent Yao] Merge branch 'master' into 397 563d69c [Kent Yao] mute flacky test 9d834b0 [Kent Yao] Merge branch 'master' into 397 e8e05b5 [Kent Yao] nit 2a2d47b [Kent Yao] Implement a simpler toHiveString to convert spark row Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
8e83245d7a |
[KYUUBI #402] Remove hadoop suffix from tarball name when dist —without-spark
 [](https://github.com/yaooqinn/kyuubi/pull/402)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request `build/dist --tgz --spark-provided` previous: ``` kyuubi-1.1.0-SNAPSHOT-bin-without-spark-hadoop3.2.tar.gz ``` after patch: ``` kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #402 from pan3793/build. a534426 [Cheng Pan] update release workflow 1629489 [Cheng Pan] Remove hadoop suffix from tarball name when dist —without-spark Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
562c45865f
|
[KYUUBI #398][FOLLOWUP] exclude unused jars
 [](https://github.com/yaooqinn/kyuubi/pull/401)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Current kyuubi server dist jars ``` ➜ kyuubi git:(KYUUBI-398-followup) ll dist/jars total 122064 -rw-r--r-- 1 chengpan staff 346K Mar 5 01:00 commons-codec-1.15.jar -rw-r--r-- 1 chengpan staff 511K Mar 5 01:00 commons-lang3-3.10.jar -rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-client-2.12.0.jar -rw-r--r-- 1 chengpan staff 197K Mar 5 01:00 curator-framework-2.12.0.jar -rw-r--r-- 1 chengpan staff 277K Mar 5 01:00 curator-recipes-2.12.0.jar -rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-test-2.12.0.jar -rw-r--r-- 1 chengpan staff 4.5K Mar 5 01:00 failureaccess-1.0.1.jar -rw-r--r-- 1 chengpan staff 2.7M Mar 5 01:00 guava-30.1-jre.jar -rw-r--r-- 1 chengpan staff 17M Mar 5 01:00 hadoop-client-api-3.2.2.jar -rw-r--r-- 1 chengpan staff 22M Mar 5 01:00 hadoop-client-runtime-3.2.2.jar -rw-r--r-- 1 chengpan staff 1.5M Mar 5 01:00 hive-service-rpc-2.3.7.jar -rw-r--r-- 1 chengpan staff 1.4M Mar 5 01:00 htrace-core4-4.1.0-incubating.jar -rw-r--r-- 1 chengpan staff 697K Mar 5 01:00 javassist-3.18.1-GA.jar -rw-r--r-- 1 chengpan staff 16K Mar 5 01:00 jcl-over-slf4j-1.7.30.jar -rw-r--r-- 1 chengpan staff 325K Mar 5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 34K Mar 5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 81K Mar 5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 306K Mar 5 01:00 libfb303-0.9.3.jar -rw-r--r-- 1 chengpan staff 229K Mar 5 01:00 libthrift-0.9.3.jar -rw-r--r-- 1 chengpan staff 478K Mar 5 01:00 log4j-1.2.17.jar -rw-r--r-- 1 chengpan staff 5.0M Mar 5 01:00 scala-library-2.12.10.jar -rw-r--r-- 1 chengpan staff 41K Mar 5 01:00 slf4j-api-1.7.30.jar -rw-r--r-- 1 chengpan staff 12K Mar 5 01:00 slf4j-log4j12-1.7.30.jar -rw-r--r-- 1 chengpan staff 890K Mar 5 01:00 zookeeper-3.4.14.jar ``` Closes #401 from pan3793/KYUUBI-398-followup. c96586b [Cheng Pan] reorder deps 207f0b1 [Cheng Pan] restore docs 0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
4abc3ff6a0
|
[KYUUBI #398] Move to hadoop shaded client
 [](https://github.com/yaooqinn/kyuubi/pull/399)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #398 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request manual test release workflow ``` ➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz ``` Closes #399 from pan3793/KYUUBI-398. a9294a1 [Cheng Pan] fix ut d1d816d [Cheng Pan] fix dist script 2e3bc20 [Cheng Pan] update release workflow and dist script 0428b1b [Cheng Pan] update travis.yml 4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
8dd54a29b3
|
[KYUUBI #383] Support beeline --hivevar --hiveconf
 [](https://github.com/yaooqinn/kyuubi/pull/394)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #383 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #394 from pan3793/KYUUBI-383. 2810b58 [Cheng Pan] [KYUUBI #383] remove regex match, add server side test cases 3d8b1e0 [Cheng Pan] [KYUUBI #383] Support beeline --hivevar --hiveconf Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
874c71c6fe
|
[KYUUBI #396] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1
 [](https://github.com/yaooqinn/kyuubi/pull/396)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> new ci rule to check Backward Compatibility ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #396 from yaooqinn/bc. ae55ac9 [Kent Yao] ga bf854bd [Kent Yao] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
25d2d587b0
|
[KYUUBI #393] Add Support for Hadoop 3.2
 [](https://github.com/yaooqinn/kyuubi/pull/393)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR Add Support for Hadoop 3.2 Profile to work with spark3.1.1 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #393 from yaooqinn/guava2. 1f56082 [Kent Yao] nit c8c576c [Kent Yao] nit ccdfd03 [Kent Yao] Add Support for Hadoop 3.2 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
2e3315867b
|
[KYUUBI #298]Add Support for Apache Spark 3.1.1
 [](https://github.com/yaooqinn/kyuubi/pull/366)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved. In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7` TODOS: - Add Hadoop 3.2 support with some dependency issues - Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release - Using spark-3.1.1 as default - Restore HiveResult string? fix #298 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #366 from yaooqinn/spark3.1. 28ddf93 [Kent Yao] ga 37cb49e [Kent Yao] Merge branch 'master' into spark3.1 7b332ec [Kent Yao] typo b6b33d5 [Kent Yao] set up ga 00005c0 [Kent Yao] set up travis 2c09fe4 [Kent Yao] test fix 68c7edb [Kent Yao] deps 2c65e98 [Kent Yao] deps 236c7e9 [Kent Yao] nit 8c8a783 [Kent Yao] Merge branch 'master' into spark3.1 46c318a [Kent Yao] nit 32f6511 [Kent Yao] Merge branch 'master' into spark3.1 27c836b [Kent Yao] Merge branch 'master' into spark3.1 5574a8e [Kent Yao] ga d2ec2d5 [Kent Yao] [WIP]Spark3.1 49313cc [Kent Yao] [WIP]Travis ccb1a97 [Kent Yao] [WIP]Spark3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
aa878e4d48
|
[KYUUBI #386] Clean up discovery service before engine stop in connection share level
 [](https://github.com/yaooqinn/kyuubi/pull/387)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time. ### _How was this patch tested?_ Add some new suites in spark engine moudle with zk emmbedded. Closes #387 from ulysses-you/issue-386. ccb1112 [ulysses-you] move method 1558dd9 [ulysses-you] test 6cd7e57 [ulysses-you] nit f8559d2 [ulysses-you] check level at engine discovery eedfaee [ulysses-you] split ServiceDiscovery to server and engine 8b20e6c [ulysses-you] timeout 0cf524c [ulysses-you] remove unnecessary test 41d36f9 [ulysses-you] move exception eaaa12d [ulysses-you] avoid stop twice b801c14 [ulysses-you] fix test db39960 [ulysses-you] add start check 2d1f6dd [ulysses-you] add state check e7374aa [ulysses-you] nit ddf383c [ulysses-you] fi ee088be [ulysses-you] nit 3c2013b [ulysses-you] nit 73b386f [ulysses-you] improve conf of test 2185f49 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
8e8679e92b
|
[KYUUBI #392] Remove netty dependency
 [](https://github.com/yaooqinn/kyuubi/pull/392)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> netty is unnecessary ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #392 from yaooqinn/netty. 824d84e [Kent Yao] nit 402317d [Kent Yao] Remove netty dependency Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d10b17d87f
|
[KYUUBI #391] Bump up guava to 14.0.1
 [](https://github.com/yaooqinn/kyuubi/pull/391)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> align with spark master ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #391 from yaooqinn/guava. 164501b [Kent Yao] Bump up guava to 14.0.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
e49ff88305
|
[KYUUBI #390] Replace abbr in doc
 [](https://github.com/yaooqinn/kyuubi/pull/390)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #390 from pan3793/doc. 4b7cdf2 [Cheng Pan] replace abbr in doc Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
f83faa8356 |
[KYUUBI #389] Bump up curator to 2.12.0
 [](https://github.com/yaooqinn/kyuubi/pull/389)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 2.12.0 is the last update for curator-test to support zk v3.4.x ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #389 from yaooqinn/curator. 0b12323 [Kent Yao] Bump up curator to 2.12.0 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Kent Yao
|
3d0f050034
|
[KYUUBI #388] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi
 [](https://github.com/yaooqinn/kyuubi/pull/388)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> User guide ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #388 from yaooqinn/aqe. dc2833f [Kent Yao] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
4eed2ee0a7 |
[KYUUBI #385] Improve pass config between kyuubi and spark engine
 [](https://github.com/yaooqinn/kyuubi/pull/385)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make test more flexible and readable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #385 from ulysses-you/cleanup-test. c5d2034 [ulysses-you] readable fbc0f16 [ulysses-you] readable b1008b7 [ulysses-you] fix 37f5739 [ulysses-you] comment fdf52a7 [ulysses-you] add config method c58dfdb [ulysses-you] fix tes a1e8517 [ulysses-you] improve pass config 1378131 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
7f2a24bd9f |
[KYUUBI #381] Release session if shared level is CONNECTION
 [](https://github.com/yaooqinn/kyuubi/pull/382)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In CONNECTION level, it's better to release session if the connection is closed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #382 from ulysses-you/KYUUBI-381. cddba6c [ulysses-you] type a3fd691 [ulysses-you] increase engine init timeout 5030553 [ulysses-you] reload system properties 7a2e72f [ulysses-you] clean code 1034409 [ulysses-you] stopped d2a001a [ulysses-you] address comment 4eb62de [ulysses-you] 60s 64d0f16 [ulysses-you] fix test 4f10d34 [ulysses-you] sleep 62dc9f9 [ulysses-you] add new stop method 77be261 [ulysses-you] remove config fd9b73c [ulysses-you] add time sleep 209a59e [ulysses-you] fix test 88ed744 [ulysses-you] import 90015e9 [ulysses-you] simply 311fce3 [ulysses-you] fix test 7af3f72 [ulysses-you] fix test e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381 f77ddbe [ulysses-you] nit 2173222 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
08dbe24d28
|
[KYUUBI #380] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle
 [](https://github.com/yaooqinn/kyuubi/pull/380)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This is a followup of #377, change list: - change the spark version suffix in default distribution bundle name from `spark-{major}.{minor}.{patch}` to `spark-{major}.{minor}` - release distribution bundles both with spark and without spark Consider Kyuubi 1.1.x has not released, the available distribution still in old style, so now just keep the doc as is. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --tgz` ``` ➜ kyuubi git:(master) ✗ ls | grep tar.gz kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.tar.gz ``` Closes #380 from pan3793/build. 36fa006 [Cheng Pan] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
0f62568171
|
[KYUUBI #379] Improve engine log capture
 [](https://github.com/yaooqinn/kyuubi/pull/379)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead 2. update `error` immediately when detecting errors at the very first line ASAP 3. Using `Exception:` instead of `Exception` to search each line to filter better 4. add the engine log address in the exception message in case of error bootstraping ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #379 from yaooqinn/logsize. 7cf58da [Kent Yao] mute in console b58a3a2 [Kent Yao] update fbc4a0b [Kent Yao] update 0815f77 [Kent Yao] log 93f5f2f [Kent Yao] log Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
2f8df2355d |
[KYUUBI #377] build/dist support --spark-provided
 [](https://github.com/yaooqinn/kyuubi/pull/378)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --spark-provided` ``` (kyuubi) ➜ kyuubi git:(master) ✗ tree dist dist ├── LICENSE ├── RELEASE ├── bin │ ├── kyuubi │ ├── kyuubi-logo │ └── load-kyuubi-env.sh ├── conf │ ├── kyuubi-defaults.conf.template │ ├── kyuubi-env.sh.template │ └── log4j.properties.template ├── externals │ └── engines │ └── spark │ └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar ├── jars │ ├── apacheds-i18n-2.0.0-M15.jar │ ├── apacheds-kerberos-codec-2.0.0-M15.jar │ ├── api-asn1-api-1.0.0-M20.jar │ ├── api-util-1.0.0-M20.jar │ ├── audience-annotations-0.5.0.jar │ ├── commons-beanutils-1.7.0.jar │ ├── commons-beanutils-core-1.8.0.jar │ ├── commons-codec-1.4.jar │ ├── commons-collections-3.2.2.jar │ ├── commons-configuration-1.6.jar │ ├── commons-digester-1.8.jar │ ├── commons-io-2.4.jar │ ├── commons-lang-2.6.jar │ ├── commons-lang3-3.10.jar │ ├── commons-math-2.2.jar │ ├── curator-client-2.7.1.jar │ ├── curator-framework-2.7.1.jar │ ├── curator-recipes-2.7.1.jar │ ├── curator-test-2.7.1.jar │ ├── guava-11.0.2.jar │ ├── hadoop-annotations-2.7.4.jar │ ├── hadoop-auth-2.7.4.jar │ ├── hadoop-common-2.7.4.jar │ ├── hive-service-rpc-2.3.7.jar │ ├── httpclient-4.5.6.jar │ ├── httpcore-4.4.12.jar │ ├── javassist-3.18.1-GA.jar │ ├── jcl-over-slf4j-1.7.30.jar │ ├── jsr305-3.0.2.jar │ ├── kyuubi-common-1.1.0-SNAPSHOT.jar │ ├── kyuubi-ha-1.1.0-SNAPSHOT.jar │ ├── kyuubi-main-1.1.0-SNAPSHOT.jar │ ├── libfb303-0.9.3.jar │ ├── libthrift-0.9.3.jar │ ├── log4j-1.2.17.jar │ ├── netty-3.7.0.Final.jar │ ├── scala-library-2.12.10.jar │ ├── slf4j-api-1.7.30.jar │ ├── slf4j-log4j12-1.7.30.jar │ ├── spotbugs-annotations-3.1.9.jar │ └── zookeeper-3.4.14.jar ├── logs ├── pid └── work ``` Bundle size ``` (kyuubi) ➜ kyuubi git:(KYUUBI-377) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz -rw-r--r-- 1 chengpan staff 20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #378 from pan3793/KYUUBI-377. 442569e [Cheng Pan] add profile spark-provided for skip download spark binary 0d8da24 [Cheng Pan] update doc 5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
968ed4f079
|
[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test
 [](https://github.com/yaooqinn/kyuubi/pull/376)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #376 from yaooqinn/hive. 26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
0a315f3287
|
[KYUUBI #375] THRIFT-4858 causes metadata operations being flaky
 [](https://github.com/yaooqinn/kyuubi/pull/375)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #368 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #375 from yaooqinn/368. 809c7c3 [Kent Yao] THRIFT-4858 causes metadata operations being flaky afa4692 [Kent Yao] THRIFT-4858 causes metadata operations being flaky 0693c54 [Kent Yao] Merge branch 'master' into 368 ef7a759 [Kent Yao] THRIFT-4858 causes metadata operations being flaky Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
da8d2d4dd3
|
[KYUUBI #374] [DOC] Add Collaborators page
 [](https://github.com/yaooqinn/kyuubi/pull/374)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #374 from pan3793/doc. da57a03 [Cheng Pan] remove unused import js script 7e791ee [Cheng Pan] add Collaborators page Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
81e36a1389
|
[KYUUBI #372] Auto upload unit-tests.log
 [](https://github.com/yaooqinn/kyuubi/pull/373)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #372 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #373 from pan3793/KYUUBI-372. a560827 [Cheng Pan] [KYUUBI #372] Auto upload unit-tests.log Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
ce7b122435
|
[KYUUBI #371] Avoid creating useless operation log for meta operation
 [](https://github.com/yaooqinn/kyuubi/pull/371)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ 1. Operation log for metadata is useless for client-side 2. Operation log for metadata dumps nothing useful for the engine side too 3. Operation log will create files that bring overhead for meta operations and may fail operations ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #371 from yaooqinn/oplog. ca62e74 [Kent Yao] Avoid creating useless operation log for meta operation Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
c659089bc2
|
[KYUUBI #360] Correct handle getNextRowSet with FETCH_PRIOR FETCH_FIRST
 [](https://github.com/yaooqinn/kyuubi/pull/370)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #360 Ref: https://github.com/apache/spark/pull/30600 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #370 from pan3793/KYUUBI-360. e79b8cb [Cheng Pan] [KYUUBI #360] comments 0fae3db [Cheng Pan] fix import 3d1b2a6 [Cheng Pan] [KYUUBI #360] fix ut eda3e59 [Cheng Pan] [KYUUBI #360] fix import 16178d6 [Cheng Pan] [KYUUBI #360] ut 179404d [Cheng Pan] [KYUUBI #360] nit 455af6b [Cheng Pan] [KYUUBI #360] correct getNextRowSet with FETCH_PRIOR FETCH_FIRST 2307f1f [Cheng Pan] [KYUUBI #360] move ThriftUtils to kyuubi-common Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d94b1c4519
|
[KYUUBI #369] Using a lightweight zk client for kyuubi session
 [](https://github.com/yaooqinn/kyuubi/pull/369)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit zkClient to method wide 2. a more lightweight retry policy 3. add some trace log for Kyuubi session ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #369 from yaooqinn/zk. 5cbe14f [Kent Yao] typo 731bae8 [Kent Yao] typo 51365da [Kent Yao] Using a lightweight zk client for kyuubi session Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
4b13f1c551
|
[KYUUBI #367] Engine terminating checker should start after exposed
 [](https://github.com/yaooqinn/kyuubi/pull/367)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating. In this PR, we move the checker to the end where the engine is exposed to the servers. Avoiding errors like: ``` org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #367 from yaooqinn/engine. f6a979e [Kent Yao] fix test 9cc561c [Kent Yao] address comment 3d15ab2 [Kent Yao] nit a85d2e0 [Kent Yao] Engine terminating checker should start after exposed Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
2ff851f9bf |
[KYUUBI #365] Support user specify app name and tag
 [](https://github.com/yaooqinn/kyuubi/pull/365)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request After this PR, we will submit spark app with such command. ``` ./dist/externals/spark-3.0.1-bin-hadoop2.7/bin/beeline -u "jdbc:hive2://10.242.146.27:10009/;#spark.app.name=test" ``` ``` 2021-02-24 14:40:53.606 INFO session.KyuubiSessionImpl: Launching SQL engine: /Users/cathy/Desktop/code/kyuubi/dist/externals/spark-3.0.1-bin-hadoop2.7/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.app.name=test \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_USER/anonymous \ --conf spark.kyuubi.ha.zookeeper.quorum=127.0.0.1:2181 \ --conf spark.yarn.tags=KYUUBI \ --conf spark.kyuubi.ha.zookeeper.acl.enabled=false \ ``` Closes #365 from ulysses-you/configs. fa80eb6 [ulysses-you] append 4000ae3 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
399b30bb95
|
[KYUUBI #363] exclude hive-shims unused dependencies
 [](https://github.com/yaooqinn/kyuubi/pull/364)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #363 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #364 from pan3793/KYUUBI-363. 9b2c8c7 [Cheng Pan] [KYUUBI #363] exclude hive-shims unused dependencies Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
28db93cc75
|
[KYUUBI #361] Remove Hive Suffix for built in
 [](https://github.com/yaooqinn/kyuubi/pull/361)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> the builtin hive1.2 is no need for builtin anymore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #361 from yaooqinn/hivesuffix. 99d49fc [Kent Yao] Remove Hive Suffix for built in Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
ee50890b00
|
[KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility
 [](https://github.com/yaooqinn/kyuubi/pull/358)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> close #310 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #358 from yaooqinn/310. 8cb30a4 [Kent Yao] sql wildcards to java regex 34d2c3a [Kent Yao] Merge branch 'master' into 310 d332be5 [Kent Yao] [KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
54c306b967
|
[KYUUBI #356] Fix missing default versions
 [](https://github.com/yaooqinn/kyuubi/pull/359)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> the default versions may be missing if 'MVN' is absent, ``` SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark--bin-hadoop ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request ``` SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark-3.0.2-bin-hadoop2.7 SPARK_CONF_DIR: HADOOP_CONF_DIR: Candidate Kyuubi lib /Users/kentyao/kyuubi/jars doesn't exist, searching development environment... Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/kyuubi/logs/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out Welcome to __ __ __ /\ \/\ \ /\ \ __ \ \ \/'/' __ __ __ __ __ __\ \ \____/\_\ \ \ , < /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \ \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \ \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\ \/_/\/_/`/___/> \/___/ \/___/ \/___/ \/_/ /\___/ \/__/ ``` Closes #359 from yaooqinn/356. 2014862 [Kent Yao] [KYUUBI #356] Fix missing default versions Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
bacff6c59a |
[KYUUBI #356] Update built-in Spark version to 3.0.2
 [](https://github.com/yaooqinn/kyuubi/pull/356)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.0.2 has been released now. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #356 from ulysses-you/spark-3.0.2. c791e19 [ulysses-you] fix ad6c564 [ulysses-you] 3.0.2 Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Honglun
|
681b0d188d
|
[KYUUBI #347] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned
 [](https://github.com/yaooqinn/kyuubi/pull/354)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request mvn test -Dtest=none -DwildcardSuites=org.apache.kyuubi.session.SessionManagerSuit Closes #354 from Honglun/master. f5c623c [Honglun] Use Eventually d0fa460 [Honglun] Simplify common code for test cases. b46dfa7 [Honglun] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned, Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
23e1d61e47
|
[KYUUBI #350] Avoid using HiveResult
 [](https://github.com/yaooqinn/kyuubi/pull/351)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #350 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #351 from pan3793/KYUUBI-350. 8b9ac46 [Cheng Pan] debug logs 21add73 [Cheng Pan] minor 46ceb54 [Cheng Pan] [KYUUBI-350] in favor cast string in executeStatment f4e3633 [Cheng Pan] [KYUUBI-350] Avoid using HiveResult in RowSet Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
600b014070
|
[KYUUBI #312] Support run kyuubi in foreground
 [](https://github.com/yaooqinn/kyuubi/pull/352)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #312 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request  Closes #352 from pan3793/KYUUBI-312. 1a48504 [Cheng Pan] update help 08f589d [Cheng Pan] update doc and help 563f39e [Cheng Pan] [KYUUBI-312] support run kyuubi in foreground Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yangrong688
|
5637e9105e
|
[KYUUBI #349] Fix typo
 [](https://github.com/yaooqinn/kyuubi/pull/349)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> Remove duplicate word vision. Closes #349 from yangrong688/master. d1673b5 [yangrong688] delete duplicate word vision Authored-by: yangrong688 <yangrong.jxufe@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
a2818003e6
|
[KYUUBI #309] GetTables supports DSv2 and keeps its backward compatibility
 [](https://github.com/yaooqinn/kyuubi/pull/346)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix #309 get tables through Spark DSv2 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #346 from yaooqinn/gettables. 16dbe4b [Kent Yao] nit e03b021 [Kent Yao] nit 7f3eee2 [Kent Yao] Merge branch 'master' into gettables 0180cc5 [Kent Yao] add a test 8498152 [Kent Yao] GetTables supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Honglun
|
c1a71b7d00
|
[KYUUBI #347]Closing expired operations does not require updating 'lastAccessTime'
 [](https://github.com/yaooqinn/kyuubi/pull/348)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> … and 'lastTidleTime' <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Method 'session.closeExpiredOperations' will constantly update the values of '_lastAccessTime' and '_lastIdleTime', so the session can't be closed. Method 'closeExpiredOperations' should remove 'withAcquirerease'. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request conf.set(KyuubiConf.SESSION_CHECK_INTERVAL,Duration.ofSeconds(5).toMillis) conf.set(KyuubiConf.SESSION_TIMEOUT,Duration.ofMinutes(1L).toMillis)  Closes #348 from Honglun/master. 6cff013 [Honglun] Closing expired operations does not require updating 'lastAccessTime'. a6330d7 [Honglun] Closing expired operations does not require updating 'lastAccessTime' and 'lastTidleTime' Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
e4d9390a74
|
typo | ||
|
ulysses-you
|
fbf1e3506c
|
[KYUUBI #345] Catch Throwable instead of Exception
 [](https://github.com/yaooqinn/kyuubi/pull/345)       [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We might get this error when running spark sql，so it's more reasonable to catch `Throwable`. ``` java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #345 from ulysses-you/throwable. b6ffacd [ulysses-you] throw Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
9dc24d6650
|
[KYUUBI #344] [MINOR] update kyuubi version in quick_start
 [](https://github.com/yaooqinn/kyuubi/pull/344)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #344 from pan3793/docs. eaa4eb1 [Cheng Pan] [MINOR] update kyuubi version in quick_start Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
