748 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Cheng Pan	8e83245d7a	[KYUUBI #402] Remove hadoop suffix from tarball name when dist —without-spark ...  [](https://github.com/yaooqinn/kyuubi/pull/402)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request `build/dist --tgz --spark-provided` previous: ``` kyuubi-1.1.0-SNAPSHOT-bin-without-spark-hadoop3.2.tar.gz ``` after patch: ``` kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #402 from pan3793/build. a534426 [Cheng Pan] update release workflow 1629489 [Cheng Pan] Remove hadoop suffix from tarball name when dist —without-spark Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-03-05 13:49:34 +08:00
Cheng Pan	562c45865f	[KYUUBI #398][FOLLOWUP] exclude unused jars ...  [](https://github.com/yaooqinn/kyuubi/pull/401)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Current kyuubi server dist jars ``` ➜ kyuubi git:(KYUUBI-398-followup) ll dist/jars total 122064 -rw-r--r-- 1 chengpan staff 346K Mar 5 01:00 commons-codec-1.15.jar -rw-r--r-- 1 chengpan staff 511K Mar 5 01:00 commons-lang3-3.10.jar -rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-client-2.12.0.jar -rw-r--r-- 1 chengpan staff 197K Mar 5 01:00 curator-framework-2.12.0.jar -rw-r--r-- 1 chengpan staff 277K Mar 5 01:00 curator-recipes-2.12.0.jar -rw-r--r-- 1 chengpan staff 2.3M Mar 5 01:00 curator-test-2.12.0.jar -rw-r--r-- 1 chengpan staff 4.5K Mar 5 01:00 failureaccess-1.0.1.jar -rw-r--r-- 1 chengpan staff 2.7M Mar 5 01:00 guava-30.1-jre.jar -rw-r--r-- 1 chengpan staff 17M Mar 5 01:00 hadoop-client-api-3.2.2.jar -rw-r--r-- 1 chengpan staff 22M Mar 5 01:00 hadoop-client-runtime-3.2.2.jar -rw-r--r-- 1 chengpan staff 1.5M Mar 5 01:00 hive-service-rpc-2.3.7.jar -rw-r--r-- 1 chengpan staff 1.4M Mar 5 01:00 htrace-core4-4.1.0-incubating.jar -rw-r--r-- 1 chengpan staff 697K Mar 5 01:00 javassist-3.18.1-GA.jar -rw-r--r-- 1 chengpan staff 16K Mar 5 01:00 jcl-over-slf4j-1.7.30.jar -rw-r--r-- 1 chengpan staff 325K Mar 5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 34K Mar 5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 81K Mar 5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 306K Mar 5 01:00 libfb303-0.9.3.jar -rw-r--r-- 1 chengpan staff 229K Mar 5 01:00 libthrift-0.9.3.jar -rw-r--r-- 1 chengpan staff 478K Mar 5 01:00 log4j-1.2.17.jar -rw-r--r-- 1 chengpan staff 5.0M Mar 5 01:00 scala-library-2.12.10.jar -rw-r--r-- 1 chengpan staff 41K Mar 5 01:00 slf4j-api-1.7.30.jar -rw-r--r-- 1 chengpan staff 12K Mar 5 01:00 slf4j-log4j12-1.7.30.jar -rw-r--r-- 1 chengpan staff 890K Mar 5 01:00 zookeeper-3.4.14.jar ``` Closes #401 from pan3793/KYUUBI-398-followup. c96586b [Cheng Pan] reorder deps 207f0b1 [Cheng Pan] restore docs 0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-05 08:57:23 +08:00
Cheng Pan	4abc3ff6a0	[KYUUBI #398] Move to hadoop shaded client ...  [](https://github.com/yaooqinn/kyuubi/pull/399)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #398 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request manual test release workflow ``` ➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz ``` Closes #399 from pan3793/KYUUBI-398. a9294a1 [Cheng Pan] fix ut d1d816d [Cheng Pan] fix dist script 2e3bc20 [Cheng Pan] update release workflow and dist script 0428b1b [Cheng Pan] update travis.yml 4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 23:33:31 +08:00
Cheng Pan	8dd54a29b3	[KYUUBI #383] Support beeline --hivevar --hiveconf ...  [](https://github.com/yaooqinn/kyuubi/pull/394)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #383 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #394 from pan3793/KYUUBI-383. 2810b58 [Cheng Pan] [KYUUBI #383] remove regex match, add server side test cases 3d8b1e0 [Cheng Pan] [KYUUBI #383] Support beeline --hivevar --hiveconf Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 21:28:22 +08:00
Kent Yao	874c71c6fe	[KYUUBI #396] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1 ...  [](https://github.com/yaooqinn/kyuubi/pull/396)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> new ci rule to check Backward Compatibility ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #396 from yaooqinn/bc. ae55ac9 [Kent Yao] ga bf854bd [Kent Yao] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 20:57:58 +08:00
Kent Yao	25d2d587b0	[KYUUBI #393] Add Support for Hadoop 3.2 ...  [](https://github.com/yaooqinn/kyuubi/pull/393)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR Add Support for Hadoop 3.2 Profile to work with spark3.1.1 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #393 from yaooqinn/guava2. 1f56082 [Kent Yao] nit c8c576c [Kent Yao] nit ccdfd03 [Kent Yao] Add Support for Hadoop 3.2 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 18:25:11 +08:00
Kent Yao	2e3315867b	[KYUUBI #298]Add Support for Apache Spark 3.1.1 ...  [](https://github.com/yaooqinn/kyuubi/pull/366)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved. In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7` TODOS: - Add Hadoop 3.2 support with some dependency issues - Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release - Using spark-3.1.1 as default - Restore HiveResult string? fix #298 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #366 from yaooqinn/spark3.1. 28ddf93 [Kent Yao] ga 37cb49e [Kent Yao] Merge branch 'master' into spark3.1 7b332ec [Kent Yao] typo b6b33d5 [Kent Yao] set up ga 00005c0 [Kent Yao] set up travis 2c09fe4 [Kent Yao] test fix 68c7edb [Kent Yao] deps 2c65e98 [Kent Yao] deps 236c7e9 [Kent Yao] nit 8c8a783 [Kent Yao] Merge branch 'master' into spark3.1 46c318a [Kent Yao] nit 32f6511 [Kent Yao] Merge branch 'master' into spark3.1 27c836b [Kent Yao] Merge branch 'master' into spark3.1 5574a8e [Kent Yao] ga d2ec2d5 [Kent Yao] [WIP]Spark3.1 49313cc [Kent Yao] [WIP]Travis ccb1a97 [Kent Yao] [WIP]Spark3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 13:30:05 +08:00
ulysses-you	aa878e4d48	[KYUUBI #386] Clean up discovery service before engine stop in connection share level ...  [](https://github.com/yaooqinn/kyuubi/pull/387)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time. ### _How was this patch tested?_ Add some new suites in spark engine moudle with zk emmbedded. Closes #387 from ulysses-you/issue-386. ccb1112 [ulysses-you] move method 1558dd9 [ulysses-you] test 6cd7e57 [ulysses-you] nit f8559d2 [ulysses-you] check level at engine discovery eedfaee [ulysses-you] split ServiceDiscovery to server and engine 8b20e6c [ulysses-you] timeout 0cf524c [ulysses-you] remove unnecessary test 41d36f9 [ulysses-you] move exception eaaa12d [ulysses-you] avoid stop twice b801c14 [ulysses-you] fix test db39960 [ulysses-you] add start check 2d1f6dd [ulysses-you] add state check e7374aa [ulysses-you] nit ddf383c [ulysses-you] fi ee088be [ulysses-you] nit 3c2013b [ulysses-you] nit 73b386f [ulysses-you] improve conf of test 2185f49 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 11:31:41 +08:00
Kent Yao	8e8679e92b	[KYUUBI #392] Remove netty dependency ...  [](https://github.com/yaooqinn/kyuubi/pull/392)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> netty is unnecessary ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #392 from yaooqinn/netty. 824d84e [Kent Yao] nit 402317d [Kent Yao] Remove netty dependency Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-03 22:23:38 +08:00
Kent Yao	d10b17d87f	[KYUUBI #391] Bump up guava to 14.0.1 ...  [](https://github.com/yaooqinn/kyuubi/pull/391)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> align with spark master ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #391 from yaooqinn/guava. 164501b [Kent Yao] Bump up guava to 14.0.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-03 22:05:05 +08:00
Cheng Pan	e49ff88305	[KYUUBI #390] Replace abbr in doc ...  [](https://github.com/yaooqinn/kyuubi/pull/390)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #390 from pan3793/doc. 4b7cdf2 [Cheng Pan] replace abbr in doc Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-03 22:04:32 +08:00
Kent Yao	f83faa8356	[KYUUBI #389] Bump up curator to 2.12.0 ...  [](https://github.com/yaooqinn/kyuubi/pull/389)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 2.12.0 is the last update for curator-test to support zk v3.4.x ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #389 from yaooqinn/curator. 0b12323 [Kent Yao] Bump up curator to 2.12.0 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-03-03 19:37:59 +08:00
Kent Yao	3d0f050034	[KYUUBI #388] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi ...  [](https://github.com/yaooqinn/kyuubi/pull/388)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> User guide ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #388 from yaooqinn/aqe. dc2833f [Kent Yao] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-03 13:55:52 +08:00
ulysses-you	4eed2ee0a7	[KYUUBI #385] Improve pass config between kyuubi and spark engine ...  [](https://github.com/yaooqinn/kyuubi/pull/385)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make test more flexible and readable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #385 from ulysses-you/cleanup-test. c5d2034 [ulysses-you] readable fbc0f16 [ulysses-you] readable b1008b7 [ulysses-you] fix 37f5739 [ulysses-you] comment fdf52a7 [ulysses-you] add config method c58dfdb [ulysses-you] fix tes a1e8517 [ulysses-you] improve pass config 1378131 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-03 10:01:54 +08:00
ulysses-you	7f2a24bd9f	[KYUUBI #381] Release session if shared level is CONNECTION ...  [](https://github.com/yaooqinn/kyuubi/pull/382)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In CONNECTION level, it's better to release session if the connection is closed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #382 from ulysses-you/KYUUBI-381. cddba6c [ulysses-you] type a3fd691 [ulysses-you] increase engine init timeout 5030553 [ulysses-you] reload system properties 7a2e72f [ulysses-you] clean code 1034409 [ulysses-you] stopped d2a001a [ulysses-you] address comment 4eb62de [ulysses-you] 60s 64d0f16 [ulysses-you] fix test 4f10d34 [ulysses-you] sleep 62dc9f9 [ulysses-you] add new stop method 77be261 [ulysses-you] remove config fd9b73c [ulysses-you] add time sleep 209a59e [ulysses-you] fix test 88ed744 [ulysses-you] import 90015e9 [ulysses-you] simply 311fce3 [ulysses-you] fix test 7af3f72 [ulysses-you] fix test e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381 f77ddbe [ulysses-you] nit 2173222 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-02 11:44:32 +08:00
Cheng Pan	08dbe24d28	[KYUUBI #380] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle ...  [](https://github.com/yaooqinn/kyuubi/pull/380)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This is a followup of #377, change list: - change the spark version suffix in default distribution bundle name from `spark-{major}.{minor}.{patch}` to `spark-{major}.{minor}` - release distribution bundles both with spark and without spark Consider Kyuubi 1.1.x has not released, the available distribution still in old style, so now just keep the doc as is. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --tgz` ``` ➜ kyuubi git:(master) ✗ ls | grep tar.gz kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.tar.gz ``` Closes #380 from pan3793/build. 36fa006 [Cheng Pan] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-28 23:05:36 +08:00
Kent Yao	0f62568171	[KYUUBI #379] Improve engine log capture ...  [](https://github.com/yaooqinn/kyuubi/pull/379)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead 2. update `error` immediately when detecting errors at the very first line ASAP 3. Using `Exception:` instead of `Exception` to search each line to filter better 4. add the engine log address in the exception message in case of error bootstraping ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #379 from yaooqinn/logsize. 7cf58da [Kent Yao] mute in console b58a3a2 [Kent Yao] update fbc4a0b [Kent Yao] update 0815f77 [Kent Yao] log 93f5f2f [Kent Yao] log Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 17:31:44 +08:00
Cheng Pan	2f8df2355d	[KYUUBI #377] build/dist support --spark-provided ...  [](https://github.com/yaooqinn/kyuubi/pull/378)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --spark-provided` ``` (kyuubi) ➜ kyuubi git:(master) ✗ tree dist dist ├── LICENSE ├── RELEASE ├── bin │   ├── kyuubi │   ├── kyuubi-logo │   └── load-kyuubi-env.sh ├── conf │   ├── kyuubi-defaults.conf.template │   ├── kyuubi-env.sh.template │   └── log4j.properties.template ├── externals │   └── engines │   └── spark │   └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar ├── jars │   ├── apacheds-i18n-2.0.0-M15.jar │   ├── apacheds-kerberos-codec-2.0.0-M15.jar │   ├── api-asn1-api-1.0.0-M20.jar │   ├── api-util-1.0.0-M20.jar │   ├── audience-annotations-0.5.0.jar │   ├── commons-beanutils-1.7.0.jar │   ├── commons-beanutils-core-1.8.0.jar │   ├── commons-codec-1.4.jar │   ├── commons-collections-3.2.2.jar │   ├── commons-configuration-1.6.jar │   ├── commons-digester-1.8.jar │   ├── commons-io-2.4.jar │   ├── commons-lang-2.6.jar │   ├── commons-lang3-3.10.jar │   ├── commons-math-2.2.jar │   ├── curator-client-2.7.1.jar │   ├── curator-framework-2.7.1.jar │   ├── curator-recipes-2.7.1.jar │   ├── curator-test-2.7.1.jar │   ├── guava-11.0.2.jar │   ├── hadoop-annotations-2.7.4.jar │   ├── hadoop-auth-2.7.4.jar │   ├── hadoop-common-2.7.4.jar │   ├── hive-service-rpc-2.3.7.jar │   ├── httpclient-4.5.6.jar │   ├── httpcore-4.4.12.jar │   ├── javassist-3.18.1-GA.jar │   ├── jcl-over-slf4j-1.7.30.jar │   ├── jsr305-3.0.2.jar │   ├── kyuubi-common-1.1.0-SNAPSHOT.jar │   ├── kyuubi-ha-1.1.0-SNAPSHOT.jar │   ├── kyuubi-main-1.1.0-SNAPSHOT.jar │   ├── libfb303-0.9.3.jar │   ├── libthrift-0.9.3.jar │   ├── log4j-1.2.17.jar │   ├── netty-3.7.0.Final.jar │   ├── scala-library-2.12.10.jar │   ├── slf4j-api-1.7.30.jar │   ├── slf4j-log4j12-1.7.30.jar │   ├── spotbugs-annotations-3.1.9.jar │   └── zookeeper-3.4.14.jar ├── logs ├── pid └── work ``` Bundle size ``` (kyuubi) ➜ kyuubi git:(KYUUBI-377) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz -rw-r--r-- 1 chengpan staff 20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #378 from pan3793/KYUUBI-377. 442569e [Cheng Pan] add profile spark-provided for skip download spark binary 0d8da24 [Cheng Pan] update doc 5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-02-26 15:00:42 +08:00
Kent Yao	968ed4f079	[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test ...  [](https://github.com/yaooqinn/kyuubi/pull/376)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #376 from yaooqinn/hive. 26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 00:28:17 +08:00
Kent Yao	0a315f3287	[KYUUBI #375] THRIFT-4858 causes metadata operations being flaky ...  [](https://github.com/yaooqinn/kyuubi/pull/375)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #368 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #375 from yaooqinn/368. 809c7c3 [Kent Yao] THRIFT-4858 causes metadata operations being flaky afa4692 [Kent Yao] THRIFT-4858 causes metadata operations being flaky 0693c54 [Kent Yao] Merge branch 'master' into 368 ef7a759 [Kent Yao] THRIFT-4858 causes metadata operations being flaky Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 23:36:11 +08:00
Cheng Pan	da8d2d4dd3	[KYUUBI #374] [DOC] Add Collaborators page ...  [](https://github.com/yaooqinn/kyuubi/pull/374)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #374 from pan3793/doc. da57a03 [Cheng Pan] remove unused import js script 7e791ee [Cheng Pan] add Collaborators page Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 21:47:56 +08:00
Cheng Pan	81e36a1389	[KYUUBI #372] Auto upload unit-tests.log ...  [](https://github.com/yaooqinn/kyuubi/pull/373)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #372 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #373 from pan3793/KYUUBI-372. a560827 [Cheng Pan] [KYUUBI #372] Auto upload unit-tests.log Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 20:55:06 +08:00
Kent Yao	ce7b122435	[KYUUBI #371] Avoid creating useless operation log for meta operation ...  [](https://github.com/yaooqinn/kyuubi/pull/371)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ 1. Operation log for metadata is useless for client-side 2. Operation log for metadata dumps nothing useful for the engine side too 3. Operation log will create files that bring overhead for meta operations and may fail operations ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #371 from yaooqinn/oplog. ca62e74 [Kent Yao] Avoid creating useless operation log for meta operation Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 20:47:48 +08:00
Cheng Pan	c659089bc2	[KYUUBI #360] Correct handle getNextRowSet with FETCH_PRIOR FETCH_FIRST ...  [](https://github.com/yaooqinn/kyuubi/pull/370)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #360 Ref: https://github.com/apache/spark/pull/30600 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #370 from pan3793/KYUUBI-360. e79b8cb [Cheng Pan] [KYUUBI #360] comments 0fae3db [Cheng Pan] fix import 3d1b2a6 [Cheng Pan] [KYUUBI #360] fix ut eda3e59 [Cheng Pan] [KYUUBI #360] fix import 16178d6 [Cheng Pan] [KYUUBI #360] ut 179404d [Cheng Pan] [KYUUBI #360] nit 455af6b [Cheng Pan] [KYUUBI #360] correct getNextRowSet with FETCH_PRIOR FETCH_FIRST 2307f1f [Cheng Pan] [KYUUBI #360] move ThriftUtils to kyuubi-common Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 18:19:02 +08:00
Kent Yao	d94b1c4519	[KYUUBI #369] Using a lightweight zk client for kyuubi session ...  [](https://github.com/yaooqinn/kyuubi/pull/369)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit zkClient to method wide 2. a more lightweight retry policy 3. add some trace log for Kyuubi session ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #369 from yaooqinn/zk. 5cbe14f [Kent Yao] typo 731bae8 [Kent Yao] typo 51365da [Kent Yao] Using a lightweight zk client for kyuubi session Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 16:57:55 +08:00
Kent Yao	4b13f1c551	[KYUUBI #367] Engine terminating checker should start after exposed ...  [](https://github.com/yaooqinn/kyuubi/pull/367)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating. In this PR, we move the checker to the end where the engine is exposed to the servers. Avoiding errors like: ``` org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #367 from yaooqinn/engine. f6a979e [Kent Yao] fix test 9cc561c [Kent Yao] address comment 3d15ab2 [Kent Yao] nit a85d2e0 [Kent Yao] Engine terminating checker should start after exposed Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 15:10:43 +08:00
ulysses-you	2ff851f9bf	[KYUUBI #365] Support user specify app name and tag ...  [](https://github.com/yaooqinn/kyuubi/pull/365)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request After this PR, we will submit spark app with such command. ``` ./dist/externals/spark-3.0.1-bin-hadoop2.7/bin/beeline -u "jdbc:hive2://10.242.146.27:10009/;#spark.app.name=test" ``` ``` 2021-02-24 14:40:53.606 INFO session.KyuubiSessionImpl: Launching SQL engine: /Users/cathy/Desktop/code/kyuubi/dist/externals/spark-3.0.1-bin-hadoop2.7/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.app.name=test \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_USER/anonymous \ --conf spark.kyuubi.ha.zookeeper.quorum=127.0.0.1:2181 \ --conf spark.yarn.tags=KYUUBI \ --conf spark.kyuubi.ha.zookeeper.acl.enabled=false \ ``` Closes #365 from ulysses-you/configs. fa80eb6 [ulysses-you] append 4000ae3 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-02-24 15:49:20 +08:00
Cheng Pan	399b30bb95	[KYUUBI #363] exclude hive-shims unused dependencies ...  [](https://github.com/yaooqinn/kyuubi/pull/364)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #363 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #364 from pan3793/KYUUBI-363. 9b2c8c7 [Cheng Pan] [KYUUBI #363] exclude hive-shims unused dependencies Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-24 11:38:31 +08:00
Kent Yao	28db93cc75	[KYUUBI #361] Remove Hive Suffix for built in ...  [](https://github.com/yaooqinn/kyuubi/pull/361)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> the builtin hive1.2 is no need for builtin anymore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #361 from yaooqinn/hivesuffix. 99d49fc [Kent Yao] Remove Hive Suffix for built in Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-24 00:13:20 +08:00
Kent Yao	ee50890b00	[KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/358)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> close #310 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #358 from yaooqinn/310. 8cb30a4 [Kent Yao] sql wildcards to java regex 34d2c3a [Kent Yao] Merge branch 'master' into 310 d332be5 [Kent Yao] [KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-23 20:43:02 +08:00
Kent Yao	54c306b967	[KYUUBI #356] Fix missing default versions ...  [](https://github.com/yaooqinn/kyuubi/pull/359)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> the default versions may be missing if 'MVN' is absent, ``` SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark--bin-hadoop ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request ``` SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark-3.0.2-bin-hadoop2.7 SPARK_CONF_DIR: HADOOP_CONF_DIR: Candidate Kyuubi lib /Users/kentyao/kyuubi/jars doesn't exist, searching development environment... Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/kyuubi/logs/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out Welcome to __ __ __ /\ \/\ \ /\ \ __ \ \ \/'/' __ __ __ __ __ __\ \ \____/\_\ \ \ , < /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \ \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \ \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\ \/_/\/_/`/___/> \/___/ \/___/ \/___/ \/_/ /\___/ \/__/ ``` Closes #359 from yaooqinn/356. 2014862 [Kent Yao] [KYUUBI #356] Fix missing default versions Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-23 16:15:49 +08:00
ulysses-you	bacff6c59a	[KYUUBI #356] Update built-in Spark version to 3.0.2 ...  [](https://github.com/yaooqinn/kyuubi/pull/356)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.0.2 has been released now. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #356 from ulysses-you/spark-3.0.2. c791e19 [ulysses-you] fix ad6c564 [ulysses-you] 3.0.2 Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-02-20 18:26:26 +08:00
Honglun	681b0d188d	[KYUUBI #347] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned ...  [](https://github.com/yaooqinn/kyuubi/pull/354)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request mvn test -Dtest=none -DwildcardSuites=org.apache.kyuubi.session.SessionManagerSuit Closes #354 from Honglun/master. f5c623c [Honglun] Use Eventually d0fa460 [Honglun] Simplify common code for test cases. b46dfa7 [Honglun] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned, Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-17 00:50:12 +08:00
Cheng Pan	23e1d61e47	[KYUUBI #350] Avoid using HiveResult ...  [](https://github.com/yaooqinn/kyuubi/pull/351)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #350 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #351 from pan3793/KYUUBI-350. 8b9ac46 [Cheng Pan] debug logs 21add73 [Cheng Pan] minor 46ceb54 [Cheng Pan] [KYUUBI-350] in favor cast string in executeStatment f4e3633 [Cheng Pan] [KYUUBI-350] Avoid using HiveResult in RowSet Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-12 14:00:39 +08:00
Cheng Pan	600b014070	[KYUUBI #312] Support run kyuubi in foreground ...  [](https://github.com/yaooqinn/kyuubi/pull/352)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #312 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request  Closes #352 from pan3793/KYUUBI-312. 1a48504 [Cheng Pan] update help 08f589d [Cheng Pan] update doc and help 563f39e [Cheng Pan] [KYUUBI-312] support run kyuubi in foreground Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-08 23:53:06 +08:00
yangrong688	5637e9105e	[KYUUBI #349] Fix typo ...  [](https://github.com/yaooqinn/kyuubi/pull/349)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> Remove duplicate word vision. Closes #349 from yangrong688/master. d1673b5 [yangrong688] delete duplicate word vision Authored-by: yangrong688 <yangrong.jxufe@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-08 15:43:36 +08:00
Kent Yao	a2818003e6	[KYUUBI #309] GetTables supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/346)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix #309 get tables through Spark DSv2 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #346 from yaooqinn/gettables. 16dbe4b [Kent Yao] nit e03b021 [Kent Yao] nit 7f3eee2 [Kent Yao] Merge branch 'master' into gettables 0180cc5 [Kent Yao] add a test 8498152 [Kent Yao] GetTables supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-07 13:45:53 +08:00
Honglun	c1a71b7d00	[KYUUBI #347]Closing expired operations does not require updating 'lastAccessTime' ...  [](https://github.com/yaooqinn/kyuubi/pull/348)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> … and 'lastTidleTime' <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Method 'session.closeExpiredOperations' will constantly update the values of '_lastAccessTime' and '_lastIdleTime', so the session can't be closed. Method 'closeExpiredOperations' should remove 'withAcquirerease'. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request conf.set(KyuubiConf.SESSION_CHECK_INTERVAL,Duration.ofSeconds(5).toMillis) conf.set(KyuubiConf.SESSION_TIMEOUT,Duration.ofMinutes(1L).toMillis)  Closes #348 from Honglun/master. 6cff013 [Honglun] Closing expired operations does not require updating 'lastAccessTime'. a6330d7 [Honglun] Closing expired operations does not require updating 'lastAccessTime' and 'lastTidleTime' Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-04 20:56:04 +08:00
Kent Yao	e4d9390a74	typo	2021-02-03 23:39:50 +08:00
ulysses-you	fbf1e3506c	[KYUUBI #345] Catch Throwable instead of Exception ...  [](https://github.com/yaooqinn/kyuubi/pull/345)       [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We might get this error when running spark sql，so it's more reasonable to catch `Throwable`. ``` java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #345 from ulysses-you/throwable. b6ffacd [ulysses-you] throw Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-03 19:31:26 +08:00
Cheng Pan	9dc24d6650	[KYUUBI #344] [MINOR] update kyuubi version in quick_start ...  [](https://github.com/yaooqinn/kyuubi/pull/344)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #344 from pan3793/docs. eaa4eb1 [Cheng Pan] [MINOR] update kyuubi version in quick_start Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-03 19:30:45 +08:00
Cheng Pan	aed27d7a65	[KYUUBI #342] Exclude unused hive-jdbc transitive dependencies ... fix #343  [](https://github.com/yaooqinn/kyuubi/pull/343)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #342 - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Squashed commit of the following: commit c1b2eeae02dbca56ddf44d140ccc5f558927b618 Author: Cheng Pan <379377944@qq.com> Date: Wed Feb 3 10:54:27 2021 +0800 remove unused hive-cli, hive-beeline commit 9f1a9de05ade4d155b44d817f6a36edcf4f27f03 Author: Cheng Pan <379377944@qq.com> Date: Wed Feb 3 10:43:07 2021 +0800 tune pom commit 5d63596abc91412b8f60589deb46e0f92e13cc00 Author: Cheng Pan <379377944@qq.com> Date: Wed Feb 3 00:25:40 2021 +0800 update pom.xml commit 70cb27a352db38578eb2b4857c2d96afd9fc4204 Author: Cheng Pan <379377944@qq.com> Date: Tue Feb 2 22:12:29 2021 +0800 remove unused hive deps in kyuubi-common commit ba8038c8981ae4dd9a597259925a38276fdaa567 Author: Cheng Pan <379377944@qq.com> Date: Tue Feb 2 21:52:47 2021 +0800 [KYUUBI #342] Exclude unused hive-jdbc transitive dependencies	2021-02-03 12:50:16 +08:00
Cheng Pan	2dc580456a	[KYUUBI #340] Swtich to commons-lang3 ...  [](https://github.com/yaooqinn/kyuubi/pull/341)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #340 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #341 from pan3793/KYUUBI-340. e014fe9 [Cheng Pan] swtich to commons-lang3 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 20:47:45 +08:00
Kent Yao	7deb2f7fd7	[KYUUBI #336] Fix a race in Kyuubi start and stop ...  [](https://github.com/yaooqinn/kyuubi/pull/336)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> there is something wrong with Travis these days due to some newly added test cases ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #336 from yaooqinn/travisagain. 0285f5a [Kent Yao] fix a race 3624889 [Kent Yao] fix a race 895b4d5 [Kent Yao] fix a race e253b9c [Kent Yao] fix a race 75d699a [Kent Yao] fix a race 7bde6b9 [Kent Yao] nit 6959e42 [Kent Yao] nit 062a151 [Kent Yao] nit ca8af10 [Kent Yao] kyuubi-common 6e4b8b6 [jhx1008] Try to restore travis Lead-authored-by: Kent Yao <yao@apache.org> Co-authored-by: jhx1008 <jhx1008@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 20:46:06 +08:00
ulysses-you	6a4697019c	[KYUUBI #339] Add JDK8 bug troubleshootint ...  [](https://github.com/yaooqinn/kyuubi/pull/339)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Help user to find the resolution. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #339 from ulysses-you/jdk8-bug. b25a629 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 18:05:37 +08:00
Kent Yao	a4d507a9c6	[KYUUBI #337] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces with dot backtick ...  [](https://github.com/yaooqinn/kyuubi/pull/337)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> add support for '.' and '`' in a namespace part for instance, cases like https://github.com/yaooqinn/kyuubi/pull/337/files#diff-bcd0dfb958c3943a58a9705ac7053374c796449fefd6be7e9f014f9dd14b558fR94 produce the below catalog tree, we should treat `a.b`.c and a.`b.c` differently. ``` /var/folders/01/h81cs4sn3dq2dd_k4j6fhrmc0000gn/T/kyuubi-94c946f7-a64b-4980-9dd1-a63a573244dd ├── a │   └── b.c ├── a.b │   └── c ├── a.b.c ├── a.b`.c ├── db1 │   └── db2 │   └── db3 └── db4 ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #337 from yaooqinn/getschema3. fdb8b19 [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces 0091cfc [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces 5ccee06 [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 09:54:47 +08:00
Kent Yao	c9d44b482c	[KYUUBI #334] GetSchemas supports DSv2 multipart namespaces ...  [](https://github.com/yaooqinn/kyuubi/pull/334)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds support for using Spark DSv2 to get Schemas with multipart namespaces under multiple catalogs. The current works are all based on Apache Iceberg. TODOS: next step we will support get tables operation for multiple catalogs and namespaces. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #334 from yaooqinn/getschema2. 5994170 [Kent Yao] update 188258a [Kent Yao] GetSchemas supports DSv2 multipart namespaces Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-01 20:55:45 +08:00
Honglun	c169c861ff	[KYUUBI #306] SparkProcessBuilder support Windows ...  [](https://github.com/yaooqinn/kyuubi/pull/335)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> SparkProcessBuilder support Windows. Easy to test for developers using Windows. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request  Closes #335 from Honglun/master. 00eb48e [Honglun] SparkProcessBuilder support Windows Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-01 20:48:29 +08:00
jhx1008	9db0088ae1	[KYUUBI #333] Fix hanging due to disordered service stopping ...  [](https://github.com/yaooqinn/kyuubi/pull/333)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> the service discovery should stop before the embedded zk server. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ```before 2021-01-31 11:26:15.800 ERROR server.KyuubiServer: RECEIVED SIGNAL 15: TERM 2021-01-31 11:26:15.801 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:58707 which had sessionid 0x1000fbbfad70002 2021-01-31 11:26:15.801 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:58490 which had sessionid 0x1000fbbfad70000 2021-01-31 11:26:15.801 INFO server.NIOServerCnxnFactory: NIOServerCnxn factory exited run method 2021-01-31 11:26:15.801 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1000fbbfad70000, likely server has closed socket, closing socket connection and attempting reconnect 2021-01-31 11:26:15.801 INFO server.ZooKeeperServer: shutting down 2021-01-31 11:26:15.801 INFO server.SessionTrackerImpl: Shutting down 2021-01-31 11:26:15.801 INFO server.PrepRequestProcessor: Shutting down 2021-01-31 11:26:15.802 INFO server.KyuubiServer: Service: [ServiceDiscovery] is stopping. 2021-01-31 11:26:15.802 INFO server.SyncRequestProcessor: Shutting down 2021-01-31 11:26:15.802 INFO server.PrepRequestProcessor: PrepRequestProcessor exited loop! 2021-01-31 11:26:15.802 INFO server.SyncRequestProcessor: SyncRequestProcessor exited! 2021-01-31 11:26:15.802 INFO server.FinalRequestProcessor: shutdown of request processor complete 2021-01-31 11:26:15.804 INFO server.EmbeddedZkServer: Service[EmbeddedZkServer] is stopped. 2021-01-31 11:26:15.906 INFO state.ConnectionStateManager: State change: SUSPENDED 2021-01-31 11:26:15.907 INFO client.ServiceDiscovery: Zookeeper client connection state changed to: SUSPENDED 2021-01-31 11:26:17.615 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:17.615 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:18.793 INFO server.SessionTrackerImpl: SessionTrackerImpl exited loop! 2021-01-31 11:26:19.126 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:19.126 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:20.815 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:20.815 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:21.947 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:21.948 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:23.288 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:23.289 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:25.075 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:25.075 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:26.965 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:26.965 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:28.412 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:28.412 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:30.356 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2021-01-31 11:26:30.356 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused 2021-01-31 11:26:31.751 ERROR curator.ConnectionState: Connection timed out for connection string (127.0.0.1:2181) and timeout (15000) / elapsed (15841) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss ``` ```after 2021-01-31 11:49:15.070 ERROR server.KyuubiServer: RECEIVED SIGNAL 15: TERM 2021-01-31 11:49:15.071 INFO server.KyuubiServer: Service: [ServiceDiscovery] is stopping. 2021-01-31 11:49:15.074 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1000fd1db370000 2021-01-31 11:49:15.074 WARN client.ServiceDiscovery: This Kyuubi instance localhost:10009 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes. 2021-01-31 11:49:15.074 INFO client.ServiceDiscovery: Service[ServiceDiscovery] is stopped. 2021-01-31 11:49:15.075 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:63940 which had sessionid 0x1000fd1db370000 2021-01-31 11:49:15.075 INFO zookeeper.ZooKeeper: Session: 0x1000fd1db370000 closed 2021-01-31 11:49:15.076 WARN client.ServiceDiscovery: Service[ServiceDiscovery] is not started(STOPPED) yet. 2021-01-31 11:49:15.076 INFO server.KyuubiServer: Service: [FrontendService] is stopping. 2021-01-31 11:49:15.078 INFO service.FrontendService: FrontendService has stopped 2021-01-31 11:49:15.078 INFO service.FrontendService: Service[FrontendService] is stopped. 2021-01-31 11:49:15.078 INFO server.KyuubiServer: Service: [KyuubiBackendService] is stopping. 2021-01-31 11:49:15.078 INFO server.KyuubiBackendService: Service: [KyuubiSessionManager] is stopping. 2021-01-31 11:49:15.078 INFO session.KyuubiSessionManager: Service: [KyuubiOperationManager] is stopping. 2021-01-31 11:49:15.078 INFO operation.KyuubiOperationManager: Service[KyuubiOperationManager] is stopped. 2021-01-31 11:49:15.078 INFO session.KyuubiSessionManager: Service[KyuubiSessionManager] is stopped. 2021-01-31 11:49:15.079 INFO server.KyuubiBackendService: Service[KyuubiBackendService] is stopped. 2021-01-31 11:49:15.079 INFO server.KyuubiServer: Service: [KinitAuxiliaryService] is stopping. 2021-01-31 11:49:15.079 INFO service.KinitAuxiliaryService: Service[KinitAuxiliaryService] is stopped. 2021-01-31 11:49:15.079 INFO server.KyuubiServer: Service[KyuubiServer] is stopped. 2021-01-31 11:49:15.079 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:64039 which had sessionid 0x1000fd1db370002 2021-01-31 11:49:15.079 INFO server.NIOServerCnxnFactory: NIOServerCnxn factory exited run method 2021-01-31 11:49:15.079 INFO server.ZooKeeperServer: shutting down 2021-01-31 11:49:15.079 INFO server.SessionTrackerImpl: Shutting down 2021-01-31 11:49:15.079 INFO server.PrepRequestProcessor: Shutting down 2021-01-31 11:49:15.080 INFO server.SyncRequestProcessor: Shutting down 2021-01-31 11:49:15.080 INFO server.PrepRequestProcessor: PrepRequestProcessor exited loop! 2021-01-31 11:49:15.080 INFO server.SyncRequestProcessor: SyncRequestProcessor exited! 2021-01-31 11:49:15.080 INFO server.FinalRequestProcessor: shutdown of request processor complete 2021-01-31 11:49:15.081 INFO server.EmbeddedZkServer: Service[EmbeddedZkServer] is stopped. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #333 from yaooqinn/hang. 945e1ba [jhx1008] Fix hanging due to disordered service stopping Authored-by: jhx1008 <jhx1008@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-31 13:30:36 +08:00
Zac Zhou	b77ff07e6f	[KYUUBI #331] Try to fix some minor issues in README. ...  [](https://github.com/yaooqinn/kyuubi/pull/332)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #332 from yuanzac/master_zq. d0ddf88 [Zac Zhou] Try to fix some minor issues in README. Authored-by: Zac Zhou <“zhouquan@apache.org”> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-31 11:31:44 +08:00