Commit Graph

758 Commits

Author SHA1 Message Date
Cheng Pan
b81f53d6c2
[KYUUBI #441] [BUILD] Downgrade scala-maven-plugin to 4.3.0
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #441](https://badgen.net/badge/Preview/Closes%20%23441/blue)](https://github.com/yaooqinn/kyuubi/pull/441) ![2](https://badgen.net/badge/%2B/2/red) ![1](https://badgen.net/badge/-/1/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
There are several known issues with `scala-maven-plugin:4.4.0` about `zinc`.

On CentOS 7, with Oracle JDK 8 installed via RPM, `/bin/java` links to `jdk/jre/bin/java` but `/bin/javac` links to `jdk/bin/javac`
```
[opsbd-ck-uat-30007-vm ~]$ ll -l /bin/java
lrwxrwxrwx 1 root root 22 Mar 12 14:03 /bin/java -> /etc/alternatives/java
[opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/java
lrwxrwxrwx 1 root root 41 Mar 12 14:03 /etc/alternatives/java -> /usr/java/jdk1.8.0_251-amd64/jre/bin/java
[opsbd-ck-uat-30007-vm ~]$ ll -l /bin/javac
lrwxrwxrwx 1 root root 23 Mar 12 14:03 /bin/javac -> /etc/alternatives/javac
[opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/javac
lrwxrwxrwx 1 root root 38 Mar 12 14:03 /etc/alternatives/javac -> /usr/java/jdk1.8.0_251-amd64/bin/javac
```
when run `build/dist --tgz` cause
```
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile (scala-test-compile-first) on project kyuubi-common: wrap: java.io.IOException: Cannot run program "/usr/java/jdk1.8.0_221-amd64/jre/bin/javac" (in directory "/data/ansible/ci/kyuubi"): error=2, No such file or directory -> [Help 1]
```
And there is another issue reported by Apache Spark: https://github.com/apache/spark/pull/31031

Please also merge it into branch-1.1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #441 from pan3793/build.

6f6c34d [Cheng Pan] add comments
312b2c5 [Cheng Pan] [BUILD] Downgrade scala-maven-plugin to 4.3.0

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 794a75395f)
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-16 22:59:34 +08:00
ulysses-you
6afacdcdb8 release v1.1.0 2021-03-10 15:30:47 +08:00
MyLanPangzi
46725901c6 [KYUUBI #413] [MINOR]: remove unused class import
![MyLanPangzi](https://badgen.net/badge/Hello/MyLanPangzi/green) [![Closes #413](https://badgen.net/badge/Preview/Closes%20%23413/blue)](https://github.com/yaooqinn/kyuubi/pull/413) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

### _Why are the changes needed?_
remove unused class import

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #413 from MyLanPangzi/master.

dc6f65a [hiscat] [MINOR]: remove unused class import

Lead-authored-by: MyLanPangzi <1251723871@qq.com>
Co-authored-by: hiscat <46845236+MyLanPangzi@users.noreply.github.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
(cherry picked from commit 8f44d0b4cf)
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-10 11:39:29 +08:00
ulysses-you
5e551bdd71 [KYUUBI #408] Fix engine log does not be overwrite
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #408](https://badgen.net/badge/Preview/Closes%20%23408/blue)](https://github.com/yaooqinn/kyuubi/pull/408) ![40](https://badgen.net/badge/%2B/40/red) ![7](https://badgen.net/badge/-/7/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Bug fix, otherwise the engine log will always increase size with append mode.

### _How was this patch tested?_
Add new test.

Closes #408 from ulysses-you/fix-engine-log-overwrite.

7e4fd1d [ulysses-you] simply code
b8d92ee [ulysses-you] config
821891c [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
(cherry picked from commit 57ed76f48d)
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-09 16:29:54 +08:00
ulysses-you
7107666f1b [KYUUBI #407] Auto update release note
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #407](https://badgen.net/badge/Preview/Closes%20%23407/blue)](https://github.com/yaooqinn/kyuubi/pull/407) ![5](https://badgen.net/badge/%2B/5/red) ![0](https://badgen.net/badge/-/0/green) ![6](https://badgen.net/badge/commits/6/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_

Closes #407 from ulysses-you/auto-update-release-note.

4dc09ae [ulysses-you] Release Changelog Builder
cfbe23c [ulysses-you] remove
8fcb56c [ulysses-you] changelog
28e8618 [ulysses-you] release
e953874 [ulysses-you] config
d792eea [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
(cherry picked from commit e971732e1d)
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-09 10:39:41 +08:00
Kent Yao
48e359b4a7
[HOTFIX] User hadoop2.7 as default for spark built in 2021-03-09 09:51:42 +08:00
Kent Yao
49094f7768
[KYUUBI #406] Create GitHub Release only once
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #406](https://badgen.net/badge/Preview/Closes%20%23406/blue)](https://github.com/yaooqinn/kyuubi/pull/406) ![10](https://badgen.net/badge/%2B/10/red) ![10](https://badgen.net/badge/-/10/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #406 from yaooqinn/release.

2fc8754 [Kent Yao] Create GitHub Release only once

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 016277d183)
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-08 20:11:19 +08:00
ulysses-you
03b7112eee [KYUUBI #403] FIx flaky test in ConnectionLevelSparkEngineSuite
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #403](https://badgen.net/badge/Preview/Closes%20%23403/blue)](https://github.com/yaooqinn/kyuubi/pull/403) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![4](https://badgen.net/badge/commits/4/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
`ignore` -> `test`

Closes #403 from ulysses-you/flaky-test2.

d8aa6c7 [ulysses-you] empty
fd38e9c [ulysses-you] empty
2e5a7ee [ulysses-you] empty
444a526 [ulysses-you] fix

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-08 11:22:19 +08:00
ulysses-you
9cb507d3d1 [KYUUBI #395] [WIP] FIx flaky test in ConnectionLevelSparkEngineSuite
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #395](https://badgen.net/badge/Preview/Closes%20%23395/blue)](https://github.com/yaooqinn/kyuubi/pull/395) ![2](https://badgen.net/badge/%2B/2/red) ![1](https://badgen.net/badge/-/1/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_

Closes #395 from ulysses-you/flaky-test.

e8bdd4d [ulysses-you] empty
08febfc [ulysses-you] empty
c3805d1 [ulysses-you] empty
702ac2c [ulysses-you] empty
a544ef0 [ulysses-you] empty
0bd8878 [ulysses-you] empty
dbd53e4 [ulysses-you] fix
f4be895 [ulysses-you] empty
195933a [ulysses-you] empty
cb69e0d [ulysses-you] wip

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-05 17:14:07 +08:00
Kent Yao
352b4eb99e [KYUUBI #400] Implement a simpler toHiveString to convert spark row
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #400](https://badgen.net/badge/Preview/Closes%20%23400/blue)](https://github.com/yaooqinn/kyuubi/pull/400) ![192](https://badgen.net/badge/%2B/192/red) ![62](https://badgen.net/badge/-/62/green) ![9](https://badgen.net/badge/commits/9/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Hive compatibility is essential! This PR tries to restore the previous behavior.

1. Not use Spark's internal class HiveResult to convert spark catalyst type-values to JDBC results, the interval APIs are not APIs and changed unexpectedly. It involves a lot of overhead. (For this reason, pan3793 and I propose a Casting way to solve it.)

2. Unfortunately, the casting way breaks things like #397. So this is a reverting PR.

3. I checked the spark `HiveResult` carefully, I noticed that the `nested` logic is not needed in Kyuubi as we already matched primitives ahead.
4. Also the date time formatter used in `HiveResult` is excessively packaged with massively complicated logic inside spark. It's not clear and simple enough to call.
5. (tiny breaking change)the trailing spaces of a timestamp in milliseconds may not be trimmed. But this is a correctness issue and acceptable.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #400 from yaooqinn/397.

388c86d [Kent Yao] nit
bbf458b [Kent Yao] time stamp formatter
e5a7436 [Kent Yao] more tests
83f3142 [Kent Yao] timezone
38f5a76 [Kent Yao] Merge branch 'master' into 397
563d69c [Kent Yao] mute flacky test
9d834b0 [Kent Yao] Merge branch 'master' into 397
e8e05b5 [Kent Yao] nit
2a2d47b [Kent Yao] Implement a simpler toHiveString to convert spark row

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-05 17:12:58 +08:00
Cheng Pan
8e83245d7a [KYUUBI #402] Remove hadoop suffix from tarball name when dist —without-spark
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #402](https://badgen.net/badge/Preview/Closes%20%23402/blue)](https://github.com/yaooqinn/kyuubi/pull/402) ![3](https://badgen.net/badge/%2B/3/red) ![4](https://badgen.net/badge/-/4/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

`build/dist --tgz --spark-provided`

previous:
```
kyuubi-1.1.0-SNAPSHOT-bin-without-spark-hadoop3.2.tar.gz
```

after patch:
```
kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz
```

Closes #402 from pan3793/build.

a534426 [Cheng Pan] update release workflow
1629489 [Cheng Pan] Remove hadoop suffix from tarball name when dist —without-spark

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-05 13:49:34 +08:00
Cheng Pan
562c45865f
[KYUUBI #398][FOLLOWUP] exclude unused jars
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #401](https://badgen.net/badge/Preview/Closes%20%23401/blue)](https://github.com/yaooqinn/kyuubi/pull/401) ![47](https://badgen.net/badge/%2B/47/red) ![155](https://badgen.net/badge/-/155/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Current kyuubi server dist jars
```
➜  kyuubi git:(KYUUBI-398-followup) ll dist/jars
total 122064
-rw-r--r--  1 chengpan  staff   346K Mar  5 01:00 commons-codec-1.15.jar
-rw-r--r--  1 chengpan  staff   511K Mar  5 01:00 commons-lang3-3.10.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-client-2.12.0.jar
-rw-r--r--  1 chengpan  staff   197K Mar  5 01:00 curator-framework-2.12.0.jar
-rw-r--r--  1 chengpan  staff   277K Mar  5 01:00 curator-recipes-2.12.0.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-test-2.12.0.jar
-rw-r--r--  1 chengpan  staff   4.5K Mar  5 01:00 failureaccess-1.0.1.jar
-rw-r--r--  1 chengpan  staff   2.7M Mar  5 01:00 guava-30.1-jre.jar
-rw-r--r--  1 chengpan  staff    17M Mar  5 01:00 hadoop-client-api-3.2.2.jar
-rw-r--r--  1 chengpan  staff    22M Mar  5 01:00 hadoop-client-runtime-3.2.2.jar
-rw-r--r--  1 chengpan  staff   1.5M Mar  5 01:00 hive-service-rpc-2.3.7.jar
-rw-r--r--  1 chengpan  staff   1.4M Mar  5 01:00 htrace-core4-4.1.0-incubating.jar
-rw-r--r--  1 chengpan  staff   697K Mar  5 01:00 javassist-3.18.1-GA.jar
-rw-r--r--  1 chengpan  staff    16K Mar  5 01:00 jcl-over-slf4j-1.7.30.jar
-rw-r--r--  1 chengpan  staff   325K Mar  5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    34K Mar  5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    81K Mar  5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff   306K Mar  5 01:00 libfb303-0.9.3.jar
-rw-r--r--  1 chengpan  staff   229K Mar  5 01:00 libthrift-0.9.3.jar
-rw-r--r--  1 chengpan  staff   478K Mar  5 01:00 log4j-1.2.17.jar
-rw-r--r--  1 chengpan  staff   5.0M Mar  5 01:00 scala-library-2.12.10.jar
-rw-r--r--  1 chengpan  staff    41K Mar  5 01:00 slf4j-api-1.7.30.jar
-rw-r--r--  1 chengpan  staff    12K Mar  5 01:00 slf4j-log4j12-1.7.30.jar
-rw-r--r--  1 chengpan  staff   890K Mar  5 01:00 zookeeper-3.4.14.jar
```

Closes #401 from pan3793/KYUUBI-398-followup.

c96586b [Cheng Pan] reorder deps
207f0b1 [Cheng Pan] restore docs
0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-05 08:57:23 +08:00
Cheng Pan
4abc3ff6a0
[KYUUBI #398] Move to hadoop shaded client
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #399](https://badgen.net/badge/Preview/Closes%20%23399/blue)](https://github.com/yaooqinn/kyuubi/pull/399) ![101](https://badgen.net/badge/%2B/101/red) ![41](https://badgen.net/badge/-/41/green) ![5](https://badgen.net/badge/commits/5/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #398

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

manual test release workflow
```
➜  kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r--   1 chengpan  staff   265M Mar  4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   273M Mar  4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```

Closes #399 from pan3793/KYUUBI-398.

a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 23:33:31 +08:00
Cheng Pan
8dd54a29b3
[KYUUBI #383] Support beeline --hivevar --hiveconf
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #394](https://badgen.net/badge/Preview/Closes%20%23394/blue)](https://github.com/yaooqinn/kyuubi/pull/394) ![169](https://badgen.net/badge/%2B/169/red) ![13](https://badgen.net/badge/-/13/green) ![2](https://badgen.net/badge/commits/2/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #383

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #394 from pan3793/KYUUBI-383.

2810b58 [Cheng Pan] [KYUUBI #383] remove regex match, add server side test cases
3d8b1e0 [Cheng Pan] [KYUUBI #383] Support beeline --hivevar --hiveconf

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 21:28:22 +08:00
Kent Yao
874c71c6fe
[KYUUBI #396] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #396](https://badgen.net/badge/Preview/Closes%20%23396/blue)](https://github.com/yaooqinn/kyuubi/pull/396) ![10](https://badgen.net/badge/%2B/10/red) ![4](https://badgen.net/badge/-/4/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

new ci rule to check Backward Compatibility

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #396 from yaooqinn/bc.

ae55ac9 [Kent Yao] ga
bf854bd [Kent Yao] Add Backward Compatibility Checker for Compile with Spark 3.0 but Run on 3.1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 20:57:58 +08:00
Kent Yao
25d2d587b0
[KYUUBI #393] Add Support for Hadoop 3.2
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #393](https://badgen.net/badge/Preview/Closes%20%23393/blue)](https://github.com/yaooqinn/kyuubi/pull/393) ![42](https://badgen.net/badge/%2B/42/red) ![41](https://badgen.net/badge/-/41/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR Add Support for Hadoop 3.2 Profile to work with spark3.1.1

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #393 from yaooqinn/guava2.

1f56082 [Kent Yao] nit
c8c576c [Kent Yao] nit
ccdfd03 [Kent Yao] Add Support for Hadoop 3.2

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 18:25:11 +08:00
Kent Yao
2e3315867b
[KYUUBI #298]Add Support for Apache Spark 3.1.1
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #366](https://badgen.net/badge/Preview/Closes%20%23366/blue)](https://github.com/yaooqinn/kyuubi/pull/366) ![359](https://badgen.net/badge/%2B/359/red) ![58](https://badgen.net/badge/-/58/green) ![17](https://badgen.net/badge/commits/17/yellow) ![Feature](https://badgen.net/badge/Label/Feature/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved.

In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7`

TODOS:
- Add Hadoop 3.2 support with some dependency issues
- Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release
- Using spark-3.1.1 as default
- Restore HiveResult string?

fix #298
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #366 from yaooqinn/spark3.1.

28ddf93 [Kent Yao] ga
37cb49e [Kent Yao] Merge branch 'master' into spark3.1
7b332ec [Kent Yao] typo
b6b33d5 [Kent Yao] set up ga
00005c0 [Kent Yao] set up travis
2c09fe4 [Kent Yao] test fix
68c7edb [Kent Yao] deps
2c65e98 [Kent Yao] deps
236c7e9 [Kent Yao] nit
8c8a783 [Kent Yao] Merge branch 'master' into spark3.1
46c318a [Kent Yao] nit
32f6511 [Kent Yao] Merge branch 'master' into spark3.1
27c836b [Kent Yao] Merge branch 'master' into spark3.1
5574a8e [Kent Yao] ga
d2ec2d5 [Kent Yao] [WIP]Spark3.1
49313cc [Kent Yao] [WIP]Travis
ccb1a97 [Kent Yao] [WIP]Spark3.1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 13:30:05 +08:00
ulysses-you
aa878e4d48
[KYUUBI #386] Clean up discovery service before engine stop in connection share level
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #387](https://badgen.net/badge/Preview/Closes%20%23387/blue)](https://github.com/yaooqinn/kyuubi/pull/387) ![330](https://badgen.net/badge/%2B/330/red) ![52](https://badgen.net/badge/-/52/green) ![18](https://badgen.net/badge/commits/18/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time.

### _How was this patch tested?_
Add some new suites in spark engine moudle with zk emmbedded.

Closes #387 from ulysses-you/issue-386.

ccb1112 [ulysses-you] move method
1558dd9 [ulysses-you] test
6cd7e57 [ulysses-you] nit
f8559d2 [ulysses-you] check level at engine discovery
eedfaee [ulysses-you] split ServiceDiscovery to server and engine
8b20e6c [ulysses-you] timeout
0cf524c [ulysses-you] remove unnecessary test
41d36f9 [ulysses-you] move exception
eaaa12d [ulysses-you] avoid stop twice
b801c14 [ulysses-you] fix test
db39960 [ulysses-you] add start check
2d1f6dd [ulysses-you] add state check
e7374aa [ulysses-you] nit
ddf383c [ulysses-you] fi
ee088be [ulysses-you] nit
3c2013b [ulysses-you] nit
73b386f [ulysses-you] improve conf of test
2185f49 [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 11:31:41 +08:00
Kent Yao
8e8679e92b
[KYUUBI #392] Remove netty dependency
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #392](https://badgen.net/badge/Preview/Closes%20%23392/blue)](https://github.com/yaooqinn/kyuubi/pull/392) ![0](https://badgen.net/badge/%2B/0/red) ![23](https://badgen.net/badge/-/23/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

netty is unnecessary

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #392 from yaooqinn/netty.

824d84e [Kent Yao] nit
402317d [Kent Yao] Remove netty dependency

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:23:38 +08:00
Kent Yao
d10b17d87f
[KYUUBI #391] Bump up guava to 14.0.1
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #391](https://badgen.net/badge/Preview/Closes%20%23391/blue)](https://github.com/yaooqinn/kyuubi/pull/391) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

align with spark master

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #391 from yaooqinn/guava.

164501b [Kent Yao] Bump up guava to 14.0.1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:05:05 +08:00
Cheng Pan
e49ff88305
[KYUUBI #390] Replace abbr in doc
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #390](https://badgen.net/badge/Preview/Closes%20%23390/blue)](https://github.com/yaooqinn/kyuubi/pull/390) ![20](https://badgen.net/badge/%2B/20/red) ![20](https://badgen.net/badge/-/20/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #390 from pan3793/doc.

4b7cdf2 [Cheng Pan] replace abbr in doc

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:04:32 +08:00
Kent Yao
f83faa8356 [KYUUBI #389] Bump up curator to 2.12.0
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #389](https://badgen.net/badge/Preview/Closes%20%23389/blue)](https://github.com/yaooqinn/kyuubi/pull/389) ![1](https://badgen.net/badge/%2B/1/red) ![28](https://badgen.net/badge/-/28/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

2.12.0 is the last update for curator-test to support zk v3.4.x

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #389 from yaooqinn/curator.

0b12323 [Kent Yao] Bump up curator to 2.12.0

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-03 19:37:59 +08:00
Kent Yao
3d0f050034
[KYUUBI #388] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #388](https://badgen.net/badge/Preview/Closes%20%23388/blue)](https://github.com/yaooqinn/kyuubi/pull/388) ![298](https://badgen.net/badge/%2B/298/red) ![28](https://badgen.net/badge/-/28/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

User guide

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #388 from yaooqinn/aqe.

dc2833f [Kent Yao] How To Use Spark Adaptive Query Execution (AQE) in Kyuubi

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 13:55:52 +08:00
ulysses-you
4eed2ee0a7 [KYUUBI #385] Improve pass config between kyuubi and spark engine
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #385](https://badgen.net/badge/Preview/Closes%20%23385/blue)](https://github.com/yaooqinn/kyuubi/pull/385) ![13](https://badgen.net/badge/%2B/13/red) ![20](https://badgen.net/badge/-/20/green) ![8](https://badgen.net/badge/commits/8/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Make test more flexible and  readable.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #385 from ulysses-you/cleanup-test.

c5d2034 [ulysses-you] readable
fbc0f16 [ulysses-you] readable
b1008b7 [ulysses-you] fix
37f5739 [ulysses-you] comment
fdf52a7 [ulysses-you] add config method
c58dfdb [ulysses-you] fix tes
a1e8517 [ulysses-you] improve pass config
1378131 [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-03 10:01:54 +08:00
ulysses-you
7f2a24bd9f [KYUUBI #381] Release session if shared level is CONNECTION
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #382](https://badgen.net/badge/Preview/Closes%20%23382/blue)](https://github.com/yaooqinn/kyuubi/pull/382) ![85](https://badgen.net/badge/%2B/85/red) ![1](https://badgen.net/badge/-/1/green) ![20](https://badgen.net/badge/commits/20/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
In CONNECTION level, it's better to release session if the connection is closed.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #382 from ulysses-you/KYUUBI-381.

cddba6c [ulysses-you] type
a3fd691 [ulysses-you] increase engine init timeout
5030553 [ulysses-you] reload system properties
7a2e72f [ulysses-you] clean code
1034409 [ulysses-you] stopped
d2a001a [ulysses-you] address comment
4eb62de [ulysses-you] 60s
64d0f16 [ulysses-you] fix test
4f10d34 [ulysses-you] sleep
62dc9f9 [ulysses-you] add new stop method
77be261 [ulysses-you] remove config
fd9b73c [ulysses-you] add time sleep
209a59e [ulysses-you] fix test
88ed744 [ulysses-you] import
90015e9 [ulysses-you] simply
311fce3 [ulysses-you] fix test
7af3f72 [ulysses-you] fix test
e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381
f77ddbe [ulysses-you] nit
2173222 [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-02 11:44:32 +08:00
Cheng Pan
08dbe24d28
[KYUUBI #380] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #380](https://badgen.net/badge/Preview/Closes%20%23380/blue)](https://github.com/yaooqinn/kyuubi/pull/380) ![4](https://badgen.net/badge/%2B/4/red) ![2](https://badgen.net/badge/-/2/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This is a followup of #377, change list:

- change the spark version suffix in default distribution bundle name from `spark-{major}.{minor}.{patch}` to `spark-{major}.{minor}`
- release distribution bundles both with spark and without spark

Consider Kyuubi 1.1.x has not released, the available distribution still in old style, so now just keep the doc as is.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

- [x] Manual test

`build/dist --tgz`
```
➜  kyuubi git:(master) ✗ ls | grep tar.gz
kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.tar.gz
```

Closes #380 from pan3793/build.

36fa006 [Cheng Pan] [FOLLOWUP #377] change default distribution bundle name, and distribute without spark bundle

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-28 23:05:36 +08:00
Kent Yao
0f62568171
[KYUUBI #379] Improve engine log capture
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #379](https://badgen.net/badge/Preview/Closes%20%23379/blue)](https://github.com/yaooqinn/kyuubi/pull/379) ![143](https://badgen.net/badge/%2B/143/red) ![121](https://badgen.net/badge/-/121/green) ![5](https://badgen.net/badge/commits/5/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead
2. update `error` immediately when detecting errors at the very first line ASAP
3. Using `Exception:` instead of `Exception` to search each line to filter better
4. add the engine log address in the exception message in  case of error bootstraping

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #379 from yaooqinn/logsize.

7cf58da [Kent Yao] mute in console
b58a3a2 [Kent Yao] update
fbc4a0b [Kent Yao] update
0815f77 [Kent Yao] log
93f5f2f [Kent Yao] log

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-26 17:31:44 +08:00
Cheng Pan
2f8df2355d [KYUUBI #377] build/dist support --spark-provided
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #378](https://badgen.net/badge/Preview/Closes%20%23378/blue)](https://github.com/yaooqinn/kyuubi/pull/378) ![55](https://badgen.net/badge/%2B/55/red) ![13](https://badgen.net/badge/-/13/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #377

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

- [x] Manual test

`build/dist --spark-provided`

```
(kyuubi) ➜  kyuubi git:(master) ✗ tree dist
dist
├── LICENSE
├── RELEASE
├── bin
│   ├── kyuubi
│   ├── kyuubi-logo
│   └── load-kyuubi-env.sh
├── conf
│   ├── kyuubi-defaults.conf.template
│   ├── kyuubi-env.sh.template
│   └── log4j.properties.template
├── externals
│   └── engines
│       └── spark
│           └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar
├── jars
│   ├── apacheds-i18n-2.0.0-M15.jar
│   ├── apacheds-kerberos-codec-2.0.0-M15.jar
│   ├── api-asn1-api-1.0.0-M20.jar
│   ├── api-util-1.0.0-M20.jar
│   ├── audience-annotations-0.5.0.jar
│   ├── commons-beanutils-1.7.0.jar
│   ├── commons-beanutils-core-1.8.0.jar
│   ├── commons-codec-1.4.jar
│   ├── commons-collections-3.2.2.jar
│   ├── commons-configuration-1.6.jar
│   ├── commons-digester-1.8.jar
│   ├── commons-io-2.4.jar
│   ├── commons-lang-2.6.jar
│   ├── commons-lang3-3.10.jar
│   ├── commons-math-2.2.jar
│   ├── curator-client-2.7.1.jar
│   ├── curator-framework-2.7.1.jar
│   ├── curator-recipes-2.7.1.jar
│   ├── curator-test-2.7.1.jar
│   ├── guava-11.0.2.jar
│   ├── hadoop-annotations-2.7.4.jar
│   ├── hadoop-auth-2.7.4.jar
│   ├── hadoop-common-2.7.4.jar
│   ├── hive-service-rpc-2.3.7.jar
│   ├── httpclient-4.5.6.jar
│   ├── httpcore-4.4.12.jar
│   ├── javassist-3.18.1-GA.jar
│   ├── jcl-over-slf4j-1.7.30.jar
│   ├── jsr305-3.0.2.jar
│   ├── kyuubi-common-1.1.0-SNAPSHOT.jar
│   ├── kyuubi-ha-1.1.0-SNAPSHOT.jar
│   ├── kyuubi-main-1.1.0-SNAPSHOT.jar
│   ├── libfb303-0.9.3.jar
│   ├── libthrift-0.9.3.jar
│   ├── log4j-1.2.17.jar
│   ├── netty-3.7.0.Final.jar
│   ├── scala-library-2.12.10.jar
│   ├── slf4j-api-1.7.30.jar
│   ├── slf4j-log4j12-1.7.30.jar
│   ├── spotbugs-annotations-3.1.9.jar
│   └── zookeeper-3.4.14.jar
├── logs
├── pid
└── work
```

Bundle size
```
(kyuubi) ➜  kyuubi git:(KYUUBI-377) ll | grep tar.gz
-rw-r--r--   1 chengpan  staff   230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz
-rw-r--r--   1 chengpan  staff    20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz
```

Closes #378 from pan3793/KYUUBI-377.

442569e [Cheng Pan] add profile spark-provided for skip download spark binary
0d8da24 [Cheng Pan] update doc
5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-02-26 15:00:42 +08:00
Kent Yao
968ed4f079
[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #376](https://badgen.net/badge/Preview/Closes%20%23376/blue)](https://github.com/yaooqinn/kyuubi/pull/376) ![23](https://badgen.net/badge/%2B/23/red) ![6](https://badgen.net/badge/-/6/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #376 from yaooqinn/hive.

26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-26 00:28:17 +08:00
Kent Yao
0a315f3287
[KYUUBI #375] THRIFT-4858 causes metadata operations being flaky
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #375](https://badgen.net/badge/Preview/Closes%20%23375/blue)](https://github.com/yaooqinn/kyuubi/pull/375) ![19](https://badgen.net/badge/%2B/19/red) ![8](https://badgen.net/badge/-/8/green) ![4](https://badgen.net/badge/commits/4/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fix #368

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #375 from yaooqinn/368.

809c7c3 [Kent Yao] THRIFT-4858 causes metadata operations being flaky
afa4692 [Kent Yao] THRIFT-4858 causes metadata operations being flaky
0693c54 [Kent Yao] Merge branch 'master' into 368
ef7a759 [Kent Yao] THRIFT-4858 causes metadata operations being flaky

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 23:36:11 +08:00
Cheng Pan
da8d2d4dd3
[KYUUBI #374] [DOC] Add Collaborators page
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #374](https://badgen.net/badge/Preview/Closes%20%23374/blue)](https://github.com/yaooqinn/kyuubi/pull/374) ![10](https://badgen.net/badge/%2B/10/red) ![0](https://badgen.net/badge/-/0/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #374 from pan3793/doc.

da57a03 [Cheng Pan] remove unused import js script
7e791ee [Cheng Pan] add Collaborators page

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 21:47:56 +08:00
Cheng Pan
81e36a1389
[KYUUBI #372] Auto upload unit-tests.log
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #373](https://badgen.net/badge/Preview/Closes%20%23373/blue)](https://github.com/yaooqinn/kyuubi/pull/373) ![12](https://badgen.net/badge/%2B/12/red) ![0](https://badgen.net/badge/-/0/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #372

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #373 from pan3793/KYUUBI-372.

a560827 [Cheng Pan] [KYUUBI #372] Auto upload unit-tests.log

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 20:55:06 +08:00
Kent Yao
ce7b122435
[KYUUBI #371] Avoid creating useless operation log for meta operation
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #371](https://badgen.net/badge/Preview/Closes%20%23371/blue)](https://github.com/yaooqinn/kyuubi/pull/371) ![8](https://badgen.net/badge/%2B/8/red) ![11](https://badgen.net/badge/-/11/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_

1. Operation log for metadata is useless for client-side
2. Operation log for metadata dumps nothing useful for the engine side too
3. Operation log will create files that bring overhead for meta operations and may fail operations

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #371 from yaooqinn/oplog.

ca62e74 [Kent Yao] Avoid creating useless operation log for meta operation

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 20:47:48 +08:00
Cheng Pan
c659089bc2
[KYUUBI #360] Correct handle getNextRowSet with FETCH_PRIOR FETCH_FIRST
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #370](https://badgen.net/badge/Preview/Closes%20%23370/blue)](https://github.com/yaooqinn/kyuubi/pull/370) ![332](https://badgen.net/badge/%2B/332/red) ![24](https://badgen.net/badge/-/24/green) ![8](https://badgen.net/badge/commits/8/yellow) ![Feature](https://badgen.net/badge/Label/Feature/) ![Bug](https://badgen.net/badge/Label/Bug/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #360

Ref: https://github.com/apache/spark/pull/30600

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #370 from pan3793/KYUUBI-360.

e79b8cb [Cheng Pan] [KYUUBI #360] comments
0fae3db [Cheng Pan] fix import
3d1b2a6 [Cheng Pan] [KYUUBI #360] fix ut
eda3e59 [Cheng Pan] [KYUUBI #360] fix import
16178d6 [Cheng Pan] [KYUUBI #360] ut
179404d [Cheng Pan] [KYUUBI #360] nit
455af6b [Cheng Pan] [KYUUBI #360] correct getNextRowSet with FETCH_PRIOR FETCH_FIRST
2307f1f [Cheng Pan] [KYUUBI #360] move ThriftUtils to kyuubi-common

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 18:19:02 +08:00
Kent Yao
d94b1c4519
[KYUUBI #369] Using a lightweight zk client for kyuubi session
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #369](https://badgen.net/badge/Preview/Closes%20%23369/blue)](https://github.com/yaooqinn/kyuubi/pull/369) ![60](https://badgen.net/badge/%2B/60/red) ![34](https://badgen.net/badge/-/34/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. limit zkClient to method wide
2. a more lightweight retry policy
3. add some trace log for Kyuubi session
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #369 from yaooqinn/zk.

5cbe14f [Kent Yao] typo
731bae8 [Kent Yao] typo
51365da [Kent Yao] Using a lightweight zk client for kyuubi session

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 16:57:55 +08:00
Kent Yao
4b13f1c551
[KYUUBI #367] Engine terminating checker should start after exposed
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #367](https://badgen.net/badge/Preview/Closes%20%23367/blue)](https://github.com/yaooqinn/kyuubi/pull/367) ![33](https://badgen.net/badge/%2B/33/red) ![33](https://badgen.net/badge/-/33/green) ![4](https://badgen.net/badge/commits/4/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating.

In this PR, we move the checker to the end where the engine is exposed to the servers.

Avoiding errors like:
```
org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset
  at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
  at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258)
  at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142)
  at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50)
  at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48)
  at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43)
  at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #367 from yaooqinn/engine.

f6a979e [Kent Yao] fix test
9cc561c [Kent Yao] address comment
3d15ab2 [Kent Yao] nit
a85d2e0 [Kent Yao] Engine terminating checker should start after exposed

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 15:10:43 +08:00
ulysses-you
2ff851f9bf [KYUUBI #365] Support user specify app name and tag
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #365](https://badgen.net/badge/Preview/Closes%20%23365/blue)](https://github.com/yaooqinn/kyuubi/pull/365) ![13](https://badgen.net/badge/%2B/13/red) ![1](https://badgen.net/badge/-/1/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

After this PR, we will submit spark app with such command.
```
./dist/externals/spark-3.0.1-bin-hadoop2.7/bin/beeline -u "jdbc:hive2://10.242.146.27:10009/;#spark.app.name=test"
```
```
2021-02-24 14:40:53.606 INFO session.KyuubiSessionImpl: Launching SQL engine: /Users/cathy/Desktop/code/kyuubi/dist/externals/spark-3.0.1-bin-hadoop2.7/bin/spark-submit \
	--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
	--conf spark.app.name=test \
	--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
	--conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_USER/anonymous \
	--conf spark.kyuubi.ha.zookeeper.quorum=127.0.0.1:2181 \
	--conf spark.yarn.tags=KYUUBI \
	--conf spark.kyuubi.ha.zookeeper.acl.enabled=false \
```

Closes #365 from ulysses-you/configs.

fa80eb6 [ulysses-you] append
4000ae3 [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-02-24 15:49:20 +08:00
Cheng Pan
399b30bb95
[KYUUBI #363] exclude hive-shims unused dependencies
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #364](https://badgen.net/badge/Preview/Closes%20%23364/blue)](https://github.com/yaooqinn/kyuubi/pull/364) ![48](https://badgen.net/badge/%2B/48/red) ![0](https://badgen.net/badge/-/0/green) ![1](https://badgen.net/badge/commits/1/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix #363

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #364 from pan3793/KYUUBI-363.

9b2c8c7 [Cheng Pan] [KYUUBI #363] exclude hive-shims unused dependencies

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-24 11:38:31 +08:00
Kent Yao
28db93cc75
[KYUUBI #361] Remove Hive Suffix for built in
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #361](https://badgen.net/badge/Preview/Closes%20%23361/blue)](https://github.com/yaooqinn/kyuubi/pull/361) ![5](https://badgen.net/badge/%2B/5/red) ![11](https://badgen.net/badge/-/11/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

the builtin hive1.2 is no need for builtin anymore

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #361 from yaooqinn/hivesuffix.

99d49fc [Kent Yao] Remove Hive Suffix for built in

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-24 00:13:20 +08:00
Kent Yao
ee50890b00
[KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #358](https://badgen.net/badge/Preview/Closes%20%23358/blue)](https://github.com/yaooqinn/kyuubi/pull/358) ![351](https://badgen.net/badge/%2B/351/red) ![223](https://badgen.net/badge/-/223/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
close #310
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #358 from yaooqinn/310.

8cb30a4 [Kent Yao] sql wildcards to java regex
34d2c3a [Kent Yao] Merge branch 'master' into 310
d332be5 [Kent Yao] [KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-23 20:43:02 +08:00
Kent Yao
54c306b967
[KYUUBI #356] Fix missing default versions
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #359](https://badgen.net/badge/Preview/Closes%20%23359/blue)](https://github.com/yaooqinn/kyuubi/pull/359) ![3](https://badgen.net/badge/%2B/3/red) ![2](https://badgen.net/badge/-/2/green) ![1](https://badgen.net/badge/commits/1/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

the default versions may be missing if 'MVN' is absent,

```
SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark--bin-hadoop
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

```
SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark-3.0.2-bin-hadoop2.7
SPARK_CONF_DIR:
HADOOP_CONF_DIR:

Candidate Kyuubi lib /Users/kentyao/kyuubi/jars doesn't exist, searching development environment...
Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/kyuubi/logs/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out
Welcome to
  __  __                           __
 /\ \/\ \                         /\ \      __
 \ \ \/'/'  __  __  __  __  __  __\ \ \____/\_\
  \ \ , <  /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \
   \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \
    \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\
     \/_/\/_/`/___/> \/___/  \/___/   \/___/  \/_/
                /\___/
                \/__/
```

Closes #359 from yaooqinn/356.

2014862 [Kent Yao] [KYUUBI #356] Fix missing default versions

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-23 16:15:49 +08:00
ulysses-you
bacff6c59a [KYUUBI #356] Update built-in Spark version to 3.0.2
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #356](https://badgen.net/badge/Preview/Closes%20%23356/blue)](https://github.com/yaooqinn/kyuubi/pull/356) ![11](https://badgen.net/badge/%2B/11/red) ![3](https://badgen.net/badge/-/3/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spark 3.0.2 has been released now.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #356 from ulysses-you/spark-3.0.2.

c791e19 [ulysses-you] fix
ad6c564 [ulysses-you] 3.0.2

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-02-20 18:26:26 +08:00
Honglun
681b0d188d
[KYUUBI #347] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned
![Honglun](https://badgen.net/badge/Hello/Honglun/green) [![Closes #354](https://badgen.net/badge/Preview/Closes%20%23354/blue)](https://github.com/yaooqinn/kyuubi/pull/354) ![103](https://badgen.net/badge/%2B/103/red) ![14](https://badgen.net/badge/-/14/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

mvn test -Dtest=none -DwildcardSuites=org.apache.kyuubi.session.SessionManagerSuit

Closes #354 from Honglun/master.

f5c623c [Honglun] Use Eventually
d0fa460 [Honglun] Simplify common code for test cases.
b46dfa7 [Honglun] The 'lastIdleTime' needs to be updated after the last expired Handle has been cleaned,

Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-17 00:50:12 +08:00
Cheng Pan
23e1d61e47
[KYUUBI #350] Avoid using HiveResult
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #351](https://badgen.net/badge/Preview/Closes%20%23351/blue)](https://github.com/yaooqinn/kyuubi/pull/351) ![52](https://badgen.net/badge/%2B/52/red) ![59](https://badgen.net/badge/-/59/green) ![4](https://badgen.net/badge/commits/4/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #350

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #351 from pan3793/KYUUBI-350.

8b9ac46 [Cheng Pan] debug logs
21add73 [Cheng Pan] minor
46ceb54 [Cheng Pan] [KYUUBI-350] in favor cast string in executeStatment
f4e3633 [Cheng Pan] [KYUUBI-350] Avoid using HiveResult in RowSet

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-12 14:00:39 +08:00
Cheng Pan
600b014070
[KYUUBI #312] Support run kyuubi in foreground
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #352](https://badgen.net/badge/Preview/Closes%20%23352/blue)](https://github.com/yaooqinn/kyuubi/pull/352) ![26](https://badgen.net/badge/%2B/26/red) ![3](https://badgen.net/badge/-/3/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Feature](https://badgen.net/badge/Label/Feature/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #312

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

![image](https://user-images.githubusercontent.com/26535726/107225226-81f3cb00-6a53-11eb-8c1e-1f0d5f8275b4.png)

Closes #352 from pan3793/KYUUBI-312.

1a48504 [Cheng Pan] update help
08f589d [Cheng Pan] update doc and help
563f39e [Cheng Pan] [KYUUBI-312] support run kyuubi in foreground

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-08 23:53:06 +08:00
yangrong688
5637e9105e
[KYUUBI #349] Fix typo
![yangrong688](https://badgen.net/badge/Hello/yangrong688/green) [![Closes #349](https://badgen.net/badge/Preview/Closes%20%23349/blue)](https://github.com/yaooqinn/kyuubi/pull/349) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

Remove duplicate word vision.

Closes #349 from yangrong688/master.

d1673b5 [yangrong688] delete duplicate word vision

Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-08 15:43:36 +08:00
Kent Yao
a2818003e6
[KYUUBI #309] GetTables supports DSv2 and keeps its backward compatibility
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #346](https://badgen.net/badge/Preview/Closes%20%23346/blue)](https://github.com/yaooqinn/kyuubi/pull/346) ![262](https://badgen.net/badge/%2B/262/red) ![102](https://badgen.net/badge/-/102/green) ![5](https://badgen.net/badge/commits/5/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

fix #309

get tables through Spark DSv2

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #346 from yaooqinn/gettables.

16dbe4b [Kent Yao] nit
e03b021 [Kent Yao] nit
7f3eee2 [Kent Yao] Merge branch 'master' into gettables
0180cc5 [Kent Yao] add a test
8498152 [Kent Yao] GetTables supports DSv2 and keeps its backward compatibility

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-07 13:45:53 +08:00
Honglun
c1a71b7d00
[KYUUBI #347]Closing expired operations does not require updating 'lastAccessTime'
![Honglun](https://badgen.net/badge/Hello/Honglun/green) [![Closes #348](https://badgen.net/badge/Preview/Closes%20%23348/blue)](https://github.com/yaooqinn/kyuubi/pull/348) ![5](https://badgen.net/badge/%2B/5/red) ![4](https://badgen.net/badge/-/4/green) ![2](https://badgen.net/badge/commits/2/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

… and 'lastTidleTime'

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Method 'session.closeExpiredOperations' will constantly update the values of '_lastAccessTime' and '_lastIdleTime', so the session can't be closed. Method 'closeExpiredOperations' should remove 'withAcquirerease'.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
conf.set(KyuubiConf.SESSION_CHECK_INTERVAL,Duration.ofSeconds(5).toMillis)
conf.set(KyuubiConf.SESSION_TIMEOUT,Duration.ofMinutes(1L).toMillis)
![image](https://user-images.githubusercontent.com/3722800/106875807-0cfe5980-6712-11eb-8a2e-d899d9c96bc1.png)

Closes #348 from Honglun/master.

6cff013 [Honglun] Closing expired operations does not require updating 'lastAccessTime'.
a6330d7 [Honglun] Closing expired operations does not require updating 'lastAccessTime' and 'lastTidleTime'

Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-04 20:56:04 +08:00
Kent Yao
e4d9390a74
typo 2021-02-03 23:39:50 +08:00
ulysses-you
fbf1e3506c
[KYUUBI #345] Catch Throwable instead of Exception
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #345](https://badgen.net/badge/Preview/Closes%20%23345/blue)](https://github.com/yaooqinn/kyuubi/pull/345) ![3](https://badgen.net/badge/%2B/3/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

We might get this error when running spark sql,so it's more reasonable to catch `Throwable`.
```
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #345 from ulysses-you/throwable.

b6ffacd [ulysses-you] throw

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-03 19:31:26 +08:00