apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,500 Commits 23 Branches 0 Tags 65 MiB
4f17f1b1af
Commit Graph

238 Commits

Author SHA1 Message Date
yanghua
d63eed2749
[KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine t… 
…o decouple with kyuubi server

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1711 from yanghua/KYUUBI-1709.

Closes #1709

95466130 [yanghua] refactor code
89f9c572 [yanghua] Reorder module list in root pom file
b664a4bb [yanghua] Rename IT module names
6b7fa88f [yanghua] Rename IT module names
b133e8e9 [yanghua] Reformatted pom indent
38444416 [yanghua] [KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine to decouple with kyuubi server

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2022-01-11 12:45:41 +08:00
Kent Yao
32990b707f
[KYUUBI #1717] Extract rat configurations from pom to file 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Extract rat configurations from pom to file

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1717 from yaooqinn/rat.

Closes #1717

3f6836fe [Kent Yao] Extract rat configuation from pom to file
cb267b37 [Kent Yao] Extract rat configuation from pom to file

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-11 11:52:18 +08:00
Kent Yao
a9d8fb2e7c
[KYUUBI #1712] Bump up maven.version to 3.8.4 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1712 from yaooqinn/mvnv.

Closes #1712

93793468 [Kent Yao] Bump up maven.version to 3.8.4

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-10 19:21:10 +08:00
hongdongdong
e886524c74
[KYUUBI #1641] Add Trino client 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add trino client to communicate with trino cluster.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1642 from hddong/add-trino-runner.

Closes #1641

67c2f1f4 [hongdongdong] fix
3e4f7782 [hongdongdong] use testcontainers
3cd4d944 [hongdongdong] [KYUUBI #1641] Add Trino client

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-07 19:58:41 +08:00
Kent Yao
d1de9ca529
[KYUUBI #1658] Revamp swagger ui dependencies 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR mainly removes the swagger-scala-module which is currently fine to remove. It also introduces transitive deps with
`-jakarta` which could introduce jar conflicts like ,

```scala
Caused by: java.lang.NoClassDefFoundError: jakarta/xml/bind/annotation/XmlElement
	at io.swagger.v3.core.jackson.SwaggerAnnotationIntrospector.hasRequiredMarker(SwaggerAnnotationIntrospector.java:29)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getMetadata(POJOPropertyBuilder.java:229)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1658 from yaooqinn/dep.

Closes #1658

d5ab0765 [Kent Yao] Merge branch 'master' into dep
d1ddd5fd [Kent Yao] Revamp swagger ui dependencies
ecfcc92a [Kent Yao] Revamp swagger ui dependencies
59196b94 [Kent Yao] Revamp swagger ui dependencies
d7ca6a5f [Kent Yao] Revamp swagger ui dependencies
215b1b53 [Kent Yao] Revamp swagger ui dependencies

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-31 09:22:24 +08:00
rayliu
2af105a417 [KYUUBI #1556] Upgrade swagger-ui version from 4.1.0 to 4.1.3 
### _Why are the changes needed?_
Close https://github.com/apache/incubator-kyuubi/issues/1612

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1630 from ldzhjn/1612.

Closes #1556

71fe05bc [rayliu] Upgrade swagger-ui version from 4.1.0 to 4.1.3

Authored-by: rayliu <rayliu@cisco.com>
Signed-off-by: yanghua <yanghua1127@gmail.com>
 2021-12-25 20:50:56 +08:00
yanghua
3673399329
[KYUUBI #1579] Implement basic ability of executing statement in Flink engine 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1603 from yanghua/KYUUBI-1579.

Closes #1579

48db76b3 [Cheng Pan] cleanup
36707516 [Cheng Pan] Address comments
25ca5ae2 [yanghua] reduce code
6f18a4a0 [yanghua] [KYUUBI #1579] Implement basic ability of executing statement

Lead-authored-by: yanghua <yanghua1127@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-22 22:55:43 +08:00
Cheng Pan
d529402168
[KYUUBI #1587] [SUB-TASK][KPIP-2] Bump Flink 1.14.2 
### _Why are the changes needed?_

Bump the latest version of 1.14.x to suppress log4-shell issue

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1587 from pan3793/flink-version.

Closes #1587

9d7ca949 [Cheng Pan] Bump Flink 1.14.2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-20 18:19:30 +08:00
yanghua
01364e589f
[KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code 
…uplicated code

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1584 from yanghua/KYUUBI-1578.

Closes #1578

90c2f89e [yanghua] [KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-17 16:30:33 +08:00
yanghua
4b58b0ab9b
[KYUUBI #1555][FLINK] Bump flink version to 1.14.0 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1573 from yanghua/KYUUBI-1555.

Closes #1555

4f5ce746 [yanghua] [KYUUBI #1555] [SUB-TASK][KPIP-2] Bump flink version to 1.14.1

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-16 16:59:38 +08:00
Cheng Pan
022f52bc7e
[KYUUBI #1460] Bump Hudi 0.10.0 
### _Why are the changes needed?_

https://hudi.apache.org/releases/release-0.10.0

Switch test from Hudi 0.9.0 & Spark 3.0 to Hudi 1.10.0 & Spark 3.1. Drop test of Spark 3.0 because

1. Hudi 1.10.0 claims to support Spark 3.0 but requires to build by self with specific maven profile, the published jar only support Spark 3.1
2. Hudi has lots of transitive deps and there are different in different versions, it's hard to maintain multi hudi versions.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1460 from pan3793/hudi.

Closes #1460

bac633a7 [Cheng Pan] Update comments
0ab8784d [Cheng Pan] Pin hadoop-common
c811e679 [Cheng Pan] Exclude scala-library
63cad3b9 [Cheng Pan] Exclude hudi-aws
a13708f9 [Cheng Pan] Disable Hudi test on Spark 3.0
4b179bae [Cheng Pan] Enable Hudi test in Spark 3.1
79dca54c [Cheng Pan] Bump Hudi 0.10.0
71982890 [Cheng Pan] Add default Pk col

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-16 10:13:42 +08:00
Cheng Pan
77a278707f
[KYUUBI #1559] Bump Jersey 2.34 
### _Why are the changes needed?_

#1561 and align Spark 3.2.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1559 from pan3793/jersey.

Closes #1559

a86d4d9f [Cheng Pan] Bump Jersey 2.34

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-16 10:12:46 +08:00
Cheng Pan
f841da4b4e
[KYUUBI #1558] Bump Jackson 2.12.5 
### _Why are the changes needed?_

Because Spark 3.2, Trino 363, Jersey 2.34 (we use Jersey 2.30 now, will update to 2.34 soon) use Jackson 2.12.x

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1558 from pan3793/jackson.

Closes #1558

7bdd65b0 [Cheng Pan] Bump Jackson 2.12.5

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-15 21:21:01 +08:00
Cheng Pan
6fa1327f4e
[KYUUBI #1560] Align all Jackson libs version 
### _Why are the changes needed?_

Fix missing version of `jackson-module-jaxb-annotations` in `dependencyManagement`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1560 from pan3793/jackson-1.

Closes #1560

49529d47 [Cheng Pan] Update deps
b3610007 [Cheng Pan] fix
67044840 [Cheng Pan] Algin all Jackson libs version

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-15 12:02:10 +08:00
hongdongdong
c1a0ce8e01
[KYUUBI #1515] Add RowSet for trino engine 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add RowSet for trino engine

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1549 from hddong/add-rowset-for-trino.

Closes #1515

cc885130 [hongdongdong] remove timeZone
2fb10934 [hongdongdong] fix toHiveString
9066caff [hongdongdong] rm optionalStart
491b7682 [hongdongdong] Reset
27895ebe [hongdongdong] fix RowSetUtils
7ce4edeb [hongdongdong] fix
34dd01b8 [hongdongdong] move functions to Utils
79e0621a [hongdongdong] fix
2dcd91e5 [hongdongdong] fix
71bdcfd2 [hongdongdong] [KYUUBI #1515] Add RowSet for trino engine

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-14 20:52:20 +08:00
yanghua
53ff56be29
[KYUUBI #1525][FLINK] Implement GetCatalogs operation and do the further initialization 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1529 from yanghua/KYUUBI-1525.

Closes #1525

e166965d [yanghua] Address review suggestion
9929aa29 [yanghua] Remove some comment
eafb200b [yanghua] refactor code
58df34c7 [yanghua] refactor code
675e14e7 [yanghua] pruning unnecessary code
15e97283 [yanghua] pruning unnecessary code
c6d4f549 [yanghua] pruning unnecessary code
b099a455 [yanghua] pruning unnecessary code
316086bb [yanghua] pruning unnecessary code
fe7af8ef [yanghua] pruning unnecessary code
ecf3ccf4 [yanghua] pruning unnecessary code
4c174517 [yanghua] pruning unnecessary code
b9226c3e [yanghua] pruning unnecessary code
fb2f6a62 [yanghua] pruning unnecessary code
21e21a50 [yanghua] pruning unnecessary code
a58076ff [yanghua] pruning unnecessary code
acd24fc5 [yanghua] pruning unnecessary code
333d4275 [yanghua] pruning unnecessary code
7416b3de [yanghua] pruning unnecessary code
84d0db91 [yanghua] pruning unnecessary code and refactor code
a55db41a [yanghua] pruning unnecessary code
b9909308 [yanghua] [KYUUBI #1525] Implement GetCatalogs operation and do the further initialization

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-14 17:40:11 +08:00
Fei Wang
609bb5a878 [KYUUBI #1535] Remove duplicate hive client version property 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Remove duplicate hive.cleint.version property, it should be involved when adding kyuubi-hive-beeline module.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1535 from turboFei/typo.

Closes #1535

56e734fa [Fei Wang] Remove duplicate hive client version property

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2021-12-09 18:37:13 +08:00
hongdongdong
6ea83e040e
[KYUUBI #1511] Initialize kyuubi trino engine module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. Initialize kyuubi trino engine module
2. Add `TrinoThriftBinaryFrontendService`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1514 from hddong/init-trino-engine.

Closes #1511

9dd0e55f [hongdongdong] [KYUUBI #1511] Initialize kyuubi trino engine module

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-07 13:33:35 +08:00
yanghua
339fa7682a
[KYUUBI #1492] Initialize kyuubi flink sql engine module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1497 from yanghua/KYUUBI-1492.

Closes #1492

ba9f221a [yanghua] [KYUUBI #1492] Initialize kyuubi flink sql engine module

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-06 15:45:44 +08:00
Kent Yao
97c3835738
[KYUUBI #1490] Introduce the basic framework for running scala 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Introduce the basic framework for running scala, see #1490 for the detail
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
```
Beeline version 1.5.0-SNAPSHOT by Apache Kyuubi (Incubating)
0: jdbc:hive2://10.242.189.214:10009/> spark.version;
2021-12-03 13:47:07.556 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:07.558 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:07.559 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:07.560 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed
                 application ID: local-1638510289918
                 application web UI: http://10.242.189.214:65027
                 master: local[*]
                 deploy mode: client
                 version: 3.1.2
           Start time: 2021-12-03T13:44:49.313
           User: kent
2021-12-03 13:47:07.562 INFO scheduler.DAGScheduler: Asked to cancel job group 321dc15d-68d0-4f91-9216-1e08f09842df
2021-12-03 13:47:07.565 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.006 seconds
2021-12-03 13:47:07.565 ERROR operation.ExecuteStatement: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Query[08b8b6da-d434-4296-b613-2027e3518441] in ERROR_STATE
2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.009 seconds
Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748) (state=,code=0)
0: jdbc:hive2://10.242.189.214:10009/> set kyuubi.operation.language=scala;
2021-12-03 13:47:11.982 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.983 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.984 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.985 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed
                 application ID: local-1638510289918
                 application web UI: http://10.242.189.214:65027
                 master: local[*]
                 deploy mode: client
                 version: 3.1.2
           Start time: 2021-12-03T13:44:49.313
           User: kent
2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Execute in full collect mode
2021-12-03 13:47:12.006 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds
2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b] in FINISHED_STATE
2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds
+----------------------------+--------+
|            key             | value  |
+----------------------------+--------+
| kyuubi.operation.language  | scala  |
+----------------------------+--------+
1 row selected (0.052 seconds)
0: jdbc:hive2://10.242.189.214:10009/> spark.version;
2021-12-03 13:47:13.685 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:15.541 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:15.543 INFO operation.ExecuteStatement: Query[178ff72d-b870-44d4-99d8-08a0fa0e8efa] in FINISHED_STATE
2021-12-03 13:47:15.544 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: RUNNING_STATE -> FINISHED_STATE, statement: spark.version, time taken: 0.003 seconds
+-----------------------+
|        output         |
+-----------------------+
| res0: String = 3.1.2  |
+-----------------------+
1 row selected (1.871 seconds)
0: jdbc:hive2://10.242.189.214:10009/> spark.sql("select current_date()")
. . . . . . . . . . . . . . . . . . .> ;
2021-12-03 13:47:36.512 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.sql("select current_date()")
2021-12-03 13:47:36.689 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: PENDING_STATE -> RUNNING_STATE, statement: spark.sql("select current_date()")
2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Query[602c2d88-7e8a-4175-9b53-e34adfff007a] in FINISHED_STATE
2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: RUNNING_STATE -> FINISHED_STATE, statement: spark.sql("select current_date()"), time taken: 0.003 seconds
+----------------------------------------------------+
|                       output                       |
+----------------------------------------------------+
| res1: org.apache.spark.sql.DataFrame = [current_date(): date] |
+----------------------------------------------------+
1 row selected (0.187 seconds)
0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3);
Error: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error:
results += spark.range(1, 5, 2, 3)
 <console>:26: error: type mismatch;
 found   : org.apache.spark.sql.Dataset[Long]
 required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
       results += spark.range(1, 5, 2, 3)
                             ^
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
	at org.apache.kyuubi.engine.spark.operation.ExecuteScala.runInternal(ExecuteScala.scala:70)
	at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130)
	at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93)
	at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:62)
	at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:121)
	at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75)
	at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:118)
	at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:61)
	at org.apache.kyuubi.service.ThriftBinaryFrontendService.ExecuteStatement(ThriftBinaryFrontendService.scala:265)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748) (state=,code=0)
0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3).toDF;
2021-12-03 13:48:30.327 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: INITIALIZED_STATE -> PENDING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF
2021-12-03 13:48:31.700 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: PENDING_STATE -> RUNNING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF
2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Query[daccd163-978a-4d66-af85-af466086d1db] in FINISHED_STATE
2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: RUNNING_STATE -> FINISHED_STATE, statement: results += spark.range(1, 5, 2, 3).toDF, time taken: 0.003 seconds
+-----+
| id  |
+-----+
| 1   |
| 3   |
+-----+
2 rows selected (1.387 seconds)
```

** Session level isolated **

![image](https://user-images.githubusercontent.com/8326978/144553902-fc390c30-06de-453b-af3d-cf8def577aa9.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1491 from yaooqinn/scala.

Closes #1490

af4d0a13 [Kent Yao] Merge branch 'master' into scala
2ebdf6a3 [Kent Yao] provided scala dep
d58b2bfe [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala
d256bfde [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala
52090434 [Kent Yao] init

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-06 11:17:59 +08:00
Cheng Pan
5dba80ccfd
[KYUUBI #1486] Enable delta test on Spark 3.2 with delta 1.1.0 
### _Why are the changes needed?_

Delta 1.1.0 is out, compatible with Spark 3.2

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1486 from pan3793/delta.

Closes #1486

8dce5373 [Cheng Pan] Enable delta test on Spark 3.2 with delta 1.1.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-03 13:36:14 +08:00
fwang12
119e7f2c2c [KYUUBI #1441] [BUILD] Bump 1.5.0-SNAPSHOT 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1441 from turboFei/bump_1.5.0.

Closes #1441

0c03c75d [fwang12] [BUILD] Bump 1.5.0-SNAPSHOT

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-23 22:22:26 +08:00
Cheng Pan
6bb5af609c
[KYUUBI #1433] Enable scalafmt check in spotless 
### _Why are the changes needed?_

fail compile if violating scalafmt rules.

```
[ERROR] Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.17.4:check (default) on project kyuubi-common_2.12: The following files had format violations:
[ERROR]     src/test/scala/org/apache/kyuubi/operation/JDBCTestHelper.scala
[ERROR]          -52,7 +52,7
[ERROR]          ··}
[ERROR]
[ERROR]          ··def·withMultipleConnectionJdbcStatement(
[ERROR]         -····tableNames:·String*)(fs:·(Statement·=>·Unit)*):·Unit·=·{
[ERROR]         +······tableNames:·String*)(fs:·(Statement·=>·Unit)*):·Unit·=·{
[ERROR]          ····val·connections·=·fs.map·{·_·=>·DriverManager.getConnection(jdbcUrlWithConf,·user,·password)·}
[ERROR]          ····val·statements·=·connections.map(_.createStatement())
[ERROR]
[ERROR] Run 'mvn spotless:apply' to fix these violations.
```
Can auto-fix by `dev/reformat` or `build/mvn spotless:apply`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1433 from pan3793/scalafmt.

Closes #1433

26188a54 [Cheng Pan] Enable scalafmt check in spotless

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-23 15:23:36 +08:00
fwang12
55a7347b6f
[KYUUBI #1425] Remove hardcode skipTests for maven-surefire-plugin 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

So that we can transfer skipTests variable when build kyuubi binary.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1425 from turboFei/java_tst.

Closes #1425

094904d2 [fwang12] typo
af447e8c [fwang12] prevent hardcode

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-22 09:50:46 +08:00
fwang12
21cc217281 [KYUUBI #1419] Refactor hive client related property names 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Refactor the kyuubi client related property names from `hive.cli` to `hive.client`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1419 from turboFei/refactor_hive_cli.

Closes #1419

de532387 [fwang12] cli -> client

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-19 16:09:43 +08:00
fwang12
e5e49b5b1d
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1414 from turboFei/rebase-kyuubi_beeline.

Closes #1414

4694bbca [fwang12] return engine launch handle anyway
20ca1400 [fwang12] address comments
2999c30a [fwang12] fix ut
4d3abf4f [fwang12] jars.target.dir to target
4659f1f6 [fwang12] add readme
7e74c5c8 [fwang12] integrate beeline and server
969af045 [fwang12] fix beeline shell
90395ade [fwang12] BeeLine version By Apache Kyuubi
4e63695e [fwang12] fix ut
026e41ef [fwang12] refactor properties
a49677af [fwang12] refactor build and beeline script
ca68eba1 [fwang12] use beeline mode property instead of static method
4a1932e7 [fwang12] cli -> beeline
624281e3 [fwang12] nit
d7d27a09 [fwang12] refactor kyuubi cli assembly
9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete
cc025ecf [fwang12] Rebase PR_1407

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-19 15:05:10 +08:00
sunfangbin
e54fabab68
[KYUUBI #1416] Refine jline dependency 
### _Why are the changes needed?_
Follows on #1413.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1416 from murong00/branch-1415.

Closes #1416

05335c45 [sunfangbin] Refine jline dependency

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-19 10:40:40 +08:00
sunfangbin
fb8bf49dbc
[KYUUBI #1413] Improve usage of zookeeper cli 
### _Why are the changes needed?_
This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1413 from murong00/branch-1413.

Closes #1413

2d1ec084 [sunfangbin] Rename shell script to zk-cli
d09f64dc [sunfangbin] Fixes failed ci
ce88357a [sunfangbin] Improve usage of zookeeper cli

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 19:17:11 +08:00
yanghua
2baabf46ca
[KYUUBI #1384] Introduce spotless and google java format for java code 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1403 from yanghua/KYUUBI-1384.

Closes #1384

e4e12088 [yanghua] address review suggestion
17a4cb86 [yanghua] reformat all the java files based on the offical google java code style
10a2ca04 [yanghua] [KYUUBI #1384] Introduce Java code style and formatter

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 10:45:51 +08:00
Fu Chen
a5ef94accc
[KYUUBI #1354] Expose OpenAPI Specifications 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi.
2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui)

```
curl http://localhost:10099/openapi.json | jq .
```

```json
{
  "openapi": "3.0.1",
  "info": {
    "title": "Apache Kyuubi RESTful Api Documentation",
    "description": "Apache Kyuubi RESTful Api Documentation",
    "contact": {
      "name": "Apache Kyuubi Community",
      "url": "https://kyuubi.apache.org/issue_tracking.html",
      "email": "devkyuubi.apache.org"
    },
    "license": {
      "name": "Apache 2.0",
      "url": "https://www.apache.org/licenses/LICENSE-2.0.html"
    },
    "version": "1.4.0"
  },
  "tags": [
    {
      "name": "Session"
    }
  ],
  "paths": {
    "/api/v1/ping": {
      "get": {
        "operationId": "ping",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a session via session handle identifier",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "delete": {
        "tags": [
          "Session"
        ],
        "operationId": "closeSession",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "close a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/execPool/statistic": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "execPoolStatistic",
        "responses": {
          "200": {
            "description": "get some statistic info of sessions",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/info/{infoType}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "getInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "infoType",
            "in": "path",
            "required": true,
            "schema": {
              "type": "integer",
              "format": "int32"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a information detail via session handle identifier and a specific information type",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfoList",
        "responses": {
          "200": {
            "description": "get all the session list hosted in SessionManager",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "post": {
        "tags": [
          "Session"
        ],
        "operationId": "openSession",
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/SessionOpenRequest"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "Open(create) a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/count": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionCount",
        "responses": {
          "200": {
            "description": "get open session count",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/exception": {
      "get": {
        "operationId": "test",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {}
            }
          }
        }
      }
    },
    "/application.wadl/{path}": {
      "get": {
        "operationId": "getExternalGrammar",
        "parameters": [
          {
            "name": "path",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/xml": {}
            }
          }
        }
      }
    },
    "/application.wadl": {
      "get": {
        "operationId": "getWadl",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/vnd.sun.wadl+xml": {},
              "application/xml": {}
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "ExecPoolStatistic": {
        "required": [
          "execPoolSize"
        ],
        "type": "object",
        "properties": {
          "execPoolSize": {
            "type": "integer",
            "format": "int32"
          },
          "execPoolActiveCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "InfoDetail": {
        "required": [
          "infoType"
        ],
        "type": "object",
        "properties": {
          "infoType": {
            "type": "string"
          },
          "infoValue": {
            "type": "string"
          }
        }
      },
      "HandleIdentifier": {
        "required": [
          "publicId"
        ],
        "type": "object",
        "properties": {
          "publicId": {
            "type": "string",
            "format": "uuid"
          },
          "secretId": {
            "type": "string",
            "format": "uuid"
          }
        }
      },
      "SessionHandle": {
        "required": [
          "identifier",
          "protocol"
        ],
        "type": "object",
        "properties": {
          "identifier": {
            "$ref": "#/components/schemas/HandleIdentifier"
          },
          "protocol": {
            "type": "string",
            "enum": [
              "HIVE_CLI_SERVICE_PROTOCOL_V1",
              "HIVE_CLI_SERVICE_PROTOCOL_V2",
              "HIVE_CLI_SERVICE_PROTOCOL_V3",
              "HIVE_CLI_SERVICE_PROTOCOL_V4",
              "HIVE_CLI_SERVICE_PROTOCOL_V5",
              "HIVE_CLI_SERVICE_PROTOCOL_V6",
              "HIVE_CLI_SERVICE_PROTOCOL_V7",
              "HIVE_CLI_SERVICE_PROTOCOL_V8",
              "HIVE_CLI_SERVICE_PROTOCOL_V9",
              "HIVE_CLI_SERVICE_PROTOCOL_V10"
            ]
          }
        }
      },
      "MapStringString": {
        "type": "object",
        "additionalProperties": {
          "type": "string"
        }
      },
      "SessionOpenRequest": {
        "required": [
          "configs",
          "protocolVersion",
          "user"
        ],
        "type": "object",
        "properties": {
          "protocolVersion": {
            "type": "integer",
            "format": "int32"
          },
          "user": {
            "type": "string"
          },
          "password": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SessionOpenCount": {
        "required": [
          "openSessionCount"
        ],
        "type": "object",
        "properties": {
          "openSessionCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "SessionDetail": {
        "required": [
          "configs",
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          },
          "lastAccessTime": {
            "type": "integer",
            "format": "int64"
          },
          "lastIdleTime": {
            "type": "integer",
            "format": "int64"
          },
          "noOperationTime": {
            "type": "integer",
            "format": "int64"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SeqSessionOverview": {
        "type": "array",
        "items": {
          "$ref": "#/components/schemas/SessionOverview"
        }
      },
      "SessionList": {
        "required": [
          "sessionList"
        ],
        "type": "object",
        "properties": {
          "sessionList": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/SessionOverview"
            }
          }
        }
      },
      "SessionOverview": {
        "required": [
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          }
        }
      }
    }
  }
}
```
http://localhost:10099/api/v1/swagger-ui

> Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json

![截屏2021-11-09 上午10 43 49](https://user-images.githubusercontent.com/8537877/141281596-e47f5552-75b9-47d1-8a36-67337cb6eb3a.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1363 from cfmcgrady/kyuubi-1354.

Closes #1354

8895d7fc [Fu Chen] typo
8f5dea9a [Fu Chen] addressed comment
4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON
0d96951e [Fu Chen] RESTful Api -> REST API
be7d02dc [Fu Chen] add LIABILITY-edl.txt
4ff1053e [Fu Chen] update dev/dependencyList
b126f7e6 [Fu Chen] swagger-ui 4.1.0
a35c2f29 [Fu Chen] addressed comment
7bde0534 [Fu Chen] update LICENSE-binary
38397dc2 [Fu Chen] align jackson version
923f3e7e [Fu Chen] update dev/dependencyList
7d5338c8 [Fu Chen] remove kyuubi-server static files
5cb66b33 [Fu Chen] update dev/dependencyList
e8a8c421 [Fu Chen] fix style
5b13f69c [Fu Chen] add license and remove .map files
c04d7a75 [Fu Chen] rebase master
2388dae8 [Fu Chen] fix style
8dcbd3db [Fu Chen] add static resource
4b0d6d0a [Fu Chen] expose OpenAPI

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-17 12:17:58 +08:00
fwang12
ed43391f8c [KYUUBI #1381] Rewrite kyuubi hive jdbc module based on hive branch-2.3 code base 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Hive code base:
8e7f23f34b

commit 8e7f23f34b2ce7328c9d571a13c336f0c8cdecb6 (HEAD, origin/branch-2.3)
Author: Neelesh Srinivas Salian <nssalianusers.noreply.github.com>
Date:   Tue Oct 19 12:50:31 2021 -0700

    HIVE-25616: Hive-24741 backport to 2.3 (Neelesh Srinivas Salian reviewed by Vihang Karajgaonkar)

    Closes (#2730)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1381 from turboFei/kyuubi_hive_jdbc.

Closes #1381

f39f1a1d [fwang12] adreess comments
5612f490 [fwang12] remove kyuubi-hive-jdbc from code cov
f5a6376e [fwang12] do not fail if no tests
4e99ffcb [fwang12] check kyuubi-hive-jdbc code coverage
1c5f2e16 [fwang12] refactor the test classes
d7f9c7d7 [fwang12] copy test classes from hive branch-2.3
a28c077a [fwang12] rename classname in comments
42f7c5ea [fwang12] try to remove hive-jdbc dependency
fcf5953a [fwang12] use kyuubi hive jdbc utils
17402f4d [fwang12] replace licenses
5cbca43d [fwang12] add missing class ignored by .gitignore
b2c19334 [fwang12] revert docs change
bdc860b4 [fwang12] backport kyuubiHiveDriver
04731e72 [fwang12] make some rename
9d57076b [fwang12] remove from org.apache.hive.jdbc to org.apache.kyuubi.jdbc.hive
86813e6b [fwang12] Backport KyuubiDriver
42f748d6 [fwang12] Backport KyuubiDatabaseMetaData.getTables and for KyuubiConnection.getMetaData, it is not needed
f0d2aedb [fwang12] rename hive classes to kyuubi version
3ea8ba73 [fwang12] copy the source code from hive branch-2.3
7db4e5ac [fwang12] remove at first

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-15 15:38:38 +08:00
Cheng Pan
c122a23a0d
[KYUUBI #1348] [DEPS] Add netty and mysql jdbc driver to kyuubi-server module 
### _Why are the changes needed?_

This PR is one part of #1334.

Add netty to `kyuubi-server` module compile scope, and mysql jdbc driver to test scope, it's required for implementing MySQL frontend protocol

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1348 from pan3793/mysql-fe-deps.

Closes #1348

288ad9c0 [Cheng Pan] [DEPS] Add netty to kyuubi-server module

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 20:31:09 +08:00
Cheng Pan
0b43a8bb78
[KYUUBI #1338] Bump Iceberg 0.12.1 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Test Iceberg 0.12.1 RC0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1338 from pan3793/iceberg-0.12.1.

Closes #1338

d2dd4c1f [Cheng Pan] Remove staging repo
a16103e8 [Cheng Pan] Test Iceberg 0.12.1 RC0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 12:28:36 +08:00
yanghua
aad575ce3e
[KYUUBI #1329] Support download Flink binary package in kyuubi-download module 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1339 from yanghua/KYUUBI-1329.

Closes #1329

895815fb [yanghua] address review suggestion
eddf0605 [yanghua] [KYUUBI #1329] Support download Flink binary package in kyuubi-download module

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-05 18:03:57 +08:00
Brian Yue
7665339244
[KYUUBI #460] Upgrade Hive dependency to 2.3.9 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #460.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1179 from byyue/feature-460.

Closes #460

92eb71b8 [Brian Yue] Upgrade Hive dependency to 2.3.9

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-23 13:47:22 +08:00
zhouyifan279
b1cf51a68f
[KYUUBI #1249][DEPS] Do not add hive dependencies in root pom.xml if they are spark's dependencies 
### _Why are the changes needed?_
We have hive dependencies defined in root pom.xml.
Some of them are also used by spark.
As spark-hive module couples tightly with hive version, we should leave these dependencies' versions as they are defined in spark by moving these dependencies into sub module pom.xml.
Otherwise, we have to sync kyuubi hive version with spark to avoid hive version compatibility issue in kyuubi-spark-sql-engine tests.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1261 from zhouyifan279/#1249.

Closes #1249

e9e69c20 [zhouyifan279] [KYUUBI #1249][DEPS] Do not add hive dependencies in root pom.xml if they are org.apache.spark:spark-hive's dependencies
6333f89b [zhouyifan279] [KYUUBI #1249][DEPS] Do not add hive dependencies in root pom.xml if they are org.apache.spark:spark-hive's dependencies

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-22 15:25:04 +08:00
Cheng Pan
589b23f578
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767

In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`.

After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1257 from pan3793/pom.

Closes #1257

1d871b6c [Cheng Pan] nit
874f30c1 [Cheng Pan] Address comments
00a32b25 [Cheng Pan] nit
445666ee [Cheng Pan] typo
bd96097e [Cheng Pan] nit
e85116cd [Cheng Pan] nit
98347c8c [Cheng Pan] Update doc
6615d7cc [Cheng Pan] Exclude HudiTest in cross test
087d3fde [Cheng Pan] Add spark 3.2 binary test
ccff63c1 [Cheng Pan] update GA
23b81dcd [Cheng Pan] Address comments
3055c2a8 [Cheng Pan] nit
49a41e86 [Cheng Pan] nit
8d400798 [Cheng Pan] Reduce unnecessary maven profile
ced8d987 [Cheng Pan] Reduce unnecessary maven profile

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-22 14:51:42 +08:00
Cheng Pan
48f8d292d6
[KYUUBI #1228] [BUILD] Add maven profile spark-3.2 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support Spark 3.2.0.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1228 from pan3793/32.

Closes #1228

8f2dfd07 [Cheng Pan] Skip Interval test
fcc4efc9 [Cheng Pan] Handle SPARK-35378
a7061650 [Cheng Pan] Add maven profile spark-3.2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-19 14:46:34 +08:00
dependabot[bot]
78399caef5
[KYUUBI #1251] Bump bcprov-jdk15on from 1.60 to 1.67 
Bumps [bcprov-jdk15on](https://github.com/bcgit/bc-java) from 1.60 to 1.67.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html">bcprov-jdk15on's changelog</a>.</em></p>
<blockquote>
<!-- raw HTML omitted -->
<!-- raw HTML omitted -->
<!-- raw HTML omitted -->
<p><!-- raw HTML omitted --><!-- raw HTML omitted -->2.1.1 Version<!-- raw HTML omitted --><!-- raw HTML omitted -->
Release: 1.70<!-- raw HTML omitted -->
Date:      TBD</p>
<!-- raw HTML omitted -->
<p><!-- raw HTML omitted --><!-- raw HTML omitted -->2.2.1 Version<!-- raw HTML omitted --><!-- raw HTML omitted -->
Release: 1.69<!-- raw HTML omitted -->
Date:      2021, June 7th.</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href="https://github.com/bcgit/bc-java/commits">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.bouncycastle:bcprov-jdk15on&package-manager=maven&previous-version=1.60&new-version=1.67)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-kyuubi/network/alerts).

</details>

Closes #1251 from dependabot[bot]/dependabot/maven/org.bouncycastle-bcprov-jdk15on-1.67.

Closes #1251

1f0c353f [dependabot[bot]] Bump bcprov-jdk15on from 1.60 to 1.67

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-19 13:17:48 +08:00
Cheng Pan
f03a1443a3
[KYUUBI #757] [DEPS] Bump Hadoop 3.3.1 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spark 3.2.0 is out, which bundles Hadoop 3.3.1 shaded client in default. https://github.com/apache/spark/pull/30135

The test failed when using Hadoop 3.3.1 client connects to Yarn Mini Cluster 3.2.2

```
Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetClusterMetricsRequestProto cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
	at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:271)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy13.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:638)
	at org.apache.spark.deploy.yarn.Client.$anonfun$submitApplication$1(Client.scala:179)
	at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
	at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
	at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:65)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:103)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:155)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #757 from pan3793/hadoop-3.3.

Closes #757

7ec9313e [Cheng Pan] [DEPS] Bump Hadoop 3.3.1

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-19 12:39:32 +08:00
Cheng Pan
eb5749d34b
[KYUUBI #1238] [BUILD] Add back Scala stuff to root POM 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This partial revert [KYUUBI #1203], recover codecov

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1238 from pan3793/mvn.

Closes #1238

b3ba87b2 [Cheng Pan] remove unnecessary changes
c124340a [Cheng Pan] nit
39c436be [Cheng Pan] pom
28546cbd [Cheng Pan] CI
f22e9c91 [Cheng Pan] nit
72e9542d [Cheng Pan] Rebase master
7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-15 17:49:48 +08:00
Cheng Pan
86b8e1ce62
[KYUUBI #1236] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The idea based on the analysis of https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/26535726/137326522-be071bc0-523c-436c-8981-01c97ada777f.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1236 from pan3793/shade.

Closes #1236

7a4e3ccc [Cheng Pan] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-15 15:27:26 +08:00
ulysses-you
8a5134e322
[KYUUBI #1225] Add kyuubi-extension-spark-3-2 module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

see https://github.com/apache/incubator-kyuubi/issues/1225

Note that:
- this PR only supports `Zorder` and some common rules with Spark 3.2.0. The others `watchdog` and `sqlclassification` should be supported in future.
- this PR dones't supprpt shade jar and we also need to update build/dist script in a new PR.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1226 from ulysses-you/extension-3.2.

Closes #1225

384a0711 [ulysses-you] address comment
08629f20 [ulysses-you] address comment
15f15f6d [ulysses-you] Add kyuubi-extension-spark-3-2 module

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-15 10:26:12 +08:00
Cheng Pan
238b16fd58
[KYUUBI #1237] [BUILD] Bump spark master version 3.3.0-SNAPSHOT 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spark has been cut `branch-3.2` and bump 3.3.0-SNAPSHOT in master branch several days, we need update to sync the change.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1237 from pan3793/spark-master.

Closes #1237

054818fc [Cheng Pan] bump spark master version to 3.3.0-SNAPSHOT
fc1ffbd7 [Cheng Pan] Enable SNAPSHOT repo only in spark-master profile

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-15 10:09:22 +08:00
Kent Yao
556b5ad57e
[KYUUBI #1223] Remove dead module kyuubi-spark-monitor 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

clean dead code

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1223 from yaooqinn/mon.

Closes #1223

3b7a9a57 [Kent Yao] Remove dead module kyuubi-spark-monitor
b1e2239e [Kent Yao] Remove dead module kyuubi-spark-monitor

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-13 13:15:33 +08:00
Cheng Pan
1f8fa86644
[KYUUBI #1152] [SUB-TASK][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Sub task of #1131

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1152 from pan3793/ut-kyuubi-hive-jdbc.

Closes #1152

2f508356 [Cheng Pan] Avoid hardcode Hive Jdbc class
5032d23d [Cheng Pan] [KYUUBI #1131][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-12 14:47:01 +08:00
Cheng Pan
9e6eb086b1
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
#1131

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1203 from pan3793/pom.

Closes #1203

8e6b8f9b [Cheng Pan] fix
d31fd198 [Cheng Pan] fix
9da82843 [Cheng Pan] fix
6d150de9 [Cheng Pan] fix
9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-09 19:24:05 +08:00
Cheng Pan
ac3e6c36ac
[KYUUBI #1149] [SUB-TASK][BUILD] Create shaded jar for kyuubi-hive-jdbc module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Create a shaded jar for `kyuubi-hive-jdbc` module, which relocate every classes except `org.apache.hadoop`, `org.slf4j`.

Test passed on DataGrip w/o Kerberos.

```
➜  kyuubi-hive-jdbc ll kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    12M Sep 24 16:46 kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar
```
```
➜  kyuubi-hive-jdbc tree -dL 5
.
├── META-INF
│   └── services
└── org
    └── apache
        ├── hadoop
        │   ├── fs
        │   ├── mapred
        │   └── security
        │       └── token
        └── kyuubi
            ├── jdbc
            │   └── hive
            └── shade
                ├── com
                └── org
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/26535726/134647840-23bafa82-deb5-44cb-acf6-bda4df108f3c.png)

![image](https://user-images.githubusercontent.com/26535726/134647364-54cec223-f67f-434e-8501-02d3afc34382.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

- [x] Pass #1152

Closes #1149 from pan3793/jdbc-shade.

Closes #1149

448eaa6a [Cheng Pan] [BUILD] Create shaded jar for kyuubi-hive-jdbc module

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-08 10:35:14 +08:00
fwang12
4f74aba50e [KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support to config operation log root dir with kyuubi configuration both for server and engine sides.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1160 from turboFei/support_config_operationlog_dir.

Closes #1160

ed424f46 [fwang12] fix ut
5edfd4ae [fwang12] refactor ut
b7026b2f [fwang12] add ut
6ff4d891 [fwang12] save
30cd5952 [fwang12] fix ut
f058619a [fwang12] fix ut
96eb020d [fwang12] refactor
ae0ded96 [fwang12] set config value for testing
83d932f4 [fwang12] exclude
33e3e214 [fwang12] update
6f030cba [fwang12] fix
77ec9ca2 [fwang12] save
711ececf [fwang12] Fix code style
cfa3bfac [fwang12] refactor
b299f8cf [fwang12] complete ut
26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-09-29 00:43:23 +08:00
ulysses-you
627d271fc6
[KYUUBI #1167] Add zorder core benchmark 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add zorder core benchmark, so we can get the perf number of zorder.

This PR adds two column type `int` and `long` benchmark:
- 2 int cols
- 3 int cols
- 4 int cols
- 2 long cols
- 3 long cols
- 4 long cols

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1167 from ulysses-you/benchmark.

Closes #1167

27ecf51f [ulysses-you] Add zorder core benchmark

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-28 09:24:19 +08:00