Commit Graph

1444 Commits

Author SHA1 Message Date
yanghua
01364e589f
[KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code
…uplicated code

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1584 from yanghua/KYUUBI-1578.

Closes #1578

90c2f89e [yanghua] [KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-17 16:30:33 +08:00
Kent Yao
b2e679d5c7
[KYUUBI #1499] Introduce DataFrameHolder for cli result fetching
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Replace ArrayList with DataFrameHolder

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1507 from yaooqinn/1499.

Closes #1499

7fd2b0ef [Kent Yao] fix it
6278009b [Kent Yao] root
de3c601d [Kent Yao] fi
358d7a68 [Kent Yao] refine
8373b9b3 [Kent Yao] refine
ab95f7dc [Kent Yao] refine
86d90b80 [Kent Yao] loader
a07117c4 [Kent Yao] nit
90a7dd4f [Kent Yao] [KYUUBI #1499] Introduce DataFrameHolder for cli result fetching
8d97b51d [Kent Yao] [KYUUBI #1499] Introduce DataFrameHolder for cli result fetching

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-17 10:45:05 +08:00
zhouyifan279
6e10eec4ca
[KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi
### _Why are the changes needed?_
Many kyuubi users are not familiar with kerberos authentication mechanism and encountered troubles connecting to kerberized Kyuubi.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
Configurations are verified in #1432

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1552 from zhouyifan279/1550.

Closes #1550

58e324a8 [zhouyifan279] [KYUUBI #1550] Add kerberos authentication flow
3b030a91 [zhouyifan279] [KYUUBI #1550] Add kerberos authentication flow
4449b689 [zhouyifan279] [KYUUBI #1550] Improve doc style
3aa8e20c [zhouyifan279] [KYUUBI #1550] Add Windows configuration steps
7e67cdc5 [zhouyifan279] [KYUUBI #1550] Capitalize title and emphasize the relation between KinitAuxiliaryService and Kerberos authentication
424e56b3 [zhouyifan279] [KYUUBI #1550] Rename client/kerberos.md to client/kerberized_kyuubi.md
d8977cea [zhouyifan279] [KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-16 17:21:31 +08:00
yanghua
4b58b0ab9b
[KYUUBI #1555][FLINK] Bump flink version to 1.14.0
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1573 from yanghua/KYUUBI-1555.

Closes #1555

4f5ce746 [yanghua] [KYUUBI #1555] [SUB-TASK][KPIP-2] Bump flink version to 1.14.1

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-16 16:59:38 +08:00
Wang Zhen
be3a964f37
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add kyuubi.engine.pool.name configuration. #1554

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1572 from wForget/KYUUBI-1554.

Closes #1554

40850f5a [Wang Zhen] [KYUUBI-1554] Add kyuubi.engine.pool.name configuration

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-16 16:33:51 +08:00
Cheng Pan
022f52bc7e
[KYUUBI #1460] Bump Hudi 0.10.0
### _Why are the changes needed?_

https://hudi.apache.org/releases/release-0.10.0

Switch test from Hudi 0.9.0 & Spark 3.0 to Hudi 1.10.0 & Spark 3.1. Drop test of Spark 3.0 because

1. Hudi 1.10.0 claims to support Spark 3.0 but requires to build by self with specific maven profile, the published jar only support Spark 3.1
2. Hudi has lots of transitive deps and there are different in different versions, it's hard to maintain multi hudi versions.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1460 from pan3793/hudi.

Closes #1460

bac633a7 [Cheng Pan] Update comments
0ab8784d [Cheng Pan] Pin hadoop-common
c811e679 [Cheng Pan] Exclude scala-library
63cad3b9 [Cheng Pan] Exclude hudi-aws
a13708f9 [Cheng Pan] Disable Hudi test on Spark 3.0
4b179bae [Cheng Pan] Enable Hudi test in Spark 3.1
79dca54c [Cheng Pan] Bump Hudi 0.10.0
71982890 [Cheng Pan] Add default Pk col

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-16 10:13:42 +08:00
Cheng Pan
77a278707f
[KYUUBI #1559] Bump Jersey 2.34
### _Why are the changes needed?_

#1561 and align Spark 3.2.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1559 from pan3793/jersey.

Closes #1559

a86d4d9f [Cheng Pan] Bump Jersey 2.34

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-16 10:12:46 +08:00
simon
bcd8c06e7f
[KYUUBI #1568] Replace sessionDetail with sessionEvent
### _Why are the changes needed?_
Replace sessionDetail with sessionEvent, because sessionEvent has more message about session.
#1568

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1569 from simon824/api.

Closes #1568

00a0ddfe [simon] Replace sessionDetail with sessionEvent
efc84a2e [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-16 10:10:56 +08:00
hongdongdong
11e208a688
[KYUUBI #1565] Move time functions to RowSetUtils and move schema to spark package
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. move time functions from Rowset to RowSetUtils
2. move schema to spark package

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1566 from hddong/move-time-functions.

Closes #1565

75de2e8d [hongdongdong] private
8b16217b [hongdongdong] [KYUUBI #1565] Move time functions to RowSetUtils and move schema to spark package

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 21:31:06 +08:00
zhouyifan279
6c235e5925
[KYUUBI #1567] [Bug] Expose HADOOP_CONF_DIR to Kyuubi beeline to fix "GSS initiate failed" error
### _Why are the changes needed?_
Failed to connect to kerberized Kyuubi with Kyuubi beeline.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

Before:
![image](https://user-images.githubusercontent.com/88070094/146178642-d02748f9-ba7c-4fd0-8375-d3dc1adcfdc1.png)

After:
![image](https://user-images.githubusercontent.com/88070094/146179154-c6f5b133-b533-4649-bd4f-c0c757f1e10c.png)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1570 from zhouyifan279/1567.

Closes #1567

1ee66b08 [zhouyifan279] [KYUUBI #1567] [Bug] Kyuubi beeline can not connect to kerberized Kyuubi

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 21:27:12 +08:00
Cheng Pan
2f45e34dc3
[KYUUBI #1571] [BUILD] Correct build command in build/dist
### _Why are the changes needed?_

`mvn install` is required because of `kyuubi-hive-jdbc-shaded`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1571 from pan3793/dist.

Closes #1571

c905411c [Cheng Pan] Correct build command in build/dist

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 21:23:10 +08:00
Cheng Pan
f841da4b4e
[KYUUBI #1558] Bump Jackson 2.12.5
### _Why are the changes needed?_

Because Spark 3.2, Trino 363, Jersey 2.34 (we use Jersey 2.30 now, will update to 2.34 soon) use Jackson 2.12.x

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1558 from pan3793/jackson.

Closes #1558

7bdd65b0 [Cheng Pan] Bump Jackson 2.12.5

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 21:21:01 +08:00
Brian Yue
01da9b432e
[KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1556.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1562 from byyue/dev-1556.

Closes #1556

8ceb244a [Brian Yue] Update the doc
fdb73c4d [Brian Yue] Shorten kyuubi.session.engine.check.interval to one minute

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-15 17:32:22 +08:00
Cheng Pan
6fa1327f4e
[KYUUBI #1560] Align all Jackson libs version
### _Why are the changes needed?_

Fix missing version of `jackson-module-jaxb-annotations` in `dependencyManagement`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1560 from pan3793/jackson-1.

Closes #1560

49529d47 [Cheng Pan] Update deps
b3610007 [Cheng Pan] fix
67044840 [Cheng Pan] Algin all Jackson libs version

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 12:02:10 +08:00
simon
22d967e9c0
[KYUUBI #1539] Replace operationDetail with KyuubiEvent
### _Why are the changes needed?_
#1539
Replace operationDetail with KyuubiEvent

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1540 from simon824/event.

Closes #1539

9d4e2a04 [Simon] Merge branch 'apache:master' into event
0cd7d15a [simon] replace operationDetail with OperationEvent
adf327e8 [simon] replace operationDetail with OperationEvent
06b6ef8c [simon] replace operationDetail with OperationEvent
b0973e20 [simon] replace operationDetail with OperationEvent
cf83fafb [simon] replace operationDetail with OperationEvent
bf955fbf [simon] replace operationDetail with OperationEvent
49c17988 [simon] replace operationDetail with OperationEvent
8990b701 [simon] replace operationDetail with OperationEvent
03937c2c [simon] replace operationDetail with OperationEvent
a3981776 [simon] replace operationDetail with KyuubiEvent
fc5d747d [simon] replace operationDetail with KyuubiEvent
344eba6a [simon] Revert
b4461411 [simon] codestyle
1d758920 [simon] replace operationDetail with KyuubiEvent
258303ae [simon] init

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-15 10:03:05 +08:00
Kent Yao
d805f49fe7
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

![image](https://user-images.githubusercontent.com/8326978/146000343-44c7b30c-28ba-4434-ab6e-9fb4d0a97d75.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1557 from yaooqinn/ttk.

Closes #1557

8446180a [Kent Yao] [DOC] The TTL Of Kyuubi Engines

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-15 10:02:02 +08:00
sychen
602bb23d36
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
### _Why are the changes needed?_
Compatible with lower version of hive driver.
https://github.com/apache/incubator-kyuubi/issues/1174

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1553 from cxzl25/KYUUBI-1174-followup.

Closes #1174

39cbdf4c [sychen] trigger test
908c4750 [sychen] gen settings
f9e2eda0 [sychen] typo
25300d56 [sychen] Modify the default value to false

Authored-by: sychen <sychen@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-15 00:09:30 +08:00
hongdongdong
c1a0ce8e01
[KYUUBI #1515] Add RowSet for trino engine
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add RowSet for trino engine

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1549 from hddong/add-rowset-for-trino.

Closes #1515

cc885130 [hongdongdong] remove timeZone
2fb10934 [hongdongdong] fix toHiveString
9066caff [hongdongdong] rm optionalStart
491b7682 [hongdongdong] Reset
27895ebe [hongdongdong] fix RowSetUtils
7ce4edeb [hongdongdong] fix
34dd01b8 [hongdongdong] move functions to Utils
79e0621a [hongdongdong] fix
2dcd91e5 [hongdongdong] fix
71bdcfd2 [hongdongdong] [KYUUBI #1515] Add RowSet for trino engine

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-14 20:52:20 +08:00
yanghua
53ff56be29
[KYUUBI #1525][FLINK] Implement GetCatalogs operation and do the further initialization
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1529 from yanghua/KYUUBI-1525.

Closes #1525

e166965d [yanghua] Address review suggestion
9929aa29 [yanghua] Remove some comment
eafb200b [yanghua] refactor code
58df34c7 [yanghua] refactor code
675e14e7 [yanghua] pruning unnecessary code
15e97283 [yanghua] pruning unnecessary code
c6d4f549 [yanghua] pruning unnecessary code
b099a455 [yanghua] pruning unnecessary code
316086bb [yanghua] pruning unnecessary code
fe7af8ef [yanghua] pruning unnecessary code
ecf3ccf4 [yanghua] pruning unnecessary code
4c174517 [yanghua] pruning unnecessary code
b9226c3e [yanghua] pruning unnecessary code
fb2f6a62 [yanghua] pruning unnecessary code
21e21a50 [yanghua] pruning unnecessary code
a58076ff [yanghua] pruning unnecessary code
acd24fc5 [yanghua] pruning unnecessary code
333d4275 [yanghua] pruning unnecessary code
7416b3de [yanghua] pruning unnecessary code
84d0db91 [yanghua] pruning unnecessary code and refactor code
a55db41a [yanghua] pruning unnecessary code
b9909308 [yanghua] [KYUUBI #1525] Implement GetCatalogs operation and do the further initialization

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-14 17:40:11 +08:00
Fei Wang
3594e08d1b [KYUUBI #1548] Get docs version from project.version
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix the docs version issue

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1548 from turboFei/docs_version.

Closes #1548

dfee7b6f [Fei Wang] Fix docs version

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2021-12-12 12:27:00 +08:00
Fei Wang
631bfe35d4 [KYUUBI #1544] Update the mailing list to send announcement
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Update the mailing list to send announcement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1544 from turboFei/finalize_release.

Closes #1544

d5cd5c36 [Fei Wang] Update finalize release doc

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2021-12-12 12:20:18 +08:00
yanghua
6045104092
[KYUUBI #1542] Move FetchIterator into kyuubi-common for multi-engine support
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1543 from yanghua/KYUUBI-1542.

Closes #1542

1ca1c08a [yanghua] Move FetchIteratorSuite into kyuubi-common module
1307d886 [yanghua] [KYUUBI #1542] Move FetchIterator into kyuubi-common for multi-engine support

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-10 16:29:05 +08:00
Fei Wang
3942cf3030
[KYUUBI #1541] Remove keys sync step for Kyuubi finalize release
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Remove keys sync step for Kyuubi finalize release.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1541 from turboFei/remove_keys_sync.

Closes #1541

79a4925d [Fei Wang] Remove unused keys sync step for kyuubi finalize release

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-10 12:38:30 +08:00
zwangsheng
2f46d4cc49
[KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
#1533
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1534 from zwangsheng/KYUUBI/1533.

Closes #1533

803afeed [zwangsheng] fix
9fbc82e0 [zwangsheng] add doc

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-10 11:12:45 +08:00
Fei Wang
609bb5a878 [KYUUBI #1535] Remove duplicate hive client version property
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Remove duplicate hive.cleint.version property, it should be involved when adding kyuubi-hive-beeline module.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1535 from turboFei/typo.

Closes #1535

56e734fa [Fei Wang] Remove duplicate hive client version property

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2021-12-09 18:37:13 +08:00
simon
239277e78a
[KYUUBI #1531] Add kyuubi version in engineSpace
### _Why are the changes needed?_
#1531
Add kyuubi version in engineSpace

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1532 from simon824/ns.

Closes #1531

335906c4 [simon] code
57968cf8 [simon] nameSpace
200f0104 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-09 16:40:34 +08:00
Wang Zhen
c363311e39
[KYUUBI #1528] Record the kyuubi server ip address in event log
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Record the kyuubi server ip address in event log. For details: #1528

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1530 from wForget/KYUUBI-1528.

Closes #1528

6bc18571 [Wang Zhen] [KYUUBI-1528] Record the kyuubi server ip address in event log, set serverIpAddress in SessionEvent.apply
5f0307d1 [Wang Zhen] [KYUUBI-1528] Record the kyuubi server ip address in event log

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-09 15:31:14 +08:00
Kent Yao
c30d580686
[KYUUBI #1527] [DOC] Improve High Availability Guide
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

![image](https://user-images.githubusercontent.com/8326978/145179241-050e66aa-32d6-471d-82bd-a88972c3ca98.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1527 from yaooqinn/hadoc.

Closes #1527

249701b6 [Kent Yao] [DOC] Improve High Availability Guide
03244332 [Kent Yao] [DOC] Improve High Availability Guide
54c1cd9d [Kent Yao] [DOC] Improve High Availability Guide
6d8a0991 [Kent Yao] [DOC] Improve High Availability Guide
5b4dff3d [Kent Yao] [DOC] Improve High Availability Guide

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-09 13:58:00 +08:00
simon
81b14eadc8
[KYUUBI #1516] Implement api: /${version}/operations/${operation_identifier}
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

/${version}/operations/${operation_identifier}
- desc: apply an action for an operation based on a given session identifier and operation identifier
- method: PUT
- params:
- action: "CANCEL" or "ADD" or "CLOSE"
- returns: status code

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1517 from simon824/oa.

Closes #1516

94742c67 [simon] fix desc
fffe3e02 [simon] fix desc
f9415507 [simon] fix
f455995f [simon] operationAction
72bcf5f3 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-08 18:12:02 +08:00
zwangsheng
520bd2653c
[KYUUBI #1480] Fix bug in docker-image-tool & Provider image way to config SPARK instead of local copy
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. fix when using `-t` may cause build fail, because tag invalid reference format
2. add new option `-S`; Declare SPARK_HOME in Docker Image. When you configured -S, you need to provide an image
  with Spark as BASE_IMAGE.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1495 from zwangsheng/KYUUBI/1480.

Closes #1480

0b43985a [zwangsheng] replace -d & add usage
7bc5d946 [zwangsheng] use SPARK_HOME_IN_DOCKER
7b68a8ba [zwangsheng] remove spark_binary
765dc284 [zwangsheng] rename
387590d8 [zwangsheng] use as build
8f2420c2 [zwangsheng] Merge branch 'KYUUBI/1480' of https://github.com/zwangsheng/incubator-kyuubi into KYUUBI/1480
e3407060 [zwangsheng] replace
1486b1ee [zwangsheng] replace
a775006c [zwangsheng] add comments
f98a7c23 [zwangsheng] 1480

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-08 17:34:57 +08:00
zhenjiaguo
7beee52448
[KYUUBI #1393][TASK-2] Add operation number metric
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Current connection number is already had. I add current operation number.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1485 from zhenjiaguo/add-operation-number.

Closes #1393

beccdfed [zhenjiaguo] when cancel we need decCount the open operation
773d01ee [zhenjiaguo] when cancel we need decCount the open operation
59001995 [zhenjiaguo] change variable name
83f64fb8 [zhenjiaguo] unify operation metric
daf0b6a7 [zhenjiaguo] add operation number

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-08 17:13:30 +08:00
sunfangbin
8fc83e4e69
[KYUUBI #1523] Refine usage about option '--namespace' in kyuubi-ctl
### _Why are the changes needed?_
The current option `-n` of command `kyuubi-ctl create` is confusing for new users. For example, when using the below command without setting `kyuubi.ha.zookeeper.namespace` it throws following exception which seems confused:
```
$ bin/kyuubi-ctl create -s localhost -p 10009 -n test
Only support expose Kyuubi server instance to another domain, but the default
namespace is [None] and specified namespace is [test]

        at org.apache.kyuubi.ctl.ServiceControlCliArguments.fail(ServiceControlCliArguments.scala:294)
        at org.apache.kyuubi.ctl.ServiceControlCliArguments.validateCreateActionArguments(ServiceControlCliArguments.scala:195)
        at org.apache.kyuubi.ctl.ServiceControlCliArguments.validateArguments(ServiceControlCliArguments.scala:176)
        at org.apache.kyuubi.ctl.ServiceControlCliArguments.parse(ServiceControlCliArguments.scala:136)
        at org.apache.kyuubi.ctl.ServiceControlCliArguments.<init>(ServiceControlCliArguments.scala:36)
        ...
```
In addtion, the description of option `-n` in `kyuubi-ctl --help` is `using kyuubi-defaults/conf if absent`, however when using below without option `-n` throws like:
```
$ bin/kyuubi-ctl create -s localhost -p 10009
Zookeeper namespace is not specified
...
```
this because the default value is not copied to `cli.args` for `create` action. Users may confuse about this.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1523 from murong00/branch-1523.

Closes #1523

2439d467 [sunfangbin] Refine usage about option '--namespace' in kyuubi-ctl

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-08 17:09:55 +08:00
Wang Zhen
16add92299
[KYUUBI #1522] Pass the client ip to engine
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Pass the client ip to engine. For details: #1522

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1524 from wForget/KYUUBI-1522.

Closes #1522

c9d54ed9 [Wang Zhen] fix word error
a16d914d [Wang Zhen] [KYUUBI-1522] Pass the client ip to engine

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-08 17:08:09 +08:00
Kent Yao
078edcd231
[KYUUBI #1526] [DOC] Update kyuubi_layers.drawio to add driver layer
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

![image](https://user-images.githubusercontent.com/8326978/145150050-08b6a13f-9809-44ab-ae41-9beb825f6ff7.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1526 from yaooqinn/img_kyuubi_layers.

Closes #1526

b769930f [Kent Yao] address comments
2c7de469 [Kent Yao] [DOC] Update kyuubi_layers.drawio to add driver layer

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-08 17:07:11 +08:00
Cheng Pan
e100cba2ba
[KYUUBI #1521] [BUILD] Migrate to actions/cache@v2
### _Why are the changes needed?_

https://github.com/actions/cache

https://github.com/actions/cache/releases/tag/v2.0.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1521 from pan3793/ga-cache.

Closes #1521

87cad572 [Cheng Pan] Migrate to actions/cache@v2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-07 18:24:00 +08:00
Kent Yao
fe31c3b62f
[KYUUBI #1489] Support operation log for ExecuteScala
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1518 from yaooqinn/1498.

Closes #1489

890d1561 [Kent Yao] [KYUUBI #1489] Support operation log for ExecuteScala

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-07 18:22:41 +08:00
Cheng Pan
627d66b171
[KYUUBI #1520] [BUILD] Switch to setup-java@v2
### _Why are the changes needed?_

Migrate to GA setup-java from v1 to v2

https://github.com/actions/setup-java

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1520 from pan3793/ga.

Closes #1520

7aad9d29 [Cheng Pan] nit
5b312423 [Cheng Pan] Use setup-java@v2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-07 16:36:08 +08:00
Wang Zhen
986d9830db
[KYUUBI #1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Make KYUUBI_WORK_DIR_ROOT as the default root path. For details: #1509.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1519 from wForget/KYUUBI-1509.

Closes #1509

ff078e5f [Wang Zhen] [KYUUBI-1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-07 16:05:44 +08:00
hongdongdong
6ea83e040e
[KYUUBI #1511] Initialize kyuubi trino engine module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. Initialize kyuubi trino engine module
2. Add `TrinoThriftBinaryFrontendService`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1514 from hddong/init-trino-engine.

Closes #1511

9dd0e55f [hongdongdong] [KYUUBI #1511] Initialize kyuubi trino engine module

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-07 13:33:35 +08:00
sunfangbin
42674aee91
[KYUUBI #1513] Added support liveness and readiness for kyuubi pod
### _Why are the changes needed?_
It is better to support `livenessProbe` and `readinessProbe` for indicating whether the container is running.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1513 from murong00/branch-1513.

Closes #1513

ea1ff6f6 [sunfangbin] keep the same ordering
6e7d7f07 [sunfangbin] Added support liveness and readiness for kyuubi pod

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-07 09:30:38 +08:00
simon
4319df744c
[KYUUBI #1501] Introduce operationsResource
### _Why are the changes needed?_
#1501
Introduce operationsResource
mv parseSessionHandle() to SessionHandle
mv parseOperationHandle() to OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1502 from simon824/operationresource.

Closes #1501

c4b1b646 [simon] fix
72446d36 [simon] introduce operationsResource
06d12014 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:43:26 +08:00
yanghua
bd3d4d7421
[KYUUBI #1503] Introduce FlinkThriftBinaryFrontendService to do the further initialization
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1512 from yanghua/KYUUBI-1503.

Closes #1503

b41fa735 [yanghua] [KYUUBI #1503] Introduce FlinkThriftBinaryFrontendService to do the further initialization

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:39:22 +08:00
Fei Wang
940c6c1b5c
[KYUUBI #1487] Correct the KEYS link and add script to generate vote and announcement
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Correct the KEYS link in release template.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1487 from turboFei/release_template.

Closes #1487

93e5fc62 [Fei Wang] fix
e06104ba [Fei Wang] refactor
feaab6e2 [Fei Wang] address comments
7675f0c7 [Fei Wang] add shell scripts
83601752 [Fei Wang] Correct the KEYS link in release template

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:31:59 +08:00
ulysses-you
37a4e5c0da
[KYUUBI #1496] Support tpcds benchmark
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support tpcds benchmark in `dev/kyuubi-tpcds` module.

Add a `README.md` in `dev/kyuubi-tpcds` module to show how to use.

The mian code is from [databricks-spark-sql-perf](https://github.com/databricks/spark-sql-perf)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1496 from ulysses-you/tpcds-benchmark.

Closes #1496

d4afe2d1 [ulysses-you] comment
54a146ef [ulysses-you] pom
91e71692 [ulysses-you] docs
20eadc49 [ulysses-you] benchmark

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:08:08 +08:00
ulysses-you
dad48c9d74
[KYUUBI #1508] Add Kyuubi version in UI
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Help to get the Kyuubi version.

### _How was this patch tested?_
![image](https://user-images.githubusercontent.com/12025282/144817101-de9526f7-6467-4355-833d-68a2f31443b9.png)

Closes #1508 from ulysses-you/version.

Closes #1508

d2567664 [ulysses-you] version

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:07:08 +08:00
Fei Wang
920c8e5ad6 [KYUUBI #1504] Support to capture console out/error into interpreter output for ExecuteScala
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Redirect console output and error when interpreting scala code, see details in #1504

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1500 from turboFei/scala_out.

Closes #1504

c3a7df1b [Fei Wang] refactor
d85b6791 [Fei Wang] address comments
f19be18f [Fei Wang] redirect output error

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2021-12-06 19:51:13 +08:00
hongdongdong
542ceadcf3
[KYUUBI #1505] Move ServiceNodeInfo to a separate file
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Decouple zookeeper #1392

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1506 from hddong/move-serviceNodeInfo.

Closes #1505

7bd7f18d [hongdongdong] [KYUUBI #1505] Move ServiceNodeInfo to a separate file

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-06 17:07:11 +08:00
yanghua
339fa7682a
[KYUUBI #1492] Initialize kyuubi flink sql engine module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1497 from yanghua/KYUUBI-1492.

Closes #1492

ba9f221a [yanghua] [KYUUBI #1492] Initialize kyuubi flink sql engine module

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-12-06 15:45:44 +08:00
simon
73ac1b63db
[KYUUBI #1445] [Kyuubi #1444] Implement api: /${version}/sessions/${session_identifier}/operations/${operation_identifier}
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
- desc: get an operation with a given session identifier and operation identifier
- method: GET
- params: none
- returns: an instance of OperationHandle

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
- desc: remove operation based on a given session identifier and operation identifier
- method: DELETE
- params: none
- returns: an instance of OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1445 from simon824/r5.

Closes #1445

196ea4c7 [simon] codestyle
df1c20e8 [simon] remove isHasOperation
6bdebab8 [simon] fix
609aba1b [simon] codestyle
dbae387d [simon] codestyle
7eb9eeb8 [simon] Merge remote-tracking branch 'upstream/master' into r5
8a9e78ca [simon] codestyle
15b00aa9 [simon] operation
b154cbc7 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-06 13:17:33 +08:00
Kent Yao
97c3835738
[KYUUBI #1490] Introduce the basic framework for running scala
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Introduce the basic framework for running scala, see #1490 for the detail
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
```
Beeline version 1.5.0-SNAPSHOT by Apache Kyuubi (Incubating)
0: jdbc:hive2://10.242.189.214:10009/> spark.version;
2021-12-03 13:47:07.556 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:07.558 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:07.559 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:07.560 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed
                 application ID: local-1638510289918
                 application web UI: http://10.242.189.214:65027
                 master: local[*]
                 deploy mode: client
                 version: 3.1.2
           Start time: 2021-12-03T13:44:49.313
           User: kent
2021-12-03 13:47:07.562 INFO scheduler.DAGScheduler: Asked to cancel job group 321dc15d-68d0-4f91-9216-1e08f09842df
2021-12-03 13:47:07.565 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.006 seconds
2021-12-03 13:47:07.565 ERROR operation.ExecuteStatement: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Query[08b8b6da-d434-4296-b613-2027e3518441] in ERROR_STATE
2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.009 seconds
Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
spark.version
^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92)
	at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748) (state=,code=0)
0: jdbc:hive2://10.242.189.214:10009/> set kyuubi.operation.language=scala;
2021-12-03 13:47:11.982 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.983 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.984 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.985 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed
                 application ID: local-1638510289918
                 application web UI: http://10.242.189.214:65027
                 master: local[*]
                 deploy mode: client
                 version: 3.1.2
           Start time: 2021-12-03T13:44:49.313
           User: kent
2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala
2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Execute in full collect mode
2021-12-03 13:47:12.006 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds
2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b] in FINISHED_STATE
2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds
+----------------------------+--------+
|            key             | value  |
+----------------------------+--------+
| kyuubi.operation.language  | scala  |
+----------------------------+--------+
1 row selected (0.052 seconds)
0: jdbc:hive2://10.242.189.214:10009/> spark.version;
2021-12-03 13:47:13.685 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version
2021-12-03 13:47:15.541 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: PENDING_STATE -> RUNNING_STATE, statement: spark.version
2021-12-03 13:47:15.543 INFO operation.ExecuteStatement: Query[178ff72d-b870-44d4-99d8-08a0fa0e8efa] in FINISHED_STATE
2021-12-03 13:47:15.544 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: RUNNING_STATE -> FINISHED_STATE, statement: spark.version, time taken: 0.003 seconds
+-----------------------+
|        output         |
+-----------------------+
| res0: String = 3.1.2  |
+-----------------------+
1 row selected (1.871 seconds)
0: jdbc:hive2://10.242.189.214:10009/> spark.sql("select current_date()")
. . . . . . . . . . . . . . . . . . .> ;
2021-12-03 13:47:36.512 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.sql("select current_date()")
2021-12-03 13:47:36.689 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: PENDING_STATE -> RUNNING_STATE, statement: spark.sql("select current_date()")
2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Query[602c2d88-7e8a-4175-9b53-e34adfff007a] in FINISHED_STATE
2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: RUNNING_STATE -> FINISHED_STATE, statement: spark.sql("select current_date()"), time taken: 0.003 seconds
+----------------------------------------------------+
|                       output                       |
+----------------------------------------------------+
| res1: org.apache.spark.sql.DataFrame = [current_date(): date] |
+----------------------------------------------------+
1 row selected (0.187 seconds)
0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3);
Error: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error:
results += spark.range(1, 5, 2, 3)
 <console>:26: error: type mismatch;
 found   : org.apache.spark.sql.Dataset[Long]
 required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
       results += spark.range(1, 5, 2, 3)
                             ^
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
	at org.apache.kyuubi.engine.spark.operation.ExecuteScala.runInternal(ExecuteScala.scala:70)
	at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130)
	at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93)
	at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:62)
	at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:121)
	at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75)
	at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:118)
	at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:61)
	at org.apache.kyuubi.service.ThriftBinaryFrontendService.ExecuteStatement(ThriftBinaryFrontendService.scala:265)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748) (state=,code=0)
0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3).toDF;
2021-12-03 13:48:30.327 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: INITIALIZED_STATE -> PENDING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF
2021-12-03 13:48:31.700 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: PENDING_STATE -> RUNNING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF
2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Query[daccd163-978a-4d66-af85-af466086d1db] in FINISHED_STATE
2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: RUNNING_STATE -> FINISHED_STATE, statement: results += spark.range(1, 5, 2, 3).toDF, time taken: 0.003 seconds
+-----+
| id  |
+-----+
| 1   |
| 3   |
+-----+
2 rows selected (1.387 seconds)
```

** Session level isolated **

![image](https://user-images.githubusercontent.com/8326978/144553902-fc390c30-06de-453b-af3d-cf8def577aa9.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1491 from yaooqinn/scala.

Closes #1490

af4d0a13 [Kent Yao] Merge branch 'master' into scala
2ebdf6a3 [Kent Yao] provided scala dep
d58b2bfe [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala
d256bfde [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala
52090434 [Kent Yao] init

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-12-06 11:17:59 +08:00