apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,365 Commits 23 Branches 0 Tags 65 MiB
ac24c8fb8c
Commit Graph

1365 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
ulysses-you
ac24c8fb8c
[KYUUBI #1429][TASK-1] Move ui package from org.apache.spark to org.apache.spark.ui 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
In order to compaible with both Spark-2 and Spark-3, we should respect package `ui`, since https://github.com/apache/spark/pull/22645, some class cope changed from package `ui` to `spark`.

### _How was this patch tested?_
Pass CI

Closes #1430 from ulysses-you/spark-2.4.

Closes #1429

8b40c600 [ulysses-you] nit
f875cda3 [ulysses-you] ui

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-23 18:31:11 +08:00
hongdongdong
e19d463e85
[KYUUBI #1404] Decouple zookeeper persistentNode from other modules 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Decouple zookeeper persistentNode from other modules

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1405 from hddong/decouple-persistentNode.

Closes #1404

fcdce91d [hongdongdong] [KYUUBI #1404] Decouple zookeeper persistentNode from other modules

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-23 17:10:54 +08:00
zhang1002
73eeea5f5c
[KYUUBI #1436] unified the field name as kyuubiStatementEvent 
### _Why are the changes needed?_
unified the field name as kyuubiStatementEvent

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1436 from zhang1002/unified-field-name.

Closes #1436

4c83e44a [zhang1002] unified the field name as kyuubiStatementEvent

Authored-by: zhang1002 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-23 17:07:05 +08:00
zhang1002
c8a7623578
[KYUUBI #1437] Fix error message on Spark UI 
### _Why are the changes needed?_
error message modification

query execution is just the sql's logical plan not the error message.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1437 from zhang1002/error-message-modification.

Closes #1437

744c218a [zhang1002] error message modification

Authored-by: zhang1002 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-23 17:05:07 +08:00
Cheng Pan
6bb5af609c
[KYUUBI #1433] Enable scalafmt check in spotless 
### _Why are the changes needed?_

fail compile if violating scalafmt rules.

```
[ERROR] Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.17.4:check (default) on project kyuubi-common_2.12: The following files had format violations:
[ERROR]     src/test/scala/org/apache/kyuubi/operation/JDBCTestHelper.scala
[ERROR]          -52,7 +52,7
[ERROR]          ··}
[ERROR]
[ERROR]          ··def·withMultipleConnectionJdbcStatement(
[ERROR]         -····tableNames:·String*)(fs:·(Statement·=>·Unit)*):·Unit·=·{
[ERROR]         +······tableNames:·String*)(fs:·(Statement·=>·Unit)*):·Unit·=·{
[ERROR]          ····val·connections·=·fs.map·{·_·=>·DriverManager.getConnection(jdbcUrlWithConf,·user,·password)·}
[ERROR]          ····val·statements·=·connections.map(_.createStatement())
[ERROR]
[ERROR] Run 'mvn spotless:apply' to fix these violations.
```
Can auto-fix by `dev/reformat` or `build/mvn spotless:apply`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1433 from pan3793/scalafmt.

Closes #1433

26188a54 [Cheng Pan] Enable scalafmt check in spotless

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-23 15:23:36 +08:00
fwang12
0489250e3b
[KYUUBI #1424] Share jars between server and beeline to reduce binary size 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Share jars between server and beeline to reduce binary size.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1424 from turboFei/share.

Closes #1424

8147a94d [fwang12] use relative path
238df243 [fwang12] refactor
e7069e0c [fwang12] share jars between server and cli

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-23 10:03:35 +08:00
Cheng Pan
116369ac84
[KYUUBI #1376] Implement MySQL Frontend based on Netty 
### _Why are the changes needed?_

Close #1219.

Implment MySQL text protocol, support basic query use mysql cli to run Spark SQL.

Test:

Add `MySQLSparkQuerySuite`.

Change `OutputSchemaTPCDSSuite` from **Thrift binary protocol** to **MySQL protocol**.

Limitations:

1) only support mysql_native_password authentication mechanism
2) only support MySQL text protocol
3) not support send back result which row larger than 16M
4) not support server-side prepared statement
5) not support overwrite engine_conf/session_conf via JDBC url parameters like Hive JDBC client
6) limited support MySQL metadata queries, thus some MySQL GUI client which send metadata queries implicitly may not work as expected
7) not support SSL

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1376 from pan3793/mysql-fe-impl.

Closes #1376

96d3efb9 [Cheng Pan] Kyuubi MySQL FE

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-22 20:35:18 +08:00
zhouyifan279
ffe41631ce
[KYUUBI #1383] Leverage Scalafmt to auto format scala code 
### _Why are the changes needed?_
#1383

As IntelliJ IDEA's `Optimize Imports` result does not meet rules in `scalastyle-config.xml`,  currently developers have to arrange package imports manually.

In this PR, we introduce a `.scalafmt.conf` file to solve the problem.
It works as follows:
1. Set scala code style formatter to `scalafmt`.
2. Execute `Reformat Code`
3. IDEA picks up configurations in  `.scalafmt.conf` and format both code lines and imports.

We also formatted all historical scala codes.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1391 from zhouyifan279/1383.

Closes #1383

e0696a9e [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code
18b8e229 [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-22 17:51:23 +08:00
fwang12
55a7347b6f
[KYUUBI #1425] Remove hardcode skipTests for maven-surefire-plugin 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

So that we can transfer skipTests variable when build kyuubi binary.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1425 from turboFei/java_tst.

Closes #1425

094904d2 [fwang12] typo
af447e8c [fwang12] prevent hardcode

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-22 09:50:46 +08:00
zhenjiaguo
f7e5e6a81b
[KYUUBI #1426] fix maven -pl option usage 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Details see issue #1426

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1427 from zhenjiaguo/building_doc_fix.

Closes #1426

2c858fbb [zhenjiaguo] fix maven -pl option usage

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-22 09:44:05 +08:00
zhenjiaguo
6b9a15a89a
[KYUUBI #1428] Add metrics reporters conf value check 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Configuration metrics reporter need to check that match kyuubi provided reporter.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1428 from zhenjiaguo/metrics_reporters_check_value.

Closes #1428

11a6f9a1 [zhenjiaguo] add metrics reporters check value

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-22 09:37:34 +08:00
Jagadesh Adireddi
4a9a853e69
[KYUUBI #1181] [BUG] Format file paths to specifications. 
Issue: https://github.com/apache/incubator-kyuubi/issues/1181
### _Why are the changes needed?_
Format file paths of local path and hdfs path to specifications

Closes #1420 from jadireddi/1181-Format.

Closes #1181

32aacc7f [Jagadesh Adireddi] update doc
679369ea [Jagadesh Adireddi] [KYUUBI #1181] [BUG] Format file paths to specifications.

Authored-by: Jagadesh Adireddi <jadiredd@conviva.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-20 14:26:39 +08:00
Jagadesh Adireddi
15844dc373 [KYUUBI #1056] [BUG] Log error if no current spark sql engine is created. 
Issue:  https://github.com/apache/incubator-kyuubi/issues/1056

### _Why are the changes needed?_
Log an error message if no current spark sql engine is created.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1423 from jadireddi/1056-no-currEng.

Closes #1056

a8d0443d [Jagadesh Adireddi] refactor
565d41a4 [Jagadesh Adireddi] [KYUUBI #1056] [BUG] Log error if no current spark sql engine is created.

Authored-by: Jagadesh Adireddi <jadiredd@conviva.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-20 12:16:55 +08:00
ulysses-you
fda295865e [KYUUBI #1421] Improve beeline print 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
We don't support isolation, so it's unnecessary to print isolation level.

before this PR
```
Connected to: Spark SQL (version 1.4.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Shaded Client (version 1.4.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.4.0-SNAPSHOT by Apache Kyuubi
```

After:
```
Connected to: Apache Kyuubi (Incubating) (version 1.4.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Shaded Client (version 1.4.0-SNAPSHOT)
Beeline version 1.4.0-SNAPSHOT by Apache Kyuubi (Incubating)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1421 from ulysses-you/improve-log.

Closes #1421

2e0e168e [ulysses-you] improve

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-19 22:58:43 +08:00
fwang12
21cc217281 [KYUUBI #1419] Refactor hive client related property names 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Refactor the kyuubi client related property names from `hive.cli` to `hive.client`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1419 from turboFei/refactor_hive_cli.

Closes #1419

de532387 [fwang12] cli -> client

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-19 16:09:43 +08:00
wForget
10582c8b96
[KYUUBI #1417] Parse instance information in a compatible way 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fix bug #1417.

After fix, run the `bin/kyuubi-ctl list server` command successfully.

![image](https://user-images.githubusercontent.com/17894939/142574175-161e40d8-ada2-4a8f-bd80-1835118ac281.png)

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1418 from wForget/KYUUBI-1417.

Closes #1417

4c662617 [wForget] [KYUUBI-1417] fix checkstyle
44759fea [wForget] [KYUUBI-1417] Parse instance information in a compatible way

Authored-by: wForget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-19 15:27:49 +08:00
fwang12
e5e49b5b1d
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1414 from turboFei/rebase-kyuubi_beeline.

Closes #1414

4694bbca [fwang12] return engine launch handle anyway
20ca1400 [fwang12] address comments
2999c30a [fwang12] fix ut
4d3abf4f [fwang12] jars.target.dir to target
4659f1f6 [fwang12] add readme
7e74c5c8 [fwang12] integrate beeline and server
969af045 [fwang12] fix beeline shell
90395ade [fwang12] BeeLine version By Apache Kyuubi
4e63695e [fwang12] fix ut
026e41ef [fwang12] refactor properties
a49677af [fwang12] refactor build and beeline script
ca68eba1 [fwang12] use beeline mode property instead of static method
4a1932e7 [fwang12] cli -> beeline
624281e3 [fwang12] nit
d7d27a09 [fwang12] refactor kyuubi cli assembly
9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete
cc025ecf [fwang12] Rebase PR_1407

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-19 15:05:10 +08:00
sunfangbin
e54fabab68
[KYUUBI #1416] Refine jline dependency 
### _Why are the changes needed?_
Follows on #1413.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1416 from murong00/branch-1415.

Closes #1416

05335c45 [sunfangbin] Refine jline dependency

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-19 10:40:40 +08:00
sunfangbin
fb8bf49dbc
[KYUUBI #1413] Improve usage of zookeeper cli 
### _Why are the changes needed?_
This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1413 from murong00/branch-1413.

Closes #1413

2d1ec084 [sunfangbin] Rename shell script to zk-cli
d09f64dc [sunfangbin] Fixes failed ci
ce88357a [sunfangbin] Improve usage of zookeeper cli

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 19:17:11 +08:00
Cheng Pan
09f04d6de8
[KYUUBI #1410] Add reformat shell script 
### _Why are the changes needed?_

Following up #1403

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1410 from pan3793/reformat.

Closes #1410

288fc1b5 [Cheng Pan] Add reformat shell script

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 18:39:22 +08:00
Cheng Pan
1a3228f132 [KYUUBI #1415] Immediately wrap the thrift client with a thread-safe proxy 
### _Why are the changes needed?_

Sync before async engine log request has been sent
```
- open session with KyuubiConnection *** FAILED ***
  java.sql.SQLException: org.apache.kyuubi.shade.org.apache.thrift.transport.TTransportException: Read a negative frame size (-2147418110)!
  at org.apache.kyuubi.jdbc.hive.KyuubiStatement.runAsyncOnServer(KyuubiStatement.java:271)
  at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:193)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.$anonfun$new$20(KyuubiOperationPerConnectionSuite.scala:145)
  at org.apache.kyuubi.operation.HiveJDBCTestHelper.withSessionConf(HiveJDBCTestHelper.scala:60)
  at org.apache.kyuubi.operation.HiveJDBCTestHelper.withSessionConf$(HiveJDBCTestHelper.scala:53)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.withSessionConf(KyuubiOperationPerConnectionSuite.scala:33)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.$anonfun$new$19(KyuubiOperationPerConnectionSuite.scala:140)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  ...
  Cause: org.apache.kyuubi.shade.org.apache.thrift.transport.TTransportException: Read a negative frame size (-2147418110)!
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:456)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
  at org.apache.kyuubi.shade.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
  at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:237)
  at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:224)
  ...
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1415 from pan3793/sync.

Closes #1415

5b7ca1bd [Cheng Pan] Immediately wrap the thrift client with a thread-safe proxy

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-18 18:25:01 +08:00
yanyu34946
93e9745a39
[KYUUBI #1293] Fix potential failure of engine discovery when mixed use subdomain 
### _Why are the changes needed?_

Fix #1293

This PR is a rework of #1303, also related to #962

We have two options to fix this issue. This PR implements option 2.

```
[option 1]
kyuubi.engine.share.level.subdomain: String -- default: "default"
kyuubi.engine.pool.size: Int                -- default: -1

val subdomain =
    if (kyuubi.engine.share.level.subdomain == "default" && kyuubi.engine.pool.size > 0) {
        s"engine-pool-[num]"
    } else {
        kyuubi.engine.share.level.subdomain
    }

[option 2]
kyuubi.engine.share.level.subdomain: Option[String] -- default: None
kyuubi.engine.pool.size: Int                        -- default: -1

val subdomain =
    kyuubi.engine.share.level.subdomain match {
        case Some(value) => value
        case None if kyuubi.engine.pool.size > 0 => s"engine-pool-[num]"
        case None => "default"
    }
```
```
[diff]
[[case1]]
kyuubi.engine.share.level.subdomain=default
kyuubi.engine.pool.size=2

option 1 result: subdomain="engine-pool-[num]"
option 2 result: subdomain="default"

[[case2]]
kyuubi.engine.share.level.subdomain=hello
kyuubi.engine.pool.size=2

option 1 result: subdomain="hello"
option 2 result: subdomain="hello"

[[case3]]
kyuubi.engine.share.level.subdomain=
kyuubi.engine.pool.size=2

option 1 result: subdomain="engine-pool-[num]"
option 2 result: subdomain="engine-pool-[num]"

[[case4]]
kyuubi.engine.share.level.subdomain=
kyuubi.engine.pool.size=

option 1 result: subdomain="default"
option 2 result: subdomain="default"

[[case5]]
kyuubi.engine.share.level.subdomain="hello"
kyuubi.engine.pool.size=

option 1 result: subdomain="hello"
option 2 result: subdomain="hello"
```

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1402 from pan3793/1293.

Closes #1293

f6c4a7db [Cheng Pan] nit
fd427bfb [yanyu34946] [KYUUBI #1293] Fix bootstrap potential failure when mixed use subdomain

Lead-authored-by: yanyu34946 <875082356@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 18:00:30 +08:00
simon
dd343fb476
[KYUUBI #1354][FOLLOWUP] Update url to camel case style 
### _Why are the changes needed?_
Update url to camel case style
[#1354](https://github.com/apache/incubator-kyuubi/pull/1408#discussion_r751948135)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1412 from simon824/r3.

Closes #1354

aa933713 [Simon] Merge branch 'apache:master' into r3
5260926a [simon] camel case style
6349ac5a [simon] ApiResponse
b1f906e5 [simon] init

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 16:40:01 +08:00
sunfangbin
995c74b8ee
[KYUUBI #1409] [MINOR] Fix the missing suffix of tarball 
### _Why are the changes needed?_
Fixes the missing suffix of tarball, which should be `$TARDIR_NAME.tgz`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1409 from murong00/branch-1409.

Closes #1409

2ba2718a [sunfangbin] Fix the missing suffix of tarball

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 14:49:45 +08:00
simon
5bf08495b9
[KYUUBI #1354][FOLLOWUP] Add ApiResponse for operation apis 
### _Why are the changes needed?_
Add ApiResponse for operation apis
#1354

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1408 from simon824/r3.

Closes #1354

6349ac5a [simon] ApiResponse
b1f906e5 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 14:48:06 +08:00
yanghua
2baabf46ca
[KYUUBI #1384] Introduce spotless and google java format for java code 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1403 from yanghua/KYUUBI-1384.

Closes #1384

e4e12088 [yanghua] address review suggestion
17a4cb86 [yanghua] reformat all the java files based on the offical google java code style
10a2ca04 [yanghua] [KYUUBI #1384] Introduce Java code style and formatter

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 10:45:51 +08:00
Cheng Pan
56a88f1e48
[KYUUBI #1395] Implement MySQL protocol codec 
### _Why are the changes needed?_

Sub task of #1219

The required `MySQLPacket`s are split into 4 groups, as Kyuubi is designed to be a "MySQL Server", only part of packets need to be encodable and others just need to be decodable.

- Generic
  - `MySQLOKPacket` with `SupportsEncode`
  - `MySQLErrPacket` with `SupportsEncode`
  - `MySQLEofPacket` with `SupportsEncode`
- Authentication
  - `MySQLHandshakePacket` with `SupportsEncode`
  - `MySQLHandshakeResponse41Packet` with `SupportsDecode`
  - `MySQLAuthSwitchRequestPacket` with `SupportsEncode`
  - `MySQLAuthSwitchResponsePacket` with `SupportsDecode`
- Command
  - `MySQLComPingPacket` with `SupportsDecode`
  - `MySQLComInitDbPacket` with `SupportsDecode`
  - `MySQLComFieldListPacket` with `SupportsDecode`
  - `MySQLComQueryPacket` with `SupportsDecode`
  - `MySQLComQuitPacket` with `SupportsDecode`
- Data
  - `MySQLFieldCountPacket`  with`SupportsEncode`
  - `MySQLColumnDefinition41Packet` with `SupportsEncode`
  - `MySQLTextResultSetRowPacket` with `SupportsEncode`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1395 from pan3793/mysql-protocol.

Closes #1395

25b7b3d8 [Cheng Pan] Nit
90637ea1 [Cheng Pan] Fix compile
7fae786e [Cheng Pan] Use while
47a1da82 [Cheng Pan] Add MySQL links
b6d91f7b [Cheng Pan] Implement MySQL protocol codec

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-17 18:42:01 +08:00
Fu Chen
a5ef94accc
[KYUUBI #1354] Expose OpenAPI Specifications 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi.
2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui)

```
curl http://localhost:10099/openapi.json | jq .
```

```json
{
  "openapi": "3.0.1",
  "info": {
    "title": "Apache Kyuubi RESTful Api Documentation",
    "description": "Apache Kyuubi RESTful Api Documentation",
    "contact": {
      "name": "Apache Kyuubi Community",
      "url": "https://kyuubi.apache.org/issue_tracking.html",
      "email": "devkyuubi.apache.org"
    },
    "license": {
      "name": "Apache 2.0",
      "url": "https://www.apache.org/licenses/LICENSE-2.0.html"
    },
    "version": "1.4.0"
  },
  "tags": [
    {
      "name": "Session"
    }
  ],
  "paths": {
    "/api/v1/ping": {
      "get": {
        "operationId": "ping",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a session via session handle identifier",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "delete": {
        "tags": [
          "Session"
        ],
        "operationId": "closeSession",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "close a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/execPool/statistic": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "execPoolStatistic",
        "responses": {
          "200": {
            "description": "get some statistic info of sessions",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/info/{infoType}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "getInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "infoType",
            "in": "path",
            "required": true,
            "schema": {
              "type": "integer",
              "format": "int32"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a information detail via session handle identifier and a specific information type",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfoList",
        "responses": {
          "200": {
            "description": "get all the session list hosted in SessionManager",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "post": {
        "tags": [
          "Session"
        ],
        "operationId": "openSession",
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/SessionOpenRequest"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "Open(create) a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/count": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionCount",
        "responses": {
          "200": {
            "description": "get open session count",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/exception": {
      "get": {
        "operationId": "test",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {}
            }
          }
        }
      }
    },
    "/application.wadl/{path}": {
      "get": {
        "operationId": "getExternalGrammar",
        "parameters": [
          {
            "name": "path",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/xml": {}
            }
          }
        }
      }
    },
    "/application.wadl": {
      "get": {
        "operationId": "getWadl",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/vnd.sun.wadl+xml": {},
              "application/xml": {}
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "ExecPoolStatistic": {
        "required": [
          "execPoolSize"
        ],
        "type": "object",
        "properties": {
          "execPoolSize": {
            "type": "integer",
            "format": "int32"
          },
          "execPoolActiveCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "InfoDetail": {
        "required": [
          "infoType"
        ],
        "type": "object",
        "properties": {
          "infoType": {
            "type": "string"
          },
          "infoValue": {
            "type": "string"
          }
        }
      },
      "HandleIdentifier": {
        "required": [
          "publicId"
        ],
        "type": "object",
        "properties": {
          "publicId": {
            "type": "string",
            "format": "uuid"
          },
          "secretId": {
            "type": "string",
            "format": "uuid"
          }
        }
      },
      "SessionHandle": {
        "required": [
          "identifier",
          "protocol"
        ],
        "type": "object",
        "properties": {
          "identifier": {
            "$ref": "#/components/schemas/HandleIdentifier"
          },
          "protocol": {
            "type": "string",
            "enum": [
              "HIVE_CLI_SERVICE_PROTOCOL_V1",
              "HIVE_CLI_SERVICE_PROTOCOL_V2",
              "HIVE_CLI_SERVICE_PROTOCOL_V3",
              "HIVE_CLI_SERVICE_PROTOCOL_V4",
              "HIVE_CLI_SERVICE_PROTOCOL_V5",
              "HIVE_CLI_SERVICE_PROTOCOL_V6",
              "HIVE_CLI_SERVICE_PROTOCOL_V7",
              "HIVE_CLI_SERVICE_PROTOCOL_V8",
              "HIVE_CLI_SERVICE_PROTOCOL_V9",
              "HIVE_CLI_SERVICE_PROTOCOL_V10"
            ]
          }
        }
      },
      "MapStringString": {
        "type": "object",
        "additionalProperties": {
          "type": "string"
        }
      },
      "SessionOpenRequest": {
        "required": [
          "configs",
          "protocolVersion",
          "user"
        ],
        "type": "object",
        "properties": {
          "protocolVersion": {
            "type": "integer",
            "format": "int32"
          },
          "user": {
            "type": "string"
          },
          "password": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SessionOpenCount": {
        "required": [
          "openSessionCount"
        ],
        "type": "object",
        "properties": {
          "openSessionCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "SessionDetail": {
        "required": [
          "configs",
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          },
          "lastAccessTime": {
            "type": "integer",
            "format": "int64"
          },
          "lastIdleTime": {
            "type": "integer",
            "format": "int64"
          },
          "noOperationTime": {
            "type": "integer",
            "format": "int64"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SeqSessionOverview": {
        "type": "array",
        "items": {
          "$ref": "#/components/schemas/SessionOverview"
        }
      },
      "SessionList": {
        "required": [
          "sessionList"
        ],
        "type": "object",
        "properties": {
          "sessionList": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/SessionOverview"
            }
          }
        }
      },
      "SessionOverview": {
        "required": [
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          }
        }
      }
    }
  }
}
```
http://localhost:10099/api/v1/swagger-ui

> Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json

![截屏2021-11-09 上午10 43 49](https://user-images.githubusercontent.com/8537877/141281596-e47f5552-75b9-47d1-8a36-67337cb6eb3a.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1363 from cfmcgrady/kyuubi-1354.

Closes #1354

8895d7fc [Fu Chen] typo
8f5dea9a [Fu Chen] addressed comment
4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON
0d96951e [Fu Chen] RESTful Api -> REST API
be7d02dc [Fu Chen] add LIABILITY-edl.txt
4ff1053e [Fu Chen] update dev/dependencyList
b126f7e6 [Fu Chen] swagger-ui 4.1.0
a35c2f29 [Fu Chen] addressed comment
7bde0534 [Fu Chen] update LICENSE-binary
38397dc2 [Fu Chen] align jackson version
923f3e7e [Fu Chen] update dev/dependencyList
7d5338c8 [Fu Chen] remove kyuubi-server static files
5cb66b33 [Fu Chen] update dev/dependencyList
e8a8c421 [Fu Chen] fix style
5b13f69c [Fu Chen] add license and remove .map files
c04d7a75 [Fu Chen] rebase master
2388dae8 [Fu Chen] fix style
8dcbd3db [Fu Chen] add static resource
4b0d6d0a [Fu Chen] expose OpenAPI

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-17 12:17:58 +08:00
fwang12
09c8a4093f [KYUUBI #1400] Expose hasMoreEngineLogs and getEngineLog for BeeLine mode 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Expose hasMoreEngineLogs and getEngineLog for BeeLine mode.

Refer the API of HiveStatement/KyuubiStatement:
- hasMoreLogs
- getQueryLog

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1400 from turboFei/refactor_kyuubi_connection.

Closes #1400

9716e8f4 [fwang12] address comments
1da636d3 [fwang12] public
acaf0c7a [fwang12] refactor
54cc1125 [fwang12] save
b66f2585 [fwang12] Make preparation for beeline mode

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-17 10:01:23 +08:00
Kent Yao
bf9736e31b
[KYUUBI #1399] [DOCS] Add doc for engine share level 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

doc improvement

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/8326978/141964500-f0121a54-4ce9-4e9d-84a5-c4a4ab1f0ac9.png)

---

![image](https://user-images.githubusercontent.com/8326978/141965482-f367e486-c68b-44d8-85d9-9c7f0e7e63aa.png)
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1399 from yaooqinn/sldoc.

Closes #1399

d8bcce8a [Kent Yao] [DOCS] Add doc for engine share level
21e2548f [Kent Yao] [DOCS] Add doc for engine share level
bba8dc9d [Kent Yao] [DOCS] Add doc for engine share level

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 19:58:25 +08:00
simon
178c39e221
[KYUUBI #1351] Implement api: /${version}/sessions/${identifier}/operations 
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

- /${version}/sessions/${identifier}/operations
  - mapping:
    - ICLIService#executeStatement
    - ICLIService#executeStatementAsync
    - ICLIService#getTypeInfo
    - ICLIService#getCatelogs
    - ICLIService#getSchemas
    - ICLIService#getTables
    - ICLIService#getTableTypes
    - ICLIService#getColumns
    - ICLIService#getFunctions
  - desc: create a new operation under a session
  - method: POST
  - params:
  - operation: e.g. GetTableTypes
  - addition: required params to create an Operation
  - returns: an instance of OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1353 from simon824/r1.

Closes #1351

98ffcda4 [simon] rename
7c0c3b40 [simon] ut fix
586a0b8a [simon] rename url
1ea6f322 [simon] codestyle
48b5d79d [simon] fix
543caf3c [simon] separate
0ff30af2 [simon] codestyle
203cb164 [simon] Merge remote-tracking branch 'upstream/master' into r1
e578cb42 [simon] fix codestype
006ce4d4 [simon] bugfix
aeb0de63 [simon] operationtype
3b6dceec [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 19:55:36 +08:00
hongdongdong
3fa3004aef
[KYUUBI #1397] Move bind host and port to configmap 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Setting `frontend.bind.host` and `frontend.bind.port` in `kyuubi-configmap.yaml`, instead of put them in ENV.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1398 from hddong/move-bind.

Closes #1397

0741f08a [hongdongdong] [KYUUBI#1397] Move bind host and port to configmap

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-16 18:05:12 +08:00
fwang12
d187391a1e [KYUUBI #1377] Support to get launch engine log asynchronously with KyuubiConnection 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Support to get launch engine log in client side.

When open session with async mode, return the launch engine operation handle by open session response configuration.

For the KyuubiConnection, it will launch a thread to fetch engine log.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/6757692/141752286-95fbcc91-d7ec-4951-ba33-6b4974f21726.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1377 from turboFei/kyuubi_session_async_logs.

Closes #1377

4c6a66e1 [fwang12] remove thread local directly
cbb653ea [fwang12] revert operation idle time
2afb282f [fwang12] nit
901f0ec3 [fwang12] add comments
3c9bc70c [fwang12] delaying close launch engine when close session
4c6775c2 [fwang12] address comments
ef12757b [fwang12] rewrite opensession in KyuubiThriftBinaryFrontendService
880b5bd7 [fwang12] address comments
bb18c90d [fwang12] fix
b59cc04a [fwang12] address nit
3f4917ed [fwang12] support to add extra log path for operation log
b39e3a85 [fwang12] use unknown type for kyuubi defined type
cd0648d6 [fwang12] refactor code
0321a426 [fwang12] only return engine op handle when run async
f0251ba0 [fwang12] complete ut
1a98d2c5 [fwang12] use Base64 encode & decode
eb86c8fb [fwang12] fix ut
329818c1 [fwang12] refactor import order
b419e547 [fwang12] support Kyuubi connection to get engine log

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-16 16:00:35 +08:00
sunfangbin
f5e8152a9e
[KYUUBI #1389] Refine helm service template to support LoadBalancer type 
### _Why are the changes needed?_
The helm service template is currently hard-coded with `NodePort`, we should make it configurable to support other service type such as `LoadBalancer`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1389 from murong00/branch-1389.

Closes #1389

5855c546 [sunfangbin] Refine toYaml usage
d659ad4c [sunfangbin] Refine helm service template to support LoadBalancer type

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 12:04:55 +08:00
yanghua
20ee845c0b
[KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1387 from yanghua/KYUUBI-1386.

Closes #1386

29415904 [yanghua] [KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 09:41:55 +08:00
sunfangbin
481903fb31
[KYUUBI #1388] Added support a zookeeper shell 
### _Why are the changes needed?_
It is useful to support a zookeeper shell so that we can view some metadata if necessary. For example:
```
$ bin/kyuubi-zookeeper-shell -server localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
...
ls /
[zookeeper, kyuubi]
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1388 from murong00/branch-1387.

Closes #1388

7698e79c [sunfangbin] Added support a zookeeper shell

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 18:25:43 +08:00
dependabot[bot]
4073716ac4
[KYUUBI #1385] Bump junit from 4.11 to 4.13.1 in /kyuubi-hive-jdbc 
Bumps [junit](https://github.com/junit-team/junit4) from 4.11 to 4.13.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/junit-team/junit4/releases">junit's releases</a>.</em></p>
<blockquote>
<h2>JUnit 4.13.1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.1.md">release notes</a> for details.</p>
<h2>JUnit 4.13</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.md">release notes</a> for details.</p>
<h2>JUnit 4.13 RC 2</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 RC 1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 3</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 2</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.12</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.12.md">release notes</a> for details.</p>
<h2>JUnit 4.12 Beta 3</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.12.md">release notes</a> for details.</p>
<h2>JUnit 4.12 Beta 2</h2>
<p>No release notes provided.</p>
<h2>JUnit 4.12 Beta 1</h2>
<p>No release notes provided.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="1b683f4ec0"><code>1b683f4</code></a> [maven-release-plugin] prepare release r4.13.1</li>
<li><a href="ce6ce3aadc"><code>ce6ce3a</code></a> Draft 4.13.1 release notes</li>
<li><a href="c29dd8239d"><code>c29dd82</code></a> Change version to 4.13.1-SNAPSHOT</li>
<li><a href="1d174861f0"><code>1d17486</code></a> Add a link to assertThrows in exception testing</li>
<li><a href="543905df72"><code>543905d</code></a> Use separate line for annotation in Javadoc</li>
<li><a href="510e906b39"><code>510e906</code></a> Add sub headlines to class Javadoc</li>
<li><a href="610155b8c2"><code>610155b</code></a> Merge pull request from GHSA-269g-pwp5-87pp</li>
<li><a href="b6cfd1e3d7"><code>b6cfd1e</code></a> Explicitly wrap float parameter for consistency (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1671">#1671</a>)</li>
<li><a href="a5d205c795"><code>a5d205c</code></a> Fix GitHub link in FAQ (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1672">#1672</a>)</li>
<li><a href="3a5c6b4d08"><code>3a5c6b4</code></a> Deprecated since jdk9 replacing constructor instance of Double and Float (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1660">#1660</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/junit-team/junit4/compare/r4.11...r4.13.1">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=junit:junit&package-manager=maven&previous-version=4.11&new-version=4.13.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-kyuubi/network/alerts).

</details>

Closes #1385 from dependabot[bot]/dependabot/maven/kyuubi-hive-jdbc/junit-junit-4.13.1.

Closes #1385

adc26825 [dependabot[bot]] Bump junit from 4.11 to 4.13.1 in /kyuubi-hive-jdbc

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 17:30:24 +08:00
Cheng Pan
cb9e55a9b4
[KYUUBI #1380] Fix kyuubi download duplicated plugins 
### _Why are the changes needed?_
Fix maven pom.xml issue

```
(apache-kyuubi) ➜  apache-kyuubi git:(mysql-fe-impl) ✗ build/mvn clean install
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-download:pom:1.4.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin com.googlecode.maven-download-plugin:download-maven-plugin  line 62, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided,flink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.932 s
[INFO] Finished at: 2021-11-15T10:41:05+08:00
[INFO] ------------------------------------------------------------------------
```

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pflink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/spark-3.1.2-bin-hadoop3.2.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.486 s
[INFO] Finished at: 2021-11-15T10:42:22+08:00
[INFO] ------------------------------------------------------------------------
```

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/flink-1.12.5-bin-scala_2.12.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.114 s
[INFO] Finished at: 2021-11-15T10:41:53+08:00
[INFO] ------------------------------------------------------------------------
```

Closes #1380 from pan3793/download.

Closes #1380

21c7cc22 [Cheng Pan] Fix kyuubi download duplicated plugins

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 15:40:50 +08:00
fwang12
ed43391f8c [KYUUBI #1381] Rewrite kyuubi hive jdbc module based on hive branch-2.3 code base 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Hive code base:
8e7f23f34b

commit 8e7f23f34b2ce7328c9d571a13c336f0c8cdecb6 (HEAD, origin/branch-2.3)
Author: Neelesh Srinivas Salian <nssalianusers.noreply.github.com>
Date:   Tue Oct 19 12:50:31 2021 -0700

    HIVE-25616: Hive-24741 backport to 2.3 (Neelesh Srinivas Salian reviewed by Vihang Karajgaonkar)

    Closes (#2730)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1381 from turboFei/kyuubi_hive_jdbc.

Closes #1381

f39f1a1d [fwang12] adreess comments
5612f490 [fwang12] remove kyuubi-hive-jdbc from code cov
f5a6376e [fwang12] do not fail if no tests
4e99ffcb [fwang12] check kyuubi-hive-jdbc code coverage
1c5f2e16 [fwang12] refactor the test classes
d7f9c7d7 [fwang12] copy test classes from hive branch-2.3
a28c077a [fwang12] rename classname in comments
42f7c5ea [fwang12] try to remove hive-jdbc dependency
fcf5953a [fwang12] use kyuubi hive jdbc utils
17402f4d [fwang12] replace licenses
5cbca43d [fwang12] add missing class ignored by .gitignore
b2c19334 [fwang12] revert docs change
bdc860b4 [fwang12] backport kyuubiHiveDriver
04731e72 [fwang12] make some rename
9d57076b [fwang12] remove from org.apache.hive.jdbc to org.apache.kyuubi.jdbc.hive
86813e6b [fwang12] Backport KyuubiDriver
42f748d6 [fwang12] Backport KyuubiDatabaseMetaData.getTables and for KyuubiConnection.getMetaData, it is not needed
f0d2aedb [fwang12] rename hive classes to kyuubi version
3ea8ba73 [fwang12] copy the source code from hive branch-2.3
7db4e5ac [fwang12] remove at first

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-15 15:38:38 +08:00
Brian Yue
e4b8e961bb
[KYUUBI #1369] Remove _config.yml 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1369.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1375 from byyue/feature-1369.

Closes #1369

03007076 [Brian Yue] Remove _config.yml

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-14 12:48:30 +08:00
fwang12
1554032df2 [KYUUBI #1346] Support launch query engine asynchronously during opening session 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Support launch query engine asynchronously during opening session, make it more user-friendly.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1346 from turboFei/kyuubi_hive_jdbc_ext.

Closes #1346

513691d6 [fwang12] address comments
63b8f901 [fwang12] remove hanle to client
4590ad98 [fwang12] remove synchronized
f3b136b4 [fwang12] add ut for open session with failure
2cae9afa [fwang12] address comments
35113d0f [fwang12] add engine session id to session event
34c27f9c [fwang12] refactor log
b5f7e639 [fwang12] block
b8b0cf9d [fwang12] to support aync client
66eb07f3 [fwang12] address comments
623c0511 [fwang12] volatile
282c9f07 [fwang12] refactor conf
5047d91d [fwang12] server session handle is not same with that of engine
dc1cb8bc [fwang12] update docs
b773fd9d [fwang12] refactor
5f657e44 [fwang12] add ut
9e56cbab [fwang12] exclude engine init engine
6abdb52c [fwang12] refactor
d3e37a8c [fwang12] block until engine finished
085b324d [fwang12] save

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-13 17:25:57 +08:00
zhouyifan279
884de795c2
[KYUUBI #1361] Update available version of backported configurations to 1.3.2 
### _Why are the changes needed?_
Sub task of #1361

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1373 from zhouyifan279/1361-4.

Closes #1361

86b92e86 [zhouyifan279] [KYUUBI #1361] Update available version of backported configurations to 1.3.2

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 22:40:10 +08:00
sunfangbin
9a55bf5e40
[KYUUBI #1368] Turn off debugging in dist scripts 
### _Why are the changes needed?_
`set -x` is currently used in dist scripts for debugging and the console looks a bit messy, it seems better to turn it off and users can enable it themselves if necessary.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1368 from murong00/branch-1366.

Closes #1368

6373fe92 [sunfangbin] Turn off debugging in dist scripts

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-12 19:45:56 +08:00
sunfangbin
ff1f78fd57
[KYUUBI #1365] Refine print environment variables to console 
### _Why are the changes needed?_
Currently, the enviroment variables are also printed to console when using `bin/kyuubi status` or `bin/kyuubi stop` like below:
```
$ bin/kyuubi status
Using kyuubi environment file /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf/kyuubi-env.sh to initialize...
JAVA_HOME: /usr/jdk1.8.0_172
KYUUBI_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating
KYUUBI_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf
KYUUBI_LOG_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/logs
KYUUBI_PID_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/pid
KYUUBI_WORK_DIR_ROOT: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/work
SPARK_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2
SPARK_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2/conf
HADOOP_CONF_DIR:
Kyuubi is not running
```
In general, these variables are only needed to display when starting a cluster to keep clean style, something like below:
```
$ bin/kyuubi status
Kyuubi is running (pid: 95472)
$ bin/kyuubi stop
Stopping org.apache.kyuubi.server.KyuubiServer
...
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1365 from murong00/branch-1364.

Closes #1365

3856494f [sunfangbin] add checking run command
92686fe1 [sunfangbin] Refine print environment variables to console

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 17:39:41 +08:00
ulysses-you
1dc755e450
[KYUUBI #1366] Add a todo in zorder that we can do rebalance partitions before local sort of zorder 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Rebalance partitions can resolve data skew issue before writing, so it should be good to do before we do the local sort of zorder, and also we can support dynamic partition insertion with zorder. However Spark only support this after 3.3.0, see [PR](https://github.com/apache/spark/pull/34542).

So add a todo first.

### _How was this patch tested?_
not needed

Closes #1366 from ulysses-you/zorder.

Closes #1366

972229eb [ulysses-you] todo

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 17:36:36 +08:00
sunfangbin
9e1f9c2e98
[KYUUBI #1359] Support setting zk chroot path when initialize cluster 
### _Why are the changes needed?_
Users may encounter the following exception when trying to set `kyuubi.ha.zookeeper.quorum` to `localhost:2181/lakehouse` with chroot path `/lakehouse` nonexisted:
```
2021-11-10 10:56:34.510 ERROR server.KyuubiThriftBinaryFrontendService: Error starting service KyuubiServiceDiscovery
org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi'
        at org.apache.kyuubi.ha.client.ServiceDiscovery$.createServiceNode(ServiceDiscovery.scala:225)
        at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:101)
......
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kyuubi
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
......
```
It is wonderful to support this since zookeeper connection with chroot path is generally recommended in production environments. With this feture the znodes in zookeeper likes below:
```
[zk: localhost:2181(CONNECTED) 28] ls /lakehouse
[kyuubi, kyuubi_USER, kyuubi_USER_SPARK_SQL]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1359 from murong00/branch-1358.

Closes #1359

42459ba0 [sunfangbin] Add a test to cover the changes
0191ab66 [sunfangbin] Fail the invalid cases
d04c532b [sunfangbin] address all the comments
8e7b5a64 [sunfangbin] Support setting zk chroot path when initialize cluster

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 12:04:35 +08:00
Cheng Pan
1061d176c8
[KYUUBI #1358] Add KyuubiMySQLFrontendService stub 
### _Why are the changes needed?_

Add KyuubiMySQLFrontendService stub, without Netty pipeline, part of #1334

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1358 from pan3793/mysql-fe-stub.

Closes #1358

c091d955 [Cheng Pan] nit
61abb0f6 [Cheng Pan] Address comments
9ac70456 [Cheng Pan] Update conf
3d75acdf [Cheng Pan] nit
4a4d8a24 [Cheng Pan] Address comments
0136dd52 [Cheng Pan] nit
477474ff [Cheng Pan] KyuubiMySQLFrontendService Stub

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-11 16:50:52 +08:00
yanghua
1d9b98e119
[KYUUBI #1360] [MINOR] Change log level from warn to error when executing statement failed 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1360 from yanghua/MINOR-LOG-LEVEL.

Closes #1360

8e846217 [yanghua] [MINOR] Change log level from warn to error when executing statement failed

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-11 14:41:32 +08:00
Kent Yao
9db2e5c054
[KYUUBI #1356] Refine KyuubiStatementEvent 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. When a statement error happened, the event will be logged twice, in `setState` and `setException`, we fix it
2. We break down KyuubiStatementEvent into three parts - statement basis, status, and sessionId/user, see the improved comments
3. log the remote operation handle too

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1356 from yaooqinn/se.

Closes #1356

4b5d0de2 [Kent Yao] Refine KyuubiStatementEvent
e7228ddd [Kent Yao] Refine KyuubiStatementEvent

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-10 12:09:04 +08:00
Cheng Pan
32c3f5cb2e
[KYUUBI #1349] Add new conf kyuubi.frontend.protocols 
### _Why are the changes needed?_

Adapt to multiple frontends protocol.

This PR also marks REST API as experimental and exposes it to users.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1349 from pan3793/frontend-conf.

Closes #1349

d113c4a6 [Cheng Pan] Simplify code by SAM
129c2c9a [Cheng Pan] Fix compile
fa8f7f67 [Cheng Pan] Add new conf kyuubi.frontend.protocols

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 22:09:50 +08:00