apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,529 Commits 23 Branches 0 Tags 65 MiB
36b95d333f
Commit Graph

332 Commits

Author SHA1 Message Date
Kent Yao
b91854e33b
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR mainly focus on refactoring
1. Use Apache Thrift Naming style for thrift-based classes which is `TXyzAbc`, e.g. `TFrontendService`
2. Created `TFrontendService` and move the `TCLIService.Iface` APIs and other common fields in `TFrontendService`, later we will use it to implement the `KyuubiTHttpFrontendService` in the server module
3. introduce a new conf also for server FEs to be able to pick IP first

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1761 from yaooqinn/http1.

Closes #1761

98b4afdd [Kent Yao] address comment
3138af29 [Kent Yao] [WIP] Thrift Http protocol support 1
a06026af [Kent Yao] [WIP] Thrift Http protocol support 1
efa899fa [Kent Yao] [WIP] Thrift Http protocol support 1
d1403e30 [Kent Yao] [WIP] Thrift Http protocol support 1
d114efd4 [Kent Yao] [WIP] Thrift Http protocol support 1
86e30c47 [Kent Yao] [WIP] Thrift Http protocol support 1
8117d12d [Kent Yao] [WIP] Thrift Http protocol support 1
62194323 [Kent Yao] [WIP] Thrift Http protocol support 1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-14 17:03:44 +08:00
yanghua
e032a6218f
[KYUUBI #1757] Extract workingDir from engine specific process builde… 
…r into ProcBuilder

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1758 from yanghua/KYUUBI-1757.

Closes #1757

14d3673c [yanghua] Fix CI issue
d6696919 [yanghua] [KYUUBI #1757] Extract workingDir from engine specific process builder into ProcBuilder

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-14 12:46:15 +08:00
hongdongdong
11556034e3
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add TRINO engine type and TrinoProcessBuilder

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1695 from hddong/add-trino-process-builder.

Closes #1510

394b853e [hongdongdong] fix
044ca8cc [hongdongdong] fix
568b1b8e [hongdongdong] use java options to pass args
fbe7e60d [hongdongdong] [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-14 12:43:16 +08:00
simon
f5205c6bbb
[KYUUBI #1759] Remove duplicate close operation api 
### _Why are the changes needed?_
Remove duplicate close operation api, closes #1759

 #1445 is duplicate with #1517 where action='close'.

```
/${version}/operations/${operation_identifier}
desc: apply an action for an operation based on a given session identifier and operation identifier
method: PUT
params:
action: "CANCEL" or "ADD" or "CLOSE"
returns: status code

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
desc: remove operation based on a given session identifier and operation identifier
method: DELETE
params: none
returns: an instance of OperationHandle
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1760 from simon824/closeoperation.

Closes #1759

9af70a07 [simon] closeoperation

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-14 12:39:39 +08:00
zhouyifan279
c17dd9d490
[KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens 
### _Why are the changes needed?_

If core-site.xml contains a FileSystem uri, whose FileSystem implementation class is not present, following error will appear repeatedly in Kyuubi log after any user connects to Kyuubi and executed any SQL:

```
22/01/12 19:45:08 WARN credentials.HadoopFsDelegationTokenProvider: Failed to get Hadoop FileSystem instance by URI: alluxio://localhost
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2667)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.$anonfun$hadoopFSsToAccess$2(HadoopFsDelegationTokenProvider.scala:100)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.$anonfun$hadoopFSsToAccess$1(HadoopFsDelegationTokenProvider.scala:100)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.hadoopFSsToAccess(HadoopFsDelegationTokenProvider.scala:99)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:62)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:125)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:124)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:60)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
	... 40 more

```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1739 from zhouyifan279/1731.

Closes #1731

387036d8 [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found
6d80b8e9 [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2022-01-13 20:23:02 +08:00
zhouyifan279
a550ac705b
[KYUUBI #1737] [WIP][KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found 
### _Why are the changes needed?_
Kyuubi server fails to start when Hadoop FileSystem class not found.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1737 from zhouyifan279/1731.

Closes #1737

Closes #1731

477ca37d [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2022-01-12 16:37:18 +08:00
Kent Yao
c1a6cd587c
[KYUUBI #1716] Unify Hive deps for server and client 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR make the hive deps be consistent between server and clients for a single release, we shall not play client-server compatibility tricky in one release

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1716 from yaooqinn/hive.

Closes #1716

2eb3bdf7 [Kent Yao] Merge branch 'master' into hive
c1bf6369 [Kent Yao] dep
c7f0f26f [Kent Yao] dep
645d76ab [Kent Yao] fix test
3ac4f55c [Kent Yao] Merge branch 'master' into hive
f1ca475a [Kent Yao] delta test
01d30291 [Kent Yao] refine
1e93f33a [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into hive
5bd96313 [Kent Yao] Unify Hive deps for server and client

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2022-01-12 16:36:40 +08:00
Kent Yao
4c682df05f
[KYUUBI #1672] Add VersionInfo in REST API 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1728 from yaooqinn/1672.

Closes #1672

511d8ceb [Kent Yao] [KYUUBI #1672] Add VersionInfo in REST API

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-11 21:16:59 +08:00
yanghua
d63eed2749
[KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine t… 
…o decouple with kyuubi server

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1711 from yanghua/KYUUBI-1709.

Closes #1709

95466130 [yanghua] refactor code
89f9c572 [yanghua] Reorder module list in root pom file
b664a4bb [yanghua] Rename IT module names
6b7fa88f [yanghua] Rename IT module names
b133e8e9 [yanghua] Reformatted pom indent
38444416 [yanghua] [KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine to decouple with kyuubi server

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2022-01-11 12:45:41 +08:00
Fei Wang
e0eeab07eb
[KYUUBI #1710] Support to specify OPERATION_LANGUAGE with TExecuteStatementReq confOverlay 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Now kyuubi supports SQL and SCALA language.

Now the way to switch OPERATION_LANAUAGE:

```
// for SCALA
spark.sql("set kyuubi.operation.language=SCALA")

// for SQL
set kyuubi.operation.language=SCALA
```

It is more user friendly that user can specify the OPERATION_LANGUAGE for each ExecuteStatement.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1710 from turboFei/conf_overlay.

Closes #1710

385264dc [Fei Wang] fix ut
58fbdbdf [Fei Wang] fix npe
2718fb48 [Fei Wang] Support to get OPERATION_LANGUAGE from execute statement confOverlay

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2022-01-10 17:21:25 +08:00
yanghua
58dd4ca71f
[KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1688 from yanghua/KYUUBI-1687.

Closes #1647

d6fda463 [yanghua] [KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2022-01-10 16:18:22 +08:00
zhenjiaguo
2a7f366b69
[KYUUBI #1697] BackendService method avg call times and rate 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

BackendService method avg call times and rate. Replace Histograms with Timer to record it. Timer include both Meters and Histograms.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1708 from zhenjiaguo/use_timer_for_backend_service.

Closes #1697

3a6af4d3 [zhenjiaguo] comment
0914af2f [zhenjiaguo] use timer to replace histograms

Authored-by: zhenjiaguo <zhenjiaguo@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2022-01-10 10:39:20 +08:00
simon
c43d3a7d22
[KYUUBI #1691] Implement api: /${version}/operations/${operation_identifier}/rowset 
### _Why are the changes needed?_
closes #1691
/${version}/operations/${operation_identifier}/rowset?fetchorientation=${fetchorientation}&maxrows=${maxrows}
mapping: ICLIService#fetchResults
desc: get the row set via the given operation identifier
method: GET
params: none
returns: an instance of RowSet

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1692 from simon824/getNextRowSet.

Closes #1691

611aafe0 [simon] desc
e6712b1b [simon] getNextRowSet

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-07 19:59:56 +08:00
xifeng yang
628719fd60
[KYUUBI #1477] Use KyuubiHadoopUtils.newHadoopConf instead of new Con… 
…figuration()

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- EventLoggingServiceSuite

![image](https://user-images.githubusercontent.com/12961802/147745991-fcb14907-4d3a-4bce-a492-1b111b6e1614.png)

- HadoopFsDelegationTokenProviderSuite
![image](https://user-images.githubusercontent.com/12961802/147746114-2d2037d7-fe72-4e85-8db5-7b3244136c5b.png)

- KyuubiHadoopUtilsSuite
![image](https://user-images.githubusercontent.com/12961802/147746199-40e0ca91-9f48-4395-941b-a08affb6fbd9.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1661 from xifeng/kyuubi-1477.

Closes #1477

ad068c6e [xifeng yang] [KYUUBI #1477] Use KyuubiHadoopUtils.newHadoopConf instead of new Configuration()

Authored-by: xifeng yang <xifeng.yang@hotmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-05 15:07:50 +08:00
odone
2f1f2589cc
[KYUUBI #1678] Fixed: The value of hasResultSet is always set to true in Kyuubi Server 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

Fixes #1678

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Updating `hasResultSet` state which is determined by the remote service

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1682 from iodone/kyuubi-1678.

Closes #1678

8e553850 [odone] comment

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-05 15:06:48 +08:00
Cheng Pan
e4f5ba7010
[KYUUBI #1673] [BUILD] Improve log4j configurations 
### _Why are the changes needed?_

- Flink engine print log to console on UT, and use `$FLINK_CONF_DIR/log4j.properties` in production.
- Add missing log4j conf `kyuubi-trino-engine` module.
- Fix log4j configuration issue of THRIFT

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1673 from pan3793/log.

Closes #1673

8abe9d08 [Cheng Pan] Remove flink-sql-engine conf
b1726288 [Cheng Pan] Improve log4j configurations

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-04 19:43:14 +08:00
Simon
dc26f2128c
[KYUUBI #1568] [FOLLOWUP] Add getSessionEvent in session 
### _Why are the changes needed?_
KyuubiSessionEvent.apply will get incorrect info.
The first one is  from `Session.getSessionEvent`, the second one is `KyuubiSessionEvent.apply`

![image](https://user-images.githubusercontent.com/18065113/147657104-16247260-4b50-4485-843c-f5dbb0116c99.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1644 from simon824/1229.

Closes #1568

bc5e4b49 [Simon] Merge branch 'apache:master' into 1229
bb316e6a [simon] Add getSessionEvent in session

Lead-authored-by: Simon <3656562@qq.com>
Co-authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2022-01-04 19:40:57 +08:00
simon
4655cd8515
[KYUUBI #1654][Bug] Flaky OperationsResourceSuite 
### _Why are the changes needed?_
closes #1654

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1655 from simon824/flaky.

Closes #1654

92548e46 [simon] style
3a748969 [simon] state
3c507da3 [simon] state
4436bb89 [simon] rm
e12d483c [simon] flakyut

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-31 19:28:57 +08:00
yanghua
4e3c718108
[KYUUBI #1665][FOLLOWUP] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1667 from yanghua/KYUUBI-1665-FOLLOWUP.

Closes #1665

aa6f6120 [yanghua] retrigger ci
013005c3 [yanghua] [KYUUBI #1665][FOLLOWUP] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-31 19:27:58 +08:00
zwangsheng
872551d3aa
[KYUUBI #1594] SparkProcessBuild submit add --conf spark.driver.host= 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
detail #1594
When Kyuubi On Kubernetes submit spark engine by client mode, executor pod can not connect to driver by kyuubi pod name, which is default set.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1596 from zwangsheng/kyuubi/1594.

Closes #1594

d4b41005 [zwangsheng] add comment
f7bcffb6 [zwangsheng] fix test name
581eb508 [zwangsheng] fix test name
b19a2c21 [zwangsheng] simple & add testsuit
83979935 [zwangsheng] fix
1a9ae9b1 [zwangsheng] fix doc
d857eac7 [zwangsheng] Merge branch 'kyuubi/1594' of https://github.com/zwangsheng/incubator-kyuubi into kyuubi/1594
b723f4ed [zwangsheng] retest
8210de56 [zwangsheng] make import normal
92ec9cf4 [zwangsheng] 1594

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-31 19:25:18 +08:00
yanghua
3eaa1d86c8 [KYUUBI #1665] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder 
…r to align with SparkProcessBuilder

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1666 from yanghua/KYUUBI-1665.

Closes #1665

569becd3 [yanghua] [KYUUBI #1665] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: yanghua <yanghua1127@gmail.com>
 2021-12-31 17:12:53 +08:00
Kent Yao
d1de9ca529
[KYUUBI #1658] Revamp swagger ui dependencies 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR mainly removes the swagger-scala-module which is currently fine to remove. It also introduces transitive deps with
`-jakarta` which could introduce jar conflicts like ,

```scala
Caused by: java.lang.NoClassDefFoundError: jakarta/xml/bind/annotation/XmlElement
	at io.swagger.v3.core.jackson.SwaggerAnnotationIntrospector.hasRequiredMarker(SwaggerAnnotationIntrospector.java:29)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319)
	at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getMetadata(POJOPropertyBuilder.java:229)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1658 from yaooqinn/dep.

Closes #1658

d5ab0765 [Kent Yao] Merge branch 'master' into dep
d1ddd5fd [Kent Yao] Revamp swagger ui dependencies
ecfcc92a [Kent Yao] Revamp swagger ui dependencies
59196b94 [Kent Yao] Revamp swagger ui dependencies
d7ca6a5f [Kent Yao] Revamp swagger ui dependencies
215b1b53 [Kent Yao] Revamp swagger ui dependencies

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-31 09:22:24 +08:00
Kent Yao
c4fd287c8b
[KYUUBI #1639] Make ApiRequestContext provide KyuubiRestFrontendService directly 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Make full use of KyuubiRestFrontendService without hacking, e.g. for the swagger UI base.

We can do further improvement based on this PR.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1639 from yaooqinn/ApiRequestContext.

Closes #1639

3c0c7061 [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly
10506c2c [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly
1482bd78 [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly
01a0b023 [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly
d3119fa4 [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly
0f45258c [Kent Yao] Make ApiRequestContext provides KyuubiRestFrontendService directly

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-29 13:10:22 +08:00
Cheng Pan
cb8721fef1
[KYUUBI #1629] Flink backend implementation 
### _Why are the changes needed?_

This PR covers #1619.

Overall, this PR contains the following changs,

1. change `build/dist` script to support flink sql engine
2. enhance `externals/flink-sql-engine/pom.xml` to support create a shaded jar
3. simplify `externals/kyuubi-flink-sql-engine/bin/flink-sql-engine.sh`
4. introduce `FlinkSQLEngine`(flink sql engine entrypoint) and `FlinkProcessBuilder`(kyuubi server launcher)
5. add ut in kyuubi server side

After this PR, we can run the basic query e.g. `select now()` from beeline and get result, and Kyuubi Server can auto launch flink engine if there is no proper one. The Flink engine also supports other engine share levels defined in Kyuubi.

The implementation based on Flink 1.14 codebase.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1629 from pan3793/flink-backend.

Closes #1629

b7e5f0e3 [Cheng Pan] revert tgz name change
a4496c37 [Cheng Pan] Fix reflection
3b4e86a1 [Cheng Pan] deps
68efa42c [Cheng Pan] log
8a9e37f3 [Cheng Pan] nit
10fb2bc3 [Cheng Pan] CI
c1560fdb [Cheng Pan] nit
303e2f1e [Cheng Pan] Restore log conf
d84720b1 [Cheng Pan] SessionContext
b258d81a [Cheng Pan] cleanup
16edd528 [Cheng Pan] Cleanup
9ae54557 [Cheng Pan] Fix CI
25b6b57d [Cheng Pan] hadoop-client-api
c12b5ca4 [Cheng Pan] Server UT pass
502d3f08 [Cheng Pan] pass
dac4323b [yanghua] Laungh local flink engine container successfully

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-28 17:20:26 +08:00
Kent Yao
e1587eeaf4
[KYUUBI #1631] Migrating existing rest fe tests to real cases 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

In this PR, we target the existing UTs from a noop server to a real shared Kyuubi server.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1631 from yaooqinn/resttest.

Closes #1631

2726ab44 [Kent Yao] address comments
c0da8090 [Kent Yao] ci
5d12f70f [Kent Yao] address comments
80e14044 [Kent Yao] Migrating existing rest fe test to real cases
a31ab506 [Kent Yao] Migrating existing rest fe test to real cases
0c1feb45 [Kent Yao] Migrating existing rest fe test to real cases
fbd9769e [Kent Yao] Migrating existing rest fe test to real cases
75cca5ff [Kent Yao] Migrating existing rest fe test to real cases
119712e8 [Kent Yao] Migrating existing rest fe test to real cases
8196e4df [Kent Yao] Migrating existing rest fe test to real cases

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-28 17:17:27 +08:00
rayliu
2af105a417 [KYUUBI #1556] Upgrade swagger-ui version from 4.1.0 to 4.1.3 
### _Why are the changes needed?_
Close https://github.com/apache/incubator-kyuubi/issues/1612

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1630 from ldzhjn/1612.

Closes #1556

71fe05bc [rayliu] Upgrade swagger-ui version from 4.1.0 to 4.1.3

Authored-by: rayliu <rayliu@cisco.com>
Signed-off-by: yanghua <yanghua1127@gmail.com>
 2021-12-25 20:50:56 +08:00
Fu Chen
21f10d752d
[KYUUBI #1614] Fix hardcoded version in OpenAPIDefinition 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

close #1614

curl http://0.0.0.0:10099/openapi.json
```json
{
  "openapi" : "3.0.1",
  "info" : {
    "title" : "Apache Kyuubi REST API Documentation",
    "description" : "Apache Kyuubi REST API Documentation",
    "contact" : {
      "name" : "Apache Kyuubi Community",
      "url" : "https://kyuubi.apache.org/issue_tracking.html",
      "email" : "devkyuubi.apache.org"
    },
    "license" : {
      "name" : "Apache 2.0",
      "url" : "https://www.apache.org/licenses/LICENSE-2.0.html"
    },
    "version" : "1.5.0-SNAPSHOT"
  },
  "tags" : [ {
    "name" : "Session"
  }, {
    "name" : "Operation"
  } ],
  "paths" : {
    "/api/v1/operations/{operationHandle}" : {
      "put" : {
        "tags" : [ "Operation" ],
        "operationId" : "applyOpAction",
        "parameters" : [ {
          "name" : "operationHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/OpActionRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "apply an action for an operation",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/operations/{operationHandle}/event" : {
      "get" : {
        "tags" : [ "Operation" ],
        "operationId" : "getOperationEvent",
        "parameters" : [ {
          "name" : "operationHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Get an operation event",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/operations/{operationHandle}/resultsetmetadata" : {
      "get" : {
        "tags" : [ "Operation" ],
        "operationId" : "getResultSetMetadata",
        "parameters" : [ {
          "name" : "operationHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "get result set metadata",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/ping" : {
      "get" : {
        "operationId" : "ping",
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "text/plain" : {
                "schema" : {
                  "type" : "string"
                }
              }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/{operationHandle}" : {
      "delete" : {
        "tags" : [ "Session" ],
        "operationId" : "closeOperation",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        }, {
          "name" : "operationHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Close an operation",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}" : {
      "get" : {
        "tags" : [ "Session" ],
        "operationId" : "sessionInfo",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "get a session event via session handle identifier",
            "content" : {
              "application/json" : { }
            }
          }
        }
      },
      "delete" : {
        "tags" : [ "Session" ],
        "operationId" : "closeSession",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Close a session",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/execPool/statistic" : {
      "get" : {
        "tags" : [ "Session" ],
        "operationId" : "execPoolStatistic",
        "responses" : {
          "200" : {
            "description" : "Get statistic info of background executors",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/statement" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "executeStatement",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/StatementRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Create an operation with EXECUTE_STATEMENT type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/catalogs" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getCatalogs",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_CATALOGS type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/columns" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getColumns",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/GetColumnsRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_COLUMNS type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/functions" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getFunctions",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/GetFunctionsRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_FUNCTIONS type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/info/{infoType}" : {
      "get" : {
        "tags" : [ "Session" ],
        "operationId" : "getInfo",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        }, {
          "name" : "infoType",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "integer",
            "format" : "int32"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "get a information detail via session handle identifier and a specific information type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/schemas" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getSchemas",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/GetSchemasRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_SCHEMAS type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/tableTypes" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getTableTypes",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_TABLE_TYPES type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/tables" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getTables",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "requestBody" : {
          "content" : {
            "*/*" : {
              "schema" : {
                "$ref" : "#/components/schemas/GetTablesRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_TABLES type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/operations/typeInfo" : {
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "getTypeInfo",
        "parameters" : [ {
          "name" : "sessionHandle",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "200" : {
            "description" : "Create an operation with GET_TYPE_INFO type",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions" : {
      "get" : {
        "tags" : [ "Session" ],
        "operationId" : "sessionInfoList",
        "responses" : {
          "200" : {
            "description" : "get all the session list hosted in SessionManager",
            "content" : {
              "application/json" : { }
            }
          }
        }
      },
      "post" : {
        "tags" : [ "Session" ],
        "operationId" : "openSession",
        "requestBody" : {
          "content" : {
            "application/json" : {
              "schema" : {
                "$ref" : "#/components/schemas/SessionOpenRequest"
              }
            }
          }
        },
        "responses" : {
          "200" : {
            "description" : "Open(create) a session",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/sessions/count" : {
      "get" : {
        "tags" : [ "Session" ],
        "operationId" : "sessionCount",
        "responses" : {
          "200" : {
            "description" : "Get the current open session count",
            "content" : {
              "application/json" : { }
            }
          }
        }
      }
    },
    "/api/v1/swagger-ui" : {
      "get" : {
        "operationId" : "swaggerUi",
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "text/html" : { }
            }
          }
        }
      }
    },
    "/api/v1/exception" : {
      "get" : {
        "operationId" : "test",
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "text/plain" : { }
            }
          }
        }
      }
    },
    "/application.wadl/{path}" : {
      "get" : {
        "operationId" : "getExternalGrammar",
        "parameters" : [ {
          "name" : "path",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "string"
          }
        } ],
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "application/xml" : { }
            }
          }
        }
      }
    },
    "/application.wadl" : {
      "get" : {
        "operationId" : "getWadl",
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "application/vnd.sun.wadl+xml" : { },
              "application/xml" : { }
            }
          }
        }
      }
    }
  },
  "components" : {
    "schemas" : {
      "OpActionRequest" : {
        "required" : [ "action" ],
        "type" : "object",
        "properties" : {
          "action" : {
            "type" : "string"
          }
        }
      },
      "KyuubiOperationEvent" : {
        "required" : [ "eventTime", "shouldRunAsync", "statementId" ],
        "type" : "object",
        "properties" : {
          "statementId" : {
            "type" : "string"
          },
          "remoteId" : {
            "type" : "string"
          },
          "statement" : {
            "type" : "string"
          },
          "shouldRunAsync" : {
            "type" : "boolean"
          },
          "state" : {
            "type" : "string"
          },
          "eventTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "createTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "startTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "completeTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "exception" : {
            "type" : "object",
            "properties" : {
              "cause" : {
                "required" : [ "stackTrace", "suppressed" ],
                "type" : "object",
                "properties" : {
                  "stackTrace" : {
                    "type" : "array",
                    "items" : {
                      "required" : [ "lineNumber" ],
                      "type" : "object",
                      "properties" : {
                        "methodName" : {
                          "type" : "string"
                        },
                        "fileName" : {
                          "type" : "string"
                        },
                        "lineNumber" : {
                          "type" : "integer",
                          "format" : "int32"
                        },
                        "className" : {
                          "type" : "string"
                        },
                        "nativeMethod" : {
                          "type" : "boolean"
                        }
                      }
                    }
                  },
                  "message" : {
                    "type" : "string"
                  },
                  "localizedMessage" : {
                    "type" : "string"
                  },
                  "suppressed" : {
                    "type" : "array",
                    "items" : {
                      "type" : "object",
                      "properties" : {
                        "stackTrace" : {
                          "type" : "array",
                          "items" : {
                            "required" : [ "lineNumber" ],
                            "type" : "object",
                            "properties" : {
                              "methodName" : {
                                "type" : "string"
                              },
                              "fileName" : {
                                "type" : "string"
                              },
                              "lineNumber" : {
                                "type" : "integer",
                                "format" : "int32"
                              },
                              "className" : {
                                "type" : "string"
                              },
                              "nativeMethod" : {
                                "type" : "boolean"
                              }
                            }
                          }
                        },
                        "message" : {
                          "type" : "string"
                        },
                        "localizedMessage" : {
                          "type" : "string"
                        }
                      }
                    }
                  }
                }
              },
              "stackTrace" : {
                "type" : "array",
                "items" : {
                  "required" : [ "lineNumber" ],
                  "type" : "object",
                  "properties" : {
                    "methodName" : {
                      "type" : "string"
                    },
                    "fileName" : {
                      "type" : "string"
                    },
                    "lineNumber" : {
                      "type" : "integer",
                      "format" : "int32"
                    },
                    "className" : {
                      "type" : "string"
                    },
                    "nativeMethod" : {
                      "type" : "boolean"
                    }
                  }
                }
              },
              "message" : {
                "type" : "string"
              },
              "localizedMessage" : {
                "type" : "string"
              },
              "suppressed" : {
                "type" : "array",
                "items" : {
                  "type" : "object",
                  "properties" : {
                    "stackTrace" : {
                      "type" : "array",
                      "items" : {
                        "required" : [ "lineNumber" ],
                        "type" : "object",
                        "properties" : {
                          "methodName" : {
                            "type" : "string"
                          },
                          "fileName" : {
                            "type" : "string"
                          },
                          "lineNumber" : {
                            "type" : "integer",
                            "format" : "int32"
                          },
                          "className" : {
                            "type" : "string"
                          },
                          "nativeMethod" : {
                            "type" : "boolean"
                          }
                        }
                      }
                    },
                    "message" : {
                      "type" : "string"
                    },
                    "localizedMessage" : {
                      "type" : "string"
                    }
                  }
                }
              }
            }
          },
          "sessionId" : {
            "type" : "string"
          },
          "sessionUser" : {
            "type" : "string"
          },
          "eventType" : {
            "type" : "string"
          }
        }
      },
      "ColumnDesc" : {
        "required" : [ "columnIndex", "columnName" ],
        "type" : "object",
        "properties" : {
          "columnName" : {
            "type" : "string"
          },
          "dataType" : {
            "type" : "string"
          },
          "columnIndex" : {
            "type" : "integer",
            "format" : "int32"
          },
          "precision" : {
            "type" : "integer",
            "format" : "int32"
          },
          "scale" : {
            "type" : "integer",
            "format" : "int32"
          },
          "comment" : {
            "type" : "string"
          }
        }
      },
      "ResultSetMetaData" : {
        "required" : [ "columns" ],
        "type" : "object",
        "properties" : {
          "columns" : {
            "type" : "array",
            "items" : {
              "$ref" : "#/components/schemas/ColumnDesc"
            }
          }
        }
      },
      "SeqColumnDesc" : {
        "type" : "array",
        "items" : {
          "$ref" : "#/components/schemas/ColumnDesc"
        }
      },
      "HandleIdentifier" : {
        "required" : [ "publicId" ],
        "type" : "object",
        "properties" : {
          "publicId" : {
            "type" : "string",
            "format" : "uuid"
          },
          "secretId" : {
            "type" : "string",
            "format" : "uuid"
          }
        }
      },
      "OperationHandle" : {
        "required" : [ "hasResultSet", "identifier", "protocol", "typ" ],
        "type" : "object",
        "properties" : {
          "identifier" : {
            "$ref" : "#/components/schemas/HandleIdentifier"
          },
          "typ" : {
            "$ref" : "#/components/schemas/Value"
          },
          "protocol" : {
            "type" : "string",
            "enum" : [ "HIVE_CLI_SERVICE_PROTOCOL_V1", "HIVE_CLI_SERVICE_PROTOCOL_V2", "HIVE_CLI_SERVICE_PROTOCOL_V3", "HIVE_CLI_SERVICE_PROTOCOL_V4", "HIVE_CLI_SERVICE_PROTOCOL_V5", "HIVE_CLI_SERVICE_PROTOCOL_V6", "HIVE_CLI_SERVICE_PROTOCOL_V7", "HIVE_CLI_SERVICE_PROTOCOL_V8", "HIVE_CLI_SERVICE_PROTOCOL_V9", "HIVE_CLI_SERVICE_PROTOCOL_V10" ]
          },
          "hasResultSet" : {
            "type" : "boolean",
            "writeOnly" : true
          }
        }
      },
      "Value" : {
        "type" : "object"
      },
      "ExecPoolStatistic" : {
        "required" : [ "execPoolSize" ],
        "type" : "object",
        "properties" : {
          "execPoolSize" : {
            "type" : "integer",
            "format" : "int32"
          },
          "execPoolActiveCount" : {
            "type" : "integer",
            "format" : "int32"
          }
        }
      },
      "StatementRequest" : {
        "required" : [ "queryTimeout", "runAsync", "statement" ],
        "type" : "object",
        "properties" : {
          "statement" : {
            "type" : "string"
          },
          "runAsync" : {
            "type" : "boolean"
          },
          "queryTimeout" : {
            "type" : "integer",
            "format" : "int64"
          }
        }
      },
      "GetColumnsRequest" : {
        "required" : [ "catalogName" ],
        "type" : "object",
        "properties" : {
          "catalogName" : {
            "type" : "string"
          },
          "schemaName" : {
            "type" : "string"
          },
          "tableName" : {
            "type" : "string"
          },
          "columnName" : {
            "type" : "string"
          }
        }
      },
      "GetFunctionsRequest" : {
        "required" : [ "catalogName" ],
        "type" : "object",
        "properties" : {
          "catalogName" : {
            "type" : "string"
          },
          "schemaName" : {
            "type" : "string"
          },
          "functionName" : {
            "type" : "string"
          }
        }
      },
      "InfoDetail" : {
        "required" : [ "infoType" ],
        "type" : "object",
        "properties" : {
          "infoType" : {
            "type" : "string"
          },
          "infoValue" : {
            "type" : "string"
          }
        }
      },
      "GetSchemasRequest" : {
        "required" : [ "catalogName" ],
        "type" : "object",
        "properties" : {
          "catalogName" : {
            "type" : "string"
          },
          "schemaName" : {
            "type" : "string"
          }
        }
      },
      "GetTablesRequest" : {
        "required" : [ "catalogName", "tableTypes" ],
        "type" : "object",
        "properties" : {
          "catalogName" : {
            "type" : "string"
          },
          "schemaName" : {
            "type" : "string"
          },
          "tableName" : {
            "type" : "string"
          },
          "tableTypes" : {
            "type" : "array",
            "items" : {
              "type" : "string"
            }
          }
        }
      },
      "SessionHandle" : {
        "required" : [ "identifier", "protocol" ],
        "type" : "object",
        "properties" : {
          "identifier" : {
            "$ref" : "#/components/schemas/HandleIdentifier"
          },
          "protocol" : {
            "type" : "string",
            "enum" : [ "HIVE_CLI_SERVICE_PROTOCOL_V1", "HIVE_CLI_SERVICE_PROTOCOL_V2", "HIVE_CLI_SERVICE_PROTOCOL_V3", "HIVE_CLI_SERVICE_PROTOCOL_V4", "HIVE_CLI_SERVICE_PROTOCOL_V5", "HIVE_CLI_SERVICE_PROTOCOL_V6", "HIVE_CLI_SERVICE_PROTOCOL_V7", "HIVE_CLI_SERVICE_PROTOCOL_V8", "HIVE_CLI_SERVICE_PROTOCOL_V9", "HIVE_CLI_SERVICE_PROTOCOL_V10" ]
          }
        }
      },
      "MapStringString" : {
        "type" : "object",
        "additionalProperties" : {
          "type" : "string"
        }
      },
      "SessionOpenRequest" : {
        "required" : [ "configs", "protocolVersion", "user" ],
        "type" : "object",
        "properties" : {
          "protocolVersion" : {
            "type" : "integer",
            "format" : "int32"
          },
          "user" : {
            "type" : "string"
          },
          "password" : {
            "type" : "string"
          },
          "ipAddr" : {
            "type" : "string"
          },
          "configs" : {
            "type" : "object",
            "additionalProperties" : {
              "type" : "string"
            }
          }
        }
      },
      "SessionOpenCount" : {
        "required" : [ "openSessionCount" ],
        "type" : "object",
        "properties" : {
          "openSessionCount" : {
            "type" : "integer",
            "format" : "int32"
          }
        }
      },
      "KyuubiSessionEvent" : {
        "required" : [ "clientVersion", "conf", "sessionName", "startTime" ],
        "type" : "object",
        "properties" : {
          "sessionName" : {
            "type" : "string"
          },
          "user" : {
            "type" : "string"
          },
          "clientIP" : {
            "type" : "string"
          },
          "serverIP" : {
            "type" : "string"
          },
          "conf" : {
            "type" : "object",
            "additionalProperties" : {
              "type" : "string"
            }
          },
          "startTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "sessionId" : {
            "type" : "string"
          },
          "remoteSessionId" : {
            "type" : "string"
          },
          "clientVersion" : {
            "type" : "integer",
            "format" : "int32"
          },
          "openedTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "endTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "totalOperations" : {
            "type" : "integer",
            "format" : "int32"
          },
          "eventType" : {
            "type" : "string"
          }
        }
      },
      "SeqSessionOverview" : {
        "type" : "array",
        "items" : {
          "$ref" : "#/components/schemas/SessionOverview"
        }
      },
      "SessionList" : {
        "required" : [ "sessionList" ],
        "type" : "object",
        "properties" : {
          "sessionList" : {
            "type" : "array",
            "items" : {
              "$ref" : "#/components/schemas/SessionOverview"
            }
          }
        }
      },
      "SessionOverview" : {
        "required" : [ "createTime", "user" ],
        "type" : "object",
        "properties" : {
          "user" : {
            "type" : "string"
          },
          "ipAddr" : {
            "type" : "string"
          },
          "createTime" : {
            "type" : "integer",
            "format" : "int64"
          },
          "sessionHandle" : {
            "$ref" : "#/components/schemas/SessionHandle"
          }
        }
      }
    }
  }
}
```

curl http://0.0.0.0:10099/openapi.yaml

```yaml
openapi: 3.0.1
info:
  title: Apache Kyuubi REST API Documentation
  description: Apache Kyuubi REST API Documentation
  contact:
    name: Apache Kyuubi Community
    url: https://kyuubi.apache.org/issue_tracking.html
    email: devkyuubi.apache.org
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  version: 1.5.0-SNAPSHOT
tags:
- name: Session
- name: Operation
paths:
  /api/v1/operations/{operationHandle}:
    put:
      tags:
      - Operation
      operationId: applyOpAction
      parameters:
      - name: operationHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/OpActionRequest'
      responses:
        "200":
          description: apply an action for an operation
          content:
            application/json: {}
  /api/v1/operations/{operationHandle}/event:
    get:
      tags:
      - Operation
      operationId: getOperationEvent
      parameters:
      - name: operationHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Get an operation event
          content:
            application/json: {}
  /api/v1/operations/{operationHandle}/resultsetmetadata:
    get:
      tags:
      - Operation
      operationId: getResultSetMetadata
      parameters:
      - name: operationHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: get result set metadata
          content:
            application/json: {}
  /api/v1/ping:
    get:
      operationId: ping
      responses:
        default:
          description: default response
          content:
            text/plain:
              schema:
                type: string
  /api/v1/sessions/{sessionHandle}/operations/{operationHandle}:
    delete:
      tags:
      - Session
      operationId: closeOperation
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      - name: operationHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Close an operation
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}:
    get:
      tags:
      - Session
      operationId: sessionInfo
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: get a session event via session handle identifier
          content:
            application/json: {}
    delete:
      tags:
      - Session
      operationId: closeSession
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Close a session
          content:
            application/json: {}
  /api/v1/sessions/execPool/statistic:
    get:
      tags:
      - Session
      operationId: execPoolStatistic
      responses:
        "200":
          description: Get statistic info of background executors
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/statement:
    post:
      tags:
      - Session
      operationId: executeStatement
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/StatementRequest'
      responses:
        "200":
          description: Create an operation with EXECUTE_STATEMENT type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/catalogs:
    post:
      tags:
      - Session
      operationId: getCatalogs
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Create an operation with GET_CATALOGS type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/columns:
    post:
      tags:
      - Session
      operationId: getColumns
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/GetColumnsRequest'
      responses:
        "200":
          description: Create an operation with GET_COLUMNS type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/functions:
    post:
      tags:
      - Session
      operationId: getFunctions
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/GetFunctionsRequest'
      responses:
        "200":
          description: Create an operation with GET_FUNCTIONS type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/info/{infoType}:
    get:
      tags:
      - Session
      operationId: getInfo
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      - name: infoType
        in: path
        required: true
        schema:
          type: integer
          format: int32
      responses:
        "200":
          description: get a information detail via session handle identifier and
            a specific information type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/schemas:
    post:
      tags:
      - Session
      operationId: getSchemas
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/GetSchemasRequest'
      responses:
        "200":
          description: Create an operation with GET_SCHEMAS type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/tableTypes:
    post:
      tags:
      - Session
      operationId: getTableTypes
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Create an operation with GET_TABLE_TYPES type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/tables:
    post:
      tags:
      - Session
      operationId: getTables
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      requestBody:
        content:
          '*/*':
            schema:
              $ref: '#/components/schemas/GetTablesRequest'
      responses:
        "200":
          description: Create an operation with GET_TABLES type
          content:
            application/json: {}
  /api/v1/sessions/{sessionHandle}/operations/typeInfo:
    post:
      tags:
      - Session
      operationId: getTypeInfo
      parameters:
      - name: sessionHandle
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: Create an operation with GET_TYPE_INFO type
          content:
            application/json: {}
  /api/v1/sessions:
    get:
      tags:
      - Session
      operationId: sessionInfoList
      responses:
        "200":
          description: get all the session list hosted in SessionManager
          content:
            application/json: {}
    post:
      tags:
      - Session
      operationId: openSession
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SessionOpenRequest'
      responses:
        "200":
          description: Open(create) a session
          content:
            application/json: {}
  /api/v1/sessions/count:
    get:
      tags:
      - Session
      operationId: sessionCount
      responses:
        "200":
          description: Get the current open session count
          content:
            application/json: {}
  /api/v1/swagger-ui:
    get:
      operationId: swaggerUi
      responses:
        default:
          description: default response
          content:
            text/html: {}
  /api/v1/exception:
    get:
      operationId: test
      responses:
        default:
          description: default response
          content:
            text/plain: {}
  /application.wadl/{path}:
    get:
      operationId: getExternalGrammar
      parameters:
      - name: path
        in: path
        required: true
        schema:
          type: string
      responses:
        default:
          description: default response
          content:
            application/xml: {}
  /application.wadl:
    get:
      operationId: getWadl
      responses:
        default:
          description: default response
          content:
            application/vnd.sun.wadl+xml: {}
            application/xml: {}
components:
  schemas:
    OpActionRequest:
      required:
      - action
      type: object
      properties:
        action:
          type: string
    KyuubiOperationEvent:
      required:
      - eventTime
      - shouldRunAsync
      - statementId
      type: object
      properties:
        statementId:
          type: string
        remoteId:
          type: string
        statement:
          type: string
        shouldRunAsync:
          type: boolean
        state:
          type: string
        eventTime:
          type: integer
          format: int64
        createTime:
          type: integer
          format: int64
        startTime:
          type: integer
          format: int64
        completeTime:
          type: integer
          format: int64
        exception:
          type: object
          properties:
            cause:
              required:
              - stackTrace
              - suppressed
              type: object
              properties:
                stackTrace:
                  type: array
                  items:
                    required:
                    - lineNumber
                    type: object
                    properties:
                      methodName:
                        type: string
                      fileName:
                        type: string
                      lineNumber:
                        type: integer
                        format: int32
                      className:
                        type: string
                      nativeMethod:
                        type: boolean
                message:
                  type: string
                localizedMessage:
                  type: string
                suppressed:
                  type: array
                  items:
                    type: object
                    properties:
                      stackTrace:
                        type: array
                        items:
                          required:
                          - lineNumber
                          type: object
                          properties:
                            methodName:
                              type: string
                            fileName:
                              type: string
                            lineNumber:
                              type: integer
                              format: int32
                            className:
                              type: string
                            nativeMethod:
                              type: boolean
                      message:
                        type: string
                      localizedMessage:
                        type: string
            stackTrace:
              type: array
              items:
                required:
                - lineNumber
                type: object
                properties:
                  methodName:
                    type: string
                  fileName:
                    type: string
                  lineNumber:
                    type: integer
                    format: int32
                  className:
                    type: string
                  nativeMethod:
                    type: boolean
            message:
              type: string
            localizedMessage:
              type: string
            suppressed:
              type: array
              items:
                type: object
                properties:
                  stackTrace:
                    type: array
                    items:
                      required:
                      - lineNumber
                      type: object
                      properties:
                        methodName:
                          type: string
                        fileName:
                          type: string
                        lineNumber:
                          type: integer
                          format: int32
                        className:
                          type: string
                        nativeMethod:
                          type: boolean
                  message:
                    type: string
                  localizedMessage:
                    type: string
        sessionId:
          type: string
        sessionUser:
          type: string
        eventType:
          type: string
    ColumnDesc:
      required:
      - columnIndex
      - columnName
      type: object
      properties:
        columnName:
          type: string
        dataType:
          type: string
        columnIndex:
          type: integer
          format: int32
        precision:
          type: integer
          format: int32
        scale:
          type: integer
          format: int32
        comment:
          type: string
    ResultSetMetaData:
      required:
      - columns
      type: object
      properties:
        columns:
          type: array
          items:
            $ref: '#/components/schemas/ColumnDesc'
    SeqColumnDesc:
      type: array
      items:
        $ref: '#/components/schemas/ColumnDesc'
    HandleIdentifier:
      required:
      - publicId
      type: object
      properties:
        publicId:
          type: string
          format: uuid
        secretId:
          type: string
          format: uuid
    OperationHandle:
      required:
      - hasResultSet
      - identifier
      - protocol
      - typ
      type: object
      properties:
        identifier:
          $ref: '#/components/schemas/HandleIdentifier'
        typ:
          $ref: '#/components/schemas/Value'
        protocol:
          type: string
          enum:
          - HIVE_CLI_SERVICE_PROTOCOL_V1
          - HIVE_CLI_SERVICE_PROTOCOL_V2
          - HIVE_CLI_SERVICE_PROTOCOL_V3
          - HIVE_CLI_SERVICE_PROTOCOL_V4
          - HIVE_CLI_SERVICE_PROTOCOL_V5
          - HIVE_CLI_SERVICE_PROTOCOL_V6
          - HIVE_CLI_SERVICE_PROTOCOL_V7
          - HIVE_CLI_SERVICE_PROTOCOL_V8
          - HIVE_CLI_SERVICE_PROTOCOL_V9
          - HIVE_CLI_SERVICE_PROTOCOL_V10
        hasResultSet:
          type: boolean
          writeOnly: true
    Value:
      type: object
    ExecPoolStatistic:
      required:
      - execPoolSize
      type: object
      properties:
        execPoolSize:
          type: integer
          format: int32
        execPoolActiveCount:
          type: integer
          format: int32
    StatementRequest:
      required:
      - queryTimeout
      - runAsync
      - statement
      type: object
      properties:
        statement:
          type: string
        runAsync:
          type: boolean
        queryTimeout:
          type: integer
          format: int64
    GetColumnsRequest:
      required:
      - catalogName
      type: object
      properties:
        catalogName:
          type: string
        schemaName:
          type: string
        tableName:
          type: string
        columnName:
          type: string
    GetFunctionsRequest:
      required:
      - catalogName
      type: object
      properties:
        catalogName:
          type: string
        schemaName:
          type: string
        functionName:
          type: string
    InfoDetail:
      required:
      - infoType
      type: object
      properties:
        infoType:
          type: string
        infoValue:
          type: string
    GetSchemasRequest:
      required:
      - catalogName
      type: object
      properties:
        catalogName:
          type: string
        schemaName:
          type: string
    GetTablesRequest:
      required:
      - catalogName
      - tableTypes
      type: object
      properties:
        catalogName:
          type: string
        schemaName:
          type: string
        tableName:
          type: string
        tableTypes:
          type: array
          items:
            type: string
    SessionHandle:
      required:
      - identifier
      - protocol
      type: object
      properties:
        identifier:
          $ref: '#/components/schemas/HandleIdentifier'
        protocol:
          type: string
          enum:
          - HIVE_CLI_SERVICE_PROTOCOL_V1
          - HIVE_CLI_SERVICE_PROTOCOL_V2
          - HIVE_CLI_SERVICE_PROTOCOL_V3
          - HIVE_CLI_SERVICE_PROTOCOL_V4
          - HIVE_CLI_SERVICE_PROTOCOL_V5
          - HIVE_CLI_SERVICE_PROTOCOL_V6
          - HIVE_CLI_SERVICE_PROTOCOL_V7
          - HIVE_CLI_SERVICE_PROTOCOL_V8
          - HIVE_CLI_SERVICE_PROTOCOL_V9
          - HIVE_CLI_SERVICE_PROTOCOL_V10
    MapStringString:
      type: object
      additionalProperties:
        type: string
    SessionOpenRequest:
      required:
      - configs
      - protocolVersion
      - user
      type: object
      properties:
        protocolVersion:
          type: integer
          format: int32
        user:
          type: string
        password:
          type: string
        ipAddr:
          type: string
        configs:
          type: object
          additionalProperties:
            type: string
    SessionOpenCount:
      required:
      - openSessionCount
      type: object
      properties:
        openSessionCount:
          type: integer
          format: int32
    KyuubiSessionEvent:
      required:
      - clientVersion
      - conf
      - sessionName
      - startTime
      type: object
      properties:
        sessionName:
          type: string
        user:
          type: string
        clientIP:
          type: string
        serverIP:
          type: string
        conf:
          type: object
          additionalProperties:
            type: string
        startTime:
          type: integer
          format: int64
        sessionId:
          type: string
        remoteSessionId:
          type: string
        clientVersion:
          type: integer
          format: int32
        openedTime:
          type: integer
          format: int64
        endTime:
          type: integer
          format: int64
        totalOperations:
          type: integer
          format: int32
        eventType:
          type: string
    SeqSessionOverview:
      type: array
      items:
        $ref: '#/components/schemas/SessionOverview'
    SessionList:
      required:
      - sessionList
      type: object
      properties:
        sessionList:
          type: array
          items:
            $ref: '#/components/schemas/SessionOverview'
    SessionOverview:
      required:
      - createTime
      - user
      type: object
      properties:
        user:
          type: string
        ipAddr:
          type: string
        createTime:
          type: integer
          format: int64
        sessionHandle:
          $ref: '#/components/schemas/SessionHandle'
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1628 from cfmcgrady/kyuubi-1614.

Closes #1614

9310f5a0 [Fu Chen] address comment
ef3cb812 [Fu Chen] fix hardcoded version in OpenAPIDefinition

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-24 19:29:14 +08:00
Kent Yao
99e278ce76
[KYUUBI #1623] KyuubiSessionEvent shall always have Id 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

session id is a val

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1623 from yaooqinn/val.

Closes #1623

23fc3b2b [Kent Yao] test
6cd86610 [Kent Yao] npe
6fb75d47 [Kent Yao] KyuubiSessionEvent shall always have Id

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-24 19:27:48 +08:00
simon
c42f31e211
[KYUUBI #1620] Implement api: /${version}/operations/${operation_identifier}/log 
### _Why are the changes needed?_
closes #1620
 Implement api: /${version}/operations/${operation_identifier}/log

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1621 from simon824/1223.

Closes #1620

5f211473 [simon] rm
f6d0ce7c [simon] fix
346df798 [simon] operationLog
c60d3447 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-24 17:02:05 +08:00
Wang Zhen
fbccba90f4
[KYUUBI #1613] Record the engine's applicationId in KyuubiSessionEvent 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Record the engine's applicationId in KyuubiSessionEvent. #1613

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/17894939/147318970-d939b3c0-c3c6-4f45-9e5b-c2de6402b5de.png)

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1622 from wForget/KYUUBI-1613.

Closes #1613

24e001f6 [Wang Zhen] comment
29453fc3 [Wang Zhen] [KYUUBI-1613] Record the engine's applicationId in KyuubiSessionEvent

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-24 14:01:01 +08:00
Fu Chen
54a1b88245
[KYUUBI #1611] Fix swagger-ui redirect wrong to other service address 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

close #1611

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1615 from cfmcgrady/kyuubi-1611.

Closes #1611

f44bf3b1 [Fu Chen] fix swagger-ui

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-23 16:58:43 +08:00
Kent Yao
1b48b1874f
[KYUUBI #1601] Align SparkStatementEvent to KyuubiOperationEvent 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Align SparkStatementEvent to KyuubiOperationEvent

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1601 from yaooqinn/soe.

Closes #1601

090dee8f [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into soe
a61ce3b7 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven
efbbdf79 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-23 16:10:43 +08:00
simon
c972139b51
[KYUUBI #1575] Implement api: /${version}/operations/${operation_identifier}/resultsetmetadata 
### _Why are the changes needed?_
#1575
Implement api: /${version}/operations/${operation_identifier}/resultsetmetadata

- /${version}/operations/${operation_identifier}/resultsetmetadata
  - mapping: ICLIService#getResultSetMetadata
  - desc: get the table schema of the result set via the given operation identifier
  - method: GET
  - params: none
  - returns: an instance of TableSchema

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1576 from simon824/api1.

Closes #1575

ecc976d1 [simon] reorder
0ef2accf [simon] add
ea635da4 [simon] fix
310ad983 [simon] resultsetmetadata
595294fa [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-22 18:17:08 +08:00
zhenjiaguo
f3dc1fdecd
[KYUUBI #1586] Add time metric on each KyuubiBackendService method 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add time metric on each` KyuubiBackendService` method can help us inspect Kyuubi Server running status inside. It can Indirect reflecting our RPC call time when using Kyuubi.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![Screenshot from 2021-12-19 12-26-39](https://user-images.githubusercontent.com/29809822/146663963-7e483f45-198c-4fd7-b039-211dcf26dde5.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1588 from zhenjiaguo/add_be_method_metric.

Closes #1586

a001eb96 [zhenjiaguo] Add time metric on each KyuubiBackendService method

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-20 09:59:57 +08:00
Wang Zhen
be3a964f37
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add kyuubi.engine.pool.name configuration. #1554

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1572 from wForget/KYUUBI-1554.

Closes #1554

40850f5a [Wang Zhen] [KYUUBI-1554] Add kyuubi.engine.pool.name configuration

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-16 16:33:51 +08:00
simon
bcd8c06e7f
[KYUUBI #1568] Replace sessionDetail with sessionEvent 
### _Why are the changes needed?_
Replace sessionDetail with sessionEvent, because sessionEvent has more message about session.
#1568

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1569 from simon824/api.

Closes #1568

00a0ddfe [simon] Replace sessionDetail with sessionEvent
efc84a2e [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-16 10:10:56 +08:00
simon
22d967e9c0
[KYUUBI #1539] Replace operationDetail with KyuubiEvent 
### _Why are the changes needed?_
#1539
Replace operationDetail with KyuubiEvent

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1540 from simon824/event.

Closes #1539

9d4e2a04 [Simon] Merge branch 'apache:master' into event
0cd7d15a [simon] replace operationDetail with OperationEvent
adf327e8 [simon] replace operationDetail with OperationEvent
06b6ef8c [simon] replace operationDetail with OperationEvent
b0973e20 [simon] replace operationDetail with OperationEvent
cf83fafb [simon] replace operationDetail with OperationEvent
bf955fbf [simon] replace operationDetail with OperationEvent
49c17988 [simon] replace operationDetail with OperationEvent
8990b701 [simon] replace operationDetail with OperationEvent
03937c2c [simon] replace operationDetail with OperationEvent
a3981776 [simon] replace operationDetail with KyuubiEvent
fc5d747d [simon] replace operationDetail with KyuubiEvent
344eba6a [simon] Revert
b4461411 [simon] codestyle
1d758920 [simon] replace operationDetail with KyuubiEvent
258303ae [simon] init

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-15 10:03:05 +08:00
simon
239277e78a
[KYUUBI #1531] Add kyuubi version in engineSpace 
### _Why are the changes needed?_
#1531
Add kyuubi version in engineSpace

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1532 from simon824/ns.

Closes #1531

335906c4 [simon] code
57968cf8 [simon] nameSpace
200f0104 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-12-09 16:40:34 +08:00
simon
81b14eadc8
[KYUUBI #1516] Implement api: /${version}/operations/${operation_identifier} 
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

/${version}/operations/${operation_identifier}
- desc: apply an action for an operation based on a given session identifier and operation identifier
- method: PUT
- params:
- action: "CANCEL" or "ADD" or "CLOSE"
- returns: status code

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1517 from simon824/oa.

Closes #1516

94742c67 [simon] fix desc
fffe3e02 [simon] fix desc
f9415507 [simon] fix
f455995f [simon] operationAction
72bcf5f3 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-08 18:12:02 +08:00
zhenjiaguo
7beee52448
[KYUUBI #1393][TASK-2] Add operation number metric 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Current connection number is already had. I add current operation number.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1485 from zhenjiaguo/add-operation-number.

Closes #1393

beccdfed [zhenjiaguo] when cancel we need decCount the open operation
773d01ee [zhenjiaguo] when cancel we need decCount the open operation
59001995 [zhenjiaguo] change variable name
83f64fb8 [zhenjiaguo] unify operation metric
daf0b6a7 [zhenjiaguo] add operation number

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-08 17:13:30 +08:00
Wang Zhen
16add92299
[KYUUBI #1522] Pass the client ip to engine 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Pass the client ip to engine. For details: #1522

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1524 from wForget/KYUUBI-1522.

Closes #1522

c9d54ed9 [Wang Zhen] fix word error
a16d914d [Wang Zhen] [KYUUBI-1522] Pass the client ip to engine

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-08 17:08:09 +08:00
Wang Zhen
986d9830db
[KYUUBI #1509] Make KYUUBI_WORK_DIR_ROOT as the default root path. 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Make KYUUBI_WORK_DIR_ROOT as the default root path. For details: #1509.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1519 from wForget/KYUUBI-1509.

Closes #1509

ff078e5f [Wang Zhen] [KYUUBI-1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-07 16:05:44 +08:00
simon
4319df744c
[KYUUBI #1501] Introduce operationsResource 
### _Why are the changes needed?_
#1501
Introduce operationsResource
mv parseSessionHandle() to SessionHandle
mv parseOperationHandle() to OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1502 from simon824/operationresource.

Closes #1501

c4b1b646 [simon] fix
72446d36 [simon] introduce operationsResource
06d12014 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-12-06 20:43:26 +08:00
simon
73ac1b63db
[KYUUBI #1445] [Kyuubi #1444] Implement api: /${version}/sessions/${session_identifier}/operations/${operation_identifier} 
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
- desc: get an operation with a given session identifier and operation identifier
- method: GET
- params: none
- returns: an instance of OperationHandle

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
- desc: remove operation based on a given session identifier and operation identifier
- method: DELETE
- params: none
- returns: an instance of OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1445 from simon824/r5.

Closes #1445

196ea4c7 [simon] codestyle
df1c20e8 [simon] remove isHasOperation
6bdebab8 [simon] fix
609aba1b [simon] codestyle
dbae387d [simon] codestyle
7eb9eeb8 [simon] Merge remote-tracking branch 'upstream/master' into r5
8a9e78ca [simon] codestyle
15b00aa9 [simon] operation
b154cbc7 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-06 13:17:33 +08:00
Fei Wang
4648b63a2c [KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1379 from turboFei/async_open_default.

Closes #1379

a7c4c6a1 [Fei Wang] fix ut
cc8d2730 [Fei Wang] Make SESSION_ENGINE_LAUNCH_ASYNC true by default

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2021-12-04 18:30:13 +08:00
Kent Yao
aecf4a8b4e [KYUUBI #1483] Revamp KyuubiSessionImpl and KyuubiSyncThriftClient 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

KyuubiSessionImpl hold the underlying transport of KyuubiSyncThriftClient, this PR move it inside KyuubiSyncThriftClient

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1483 from yaooqinn/KyuubiSyncThriftClient.

Closes #1483

497c5e4e [Kent Yao] address comment
b9f8943b [Kent Yao] Revamp KyuubiSessionImpl and KyuubiSyncThriftClient
78f4bab8 [Kent Yao] Revamp KyuubiSessionImpl and KyuubiSyncThriftClient

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2021-12-02 16:58:47 +08:00
fwang12
a6d23238cb [KYUUBI #1382] Sync the flink-provided profile in docs and testing suite comments 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Sync the flink-provided profile in docs and testing suite comments.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1382 from turboFei/flink_provided.

Closes #1382

3ae96963 [Fei Wang] address nit
ef5bba7f [fwang12] reorder
e9c9c527 [fwang12] save

Lead-authored-by: fwang12 <fwang12@ebay.com>
Co-authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
 2021-12-01 20:33:47 +08:00
zhenjiaguo
5a4e3707ca
[KYUUBI #1449] Fix when KyuubiServer initialize fail but not exit 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Kyuubi server should exit when initialize fail.

Detail see #1449.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1468 from zhenjiaguo/fix-TBFS-init-fail-not-exit.

Closes #1449

4467a249 [zhenjiaguo] add status judgment before zkServer stop
4208663e [zhenjiaguo] stop zkServer when kyuubi server init fail
1ff3a825 [zhenjiaguo] try catch server initialize

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-12-01 18:46:00 +08:00
ulysses-you
9d8476a3f2
[KYUUBI #1471] Improve log if user specify the spark principal and keytab but not same with proxy user 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add the warning log instead of hiding the behavior implicitly.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1471 from ulysses-you/log.

Closes #1471

3c9ae370 [ulysses-you] improve log

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-30 18:13:47 +08:00
zhenjiaguo
795b7e1bb2
[KYUUBI #1459] Kyuubi Server Event Test change 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Use regular matching to replace jackson Json parsing in kyuubi server event info test. Because the Json string is queried by SparkSQL, the special characters will be converted. This causes the configuration of environment variables to be read again and cannot be parsed into JSON.
For example, the environment variable contains the variable `ENV_VAR="\"string\""`, which becomes `ENV_VAR=""string""` after being read by SparkSQL, which cannot be converted into JSON.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1459 from zhenjiaguo/add-kyuubi-server-event.

Closes #1459

0282030a [jia] Merge branch 'master' into add-kyuubi-server-event
8b4f4f1a [zhenjiaguo] use regular replace JSON parser
d97e1d00 [zhenjiaguo] optimize import
56939c13 [zhenjiaguo] change import way
7632363b [zhenjiaguo] rebase master
44559154 [zhenjiaguo] add more info in kyuubi server event
beacc88a [zhenjiaguo] add sparksql case-sensitive setting
b35875fc [zhenjiaguo] add kyuubi server start event test
d3161681 [zhenjiaguo] add KyuubiServerEvent

Lead-authored-by: zhenjiaguo <zhenjia_guo@163.com>
Co-authored-by: jia <zhenjia_guo@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-29 14:13:44 +08:00