<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1 use isEmpty instead show for the init SQLs
2 add job desc for SQL tab of Spark UI
3 make it simple.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#914 from yaooqinn/isempty.
Closes#914
70785da7 [Kent Yao] Use isEmpty to trigger default init queries
438009e4 [Kent Yao] Use isEmpty to trigger default init queries
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#909 from yaooqinn/906.
Closes#906
e202b70d [Kent Yao] nit
1271e841 [Kent Yao] fix test
d696021f [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes
bdd2b875 [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#888 from pan3793/scala.
Closes#888
571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#885 from yaooqinn/engineevent.
Closes#885
6a7ac341 [Kent Yao] refine
41baf0e7 [Kent Yao] refine
2c141a06 [Kent Yao] test history server
d1b0823a [Kent Yao] test history server
3aae8417 [Kent Yao] dump the final engine status
257e50f5 [Kent Yao] Add Event for Engine
f1f1f0e8 [Kent Yao] Add Event for Engine
85addc6e [Kent Yao] Add Event for Engine
cece5888 [Kent Yao] Add Event for Engine
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#765
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#884 from pan3793/jdbc-java.
Closes#884
ca7bad48 [Cheng Pan] Rewrite kyuubi-hive-jdbc in Java
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
**ROOT CAUSE**
When server receive request of close session, will call `FrontendService#CloseSession`. When an error is encountered when closing the session, `CURRENT_SERVER_CONTEXT` will never be called.
```
2021-08-02 09:41:50.945 INFO service.FrontendService: Received request of closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.946 INFO session.KyuubiSessionManager: SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72] is closed, current opening sessions 0
2021-08-02 09:41:50.949 WARN service.FrontendService: Error closing session:
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:89)
at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:74)
at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
at org.apache.kyuubi.service.FrontendService.CloseSession(FrontendService.scala:218)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1397)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1382)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:191)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:178)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:67)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:47)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:67)
at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:86)
... 12 more
2021-08-02 09:41:50.953 INFO service.FrontendService: Finished closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.957 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.963 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.964 WARN service.FrontendService: Failed closing session
org.apache.kyuubi.KyuubiSQLException: Invalid SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:71)
at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
at org.apache.kyuubi.service.FrontendService$FeTServerEventHandler.deleteContext(FrontendService.scala:529)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-02 09:42:30.908 INFO server.ZooKeeperServer: Expiring session 0x1000da7829f0002, timeout of 60000ms exceeded
2021-08-02 09:42:30.909 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1000da7829f0002
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#879 from timothy65535/ky-821.
Closes#821
f485f580 [timothy65535] [KYUUBI #821] Improper exception for closing a non-existent session
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#869 from yaooqinn/ui.
Closes#869
aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui
0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#857 from pan3793/seesion-init-sql.
Closes#857
f6f59196 [Cheng Pan] remove unnecessary change
4960830a [Cheng Pan] run ENGINE_SESSION_INITIALIZE_SQL when engine startup
6e8100fa [Cheng Pan] address comments
08f4af62 [Cheng Pan] address comments
2bb1bb85 [Cheng Pan] fix import
880b38ec [Cheng Pan] session initialize sql will only execute once in single session mode
2a646890 [Cheng Pan] test session initialize sql will not execute in single session mode
f03f3eff [Cheng Pan] update doc
6e4a72f2 [Cheng Pan] address comments
34285e25 [Cheng Pan] format
6d978025 [Cheng Pan] revert log4j conf change
881a89e0 [Cheng Pan] improve test
b9ec18c9 [Cheng Pan] fix ut and add log
b4f586d8 [Cheng Pan] Support init SQL for each session
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We shall avoid calling cancelJobGroup if the sc is already stopped.
We shall stop the engine to stop serving before we stop the sc during the shutdown hook
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#868 from yaooqinn/shutdownhook.
Closes#868
14810875 [Kent Yao] comment
972e2741 [Kent Yao] Avoid calling a stopped SparkContext
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
* Use github action to install `MiniKube`
* Start Kyuubi Server as Pod with `MiniKube`
* Add a new module `kubernetes-integration-tests` to run basic JDBC test
Since there exists many `MiniKube` app in github market, this PR use [Setup Minikube Kubernetes Cluster](https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster) , we can change this if necessary in future.
Note that, this PR is for Kyuubi test with k8s not for Apache Spark.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#853 from ulysses-you/k8s-it.
Closes#444Closes#854
4a73a8fa [ulysses-you] ignore
a501482d [ulysses-you] ignore
242f60c4 [ulysses-you] todo
994e52e4 [ulysses-you] pom
3f60fb2e [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into k8s-it
3febdcd2 [ulysses-you] fix
f564792b [ulysses-you] Add minikuube for k8s integration test
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
https://infra.apache.org/publishing-maven-artifacts.htmlhttps://issues.apache.org/jira/browse/INFRA-22114
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#849 from pan3793/asf.
Closes#849
c609f6ba [Cheng Pan] update pom
31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent
9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
loadFromMap ignored user specific defaults that start with '___'.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#840 from hddong/kyuubi-839.
Closes#839
ba3683e3 [hongdongdong] fix comments
a7753849 [hongdongdong] fix test
07f82667 [hongdongdong] fix test
b2a4c3cb [hongdongdong] [KYUUBI#839]Fix load user specific defaults from properties file
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
add udf system_user.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#758 from zhaomin1423/661_add-system_user.
Closes#758
70eba56a [Min Zhao] update system user name
f5edc621 [Min Zhao] [KYUUBI 661] Add UDF system_user
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Store all jobInfos for each statement in mem.
- First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds.
- Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds.
Some interfaces:
- KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds.
- KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on.
- KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold.
Test case:
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#814 from zhang1002/branch-1.2_add-jobinfo.
Closes#814
7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException
0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle)
bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener
741fcd0f [张宇翔] Add interval
b61b9406 [张宇翔] Code changes
cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code
c1a1d732 [张宇翔] 1. Remove volatile for jobInfo
9547d49e [张宇翔] 1. Remove volatile for jobInfo
970c0002 [张宇翔] 1. Add volatile for jobInfo
62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations
db1de381 [张宇翔] 1. Add some log 2. Add some annotations
8deb61f2 [张宇翔] change some test
87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test
86c49ebc [张宇翔] Merge master branch
a0a99b3a [张宇翔] Merge master branch
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5d3b9afb [张宇翔] Code optimization
db3a0b6f [张宇翔] Format changes
08d6d1fc [张宇翔] Modify some annotations
f62f00e9 [张宇翔] Event Tracking: For job
85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo
3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing
b4290bfa [张宇翔] Event tracking: For jobInfo
fe1f7cce [张宇翔] Change directory structure
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
71d33b9a [张宇翔] Add some comment
6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment
f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo
8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code
f43b3c8f [张宇翔] merge master branch
55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor
6f0be547 [张宇翔] format change
bbfba274 [张宇翔] Even tracking
5a62f586 [张宇翔] remove some unused conf
b26345ac [张宇翔] Merge master branch and resolve conflict
fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished
d1676268 [张宇翔] event tracking: for SQLExecutionStart
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix wrong hint in test for update doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#827 from zhaomin1423/kyuubi_749.
Closes#749
a2b10247 [Min Zhao] fix wrong hint in test for update doc
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Add a sync client to ensure all requests sent to engine within a session are in order
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#819 from yaooqinn/283.
Closes#283
16cb8632 [Kent Yao] [KYUUBI #283] Add a sync client to ensure all requests in order
8bb2c81f [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order
5c73fd0b [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
See issue #817 , when we pull the project and do not invoke any build action, e.g. `mvn package` or `mvn generate-resources`, we would miss the `kyuubi-version-info.properties` file, it caused the NPE.
As discussed under issue #817 , this PR tries to check it and enhance the exception message to make it more friendly to newbie developers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
- [ ] Verified in the local env and no need to add test cases.
Closes#820 from yanghua/KYUUBI-817.
Closes#820
9464d07b [yanghua] [KYUUBI 817] Fix KyuubiDriverSuite UT failure because of missing kyuubi-version-info.properties
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
Fix requestTimeoutUnit in FrontendService #813
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#816 from wuchunfu/813.
Closes#816
d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 0a87a254f7)
Signed-off-by: Kent Yao <yao@apache.org>
Fix requestTimeoutUnit in FrontendService #813
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#816 from wuchunfu/813.
Closes#816
d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
#751
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#752 from wForget/dev.
Closes#751
ae9dc873 [wForget] [KYUUBI #751] Support single session mode
Authored-by: wForget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. reduce one server bootstrap
2. remove explicitly hanging a test for 1 min long to wait for an unnecessary message
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#761 from yaooqinn/test2.
Closes#761
1d5c8cb4 [Kent Yao] style
e877348a [Kent Yao] simpler
7ade8ead [Kent Yao] simpler
46a11b8c [Kent Yao] Reduce explicit hang for test
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Close#550, relate to #559#688#689.
Add `javax.xml.bind:jaxb-api` and `javax.activation:activation` since those apis has been removed by JDK 11.
In GA, we saw a significant performance downgrade in the unit tests, and even with some changes like enlarging timeout, using in-memory catalog instead of hive catalog, the JDK11(~43min) tests cost 2 times than JDK8(~22min), this performance regression is tracked in #745.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#693 from pan3793/jdk11.
Closes#693
5706daf1 [Cheng Pan] shrink scalatest memory
6ebac68e [Cheng Pan] address comments
2b479391 [Cheng Pan] address comments
5fa74aa0 [Cheng Pan] disable test in kyuubi-extension-spark_3.1
9f070a40 [Cheng Pan] maven profile for jdk versions
801e4cd1 [Cheng Pan] jdk11 in GA
e86893cc [Cheng Pan] enlarge YarnCluster timeout
c880edd1 [Cheng Pan] minor
38d2775c [Cheng Pan] enlarge engine crash timeout
a945ade3 [Cheng Pan] use in-memory catalog in EngineRefSuite
51604d73 [Cheng Pan] enlarge ENGINE_INIT_TIMEOUT
7baeb965 [Cheng Pan] minicluster
a7e4bdf1 [Cheng Pan] GA matrix
b29aa2f4 [Cheng Pan] license for jaxb
4283777b [Cheng Pan] Add dependency jaxb-api
c5b0e580 [Cheng Pan] [GA] Enable JDK 11 in GA
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Follow #670 to:
Apply errorCode 0A000: feature_not_supported to val errStatus = KyuubiSQLException("Feature is not available").toTStatus
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#729 from zhaomin1423/728.
Closes#728
396e70d8 [Min Zhao] update FrontendServiceSuite to adapt KyuubiSQLException.featureNotSupported
c06dbfd2 [Min Zhao] invoke KyuubiSQLException.featureNotSupported() in FrontendService
281aed97 [Min Zhao] add featureNotSupported to KyuubiSQLException object
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
move log 'Closing ...' to correct line
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#748 from zhaomin1423/log_fix.
Closes#748
4a433dd3 [Min Zhao] move log 'Closing ...' to correct line
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. add '=' in start of line to keep the consistent code style.
2. update log info to correct position.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#744 from zhaomin1423/log_update.
Closes#744
bf1e0d8a [Min Zhao] update log 'Closing ...' to corrent position and add 'Closed...'
6907430f [Min Zhao] update log info to keep the consistent code style
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix using the ThreadAudit in the KyuubiFunSuite. The method of doThreadPostAudit is used in the beforeAll and afterAll, I think the doThreadPreAudit should be used in the beforeAll.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#743 from zhaomin1423/bug_fix.
Closes#743
3980c2d4 [Min Zhao] fix using the ThreadAudit in the KyuubiFunSuite
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Rename `KyuubiOperationHiveSuite` to `KyuubiOperationHiveCatalogSuite`;
Simplify redundant case in `KyuubiOperationPerConnectionSuite`;
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#741 from pan3793/suite.
Closes#741
abea3aaf [Cheng Pan] code style
81ada061 [Cheng Pan] refine test suites
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
delete redundant empty line
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#730 from zhaomin1423/code_style.
Closes#730
33ec472b [Min Zhao] delete redundant empty line
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Followup #707, use in-memory catalog in most test suites to avoid derby lock to speed up test suites.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#722 from pan3793/catalog.
Closes#722
fcc38af6 [Cheng Pan] fix ut
0c131629 [Cheng Pan] trait HiveJDBCTests extends BasicJDBCTests
19bb4237 [Cheng Pan] [TEST] Use in-memory catalog in most test suites
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Save Hadoop conf file instead of appending all of them to connection url
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#696 from turboFei/mini_cluster_core-site.
Closes#696
6cc24aec [fwang12] address comments
ff7ffb39 [fwang12] refactor
f47494be [fwang12] use configuration.writeXml
b6fcc372 [fwang12] Save hadoop conf to file instead of appending all conf to jdbc url
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#692 from pan3793/construct.
Closes#692
9803aee [Cheng Pan] Simplify construct of KyuubiException
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
Enable transfer some env variables to engine launcher process
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#687 from turboFei/spark_proc_env.
Closes#687
262c320 [fwang12] address comments
b2f3539 [fwang12] use kyuubi engine env instead of env list
812b0ec [fwang12] move to general
e664814 [fwang12] Enable transfer env map to engine launcher process
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make JDK11 work step by step, in this PR, we fix the issue in `KyuubiSQLExceptionSuite` and report a new one in MetricsSystemSuite
close#559 and file a new ticket to track #689
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#688 from yaooqinn/t.
Closes#551
2e8591b9 [Kent Yao] [JDK11] Fix KyuubiSQLExceptionSuite
f4cb40f6 [Kent Yao] [JDK11] Fix KyuubiSQLExceptionSuite
6452d18a [Kent Yao] Merge branch 'master' into t
511307bd [hzmintao] #551 TEST JDK 11 in travis
Lead-authored-by: hzmintao <mintaoisjack@163.com>
Co-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: hzmintao <hzmintao@corp.netease.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
In this PR, I propose to add kyuubi_version as a user-defined function.
This is also a good example to add other new functions that needed in the Kyuubi system.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#686 from yaooqinn/657.
Closes#657
d30ac8f6 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version
e1e585e9 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
#670
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#674 from pan3793/sql-error-code.
Closes#670
08c95915 [Cheng Pan] address comments
e0808bc5 [Cheng Pan] address comments
e14e884b [Cheng Pan] KyuubiSQLException support sqlState and errorCode
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#669 from pan3793/cleanup.
Closes#669
2cdc4a3 [Cheng Pan] Cleanup code and fix typos
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
When run on k8s, engine write hostname to zk. But, server can not connect pod by hostname, we should use ip instead.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#663 from hddong/fix-k8s-connect.
Closes#662
385ba6b [hongdongdong] fix test
370201d [hongdongdong] fix comments and add test
736261a [hongdongdong] [KYUUBI #662]Fix run on k8s
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support test with mini yarn cluster.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#538 from turboFei/KYUUBI_527_mini_yarn.
Closes#527
8bedc707 [fwang12] use eventually
89794bc6 [fwang12] refactor
b955b825 [fwang12] refactor code
2f2baacf [fwang12] save
0d6359ea [fwang12] remove scope in parent pom
442c2c42 [fwang12] [KYUUBI #527] Support test with MiniYARNCluster
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Users may have their own authentication system
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#642 from hddong/support-custom-auth.
Closes#641
b4ce0a8 [hongdongdong] fix
fb2752b [hongdongdong] fix test
dbb4a96 [hongdongdong] fix test
15c93cc [hongdongdong] [KYUUBI #641]Support custom authentication
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
When you stop the KyuubiServiceDiscovery, you need to close node in zk and delete localNodePath at first. But when EmbeddedZookeeper is closed before KyuubiServiceDiscovery, the operation that stop kyuubiServer will timeout.
Now use 'org.apache.hadoop.util.ShutdownHookManager' to close each service by different priority.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#632 from zhang1002/dev.
Closes#632
ee21e68 [张宇翔] Change the relevant format
f68fc7a [张宇翔] Merge branch 'master' into dev
128f244 [张宇翔] solve the timeout problem when stop the kyuubiServer
79b3d5a [张宇翔] ## solve the folder naming problem For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
Lead-authored-by: 张宇翔 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhangyuxiang03@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid.
Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached.
With multiple engines support, end-users are able to
1. create as many engines as they want to meet the concurrency.
2. leverage different queue/namespace resource with long-running mode
3. can give different resources or priorities to their own engines for different workloads
4. can perform fast failover by creating a new sub domain
5. etc.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#627 from yaooqinn/multipleengines.
Closes#627
9f04bc3e [Kent Yao] fix tests
d7af751e [Kent Yao] address comments
be9c46d8 [Kent Yao] address comments
ccce87d7 [Kent Yao] address comments
d49116c7 [Kent Yao] Support multiple engines for non-connection share levels
f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Now we only get 'operation.idle.timeout' from user request. Kyuubi server-side should support "operation.idle. timeout".
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#626 from Honglun/feature/operation-timeout.
Closes#625
e56c11b [Honglun] Kyuubi server-side support 'operation.idle. timeout'
Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make `*SuiteMixin` resuable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#619 from pan3793/mixin.
Closes#619
fa776869 [Cheng Pan] address comments
54ac9a19 [Cheng Pan] Revert "gather KyuubiConf"
92985964 [Cheng Pan] gather KyuubiConf
61492616 [Cheng Pan] extract DataLakeSuiteMixin
66b6ffa4 [Cheng Pan] [TEST] Refactor data lake test suite hierarchical
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
