<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Imporve perf of generate config docs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#901 from ulysses-you/enhance-comment.
Closes#901
a86a9aa6 [ulysses-you] skip-download
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
```
export RELEASE_VERSION=1.3.0-SNAPSHOT # keep same version with pom.xml
export SKIP_GPG=false # optional, default false
build/release/create-package.sh source
build/release/create-package.sh binary
```
```
➜ apache-kyuubi git:(asf-pub) ls -lh work/release
total 161256
-rw-r--r-- 1 chengpan staff 60M Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz
-rw-r--r-- 1 chengpan staff 488B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.asc
-rw-r--r-- 1 chengpan staff 212B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.sha512
-rw-r--r-- 1 chengpan staff 18M Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz
-rw-r--r-- 1 chengpan staff 488B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.asc
-rw-r--r-- 1 chengpan staff 215B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.sha512
```
Planning to add release script and release doc in following PRs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#889 from pan3793/asf-package.
Closes#889
0aef5d77 [Cheng Pan] Update docs
4b0be7da [Cheng Pan] [BUILD] Package release tarball
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
When start the server, If not specified `SPARK_CONF_DIR`, it should output default.
```
KYUUBI_WORK_DIR_ROOT: /root/tmp/kyuubi/work
SPARK_HOME: /root/package/spark-3.1.2-bin-hadoop2.7
SPARK_CONF_DIR:
HADOOP_CONF_DIR: /root/package/hadoop-3.2.0/etc/hadoop
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#896 from timothy65535/ky-895.
Closes#895
9a936a77 [timothy65535] [KYUUBI #895] Enhance output uses the default spark conf
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
change the k8s-it serviceaccount to kyuubi instead of default
### _How was this patch tested?_
Pass CI
Closes#890 from ulysses-you/fix-minikube-test.
Closes#890
5844c482 [ulysses-you] kyuubi
16192d6e [ulysses-you] test
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#888 from pan3793/scala.
Closes#888
571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#885 from yaooqinn/engineevent.
Closes#885
6a7ac341 [Kent Yao] refine
41baf0e7 [Kent Yao] refine
2c141a06 [Kent Yao] test history server
d1b0823a [Kent Yao] test history server
3aae8417 [Kent Yao] dump the final engine status
257e50f5 [Kent Yao] Add Event for Engine
f1f1f0e8 [Kent Yao] Add Event for Engine
85addc6e [Kent Yao] Add Event for Engine
cece5888 [Kent Yao] Add Event for Engine
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#887 from pan3793/publish-snapshot.
Closes#887
5b2044a1 [Cheng Pan] Pin ubuntu-20.04 in GA
98945a12 [Cheng Pan] Remove Kyuubi local cache before deploy
f29cb6e4 [Cheng Pan] [GA] Daily publish snapshot
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
`kyuubi-extension-spark_3.1` is conflict with the scala naming convention, e.g. `spark-core_2.12`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#882 from pan3793/rename-spark-3-1.
Closes#882
d3297409 [Cheng Pan] Rename kyuubi-extension-spark_3.1 to kyuubi-extension-spark-3-1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#765
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#884 from pan3793/jdbc-java.
Closes#884
ca7bad48 [Cheng Pan] Rewrite kyuubi-hive-jdbc in Java
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Add kubernetes configmap yaml file
```
kubectl apply -f docker/kyuubi-configmap.yaml
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#878 from timothy65535/ky-877.
Closes#877
dff774de [timothy65535] improve patch
79e79d03 [timothy65535] keep origin configmap usage
ed28c78a [timothy65535] [KYUUBI #877] Add kubernetes configmap yaml file
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#881 from pan3793/ga-release.
Closes#881
a1f254c9 [Cheng Pan] [GA] Remove GA Release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#880 from pan3793/ga-sl-scan.
Closes#880
0f66dbce [Cheng Pan] [GA] Remove SL Scan
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
**ROOT CAUSE**
When server receive request of close session, will call `FrontendService#CloseSession`. When an error is encountered when closing the session, `CURRENT_SERVER_CONTEXT` will never be called.
```
2021-08-02 09:41:50.945 INFO service.FrontendService: Received request of closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.946 INFO session.KyuubiSessionManager: SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72] is closed, current opening sessions 0
2021-08-02 09:41:50.949 WARN service.FrontendService: Error closing session:
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:89)
at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:74)
at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
at org.apache.kyuubi.service.FrontendService.CloseSession(FrontendService.scala:218)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1397)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1382)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:191)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:178)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:67)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:47)
at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:67)
at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:86)
... 12 more
2021-08-02 09:41:50.953 INFO service.FrontendService: Finished closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.957 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.963 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.964 WARN service.FrontendService: Failed closing session
org.apache.kyuubi.KyuubiSQLException: Invalid SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:71)
at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
at org.apache.kyuubi.service.FrontendService$FeTServerEventHandler.deleteContext(FrontendService.scala:529)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-02 09:42:30.908 INFO server.ZooKeeperServer: Expiring session 0x1000da7829f0002, timeout of 60000ms exceeded
2021-08-02 09:42:30.909 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1000da7829f0002
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#879 from timothy65535/ky-821.
Closes#821
f485f580 [timothy65535] [KYUUBI #821] Improper exception for closing a non-existent session
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#869 from yaooqinn/ui.
Closes#869
aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui
0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
As we will release soon, it's better to format the module name and artifactId.
The name of kyuubi-main sounds like a bootstrap of Kyuubi but not a server.
### _How was this patch tested?_
Pass CI
Closes#872 from ulysses-you/kyuubi-server.
Closes#871
1437f9b5 [ulysses-you] address comment
117f2c7c [ulysses-you] server
86986baa [ulysses-you] master
eba660a2 [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into kyuubi-server
e649012b [ulysses-you] change module name
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#857 from pan3793/seesion-init-sql.
Closes#857
f6f59196 [Cheng Pan] remove unnecessary change
4960830a [Cheng Pan] run ENGINE_SESSION_INITIALIZE_SQL when engine startup
6e8100fa [Cheng Pan] address comments
08f4af62 [Cheng Pan] address comments
2bb1bb85 [Cheng Pan] fix import
880b38ec [Cheng Pan] session initialize sql will only execute once in single session mode
2a646890 [Cheng Pan] test session initialize sql will not execute in single session mode
f03f3eff [Cheng Pan] update doc
6e4a72f2 [Cheng Pan] address comments
34285e25 [Cheng Pan] format
6d978025 [Cheng Pan] revert log4j conf change
881a89e0 [Cheng Pan] improve test
b9ec18c9 [Cheng Pan] fix ut and add log
b4f586d8 [Cheng Pan] Support init SQL for each session
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#870 from pan3793/asf-misc.
Closes#870
ef1ca020 [Cheng Pan] [MISC] Migrate from NetEase to Apache
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We shall avoid calling cancelJobGroup if the sc is already stopped.
We shall stop the engine to stop serving before we stop the sc during the shutdown hook
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#868 from yaooqinn/shutdownhook.
Closes#868
14810875 [Kent Yao] comment
972e2741 [Kent Yao] Avoid calling a stopped SparkContext
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Now, we need to rebuild docker image when config changed. Use configmap to save kyuubi config file, then we can change config by edit configmap and restart pod.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#866 from hddong/kyuubi-859.
Closes#859
caae67d1 [hongdongdong] fix comments
13fc08aa [hongdongdong] [KYUUBI#859]Add kubernetes configmap
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The daemonset may encounter some pontential issue during long-time running. It should be robust to catch `NonFatal` exception with it.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#864 from ulysses-you/block-cleaner.
Closes#864
d3e9dd00 [ulysses-you] dir
46ec3fc0 [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
* Use github action to install `MiniKube`
* Start Kyuubi Server as Pod with `MiniKube`
* Add a new module `kubernetes-integration-tests` to run basic JDBC test
Since there exists many `MiniKube` app in github market, this PR use [Setup Minikube Kubernetes Cluster](https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster) , we can change this if necessary in future.
Note that, this PR is for Kyuubi test with k8s not for Apache Spark.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#853 from ulysses-you/k8s-it.
Closes#444Closes#854
4a73a8fa [ulysses-you] ignore
a501482d [ulysses-you] ignore
242f60c4 [ulysses-you] todo
994e52e4 [ulysses-you] pom
3f60fb2e [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into k8s-it
3febdcd2 [ulysses-you] fix
f564792b [ulysses-you] Add minikuube for k8s integration test
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Revise the unit test for "add kyuubiStatementInfo into queue"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#862 from zhang1002/update_statementinfo_ut.
Closes#862
7e5f8b50 [张宇翔] Revise the unit test for "add kyuubiStatementInfo into queue"
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
https://infra.apache.org/publishing-maven-artifacts.htmlhttps://issues.apache.org/jira/browse/INFRA-22114
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#849 from pan3793/asf.
Closes#849
c609f6ba [Cheng Pan] update pom
31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent
9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
`kubectl port-forward` usually use to test, use `NodePort` to visit kyuubi service in pod.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#858 from hddong/kyuubi-844.
Closes#844
d1492262 [hongdongdong] modify comments
72d51d61 [hongdongdong] fix comments
91727986 [hongdongdong] fix
a169332e [hongdongdong] fix comments
74cec930 [hongdongdong] [KYUUBI#844]Use NodePort to expose kyuubi port on k8s
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Enhacce Spark Block Cleaner loginfo
* Unite timeunit
* Remove period
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#856 from zwangsheng/enhance/block-cleaner.
Closes#856
c9b3fda8 [Binjie Yang] timeunit
124d1566 [Binjie Yang] enhance spark block cleaner loginfo
Authored-by: Binjie Yang <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
Fixes#842
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### 1. Describe the bug
Meet rat check error when build kyuubi project
```
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (default-cli) on project kyuubi: Too many files with unapproved license: 75 See RAT report in: /work/projects/opensource/kyuubi/target/rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
```
### 2. Steps to reproduce
1. Build kyuubi project
```
./build/dist --tgz --spark-provided -Pkyuubi-extension-spark_3.1
```
2. Untar dist package and check some jars
3. Build kyuubi again
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
```
mvn org.apache.rat:apache-rat-plugin:0.13:check
```
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
```
74 Unknown Licenses
*****************************************************
Files with unapproved licenses:
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-hadoop3.2/data/mllib/ridge-data/lpsa.data
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-hadoop3.2/data/mllib/als/test.data
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/sparkr-vignettes.R
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/index.html
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/externals/spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/R/SparkR
kyuubi-1.3.0-SNAPSHOT-bin-spark-3.1-hadoop3.2/RELEASE
kyuubi-1.3.0-SNAPSHOT-bin-without-spark/RELEASE
*****************************************************
```
Closes#843 from timothy65535/ky-842.
Closes#842
a236167d [timothy65535] improve rat exclude list
a7bc4fb0 [timothy65535] [KYUUBI #842] Fix meet rat check error when build kyuubi project
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
loadFromMap ignored user specific defaults that start with '___'.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#840 from hddong/kyuubi-839.
Closes#839
ba3683e3 [hongdongdong] fix comments
a7753849 [hongdongdong] fix test
07f82667 [hongdongdong] fix test
b2a4c3cb [hongdongdong] [KYUUBI#839]Fix load user specific defaults from properties file
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
### _Why are the changes needed?_
Spark-Block-Cleaner log files are not governed by itself.
Add log4j-defaults.properties to help input both console and file.
And use rollingFileApeender to split big logfile to small logFiles.
Create a log file under /logs/spark-block-cleaner-log/ so that you can manage the log yourself with the ability to clear the folder leading to `spark-`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#831 from zwangsheng/feature/block-cleaner.
Closes#828
83dc639e [ulysses] nit
5458f84b [Binjie Yang] make init log4j static
f0d65dd7 [Binjie Yang] import format
48514496 [Binjie Yang] optimize properties
52713f1e [Binjie Yang] assign log4j.properties
a8723f1a [Binjie Yang] fix
4d44a033 [Binjie Yang] use rollingfileappender
Lead-authored-by: Binjie Yang <2213335496@qq.com>
Co-authored-by: ulysses <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
add udf system_user.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#758 from zhaomin1423/661_add-system_user.
Closes#758
70eba56a [Min Zhao] update system user name
f5edc621 [Min Zhao] [KYUUBI 661] Add UDF system_user
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Store all jobInfos for each statement in mem.
- First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds.
- Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds.
Some interfaces:
- KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds.
- KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on.
- KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold.
Test case:
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#814 from zhang1002/branch-1.2_add-jobinfo.
Closes#814
7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException
0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle)
bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener
741fcd0f [张宇翔] Add interval
b61b9406 [张宇翔] Code changes
cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code
c1a1d732 [张宇翔] 1. Remove volatile for jobInfo
9547d49e [张宇翔] 1. Remove volatile for jobInfo
970c0002 [张宇翔] 1. Add volatile for jobInfo
62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations
db1de381 [张宇翔] 1. Add some log 2. Add some annotations
8deb61f2 [张宇翔] change some test
87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test
86c49ebc [张宇翔] Merge master branch
a0a99b3a [张宇翔] Merge master branch
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5d3b9afb [张宇翔] Code optimization
db3a0b6f [张宇翔] Format changes
08d6d1fc [张宇翔] Modify some annotations
f62f00e9 [张宇翔] Event Tracking: For job
85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo
3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing
b4290bfa [张宇翔] Event tracking: For jobInfo
fe1f7cce [张宇翔] Change directory structure
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
71d33b9a [张宇翔] Add some comment
6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment
f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo
8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code
f43b3c8f [张宇翔] merge master branch
55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor
6f0be547 [张宇翔] format change
bbfba274 [张宇翔] Even tracking
5a62f586 [张宇翔] remove some unused conf
b26345ac [张宇翔] Merge master branch and resolve conflict
fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished
d1676268 [张宇翔] event tracking: for SQLExecutionStart
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#798
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#824 from pan3793/release.
Closes#824
5339d76c [Cheng Pan] remove spark bin tgz from license-binary
bfa484e6 [Cheng Pan] [GA] Remove spark-bundled binary distributions in GA release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
fix wrong hint in test for update doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#827 from zhaomin1423/kyuubi_749.
Closes#749
a2b10247 [Min Zhao] fix wrong hint in test for update doc
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#823 from pan3793/deps.
Closes#823
51cdcb5d [Cheng Pan] restore thrift deps from spark-hive
148038a3 [Cheng Pan] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Add a sync client to ensure all requests sent to engine within a session are in order
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#819 from yaooqinn/283.
Closes#283
16cb8632 [Kent Yao] [KYUUBI #283] Add a sync client to ensure all requests in order
8bb2c81f [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order
5c73fd0b [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
See issue #817 , when we pull the project and do not invoke any build action, e.g. `mvn package` or `mvn generate-resources`, we would miss the `kyuubi-version-info.properties` file, it caused the NPE.
As discussed under issue #817 , this PR tries to check it and enhance the exception message to make it more friendly to newbie developers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
- [ ] Verified in the local env and no need to add test cases.
Closes#820 from yanghua/KYUUBI-817.
Closes#820
9464d07b [yanghua] [KYUUBI 817] Fix KyuubiDriverSuite UT failure because of missing kyuubi-version-info.properties
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Docs for kyuubi tools spark-block-cleaner.
* Explain the parameters
* Introduction to basic startup
* Give an example
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#815 from zwangsheng/doc/spark_block_cleaner.
Closes#815
1ec6795f [Binjie Yang] delete todo
bbf4d6e2 [Binjie Yang] make it common
9cf3e159 [Binjie Yang] format
0803995a [Binjie Yang] straighten out the article
f834b382 [Binjie Yang] refactor
25be318f [Binjie Yang] fix
7304e595 [Binjie Yang] docs for spark-block-cleaner
Authored-by: Binjie Yang <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
Fix requestTimeoutUnit in FrontendService #813
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#816 from wuchunfu/813.
Closes#816
d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 0a87a254f7)
Signed-off-by: Kent Yao <yao@apache.org>
Fix requestTimeoutUnit in FrontendService #813
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#816 from wuchunfu/813.
Closes#816
d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Enable code coverage report
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#811 from yaooqinn/codecov.
Closes#811
90d9f002 [Kent Yao] Update codecov token
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make the `dev/merge_kyuubi_pr.py` work as expected on reopened PR.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Test passed on merge the reopened PR #807.
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#809 from pan3793/merge_pr.
Closes#809
fd7176f8 [Cheng Pan] [INFRA] Support reopen pr in pr merge tool
cc02b5e2 [Cheng Pan] [INFRA] Fix typo in merge_kyuubi_pr script
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Make this extesion more robust.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#807 from ulysses-you/sql-extension.
Closes#807
27155c1e [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
