### _Why are the changes needed?_
Hive Backend Engine - maven-google-downloader plugin support for hive distribution.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2137 from SteNicholas/KYUUBI-2022.
Closes#2022
27df3241 [SteNicholas] [KYUUBI #2022] Hive Backend Engine - maven-google-downloader plugin support for hive distribution
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2084
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2091 from jiaoqingbo/2084.
Closes#2084
624216fa [Cheng Pan] permission
c45a71ce [jiaoqingbo] delete empty line
29987bf1 [jiaoqingbo] fix ut failed
2e7f0acc [jiaoqingbo] Merge branch 'master' into 2084
04c0f8e1 [jiaoqingbo] Merge branch 'master' into 2084
1267d195 [jiaoqingbo] Merge branch '2084' of https://github.com/jiaoqingbo/incubator-kyuubi into 2084
1bb2b6d6 [jiaoqingbo] Merge branch 'master' into 2084
db4a4aac [jiaoqingbo] Merge branch 'master' into 2084
82097388 [jiaoqingbo] ignore java/sun prefix
d4640ec3 [jiaoqingbo] add log
d5af758f [jiaoqingbo] modify ut
0f3af6c6 [jiaoqingbo] modify ut case
9ffb4177 [jiaoqingbo] Merge branch 'master' into 2084
52dd5090 [jiaoqingbo] modify ut
3f881748 [jiaoqingbo] modify test conf
705f94af [jiaoqingbo] modify test conf
096fba1d [jiaoqingbo] modify test conf
83f1aba9 [jiaoqingbo] add more example configuration
d2581534 [jiaoqingbo] fix compile problem
65c3198a [jiaoqingbo] [KYUUBI #2084] Support arbitrary parameters for KyuubiConf
Lead-authored-by: jiaoqingbo <1178404354@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/2112
Clients less than version 2.1 have no [HIVE-4924](https://issues.apache.org/jira/browse/HIVE-4924) Patch, no `queryTimeout `parameter and no `TIMEOUT` status. When the server enables `kyuubi.operation.query.timeout`, this will cause the client of the lower version to get stuck.
Check thrift protocol version <= `HIVE_CLI_SERVICE_PROTOCOL_V8`, convert `TIMEDOUT_STATE` to `CANCELED`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2113 from cxzl25/KYUUBI-2112.
Closes#2112
c50ab2c8 [sychen] fix style
76224707 [sychen] add comment
d07c945a [sychen] fix style
dbaeb1f5 [sychen] use protocol version
46307b1f [sychen] make it simple
ce9d75f9 [sychen] fix UT
e73fd744 [sychen] add UT
147059fa [sychen] add config description to documentation
0e4e3af2 [sychen] fix style
6d226aa3 [sychen] queryTimeout compatibility
Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add `Deploy Kyuubi Flink engine on Yarn`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2131 from SteNicholas/KYUUBI-1866.
Closes#1866
ba639f15 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
cc6f4d44 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now Spark supports RenewDelegationToken, but there is no log on the driver side when it is actually updated.
https://github.com/apache/incubator-kyuubi/issues/2120
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2121 from cxzl25/KYUUBI-2120.
Closes#2120
ef11f062 [sychen] add RenewDelegationToken log
Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
…initialization times out and yarn application status is accepted
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2110 from 942011334/KYUUBI-2104.
Closes#2104
6bb4f37c [jiadongdong] [KYUUBI #2104] Kill yarn job using yarn client API when kyuubi engine initialization times out and yarn application status is accepted
38118aaf [jiadongdong] [KYUUBI #2104] Kill yarn job using yarn client API when kyuubi engine initialization times out and yarn application status is accepted
2db9c458 [jiadongdong] Merge branch 'KYUUBI-2104' of https://github.com/942011334/incubator-kyuubi into KYUUBI-2104
549fab4d [jiadongdong] [KYUUBI #2104] Kill yarn job using yarn client API when kyuubi engine initialization times out and yarn application status is accepted
65c6c5cf [jiadongdong] [KYUUBI #2104] Kill yarn job using yarn client API when kyuubi engine initialization times out and yarn application status is accepted
28665f3d [jiadongdong] [Kyuubi#2104] Kill yarn job using yarn client API when kyuubi engine initialization times out and yarn application status is accepted
Authored-by: jiadongdong <jiadongdong@bilibili.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Adding and removing jars for a session at runtime is crucial for using UDFs and connectors, which we should support.
This is a sub-task of KPIP-2 https://github.com/apache/incubator-kyuubi/issues/1322.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2118 from link3280/KYUUBI-2002.
Closes#2118
fd3a4b47 [Paul Lin] [KYUUBI #2002] Support show jars operations in PlanOnlyStatement
56cdd12e [Paul Lin] [KYUUBI #2002] Support show jars operations
fbdb009e [Paul Lin] [KYUUBI #2002] Support add/remove jar operations
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
The HA_ZK_NAMESPACE and HA_ZK_ENGINE_REF_ID are set repetitively when create flink sql engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2128 from zhaomin1423/fix_set_conf.
Closes#2127
402f2f11 [Min Zhao] [KYUUBI #2127] avoid to set HA_ZK_NAMESPACE and HA_ZK_ENGINE_REF_ID repetitively when create flink sql engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/2123
Now after the openEngineSession call fails, there is no engine information output, and it is more difficult to know which engine has the problem in pool mode.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2124 from cxzl25/KYUUBI-2123.
Closes#2123
bc0d90cb [sychen] fix style
ffa4d77b [sychen] log engine info
Authored-by: sychen <sychen@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/2125
Now after the `openSession` call fails, `closeSession` is called. But because there is no `_remoteSessionHandle`, the `CloseSession` RPC request fails.
This does not need to send RPC and also increases the complexity of logging.
```
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:156)
at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:75)
at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45)
at org.apache.kyuubi.service.ThriftBinaryFrontendService.getSessionHandle(ThriftBinaryFrontendService.scala:199)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2126 from cxzl25/KYUUBI-2125.
Closes#2125
2e96f418 [sychen] optimize closeSession
Authored-by: sychen <sychen@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Move toString() to ProcBuilder trait from its implements because of their logic is same.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2117 from zhaomin1423/impr_procBuilder.
Closes#2116
0e29b7aa [Min Zhao] [Kyuubi #2116] move toString() to ProcBuilder trait from its implements
612d7a65 [Min Zhao] [Kyuubi #2116] move toString() to ProcBuilder trait from its implements
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093
We need more time to evaluate it and add more test cases to ensure stability before the next release.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2103 from pan3793/revert.
Closes#2103Closes#1948
575dfe8d [Cheng Pan] properties order
4c94481e [Cheng Pan] fix deps
bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test
bad4e90e [Cheng Pan] surefire
f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2"
cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf"
52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes"
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add logging of Flink SQL Engine in `logging.md`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2111 from SteNicholas/KYUUBI-1866.
Closes#1866
7a0bdbb8 [SteNicholas] [KYUUBI #1866] Add logging of Flink SQL Engine
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add quick start documents of the Flink SQL Engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2106 from deadwind4/KYUUBI-1866-quickstart.
Closes#1866
2533aafd [Ada Wong] remove Yarn section
6aa4db8a [Ada Wong] compress png
ff6bff72 [Ada Wong] [KYUUBI #1866][DOCS] Add flink sql engine quick start
Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add description about trino in the config of engine.type.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2109 from zhaomin1423/fix_engine_type_doc.
Closes#2108
adf898c0 [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type
8f11514e [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Using while-loop instead of map/range to improve performance in RowSet
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2107 from zhaomin1423/improve_loop.
Closes#2071
b6d70b3a [Min Zhao] Merge branch 'improve_loop' of github.com:zhaomin1423/kyuubi into improve_loop
43bca772 [Min Zhao] [Kyuubi #2071] Using while-loop instead of map/range to improve performance in RowSet
ec2cb912 [Min Zhao] [Kyuubi #2071] Using while-loop instead of map/range to improve performance in RowSet
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support submitting a job to Yarn. The 'flink-sql-client.jar' has supported submitting a job to Yarn. We only make a few changes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2074 from deadwind4/KYUUBI-2070.
Closes#2070
b9223f4a [Ada Wong] remove ${HADOOP_CLASSPATH} for supporting only yarn-session
59d692fc [Ada Wong] update HADDOP_CLASSPATH
c0496e91 [Ada Wong] [KYUUBI #2070][FLINK] Support Flink job submission on yarn-session mode
Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add debugging instructions for Flink engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2098 from SteNicholas/KYUUBI-2089.
Closes#2089
08189b85 [SteNicholas] [KYUUBI #2089] Add debugging instructions for flink engine
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Upload Test Log for CI failure shall contain trino engine log.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2097 from zhaomin1423/update_ci.
Closes#2097
5796174a [Min Zhao] [CI] Upload Test Log for CI failure shall contain trino engine log #2094
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Remove useless logic about add conf when create a new engine
This code worked before commit [575e9c9599] which unset `FRONTEND_THRIFT_BINARY_BIND_HOST` when copy conf.
The code which removed in this PR, only help kyuubi on k8s with spark client mode. Can be replace by set `spark.driver.host`.
Considering this approach is not recommended, so remove this code in this PR.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2095 from zwangsheng/fix/remove_useless_logic.
Closes#2095
6cd89d0d [zwangsheng] remove useless logic
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We should relocate fb303 classes to avoid conflict, fix#2093.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2096 from SteNicholas/KYUUBI-2093.
Closes#1948
2095f835 [SteNicholas] [KYUUBI #2093] kyuubi beeline can not connect the kyuubi server due to exception
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2082 from SteNicholas/KYUUBI-2081.
Closes#2081
c6f76c5c [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
fee5c21f [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
`logCaptureThread` does not catch sparksubmit exception.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2090 from SteNicholas/KYUUBI-2078.
Closes#2078
11ff6824 [SteNicholas] [KYUUBI #2078] logCaptureThread does not catch sparksubmit exception
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
To close#1936
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2059 from lightning-L/kyuubi-1936.
Closes#1936
e9c83ef6 [Tianlin Liao] [KYUUBI #1936] send credentials when opening session and wait for completion
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
… engine
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2080 from yanghua/KYUUBI-2079.
Closes#2079
2bf98720 [yanghua] [KYUUBI #2079] Update kyuubi layer source file to add flink and trino engine
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
closes https://github.com/apache/incubator-kyuubi/issues/1563
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2076 from ulysses-you/KYUUBI-1563.
Closes#1563
8a464f2f [ulysses] Update CONTRIBUTING.md
798913a1 [ulysses-you] sync
Lead-authored-by: ulysses-you <ulyssesyou18@gmail.com>
Co-authored-by: ulysses <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
1. use host address rather than host name, since we also use host address in thrift server
2. remove alert in web ui, it's noisy..
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2073 from ulysses-you/web-ui.
Closes#2072
91b6f5b3 [ulysses-you] rest
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
We met below issue when fetching result.
```
Caused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException:46
7022 at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:453)
7023 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2397)
7024 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
7025 at java.util.Calendar.setTimeInMillis(Calendar.java:1804)
7026 at java.util.Calendar.setTime(Calendar.java:1770)
7027 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:943)
7028 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
7029 at java.text.DateFormat.format(DateFormat.java:345)
7030 at org.apache.kyuubi.schema.RowSet$.toHiveString(RowSet.scala:245)
7031 at org.apache.kyuubi.schema.RowSet$.$anonfun$toTColumn$3(RowSet.scala:120)
7032 at scala.collection.immutable.List.map(List.scala:290)
7033 at org.apache.kyuubi.schema.RowSet$.toTColumn(RowSet.scala:115)
7034 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1(RowSet.scala:65)
7035 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1$adapted(RowSet.scala:64)
7036 at scala.collection.immutable.List.foreach(List.scala:392)
7037 at org.apache.kyuubi.schema.RowSet$.toColumnBasedSet(RowSet.scala:64)
7038 at org.apache.kyuubi.schema.RowSet$.toTRowSet(RowSet.scala:47)
7039 at org.apache.kyuubi.engine.spark.operation.SparkOperation.getNextRowSet(SparkOperation.scala:183)
7040 at org.apache.kyuubi.operation.OperationManager.getOperationNextRowSet(OperationManager.scala:116)
7041 at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:197)
7042 at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:169)
7043 at org.apache.kyuubi.service.ThriftBinaryFrontendService.FetchResults(ThriftBinaryFrontendService.scala:505)
```
The root cause is that the date time formatter used to convert the result to rowSet is not thread-safe.
In this pr, we use thread-safe FastDateFormat instead of SimpleDateFormat.
### _How was this patch tested?_
Existing UT.
Closes#2075 from turboFei/thread_safe_fat_stimeformatter.
Closes#2075
44ae8fd1 [Fei Wang] Using thread-safe FastDateFormat instead of SimpleDateFormat
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2066
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2067 from jiaoqingbo/2066.
Closes#2066
331c1812 [jiaoqingbo] [KYUUBI #2066] fix spelling mistake and appropriate naming
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2064 from yanghua/KYUUBI-2063.
Closes#2063
35ea82cf [yanghua] [KYUUBI #2063] Fix engine idle timeout lose efficacy for Flink Engine
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
thanks to the pr original author yaooqinn https://github.com/apache/incubator-kyuubi/pull/1670
After this pr, the endpoint of OpenAPI will change from`http://${ip}:${port}/openapi.json` to `http://${ip}:${port}/api/openapi.json`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2061 from cfmcgrady/pr-1670.
Closes#2061
969b7d3d [Fu Chen] update dev/dependencyList
ff4b6bc2 [Fu Chen] fix ut
207553b6 [Fu Chen] addressed comment
a3baefe0 [Fu Chen] new line at end of file
ff36f957 [Fu Chen] fix
bb6abb3f [Fu Chen] language
b22be2ec [Fu Chen] style
f0c9562f [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
d4ac4cba [Fu Chen] fix
cd5ee2b8 [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
1b108a04 [Kent Yao] ui
85e31519 [Kent Yao] ui
0c20fcff [Kent Yao] ui
c2abde93 [Kent Yao] ui
b4dff513 [Kent Yao] ui
43dffab8 [Kent Yao] Merge branch 'master' into web
7cd273a3 [Kent Yao] swagger ui enabled
518882e2 [Kent Yao] swagger ui enabled
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support a lightweight hive engine which has some advantages over hiveserver2
- separate engine/process to the multi-tenancy framework for better stability, without having a heavy server
- more engines introduced in kyuubi, hive may be aged but still widely used
- also good for us to do some cross verifications for spark/flink etc with a stable hive.
In this PR, a basic hive engine backend is introduced.
- only engine side
- only execute statement is supported
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2013 from yaooqinn/hiveengine.
Closes#2011
75d146bc [Kent Yao] Merge branch 'master' into hiveengine
a7152598 [Kent Yao] addressed
b6c8ef59 [Kent Yao] llap/shims -> jdk:tools
07a9843a [Kent Yao] hbase client -> jdk:tools
3acb4aa7 [Kent Yao] Merge branch 'master' into hiveengine
ef5bcb8a [Kent Yao] codecov
b09e46a3 [Kent Yao] hbase client -> jdk:tools
56f74d50 [Kent Yao] hive
dcacffea [Kent Yao] hive
c46ffc41 [Kent Yao] hive
2f883036 [Kent Yao] hive
186dd33d [Kent Yao] hive
5038505f [Kent Yao] hive
5bd2a849 [Kent Yao] hive
180d892d [Kent Yao] hive
568a52b9 [Kent Yao] init hive engine
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Close#2043
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2046 from RishiKumarRay/master.
Closes#2043
bf9a4a0b [--replace-all] fixed sql issue
c119e5c2 [Rishi Kumar Ray] Merge branch 'apache:master' into master
323d81c7 [--replace-all] Upgrade log4j/2.x/ to 2.17.2
Lead-authored-by: --replace-all <rishi.ray@knoldus.com>
Co-authored-by: Rishi Kumar Ray <87641376+RishiKumarRay@users.noreply.github.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We need to clear job group information for init SQL, otherwise in incremental collection mode, the fetchNext will always display wrong description.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Before this patch:
<img width="1916" alt="incremental_collect" src="https://user-images.githubusercontent.com/26535726/157165964-502ad809-9d99-44de-b08c-131d135ce33d.png">
After this patch:
<img width="1917" alt="Xnip2022-03-08_12-04-50" src="https://user-images.githubusercontent.com/26535726/157165975-68eddf24-e0ad-4db2-931c-c75541802a7b.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2060 from pan3793/clear.
Closes#2060
76a67372 [Cheng Pan] Clear job group for init SQL
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Remove thread locals to prevent memory leak.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2044 from turboFei/remove_thread_local.
Closes#2044
bcce9e31 [Fei Wang] remove thread local
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
branch-1.5 & 1.4 snapshots are missing
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2051 from yaooqinn/1955.
Closes#1955
693c79b1 [Kent Yao] Add CI for branch-1.5 & 1.4 SNAPSHOTS
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support closing Flink SQL engine process.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2054 from SteNicholas/KYUUBI-1819.
Closes#2054
992bd0b2 [SteNicholas] [KYUUBI-1819] Support closing Flink SQL engine process
69567120 [SteNicholas] [KYUUBI-1819] Support closing Flink SQL engine process
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2055
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2056 from jiaoqingbo/2055.
Closes#2055
52a2e623 [jiaoqingbo] [KYUUBI #2055] correct the log service name
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support more MySQL JDBC driver versions.
The user may use the 5.x mysql driver to connect to the kyuubi server.
https://github.com/apache/incubator-kyuubi/issues/2047
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2048 from cxzl25/KYUUBI-2047.
Closes#2047
03cd15f2 [sychen] Support more MySQL JDBC driver versions
Authored-by: sychen <sychen@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Preparing v1.6.0-SNAPSHOT with branch-1.5 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2045 from yaooqinn/pv.
Closes#2045
6a506a8f [Kent Yao] Preparing v1.6.0-SNAPSHOT
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix#2012#1180
In `EventLoggingService`, when `EventLoggerType` is set to `JSON`
Kyuubi creates an error directory, it should not be the `ENGINE_EVENT_JSON_LOG_PATH` but the `SERVER_EVENT_JSON_LOG_PATH`.
<img width="557" alt="微信图片_20220304165614" src="https://user-images.githubusercontent.com/14961757/156771039-e18fcdbe-8b99-4190-9b14-6953daa55e5f.png">
<img width="605" alt="微信图片_20220304165633" src="https://user-images.githubusercontent.com/14961757/156771053-defdee62-67f9-4f16-9d4a-8ef170304221.png">
rename `EventLoggingService` in `kyuubi-server` to `KyuubiEventLoggingService`
rename `EventLoggingService` in `kyuubi-spark-sql-engine` to `SparkEventLoggingService`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2037 from jiaoqingbo/kyuubi2012.
Closes#2012
714e5ea5 [jiaoqingbo] fix ut failed
10d1a151 [jiaoqingbo] execute dev/format
cb99a458 [jiaoqingbo] [KYUUBI #2012] rename EventLoggingService and creates an correct directory
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Update `.asf.yaml` to redirect Issues/PR Notifications to notificationskyuubi.apache.org
### _How was this patch tested?_
No test needed
Closes#2042 from zjffdu/notification.
Closes#2042
1f16cb4a [Jeff Zhang] Redirect Issues/PR Notifications to notifications@kyuubi.apache.org
Authored-by: Jeff Zhang <zjffdu@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Replaced by FlinkOperationSuite
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2039 from pan3793/flink.
Closes#2039
f9fd6674 [Cheng Pan] Remove LegacyFlinkOperationSuite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
