0cfbcd2e08
320 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jiaoqingbo
|
8f15622d9b
|
[KYUUBI #2084] Support arbitrary parameters for KyuubiConf
### _Why are the changes needed?_ fix #2084 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2091 from jiaoqingbo/2084. Closes #2084 624216fa [Cheng Pan] permission c45a71ce [jiaoqingbo] delete empty line 29987bf1 [jiaoqingbo] fix ut failed 2e7f0acc [jiaoqingbo] Merge branch 'master' into 2084 04c0f8e1 [jiaoqingbo] Merge branch 'master' into 2084 1267d195 [jiaoqingbo] Merge branch '2084' of https://github.com/jiaoqingbo/incubator-kyuubi into 2084 1bb2b6d6 [jiaoqingbo] Merge branch 'master' into 2084 db4a4aac [jiaoqingbo] Merge branch 'master' into 2084 82097388 [jiaoqingbo] ignore java/sun prefix d4640ec3 [jiaoqingbo] add log d5af758f [jiaoqingbo] modify ut 0f3af6c6 [jiaoqingbo] modify ut case 9ffb4177 [jiaoqingbo] Merge branch 'master' into 2084 52dd5090 [jiaoqingbo] modify ut 3f881748 [jiaoqingbo] modify test conf 705f94af [jiaoqingbo] modify test conf 096fba1d [jiaoqingbo] modify test conf 83f1aba9 [jiaoqingbo] add more example configuration d2581534 [jiaoqingbo] fix compile problem 65c3198a [jiaoqingbo] [KYUUBI #2084] Support arbitrary parameters for KyuubiConf Lead-authored-by: jiaoqingbo <1178404354@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
SteNicholas
|
a83cd49e1b
|
[KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
### _Why are the changes needed?_ Add `Deploy Kyuubi Flink engine on Yarn`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2131 from SteNicholas/KYUUBI-1866. Closes #1866 ba639f15 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn cc6f4d44 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f8efcb7125
|
[KYUUBI #2103] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_ This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093 We need more time to evaluate it and add more test cases to ensure stability before the next release. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2103 from pan3793/revert. Closes #2103 Closes #1948 575dfe8d [Cheng Pan] properties order 4c94481e [Cheng Pan] fix deps bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test bad4e90e [Cheng Pan] surefire f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0" 4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2" cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf" 52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes" Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
b8389dae40
|
[KYUUBI #1866][FOLLOWUP] Add logging of Flink SQL Engine
### _Why are the changes needed?_ Add logging of Flink SQL Engine in `logging.md`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2111 from SteNicholas/KYUUBI-1866. Closes #1866 7a0bdbb8 [SteNicholas] [KYUUBI #1866] Add logging of Flink SQL Engine Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Ada Wong
|
8f7b2c6640
|
[KYUUBI #1866][DOCS] Add flink sql engine quick start
### _Why are the changes needed?_ Add quick start documents of the Flink SQL Engine. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2106 from deadwind4/KYUUBI-1866-quickstart. Closes #1866 2533aafd [Ada Wong] remove Yarn section 6aa4db8a [Ada Wong] compress png ff6bff72 [Ada Wong] [KYUUBI #1866][DOCS] Add flink sql engine quick start Authored-by: Ada Wong <rsl4@foxmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Min Zhao
|
b7a5cfcf78
|
[KYUUBI #2108] Add description about trino in the config of engine.type
### _Why are the changes needed?_ Add description about trino in the config of engine.type. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2109 from zhaomin1423/fix_engine_type_doc. Closes #2108 adf898c0 [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type 8f11514e [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
SteNicholas
|
2486c5dff3
|
[KYUUBI #2089] Add debugging instructions for Flink engine
### _Why are the changes needed?_ Add debugging instructions for Flink engine. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2098 from SteNicholas/KYUUBI-2089. Closes #2089 08189b85 [SteNicholas] [KYUUBI #2089] Add debugging instructions for flink engine Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fei Wang
|
1e23e7a93c |
[KYUUBI #1978] Support NEGOTIATE/BASIC authorization for restful frontend service
### _Why are the changes needed?_ to close #1978 Support both NEGOTIATE and BASIC authentication for restful frontend service. At first, I involve two auth schemes, FYI [hadoop/HttpConstants.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FHttpConstants.java) - BASIC: mapping to NOSASL, NONE, LDAP, CUSTOM authentication - NEGOTIATE: mapping to KERBEROS authentication BTW, hadoop also supports `Digest` auth scheme. Two authentication handlers: - BasicAuthenticationHandler(reuse existing passwdAuthenticationProvider(LDAP/CUSTOM)) - KerberosAuthenticationHandler(refer [hadoop/KerberosAuthenticationHandler.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FKerberosAuthenticationHandler.java) and [hadoop/KerberosUtil.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Futil%2FKerberosUtil.java)) #### AuthenticationFilter - redirect the http request according to the authentication scheme specified in `Authorization` header. For basic authentication The value of Authorization header is `BASIC ${encodeBase64(user + ":" + password)}`. Then using the passwdAuthenticationProvider to authenticate `user` and `password`. For SPNEGO(kerberos) authentication. The value of Authorization header is `NEGOTIATE ${encodeBase64(clientToken)}`. Then checking whether the clientToken is valid. And currently, the AuthenticationFilter take affect for all the requests with pathSpec `/api/*`. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2049 from turboFei/kyuubi_1978_kerberos. Closes #1978 c8b6362b [Fei Wang] refactor cedec70f [Fei Wang] add spnego unit test c1d45cde [Fei Wang] rename package to http.authentication 3d0b220b [Fei Wang] address comments 296f181e [Fei Wang] fix ut f9371e14 [Fei Wang] spnego and basic auth Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
SteNicholas
|
a71c511ba3
|
[KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
### _Why are the changes needed?_ YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2082 from SteNicholas/KYUUBI-2081. Closes #2081 c6f76c5c [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR fee5c21f [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
a882c4bf4e
|
[KYUUBI #2079] Update kyuubi layer source file to add flink and trino…
… engine <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2080 from yanghua/KYUUBI-2079. Closes #2079 2bf98720 [yanghua] [KYUUBI #2079] Update kyuubi layer source file to add flink and trino engine Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
54dfb4bbc0
|
[KYUUBI #1215][DOC] Document incremental collection
### _Why are the changes needed?_ Document incremental collection, close #1215 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1914" alt="1" src="https://user-images.githubusercontent.com/26535726/157191008-451bee2a-eb6b-4bb6-869b-5b0f75a21448.png"> <img width="1919" alt="2" src="https://user-images.githubusercontent.com/26535726/157191016-e183bbf5-aa4a-496d-a250-5d14cf04101d.png"> <img width="1920" alt="3" src="https://user-images.githubusercontent.com/26535726/157191026-343a39d7-0ab8-4886-9a51-3670394ef6be.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2057 from pan3793/doc. Closes #1215 82677a03 [Cheng Pan] grammar 3be0e3f1 [Cheng Pan] fix b467e975 [Cheng Pan] Update d256cbaf [Cheng Pan] compress picture b3c5fb64 [Cheng Pan] Fix 4c06307c [Cheng Pan] [KYUUBI #1215][DOC] Document incremental collection Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
HYBG-1126
|
85d515dbd2
|
[KYUUBI #1999] Update quick_start_with_helm.md
cretate ns kyuubi ---> create ns kyuubi ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1999 from HYBG-1126/patch-1. Closes #1999 f68f73e9 [HYBG] Update quick_start_with_helm.md Lead-authored-by: HYBG-1126 <hybg1126@outlook.com> Co-authored-by: HYBG <44338121+HYBG-1126@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
114fefefa6
|
[KYUUBI #2004] Sync contents for CONTRIBUTING & COMMUNITY between web and main repo
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add CONTRIBUTING.md to list COMMUNITY resources that I contributor can use delete contributions and improvement-proposal in main repo ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2004 from yaooqinn/CONTRIBUTING. Closes #2004 ddbcd521 [Kent Yao] CONTRIBUTING f81c14b3 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into CONTRIBUTING 3bdbdc9b [Kent Yao] CONTRIBUTING 574ceff2 [Kent Yao] CONTRIBUTING f0a6b4c4 [Kent Yao] CONTRIBUTING c55c6a02 [Kent Yao] CONTRIBUTING Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
7887a47e02
|
[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
### _Why are the changes needed?_ [THRIFT-4805](https://issues.apache.org/jira/browse/THRIFT-4805) has been fixed in thrift 0.13.0, we can remove the workaround of log4j2 configurations because we use thrift 0.16.0 now, the change also removes the following warnings. ``` 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" ``` `maven-surefire-plugin` is only used in `kyuubi-hive-jdbc`, we should disable it in other modules. ``` [INFO] --- maven-surefire-plugin:2.22.0:test (default-test) kyuubi-ctl_2.12 --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1996 from pan3793/log. Closes #1996 291441c0 [Cheng Pan] nit f2c94246 [Cheng Pan] Tune surefire plugin in conf e1ee6402 [Cheng Pan] Clean up log4j2 conf Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
6d757a3df9
|
[KYUUBI #1963] Make plan only mode skippable for configuable plans
### _Why are the changes needed?_ Make plan only mode skippable for configuable plans. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1967 from SteNicholas/KYUUBI-1963. Closes #1963 e253aec6 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans b5cff405 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
644b917a30
|
[KYUUBI #1970] Replace mirror-cn profile with Apache officially suggested dlcdn
### _Why are the changes needed?_ Replace `mirror-cn` profile with Apache officially suggested dlcdn. mirror-cn is only suite for CN, while dlcdn is much wiser. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1970 from yaooqinn/cdn. Closes #1970 a325f828 [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn a0756b4f [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
SteNicholas
|
770499ca54
|
[KYUUBI #1948] Upgrade thrift version to 0.16.0
### _Why are the changes needed?_ Upgrade libthrift to 0.16.0 due to [CVE-2020-13949](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949) and the coming upstream change of Spark https://github.com/apache/spark/pull/34362 ### _What changed in this PR?_ - Upgrade libthrift to 0.16.0 - Shade and relocate `thrift` and `hive-service-rpc` classes in `kyuubi-spark-engine`, it's necessary to avoide conflicting with old thrift libs bundled in Spark binary releases. - Due to thrift change the method signature, the subclasses those interfaces in Kyuubi also need to modify to pass compile. We rely on Hive 2.3.9 jars in some components, e.g. `kyuubi-hive-jdbc`, `LocalMetaServer` in `kyuubi-server` test classes. Some classes in Hive jars compiled against old thrift interfaces which are not compatible with thrift 0.16.0, it causes runtime link error, we found the following classes which breaks the test and copied them with necessary modification to make it work with thrift 0.16.0. - `TFramedTransport` - `TFilterTransport` - `TUGIAssumingTransport` - `TUGIContainingTransport` - Next Steps, I think it's worth to do in separated PRs. - Recover the `HiveDelegationTokenProviderSuite`, one approach is use an isolate classloader to load HMS classes and thrift 0.9.3 classes from Maven, this approach can also be used for the planed Zoopkeeper upgrading to help us verficating the compatibility of Zookeeper Server 3.4.x. - Rewrite `kyuubi-hive-jdbc` to make it decouple with Hive jars, because there maybe other places which may not work with thrift 0.16.0 but the UTs does not cover. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1953 from SteNicholas/KYUUBI-1948. Closes #1948 de5d1ea2 [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 898effcd [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 803e270c [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Paul Lin
|
e54d431024
|
[KYUUBI #1883] Support max result rows for Flink queries
### _Why are the changes needed?_ Currently, Flink engine would pull all result rows into memory before returning it to the client. This would be problematic for large result sets and infinite result sets. This is a sub-task of KPIP-2 https://github.com/apache/incubator-kyuubi/issues/1322. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1938 from link3280/feature/FLINK-1883. Closes #1883 80020cee [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala 1b958221 [Paul Lin] [KYUUBI #1883] Avoid allocating too much buffer space 5be7535c [Paul Lin] [KYUUBI #1883] Support max result rows for Flink queries Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
1d64bb279e
|
[KYUUBI #1796] Destroy the process after the engine starts
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Destroy the process after the engine starts. #1796 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate add conf: `kyuubi.session.engine.startup.waitCompletion false`  - [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1815 from wForget/KYUUBI-1796. Closes #1796 12aea19a [Wang Zhen] fix 27cb602a [Wang Zhen] fix e70e7071 [Wang Zhen] add log ecaf828c [Wang Zhen] regenerate settings.md a5f6c5de [Wang Zhen] fix desc 09812beb [Wang Zhen] Merge remote-tracking branch 'origin/master' into KYUUBI-1796 27fab5eb [Wang Zhen] [KYUUBI-1796] Destroy the process after the engine starts Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
6c6238c2ae
|
[KYUUBI #1919] Add more enum values for OperationModes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add more enum values for `OperationModes`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1933 from SteNicholas/KYUUBI-1919. Closes #1919 adcc7f5e [SteNicholas] [KYUUBI #1919] Add more enum values for OperationModes Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhenjiaguo
|
2501f9ad83
|
[KYUUBI #1914] Update metric doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Update metric doc when metric change. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1916 from zhenjiaguo/update_metric_doc. Closes #1914 e8ea7f6c [zhenjiaguo] update metric doc a5d730ce [zhenjiaguo] update metric doc Authored-by: zhenjiaguo <zhenjiaguo@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
24e265d790
|
[KYUUBI #1876] Set Spark 3.2.1 as default
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Set Spark 3.2.1 as default. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1913 from SteNicholas/KYUUBI-1876. Closes #1876 1a6e4c6c [SteNicholas] [KYUUBI #1876] Set Spark 3.2.1 as default Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
681153036d
|
[KYUUBI #1536] Add session conf advisor
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> closes https://github.com/apache/incubator-kyuubi/issues/1536 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1892 from ulysses-you/KYUUBI-1536. Closes #1536 f0c450a9 [ulysses-you] codecov 35fac4f6 [ulysses-you] fix f34cfbf4 [ulysses-you] comment 78a54bc1 [ulysses-you] address comment 5cb574ba [ulysses-you] fix test 1033909b [ulysses-you] fix 3c925e56 [ulysses-you] address comment a74988cd [ulysses-you] style 649f02d0 [ulysses-you] Add session conf advisor debd3adb [ulysses-you] Add session conf advisor Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
yanghua
|
a6480d0e31
|
[KYUUBI #1898] Enhance release guide add more information about how to…
… generate mail content automatically <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1900 from yanghua/KYUUBI-1898. Closes #1898 782ff20b [yanghua] [KYUUBI-1898] Enhance release guide add more information about how to generate mail content automatically Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
sychen
|
677e2a40ab
|
[KYUUBI #1889] Improve the user experience of the configuration parameters of the document
### _Why are the changes needed?_ https://kyuubi.apache.org/docs/latest/deployment/settings.html The configuration parameters in the document are too long and a newline is inserted(`<br/>`), which makes it difficult to search. And if the user copies the configuration parameters, it may become several lines. Use `<code>` to replace `<br/>`. https://github.com/apache/incubator-kyuubi/issues/1889 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1890 from cxzl25/KYUUBI-1889. Closes #1889 765b0622 [sychen] format metrics.md abeb0601 [sychen] add newline 28865ad7 [sychen] increase max width 2f9c7e05 [sychen] use <code> Authored-by: sychen <sychen@trip.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
yanghua
|
d8f70e9727
|
[KYUUBI #1844] Add step for generating release notes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1864 from yanghua/KYUUBI-1844. Closes #1844 47d09c39 [yanghua] Address review comments b5a8bbc3 [yanghua] [KYUUBI #1844] Add step for generating release notes Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
ec719b82de
|
[KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Revert the dependency changes for kyuubi-hive-jdbc and kyuubi-hive-jdbc-shaded modules from #1842 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1854 from turboFei/slf4j-api. Closes #1769 dffe6c5c [Fei Wang] reserve for kyuubi-hive-beeline 20012434 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-beeline and kyuubi-hive-jdbc(shaded) modules Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
33d53db3ff
|
[KYUUBI #1813] Enable the internal secure authentication between Kyuubi server and engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Enable the internal secure access between Kyuubi server and engine. This pr use secret and cipher to encrypt a dynamic token identifier for authentication. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1813 from turboFei/engine_secure. Closes #1813 aa7ad0de [Fei Wang] run a simple query for engien security 34354d97 [Fei Wang] requre zk node must be defined for zk secret provider f84ce7c9 [Fei Wang] Require zk secret node non empty b545281d [Fei Wang] refactor d8d6dfd2 [Fei Wang] address comments 0e9273e1 [Fei Wang] refactor provider conf d3263c08 [Fei Wang] place holder e1266e32 [Fei Wang] reformat 9e252fea [Fei Wang] enhance prompt dbc9100c [Fei Wang] fix conf ff6b6dea [Fei Wang] refactor 4ea020ed [Fei Wang] refactor 20915902 [Fei Wang] refactor a7b4e41e [Fei Wang] add more ut: 1e1ceb27 [Fei Wang] use orignal cipher 8d8387b2 [Fei Wang] save ad777cb4 [Fei Wang] refactor b8997313 [Fei Wang] save b997095d [Fei Wang] refactor e05b47c5 [Fei Wang] move a322b784 [Fei Wang] refactor b14f7cbf [Fei Wang] Enable the internal secure authentication Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
53d59a02bf |
[KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This patch proposes to migrate from log4j1 to log4j2 in Kyuubi. Refer the spark patch: https://github.com/apache/spark/pull/34895 ### Does this PR introduce any user-facing change? Yes. Users may need to rewrite their log4j properties file for log4j2. As of version 2.4, log4j now supports configuration via properties files. Note that the property syntax is not the same as the syntax used in log4j 1, but during the migration I found the syntax is pretty close so the migration should be straightforward. ### _How was this patch tested?_ Passed all existing tests. Closes #1769 from turboFei/log4j2. Closes #1769 8613779c [Fei Wang] remove log4j dependencies from spark-sql-engine b2fe6dba [Fei Wang] Use String to present default log level 762e2d03 [Fei Wang] only remove org.apache.logging.log4j:log4j-slf4j-impl 8a912086 [Fei Wang] remove dependencies from spark-sql-engine 7e3a4980 [Fei Wang] address commments 051f49f5 [Fei Wang] address comments 85316a0b [Fei Wang] Keep compatible with log4j12 01d1a84e [Fei Wang] for log4j1 b9e17e1b [Fei Wang] refactor e24391ed [Fei Wang] revert log count 38803002 [Fei Wang] add log4j2.properties.template 4f0b22fc [Fei Wang] save 7ce84119 [Fei Wang] modify log level 1ea5ca53 [Fei Wang] add log4j to engine c4a86d4d [Fei Wang] use AbstractFilter 27b08b6a [Fei Wang] remove more 8cc15ae7 [Fei Wang] reformat c13ec29e [Fei Wang] save temporarily 33a38e2e [Fei Wang] exclude log4j12 from spark-sql 9129a64a [Fei Wang] refactor 5362b43d [Fei Wang] make it run at first 7f27f519 [Fei Wang] more 56f4f1ff [Fei Wang] fix logging a74b6d37 [Fei Wang] start appender dea964aa [Fei Wang] fix build erorr at first e20b7500 [Fei Wang] address comments 2ec02b4d [Fei Wang] fix LogDivertAppender dded1290 [Fei Wang] more c63e0008 [Fei Wang] add log4j2.properties Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
yanghua
|
0337da6701
|
[KYUUBI #1836] [RELEASE] Enhance release guide
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1837 from yanghua/KYUUBI-1836. Closes #1836 913221fc [yanghua] [KYUUBI #1836] [RELEASE] Enhance release guide Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
8bd7b5aae4
|
[KYUUBI #1831] [DOCS] Fix the jdbc connection url format doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the jdbc connection url format doc. In fact, the kyuubiConfs part and kyuubiVars part should have the same effect. But it is better to split the engine initialization conf and runtime vars to two parts, so here I do not modify these two parts. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1831 from turboFei/jdbc_docs. Closes #1831 40288282 [Fei Wang] Update the docs for jdbc url Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
916f81abaf
|
[KYUUBI #1775] Add kyuubi.server.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add `kyuubi.server.name` configuration. #1775 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate   - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1789 from wForget/KYUUBI-1775. Closes #1775 8b895044 [Wang Zhen] [KYUUBI-1775] Add kyuubi.server.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
b91854e33b
|
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR mainly focus on refactoring 1. Use Apache Thrift Naming style for thrift-based classes which is `TXyzAbc`, e.g. `TFrontendService` 2. Created `TFrontendService` and move the `TCLIService.Iface` APIs and other common fields in `TFrontendService`, later we will use it to implement the `KyuubiTHttpFrontendService` in the server module 3. introduce a new conf also for server FEs to be able to pick IP first ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1761 from yaooqinn/http1. Closes #1761 98b4afdd [Kent Yao] address comment 3138af29 [Kent Yao] [WIP] Thrift Http protocol support 1 a06026af [Kent Yao] [WIP] Thrift Http protocol support 1 efa899fa [Kent Yao] [WIP] Thrift Http protocol support 1 d1403e30 [Kent Yao] [WIP] Thrift Http protocol support 1 d114efd4 [Kent Yao] [WIP] Thrift Http protocol support 1 86e30c47 [Kent Yao] [WIP] Thrift Http protocol support 1 8117d12d [Kent Yao] [WIP] Thrift Http protocol support 1 62194323 [Kent Yao] [WIP] Thrift Http protocol support 1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
hongdongdong
|
11556034e3
|
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add TRINO engine type and TrinoProcessBuilder ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1695 from hddong/add-trino-process-builder. Closes #1510 394b853e [hongdongdong] fix 044ca8cc [hongdongdong] fix 568b1b8e [hongdongdong] use java options to pass args fbe7e60d [hongdongdong] [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
cb8721fef1
|
[KYUUBI #1629] Flink backend implementation
### _Why are the changes needed?_ This PR covers #1619. Overall, this PR contains the following changs, 1. change `build/dist` script to support flink sql engine 2. enhance `externals/flink-sql-engine/pom.xml` to support create a shaded jar 3. simplify `externals/kyuubi-flink-sql-engine/bin/flink-sql-engine.sh` 4. introduce `FlinkSQLEngine`(flink sql engine entrypoint) and `FlinkProcessBuilder`(kyuubi server launcher) 5. add ut in kyuubi server side After this PR, we can run the basic query e.g. `select now()` from beeline and get result, and Kyuubi Server can auto launch flink engine if there is no proper one. The Flink engine also supports other engine share levels defined in Kyuubi. The implementation based on Flink 1.14 codebase. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1629 from pan3793/flink-backend. Closes #1629 b7e5f0e3 [Cheng Pan] revert tgz name change a4496c37 [Cheng Pan] Fix reflection 3b4e86a1 [Cheng Pan] deps 68efa42c [Cheng Pan] log 8a9e37f3 [Cheng Pan] nit 10fb2bc3 [Cheng Pan] CI c1560fdb [Cheng Pan] nit 303e2f1e [Cheng Pan] Restore log conf d84720b1 [Cheng Pan] SessionContext b258d81a [Cheng Pan] cleanup 16edd528 [Cheng Pan] Cleanup 9ae54557 [Cheng Pan] Fix CI 25b6b57d [Cheng Pan] hadoop-client-api c12b5ca4 [Cheng Pan] Server UT pass 502d3f08 [Cheng Pan] pass dac4323b [yanghua] Laungh local flink engine container successfully Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhenjiaguo
|
3ffb12de24
|
[KYUUBI #1607] [DOC] Add operation and backend service metrics in metrics doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add operation and backend service metrics doc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #1607 from zhenjiaguo/add_be_metric_doc. Closes #1607 e10bf146 [zhenjiaguo] Comment 0062f3d8 [zhenjiaguo] Metric format adjustment af84534c [zhenjiaguo] Add metrics version compatibility description c6f9eb26 [zhenjiaguo] Add operation and backend service method metric info doc Authored-by: zhenjiaguo <zhenjia_guo@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
df1d9f3bb2
|
[KYUUBI #1577] Add DropIgnoreNonexistent Rule.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Like the `hive.exec.drop.ignorenonexistent` configuration in hive, do not report an error if DROP DATABASE/TABLE/VIEW/Function/Partition specifies a non-existent database/table/view/function/partition. For details: #1577. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1583 from wForget/KYUUBI-1577. Closes #1577 63382660 [Wang Zhen] delete useless comment adadf424 [Wang Zhen] add DropIgnoreNonexistent rule using injectPostHocResolutionRule 860b15d3 [Wang Zhen] add DropIgnoreNonexistent rule for spark 3.2 aad43cdb [Wang Zhen] adjust code ca1b9f11 [Wang Zhen] make some adjustments and update rules.md 804bf40e [Wang Zhen] [KYUUBI-1577] Add DropIgnoreNonexistent Rule. Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
6e10eec4ca
|
[KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi
### _Why are the changes needed?_ Many kyuubi users are not familiar with kerberos authentication mechanism and encountered troubles connecting to kerberized Kyuubi. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate Configurations are verified in #1432 - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1552 from zhouyifan279/1550. Closes #1550 58e324a8 [zhouyifan279] [KYUUBI #1550] Add kerberos authentication flow 3b030a91 [zhouyifan279] [KYUUBI #1550] Add kerberos authentication flow 4449b689 [zhouyifan279] [KYUUBI #1550] Improve doc style 3aa8e20c [zhouyifan279] [KYUUBI #1550] Add Windows configuration steps 7e67cdc5 [zhouyifan279] [KYUUBI #1550] Capitalize title and emphasize the relation between KinitAuxiliaryService and Kerberos authentication 424e56b3 [zhouyifan279] [KYUUBI #1550] Rename client/kerberos.md to client/kerberized_kyuubi.md d8977cea [zhouyifan279] [KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
be3a964f37
|
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add kyuubi.engine.pool.name configuration. #1554 ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1572 from wForget/KYUUBI-1554. Closes #1554 40850f5a [Wang Zhen] [KYUUBI-1554] Add kyuubi.engine.pool.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
01da9b432e
|
[KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1556. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1562 from byyue/dev-1556. Closes #1556 8ceb244a [Brian Yue] Update the doc fdb73c4d [Brian Yue] Shorten kyuubi.session.engine.check.interval to one minute Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d805f49fe7
|
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1557 from yaooqinn/ttk. Closes #1557 8446180a [Kent Yao] [DOC] The TTL Of Kyuubi Engines Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
602bb23d36
|
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
### _Why are the changes needed?_ Compatible with lower version of hive driver. https://github.com/apache/incubator-kyuubi/issues/1174 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1553 from cxzl25/KYUUBI-1174-followup. Closes #1174 39cbdf4c [sychen] trigger test 908c4750 [sychen] gen settings f9e2eda0 [sychen] typo 25300d56 [sychen] Modify the default value to false Authored-by: sychen <sychen@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
3594e08d1b |
[KYUUBI #1548] Get docs version from project.version
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the docs version issue ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1548 from turboFei/docs_version. Closes #1548 dfee7b6f [Fei Wang] Fix docs version Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Fei Wang
|
631bfe35d4 |
[KYUUBI #1544] Update the mailing list to send announcement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Update the mailing list to send announcement. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1544 from turboFei/finalize_release. Closes #1544 d5cd5c36 [Fei Wang] Update finalize release doc Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
zwangsheng
|
2f46d4cc49
|
[KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE #1533 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1534 from zwangsheng/KYUUBI/1533. Closes #1533 803afeed [zwangsheng] fix 9fbc82e0 [zwangsheng] add doc Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
c30d580686
|
[KYUUBI #1527] [DOC] Improve High Availability Guide
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1527 from yaooqinn/hadoc. Closes #1527 249701b6 [Kent Yao] [DOC] Improve High Availability Guide 03244332 [Kent Yao] [DOC] Improve High Availability Guide 54c1cd9d [Kent Yao] [DOC] Improve High Availability Guide 6d8a0991 [Kent Yao] [DOC] Improve High Availability Guide 5b4dff3d [Kent Yao] [DOC] Improve High Availability Guide Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
078edcd231
|
[KYUUBI #1526] [DOC] Update kyuubi_layers.drawio to add driver layer
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1526 from yaooqinn/img_kyuubi_layers. Closes #1526 b769930f [Kent Yao] address comments 2c7de469 [Kent Yao] [DOC] Update kyuubi_layers.drawio to add driver layer Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
97c3835738
|
[KYUUBI #1490] Introduce the basic framework for running scala
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce the basic framework for running scala, see #1490 for the detail ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` Beeline version 1.5.0-SNAPSHOT by Apache Kyuubi (Incubating) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:07.556 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.558 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.559 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:07.562 INFO scheduler.DAGScheduler: Asked to cancel job group 321dc15d-68d0-4f91-9216-1e08f09842df 2021-12-03 13:47:07.565 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.006 seconds 2021-12-03 13:47:07.565 ERROR operation.ExecuteStatement: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Query[08b8b6da-d434-4296-b613-2027e3518441] in ERROR_STATE 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.009 seconds Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> set kyuubi.operation.language=scala; 2021-12-03 13:47:11.982 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.983 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.984 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Execute in full collect mode 2021-12-03 13:47:12.006 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b] in FINISHED_STATE 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds +----------------------------+--------+ | key | value | +----------------------------+--------+ | kyuubi.operation.language | scala | +----------------------------+--------+ 1 row selected (0.052 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:13.685 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:15.541 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:15.543 INFO operation.ExecuteStatement: Query[178ff72d-b870-44d4-99d8-08a0fa0e8efa] in FINISHED_STATE 2021-12-03 13:47:15.544 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: RUNNING_STATE -> FINISHED_STATE, statement: spark.version, time taken: 0.003 seconds +-----------------------+ | output | +-----------------------+ | res0: String = 3.1.2 | +-----------------------+ 1 row selected (1.871 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.sql("select current_date()") . . . . . . . . . . . . . . . . . . .> ; 2021-12-03 13:47:36.512 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.689 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: PENDING_STATE -> RUNNING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Query[602c2d88-7e8a-4175-9b53-e34adfff007a] in FINISHED_STATE 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: RUNNING_STATE -> FINISHED_STATE, statement: spark.sql("select current_date()"), time taken: 0.003 seconds +----------------------------------------------------+ | output | +----------------------------------------------------+ | res1: org.apache.spark.sql.DataFrame = [current_date(): date] | +----------------------------------------------------+ 1 row selected (0.187 seconds) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3); Error: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error: results += spark.range(1, 5, 2, 3) <console>:26: error: type mismatch; found : org.apache.spark.sql.Dataset[Long] required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] results += spark.range(1, 5, 2, 3) ^ at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.spark.operation.ExecuteScala.runInternal(ExecuteScala.scala:70) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130) at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93) at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:62) at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:121) at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75) at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:118) at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:61) at org.apache.kyuubi.service.ThriftBinaryFrontendService.ExecuteStatement(ThriftBinaryFrontendService.scala:265) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3).toDF; 2021-12-03 13:48:30.327 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: INITIALIZED_STATE -> PENDING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.700 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: PENDING_STATE -> RUNNING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Query[daccd163-978a-4d66-af85-af466086d1db] in FINISHED_STATE 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: RUNNING_STATE -> FINISHED_STATE, statement: results += spark.range(1, 5, 2, 3).toDF, time taken: 0.003 seconds +-----+ | id | +-----+ | 1 | | 3 | +-----+ 2 rows selected (1.387 seconds) ``` ** Session level isolated **  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1491 from yaooqinn/scala. Closes #1490 af4d0a13 [Kent Yao] Merge branch 'master' into scala 2ebdf6a3 [Kent Yao] provided scala dep d58b2bfe [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala d256bfde [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala 52090434 [Kent Yao] init Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fei Wang
|
4648b63a2c |
[KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1379 from turboFei/async_open_default. Closes #1379 a7c4c6a1 [Fei Wang] fix ut cc8d2730 [Fei Wang] Make SESSION_ENGINE_LAUNCH_ASYNC true by default Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
zhichaoleo
|
7f3f8af902 |
[KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
### _Why are the changes needed?_ 1. After test again in my local, I found I miss a dependency jar which need to be put into ./spark/jars/ ### _How was this patch tested?_ I test thses doc by the steps provided by the doc : https://kyuubi.apache.org/docs/latest/develop_tools/build_document.html Closes #1484 from zhichaoleo/defect/Fix_Doc_For_Delta. Closes #1484 a1dff005 [zhichaoleo] Add a dependency describe for kyuubi on azure delta-lake doc. Authored-by: zhichaoleo <zhichaoleo@163.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
