0cfbcd2e08
155 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jiaoqingbo
|
8f15622d9b
|
[KYUUBI #2084] Support arbitrary parameters for KyuubiConf
### _Why are the changes needed?_ fix #2084 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2091 from jiaoqingbo/2084. Closes #2084 624216fa [Cheng Pan] permission c45a71ce [jiaoqingbo] delete empty line 29987bf1 [jiaoqingbo] fix ut failed 2e7f0acc [jiaoqingbo] Merge branch 'master' into 2084 04c0f8e1 [jiaoqingbo] Merge branch 'master' into 2084 1267d195 [jiaoqingbo] Merge branch '2084' of https://github.com/jiaoqingbo/incubator-kyuubi into 2084 1bb2b6d6 [jiaoqingbo] Merge branch 'master' into 2084 db4a4aac [jiaoqingbo] Merge branch 'master' into 2084 82097388 [jiaoqingbo] ignore java/sun prefix d4640ec3 [jiaoqingbo] add log d5af758f [jiaoqingbo] modify ut 0f3af6c6 [jiaoqingbo] modify ut case 9ffb4177 [jiaoqingbo] Merge branch 'master' into 2084 52dd5090 [jiaoqingbo] modify ut 3f881748 [jiaoqingbo] modify test conf 705f94af [jiaoqingbo] modify test conf 096fba1d [jiaoqingbo] modify test conf 83f1aba9 [jiaoqingbo] add more example configuration d2581534 [jiaoqingbo] fix compile problem 65c3198a [jiaoqingbo] [KYUUBI #2084] Support arbitrary parameters for KyuubiConf Lead-authored-by: jiaoqingbo <1178404354@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
SteNicholas
|
a83cd49e1b
|
[KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
### _Why are the changes needed?_ Add `Deploy Kyuubi Flink engine on Yarn`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2131 from SteNicholas/KYUUBI-1866. Closes #1866 ba639f15 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn cc6f4d44 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f8efcb7125
|
[KYUUBI #2103] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_ This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093 We need more time to evaluate it and add more test cases to ensure stability before the next release. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2103 from pan3793/revert. Closes #2103 Closes #1948 575dfe8d [Cheng Pan] properties order 4c94481e [Cheng Pan] fix deps bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test bad4e90e [Cheng Pan] surefire f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0" 4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2" cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf" 52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes" Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
b7a5cfcf78
|
[KYUUBI #2108] Add description about trino in the config of engine.type
### _Why are the changes needed?_ Add description about trino in the config of engine.type. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2109 from zhaomin1423/fix_engine_type_doc. Closes #2108 adf898c0 [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type 8f11514e [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fei Wang
|
1e23e7a93c |
[KYUUBI #1978] Support NEGOTIATE/BASIC authorization for restful frontend service
### _Why are the changes needed?_ to close #1978 Support both NEGOTIATE and BASIC authentication for restful frontend service. At first, I involve two auth schemes, FYI [hadoop/HttpConstants.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FHttpConstants.java) - BASIC: mapping to NOSASL, NONE, LDAP, CUSTOM authentication - NEGOTIATE: mapping to KERBEROS authentication BTW, hadoop also supports `Digest` auth scheme. Two authentication handlers: - BasicAuthenticationHandler(reuse existing passwdAuthenticationProvider(LDAP/CUSTOM)) - KerberosAuthenticationHandler(refer [hadoop/KerberosAuthenticationHandler.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FKerberosAuthenticationHandler.java) and [hadoop/KerberosUtil.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Futil%2FKerberosUtil.java)) #### AuthenticationFilter - redirect the http request according to the authentication scheme specified in `Authorization` header. For basic authentication The value of Authorization header is `BASIC ${encodeBase64(user + ":" + password)}`. Then using the passwdAuthenticationProvider to authenticate `user` and `password`. For SPNEGO(kerberos) authentication. The value of Authorization header is `NEGOTIATE ${encodeBase64(clientToken)}`. Then checking whether the clientToken is valid. And currently, the AuthenticationFilter take affect for all the requests with pathSpec `/api/*`. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2049 from turboFei/kyuubi_1978_kerberos. Closes #1978 c8b6362b [Fei Wang] refactor cedec70f [Fei Wang] add spnego unit test c1d45cde [Fei Wang] rename package to http.authentication 3d0b220b [Fei Wang] address comments 296f181e [Fei Wang] fix ut f9371e14 [Fei Wang] spnego and basic auth Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
SteNicholas
|
a71c511ba3
|
[KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
### _Why are the changes needed?_ YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2082 from SteNicholas/KYUUBI-2081. Closes #2081 c6f76c5c [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR fee5c21f [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
54dfb4bbc0
|
[KYUUBI #1215][DOC] Document incremental collection
### _Why are the changes needed?_ Document incremental collection, close #1215 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1914" alt="1" src="https://user-images.githubusercontent.com/26535726/157191008-451bee2a-eb6b-4bb6-869b-5b0f75a21448.png"> <img width="1919" alt="2" src="https://user-images.githubusercontent.com/26535726/157191016-e183bbf5-aa4a-496d-a250-5d14cf04101d.png"> <img width="1920" alt="3" src="https://user-images.githubusercontent.com/26535726/157191026-343a39d7-0ab8-4886-9a51-3670394ef6be.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2057 from pan3793/doc. Closes #1215 82677a03 [Cheng Pan] grammar 3be0e3f1 [Cheng Pan] fix b467e975 [Cheng Pan] Update d256cbaf [Cheng Pan] compress picture b3c5fb64 [Cheng Pan] Fix 4c06307c [Cheng Pan] [KYUUBI #1215][DOC] Document incremental collection Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
7887a47e02
|
[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
### _Why are the changes needed?_ [THRIFT-4805](https://issues.apache.org/jira/browse/THRIFT-4805) has been fixed in thrift 0.13.0, we can remove the workaround of log4j2 configurations because we use thrift 0.16.0 now, the change also removes the following warnings. ``` 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" ``` `maven-surefire-plugin` is only used in `kyuubi-hive-jdbc`, we should disable it in other modules. ``` [INFO] --- maven-surefire-plugin:2.22.0:test (default-test) kyuubi-ctl_2.12 --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1996 from pan3793/log. Closes #1996 291441c0 [Cheng Pan] nit f2c94246 [Cheng Pan] Tune surefire plugin in conf e1ee6402 [Cheng Pan] Clean up log4j2 conf Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
6d757a3df9
|
[KYUUBI #1963] Make plan only mode skippable for configuable plans
### _Why are the changes needed?_ Make plan only mode skippable for configuable plans. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1967 from SteNicholas/KYUUBI-1963. Closes #1963 e253aec6 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans b5cff405 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
SteNicholas
|
770499ca54
|
[KYUUBI #1948] Upgrade thrift version to 0.16.0
### _Why are the changes needed?_ Upgrade libthrift to 0.16.0 due to [CVE-2020-13949](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949) and the coming upstream change of Spark https://github.com/apache/spark/pull/34362 ### _What changed in this PR?_ - Upgrade libthrift to 0.16.0 - Shade and relocate `thrift` and `hive-service-rpc` classes in `kyuubi-spark-engine`, it's necessary to avoide conflicting with old thrift libs bundled in Spark binary releases. - Due to thrift change the method signature, the subclasses those interfaces in Kyuubi also need to modify to pass compile. We rely on Hive 2.3.9 jars in some components, e.g. `kyuubi-hive-jdbc`, `LocalMetaServer` in `kyuubi-server` test classes. Some classes in Hive jars compiled against old thrift interfaces which are not compatible with thrift 0.16.0, it causes runtime link error, we found the following classes which breaks the test and copied them with necessary modification to make it work with thrift 0.16.0. - `TFramedTransport` - `TFilterTransport` - `TUGIAssumingTransport` - `TUGIContainingTransport` - Next Steps, I think it's worth to do in separated PRs. - Recover the `HiveDelegationTokenProviderSuite`, one approach is use an isolate classloader to load HMS classes and thrift 0.9.3 classes from Maven, this approach can also be used for the planed Zoopkeeper upgrading to help us verficating the compatibility of Zookeeper Server 3.4.x. - Rewrite `kyuubi-hive-jdbc` to make it decouple with Hive jars, because there maybe other places which may not work with thrift 0.16.0 but the UTs does not cover. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1953 from SteNicholas/KYUUBI-1948. Closes #1948 de5d1ea2 [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 898effcd [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 803e270c [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0 Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Paul Lin
|
e54d431024
|
[KYUUBI #1883] Support max result rows for Flink queries
### _Why are the changes needed?_ Currently, Flink engine would pull all result rows into memory before returning it to the client. This would be problematic for large result sets and infinite result sets. This is a sub-task of KPIP-2 https://github.com/apache/incubator-kyuubi/issues/1322. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1938 from link3280/feature/FLINK-1883. Closes #1883 80020cee [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala 1b958221 [Paul Lin] [KYUUBI #1883] Avoid allocating too much buffer space 5be7535c [Paul Lin] [KYUUBI #1883] Support max result rows for Flink queries Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
1d64bb279e
|
[KYUUBI #1796] Destroy the process after the engine starts
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Destroy the process after the engine starts. #1796 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate add conf: `kyuubi.session.engine.startup.waitCompletion false`  - [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1815 from wForget/KYUUBI-1796. Closes #1796 12aea19a [Wang Zhen] fix 27cb602a [Wang Zhen] fix e70e7071 [Wang Zhen] add log ecaf828c [Wang Zhen] regenerate settings.md a5f6c5de [Wang Zhen] fix desc 09812beb [Wang Zhen] Merge remote-tracking branch 'origin/master' into KYUUBI-1796 27fab5eb [Wang Zhen] [KYUUBI-1796] Destroy the process after the engine starts Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
6c6238c2ae
|
[KYUUBI #1919] Add more enum values for OperationModes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add more enum values for `OperationModes`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1933 from SteNicholas/KYUUBI-1919. Closes #1919 adcc7f5e [SteNicholas] [KYUUBI #1919] Add more enum values for OperationModes Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
24e265d790
|
[KYUUBI #1876] Set Spark 3.2.1 as default
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Set Spark 3.2.1 as default. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1913 from SteNicholas/KYUUBI-1876. Closes #1876 1a6e4c6c [SteNicholas] [KYUUBI #1876] Set Spark 3.2.1 as default Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
681153036d
|
[KYUUBI #1536] Add session conf advisor
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> closes https://github.com/apache/incubator-kyuubi/issues/1536 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1892 from ulysses-you/KYUUBI-1536. Closes #1536 f0c450a9 [ulysses-you] codecov 35fac4f6 [ulysses-you] fix f34cfbf4 [ulysses-you] comment 78a54bc1 [ulysses-you] address comment 5cb574ba [ulysses-you] fix test 1033909b [ulysses-you] fix 3c925e56 [ulysses-you] address comment a74988cd [ulysses-you] style 649f02d0 [ulysses-you] Add session conf advisor debd3adb [ulysses-you] Add session conf advisor Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sychen
|
677e2a40ab
|
[KYUUBI #1889] Improve the user experience of the configuration parameters of the document
### _Why are the changes needed?_ https://kyuubi.apache.org/docs/latest/deployment/settings.html The configuration parameters in the document are too long and a newline is inserted(`<br/>`), which makes it difficult to search. And if the user copies the configuration parameters, it may become several lines. Use `<code>` to replace `<br/>`. https://github.com/apache/incubator-kyuubi/issues/1889 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1890 from cxzl25/KYUUBI-1889. Closes #1889 765b0622 [sychen] format metrics.md abeb0601 [sychen] add newline 28865ad7 [sychen] increase max width 2f9c7e05 [sychen] use <code> Authored-by: sychen <sychen@trip.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
ec719b82de
|
[KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Revert the dependency changes for kyuubi-hive-jdbc and kyuubi-hive-jdbc-shaded modules from #1842 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1854 from turboFei/slf4j-api. Closes #1769 dffe6c5c [Fei Wang] reserve for kyuubi-hive-beeline 20012434 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-beeline and kyuubi-hive-jdbc(shaded) modules Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
33d53db3ff
|
[KYUUBI #1813] Enable the internal secure authentication between Kyuubi server and engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Enable the internal secure access between Kyuubi server and engine. This pr use secret and cipher to encrypt a dynamic token identifier for authentication. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1813 from turboFei/engine_secure. Closes #1813 aa7ad0de [Fei Wang] run a simple query for engien security 34354d97 [Fei Wang] requre zk node must be defined for zk secret provider f84ce7c9 [Fei Wang] Require zk secret node non empty b545281d [Fei Wang] refactor d8d6dfd2 [Fei Wang] address comments 0e9273e1 [Fei Wang] refactor provider conf d3263c08 [Fei Wang] place holder e1266e32 [Fei Wang] reformat 9e252fea [Fei Wang] enhance prompt dbc9100c [Fei Wang] fix conf ff6b6dea [Fei Wang] refactor 4ea020ed [Fei Wang] refactor 20915902 [Fei Wang] refactor a7b4e41e [Fei Wang] add more ut: 1e1ceb27 [Fei Wang] use orignal cipher 8d8387b2 [Fei Wang] save ad777cb4 [Fei Wang] refactor b8997313 [Fei Wang] save b997095d [Fei Wang] refactor e05b47c5 [Fei Wang] move a322b784 [Fei Wang] refactor b14f7cbf [Fei Wang] Enable the internal secure authentication Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
53d59a02bf |
[KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This patch proposes to migrate from log4j1 to log4j2 in Kyuubi. Refer the spark patch: https://github.com/apache/spark/pull/34895 ### Does this PR introduce any user-facing change? Yes. Users may need to rewrite their log4j properties file for log4j2. As of version 2.4, log4j now supports configuration via properties files. Note that the property syntax is not the same as the syntax used in log4j 1, but during the migration I found the syntax is pretty close so the migration should be straightforward. ### _How was this patch tested?_ Passed all existing tests. Closes #1769 from turboFei/log4j2. Closes #1769 8613779c [Fei Wang] remove log4j dependencies from spark-sql-engine b2fe6dba [Fei Wang] Use String to present default log level 762e2d03 [Fei Wang] only remove org.apache.logging.log4j:log4j-slf4j-impl 8a912086 [Fei Wang] remove dependencies from spark-sql-engine 7e3a4980 [Fei Wang] address commments 051f49f5 [Fei Wang] address comments 85316a0b [Fei Wang] Keep compatible with log4j12 01d1a84e [Fei Wang] for log4j1 b9e17e1b [Fei Wang] refactor e24391ed [Fei Wang] revert log count 38803002 [Fei Wang] add log4j2.properties.template 4f0b22fc [Fei Wang] save 7ce84119 [Fei Wang] modify log level 1ea5ca53 [Fei Wang] add log4j to engine c4a86d4d [Fei Wang] use AbstractFilter 27b08b6a [Fei Wang] remove more 8cc15ae7 [Fei Wang] reformat c13ec29e [Fei Wang] save temporarily 33a38e2e [Fei Wang] exclude log4j12 from spark-sql 9129a64a [Fei Wang] refactor 5362b43d [Fei Wang] make it run at first 7f27f519 [Fei Wang] more 56f4f1ff [Fei Wang] fix logging a74b6d37 [Fei Wang] start appender dea964aa [Fei Wang] fix build erorr at first e20b7500 [Fei Wang] address comments 2ec02b4d [Fei Wang] fix LogDivertAppender dded1290 [Fei Wang] more c63e0008 [Fei Wang] add log4j2.properties Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Wang Zhen
|
916f81abaf
|
[KYUUBI #1775] Add kyuubi.server.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add `kyuubi.server.name` configuration. #1775 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate   - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1789 from wForget/KYUUBI-1775. Closes #1775 8b895044 [Wang Zhen] [KYUUBI-1775] Add kyuubi.server.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
b91854e33b
|
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR mainly focus on refactoring 1. Use Apache Thrift Naming style for thrift-based classes which is `TXyzAbc`, e.g. `TFrontendService` 2. Created `TFrontendService` and move the `TCLIService.Iface` APIs and other common fields in `TFrontendService`, later we will use it to implement the `KyuubiTHttpFrontendService` in the server module 3. introduce a new conf also for server FEs to be able to pick IP first ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1761 from yaooqinn/http1. Closes #1761 98b4afdd [Kent Yao] address comment 3138af29 [Kent Yao] [WIP] Thrift Http protocol support 1 a06026af [Kent Yao] [WIP] Thrift Http protocol support 1 efa899fa [Kent Yao] [WIP] Thrift Http protocol support 1 d1403e30 [Kent Yao] [WIP] Thrift Http protocol support 1 d114efd4 [Kent Yao] [WIP] Thrift Http protocol support 1 86e30c47 [Kent Yao] [WIP] Thrift Http protocol support 1 8117d12d [Kent Yao] [WIP] Thrift Http protocol support 1 62194323 [Kent Yao] [WIP] Thrift Http protocol support 1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
hongdongdong
|
11556034e3
|
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add TRINO engine type and TrinoProcessBuilder ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1695 from hddong/add-trino-process-builder. Closes #1510 394b853e [hongdongdong] fix 044ca8cc [hongdongdong] fix 568b1b8e [hongdongdong] use java options to pass args fbe7e60d [hongdongdong] [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
cb8721fef1
|
[KYUUBI #1629] Flink backend implementation
### _Why are the changes needed?_ This PR covers #1619. Overall, this PR contains the following changs, 1. change `build/dist` script to support flink sql engine 2. enhance `externals/flink-sql-engine/pom.xml` to support create a shaded jar 3. simplify `externals/kyuubi-flink-sql-engine/bin/flink-sql-engine.sh` 4. introduce `FlinkSQLEngine`(flink sql engine entrypoint) and `FlinkProcessBuilder`(kyuubi server launcher) 5. add ut in kyuubi server side After this PR, we can run the basic query e.g. `select now()` from beeline and get result, and Kyuubi Server can auto launch flink engine if there is no proper one. The Flink engine also supports other engine share levels defined in Kyuubi. The implementation based on Flink 1.14 codebase. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1629 from pan3793/flink-backend. Closes #1629 b7e5f0e3 [Cheng Pan] revert tgz name change a4496c37 [Cheng Pan] Fix reflection 3b4e86a1 [Cheng Pan] deps 68efa42c [Cheng Pan] log 8a9e37f3 [Cheng Pan] nit 10fb2bc3 [Cheng Pan] CI c1560fdb [Cheng Pan] nit 303e2f1e [Cheng Pan] Restore log conf d84720b1 [Cheng Pan] SessionContext b258d81a [Cheng Pan] cleanup 16edd528 [Cheng Pan] Cleanup 9ae54557 [Cheng Pan] Fix CI 25b6b57d [Cheng Pan] hadoop-client-api c12b5ca4 [Cheng Pan] Server UT pass 502d3f08 [Cheng Pan] pass dac4323b [yanghua] Laungh local flink engine container successfully Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
be3a964f37
|
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add kyuubi.engine.pool.name configuration. #1554 ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1572 from wForget/KYUUBI-1554. Closes #1554 40850f5a [Wang Zhen] [KYUUBI-1554] Add kyuubi.engine.pool.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
01da9b432e
|
[KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1556. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1562 from byyue/dev-1556. Closes #1556 8ceb244a [Brian Yue] Update the doc fdb73c4d [Brian Yue] Shorten kyuubi.session.engine.check.interval to one minute Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d805f49fe7
|
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1557 from yaooqinn/ttk. Closes #1557 8446180a [Kent Yao] [DOC] The TTL Of Kyuubi Engines Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
602bb23d36
|
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
### _Why are the changes needed?_ Compatible with lower version of hive driver. https://github.com/apache/incubator-kyuubi/issues/1174 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1553 from cxzl25/KYUUBI-1174-followup. Closes #1174 39cbdf4c [sychen] trigger test 908c4750 [sychen] gen settings f9e2eda0 [sychen] typo 25300d56 [sychen] Modify the default value to false Authored-by: sychen <sychen@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
2f46d4cc49
|
[KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE #1533 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1534 from zwangsheng/KYUUBI/1533. Closes #1533 803afeed [zwangsheng] fix 9fbc82e0 [zwangsheng] add doc Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
c30d580686
|
[KYUUBI #1527] [DOC] Improve High Availability Guide
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1527 from yaooqinn/hadoc. Closes #1527 249701b6 [Kent Yao] [DOC] Improve High Availability Guide 03244332 [Kent Yao] [DOC] Improve High Availability Guide 54c1cd9d [Kent Yao] [DOC] Improve High Availability Guide 6d8a0991 [Kent Yao] [DOC] Improve High Availability Guide 5b4dff3d [Kent Yao] [DOC] Improve High Availability Guide Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
97c3835738
|
[KYUUBI #1490] Introduce the basic framework for running scala
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce the basic framework for running scala, see #1490 for the detail ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` Beeline version 1.5.0-SNAPSHOT by Apache Kyuubi (Incubating) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:07.556 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.558 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.559 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:07.562 INFO scheduler.DAGScheduler: Asked to cancel job group 321dc15d-68d0-4f91-9216-1e08f09842df 2021-12-03 13:47:07.565 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.006 seconds 2021-12-03 13:47:07.565 ERROR operation.ExecuteStatement: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Query[08b8b6da-d434-4296-b613-2027e3518441] in ERROR_STATE 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.009 seconds Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> set kyuubi.operation.language=scala; 2021-12-03 13:47:11.982 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.983 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.984 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Execute in full collect mode 2021-12-03 13:47:12.006 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b] in FINISHED_STATE 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds +----------------------------+--------+ | key | value | +----------------------------+--------+ | kyuubi.operation.language | scala | +----------------------------+--------+ 1 row selected (0.052 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:13.685 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:15.541 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:15.543 INFO operation.ExecuteStatement: Query[178ff72d-b870-44d4-99d8-08a0fa0e8efa] in FINISHED_STATE 2021-12-03 13:47:15.544 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: RUNNING_STATE -> FINISHED_STATE, statement: spark.version, time taken: 0.003 seconds +-----------------------+ | output | +-----------------------+ | res0: String = 3.1.2 | +-----------------------+ 1 row selected (1.871 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.sql("select current_date()") . . . . . . . . . . . . . . . . . . .> ; 2021-12-03 13:47:36.512 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.689 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: PENDING_STATE -> RUNNING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Query[602c2d88-7e8a-4175-9b53-e34adfff007a] in FINISHED_STATE 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: RUNNING_STATE -> FINISHED_STATE, statement: spark.sql("select current_date()"), time taken: 0.003 seconds +----------------------------------------------------+ | output | +----------------------------------------------------+ | res1: org.apache.spark.sql.DataFrame = [current_date(): date] | +----------------------------------------------------+ 1 row selected (0.187 seconds) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3); Error: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error: results += spark.range(1, 5, 2, 3) <console>:26: error: type mismatch; found : org.apache.spark.sql.Dataset[Long] required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] results += spark.range(1, 5, 2, 3) ^ at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.spark.operation.ExecuteScala.runInternal(ExecuteScala.scala:70) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130) at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93) at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:62) at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:121) at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75) at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:118) at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:61) at org.apache.kyuubi.service.ThriftBinaryFrontendService.ExecuteStatement(ThriftBinaryFrontendService.scala:265) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3).toDF; 2021-12-03 13:48:30.327 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: INITIALIZED_STATE -> PENDING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.700 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: PENDING_STATE -> RUNNING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Query[daccd163-978a-4d66-af85-af466086d1db] in FINISHED_STATE 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: RUNNING_STATE -> FINISHED_STATE, statement: results += spark.range(1, 5, 2, 3).toDF, time taken: 0.003 seconds +-----+ | id | +-----+ | 1 | | 3 | +-----+ 2 rows selected (1.387 seconds) ``` ** Session level isolated **  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1491 from yaooqinn/scala. Closes #1490 af4d0a13 [Kent Yao] Merge branch 'master' into scala 2ebdf6a3 [Kent Yao] provided scala dep d58b2bfe [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala d256bfde [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala 52090434 [Kent Yao] init Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fei Wang
|
4648b63a2c |
[KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1379 from turboFei/async_open_default. Closes #1379 a7c4c6a1 [Fei Wang] fix ut cc8d2730 [Fei Wang] Make SESSION_ENGINE_LAUNCH_ASYNC true by default Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Wang Zhen
|
cc40c87743
|
[KYUUBI #1472] allow the engine not to self-terminate.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Allow the engine not to self-terminate. For details: #1472. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1475 from wForget/KYUUBI-1472. Closes #1472 97412e54 [Wang Zhen] Merge remote-tracking branch 'remotes/origin/master' 8ff0cf72 [Wang Zhen] [KYUUBI-1472] allow the engine not to self-terminate. Use zero or negative to represent no limit. 7cfa03c1 [Wang Zhen] [KYUUBI-1472] allow the engine not to self-terminate. Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zwangsheng
|
96963ab293
|
[KYUUBI #1462] [DOC] Use relative paths instead of absolute paths
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Use relative paths instead of absolute paths, avoid jumping to the latest document page. 2. fix `setting` to `settings` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1462 from zwangsheng/doc/replace_html_to_md. Closes #1462 09c97004 [zwangsheng] remove 70379580 [zwangsheng] use uri replace url & fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zwangsheng
|
3968832815
|
[KYUUBI #1454] Add Doc About Deploy Kyuubi On Kubernetes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The Kyuubi documentation is missing about deploying Kyuubi on Kubernetes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1457 from zwangsheng/doc/kyuubi_on_k8s. Closes #1454 f916866f [zwangsheng] modify .html to .md 707368ce [zwangsheng] fix url & add todo 1a509781 [zwangsheng] Fix spelling a624a602 [zwangsheng] fix 4ca74164 [zwangsheng] rename & add Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
wForget
|
b7e94ff45c
|
[KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. For details: #1439. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1440 from wForget/KYUUBI-1439. Closes #1439 bbbf1f23 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. update settings.md da7779a7 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. Authored-by: wForget <643348094@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
116369ac84
|
[KYUUBI #1376] Implement MySQL Frontend based on Netty
### _Why are the changes needed?_ Close #1219. Implment MySQL text protocol, support basic query use mysql cli to run Spark SQL. Test: Add `MySQLSparkQuerySuite`. Change `OutputSchemaTPCDSSuite` from **Thrift binary protocol** to **MySQL protocol**. Limitations: 1) only support mysql_native_password authentication mechanism 2) only support MySQL text protocol 3) not support send back result which row larger than 16M 4) not support server-side prepared statement 5) not support overwrite engine_conf/session_conf via JDBC url parameters like Hive JDBC client 6) limited support MySQL metadata queries, thus some MySQL GUI client which send metadata queries implicitly may not work as expected 7) not support SSL ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1376 from pan3793/mysql-fe-impl. Closes #1376 96d3efb9 [Cheng Pan] Kyuubi MySQL FE Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Jagadesh Adireddi
|
4a9a853e69
|
[KYUUBI #1181] [BUG] Format file paths to specifications.
Issue: https://github.com/apache/incubator-kyuubi/issues/1181 ### _Why are the changes needed?_ Format file paths of local path and hdfs path to specifications Closes #1420 from jadireddi/1181-Format. Closes #1181 32aacc7f [Jagadesh Adireddi] update doc 679369ea [Jagadesh Adireddi] [KYUUBI #1181] [BUG] Format file paths to specifications. Authored-by: Jagadesh Adireddi <jadiredd@conviva.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
e5e49b5b1d
|
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1414 from turboFei/rebase-kyuubi_beeline. Closes #1414 4694bbca [fwang12] return engine launch handle anyway 20ca1400 [fwang12] address comments 2999c30a [fwang12] fix ut 4d3abf4f [fwang12] jars.target.dir to target 4659f1f6 [fwang12] add readme 7e74c5c8 [fwang12] integrate beeline and server 969af045 [fwang12] fix beeline shell 90395ade [fwang12] BeeLine version By Apache Kyuubi 4e63695e [fwang12] fix ut 026e41ef [fwang12] refactor properties a49677af [fwang12] refactor build and beeline script ca68eba1 [fwang12] use beeline mode property instead of static method 4a1932e7 [fwang12] cli -> beeline 624281e3 [fwang12] nit d7d27a09 [fwang12] refactor kyuubi cli assembly 9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete cc025ecf [fwang12] Rebase PR_1407 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
yanyu34946
|
93e9745a39
|
[KYUUBI #1293] Fix potential failure of engine discovery when mixed use subdomain
### _Why are the changes needed?_ Fix #1293 This PR is a rework of #1303, also related to #962 We have two options to fix this issue. This PR implements option 2. ``` [option 1] kyuubi.engine.share.level.subdomain: String -- default: "default" kyuubi.engine.pool.size: Int -- default: -1 val subdomain = if (kyuubi.engine.share.level.subdomain == "default" && kyuubi.engine.pool.size > 0) { s"engine-pool-[num]" } else { kyuubi.engine.share.level.subdomain } [option 2] kyuubi.engine.share.level.subdomain: Option[String] -- default: None kyuubi.engine.pool.size: Int -- default: -1 val subdomain = kyuubi.engine.share.level.subdomain match { case Some(value) => value case None if kyuubi.engine.pool.size > 0 => s"engine-pool-[num]" case None => "default" } ``` ``` [diff] [[case1]] kyuubi.engine.share.level.subdomain=default kyuubi.engine.pool.size=2 option 1 result: subdomain="engine-pool-[num]" option 2 result: subdomain="default" [[case2]] kyuubi.engine.share.level.subdomain=hello kyuubi.engine.pool.size=2 option 1 result: subdomain="hello" option 2 result: subdomain="hello" [[case3]] kyuubi.engine.share.level.subdomain= kyuubi.engine.pool.size=2 option 1 result: subdomain="engine-pool-[num]" option 2 result: subdomain="engine-pool-[num]" [[case4]] kyuubi.engine.share.level.subdomain= kyuubi.engine.pool.size= option 1 result: subdomain="default" option 2 result: subdomain="default" [[case5]] kyuubi.engine.share.level.subdomain="hello" kyuubi.engine.pool.size= option 1 result: subdomain="hello" option 2 result: subdomain="hello" ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1402 from pan3793/1293. Closes #1293 f6c4a7db [Cheng Pan] nit fd427bfb [yanyu34946] [KYUUBI #1293] Fix bootstrap potential failure when mixed use subdomain Lead-authored-by: yanyu34946 <875082356@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
bf9736e31b
|
[KYUUBI #1399] [DOCS] Add doc for engine share level
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> doc improvement ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  ---  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1399 from yaooqinn/sldoc. Closes #1399 d8bcce8a [Kent Yao] [DOCS] Add doc for engine share level 21e2548f [Kent Yao] [DOCS] Add doc for engine share level bba8dc9d [Kent Yao] [DOCS] Add doc for engine share level Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
1554032df2 |
[KYUUBI #1346] Support launch query engine asynchronously during opening session
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support launch query engine asynchronously during opening session, make it more user-friendly. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1346 from turboFei/kyuubi_hive_jdbc_ext. Closes #1346 513691d6 [fwang12] address comments 63b8f901 [fwang12] remove hanle to client 4590ad98 [fwang12] remove synchronized f3b136b4 [fwang12] add ut for open session with failure 2cae9afa [fwang12] address comments 35113d0f [fwang12] add engine session id to session event 34c27f9c [fwang12] refactor log b5f7e639 [fwang12] block b8b0cf9d [fwang12] to support aync client 66eb07f3 [fwang12] address comments 623c0511 [fwang12] volatile 282c9f07 [fwang12] refactor conf 5047d91d [fwang12] server session handle is not same with that of engine dc1cb8bc [fwang12] update docs b773fd9d [fwang12] refactor 5f657e44 [fwang12] add ut 9e56cbab [fwang12] exclude engine init engine 6abdb52c [fwang12] refactor d3e37a8c [fwang12] block until engine finished 085b324d [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
884de795c2
|
[KYUUBI #1361] Update available version of backported configurations to 1.3.2
### _Why are the changes needed?_ Sub task of #1361 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1373 from zhouyifan279/1361-4. Closes #1361 86b92e86 [zhouyifan279] [KYUUBI #1361] Update available version of backported configurations to 1.3.2 Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
1061d176c8
|
[KYUUBI #1358] Add KyuubiMySQLFrontendService stub
### _Why are the changes needed?_ Add KyuubiMySQLFrontendService stub, without Netty pipeline, part of #1334 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1358 from pan3793/mysql-fe-stub. Closes #1358 c091d955 [Cheng Pan] nit 61abb0f6 [Cheng Pan] Address comments 9ac70456 [Cheng Pan] Update conf 3d75acdf [Cheng Pan] nit 4a4d8a24 [Cheng Pan] Address comments 0136dd52 [Cheng Pan] nit 477474ff [Cheng Pan] KyuubiMySQLFrontendService Stub Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
32c3f5cb2e
|
[KYUUBI #1349] Add new conf kyuubi.frontend.protocols
### _Why are the changes needed?_ Adapt to multiple frontends protocol. This PR also marks REST API as experimental and exposes it to users. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1349 from pan3793/frontend-conf. Closes #1349 d113c4a6 [Cheng Pan] Simplify code by SAM 129c2c9a [Cheng Pan] Fix compile fa8f7f67 [Cheng Pan] Add new conf kyuubi.frontend.protocols Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhenjiaguo
|
0f1d4ec0e0
|
[KYUUBI #1347] [DOC] Fix miscellaneous doc typos
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Some spelling & grammar format fix. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1347 from zhenjiaguo/typo-fix. Closes #1347 5c20ae28 [zhenjiaguo] change serial to several db3fa969 [zhenjiaguo] typo fix Authored-by: zhenjiaguo <zhenjia_guo@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
5421b56440
|
[KYUUBI #1327] Fix socket timeout error when client sync execute statement cost time longer than engine.login.timeout
### _Why are the changes needed?_ Fix #1327 #### Analysis The typical error logs are ```log 2021-11-01 11:27:54.018 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:47 69 37 B3 13 38 48 DA 87 7A 8A B6 BD 22 FA 57, secret:C1 01 AE 0B 6F 5F 48 F1 9A F0 FD 84 E3 0F 2B 1E), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: INITIALIZED_STATE -> PENDING_STATE, statement: [...SQL] 2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: PENDING_STATE -> RUNNING_STATE, statement: [...SQL] 2021-11-01 11:28:09.034 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: RUNNING_STATE -> ERROR_STATE, statement: [...SQL], time taken: 15.015 seconds 2021-11-01 11:28:09.035 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: ERROR_STATE -> CLOSED_STATE, statement: [...SQL] 2021-11-01 11:28:09.035 WARN server.KyuubiThriftBinaryFrontendService: Error executing statement: org.apache.kyuubi.KyuubiSQLException: Error operating EXECUTE_STATEMENT: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out [...omit detail stacktrace here] ``` The key points here are: 1. client execute query in sync mode, Hive JDBC client use async mode since 2.1.0 [HIVE-6535](https://issues.apache.org/jira/browse/HIVE-6535) 2. query execute cost time great than `kyuubi.session.engine.login.timeout`, which default is `15s` Kyuubi server create Thrift clients use `kyuubi.session.engine.login.timeout` as both `socket_timeout` and `connect_timeout`, and there is no heartbeat/keepalive mechanism in Thrift Protocol layer, thus if engine does not send response to Kyuubi server in `socket_timeout`, the Thrift client(Kyuubi server) will fail with `SocketTimeoutException: Read timed out`. In sync mode, when user send a execute statement request to Kyuubi server, Kyuubi server forword the request to the engine, engine return nothing until the execution finished or any other error happend. In async mode, the query request passes in same way, but engine will return an `op_handle` immediately instead of waiting for query to finish, the client should use the `op_handle` to check query status periodically, in detail 1. Kyuubi server ask engine for query status, and keep the result in memory 2. User ask Kyuubi server for query status, Kyuubi server get result from memory #### Solution Option 1: change `socket_timeout` of Thrift clients created by Kyuubi server to infinite. This approach may cause another issue. if engine crash when executing the query, Kyuubi server will wait until the socket keepalive(controlled by OS) timeout, so the query status will always be `running` in Kyuubi server memory. Option 2: always run a query in async mode, simulate sync mode in Server side. This PR implement the option 2, also introduce a new conf `kyuubi.session.engine.request.timeout` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1328 from pan3793/timeout. Closes #1327 cc15e6d5 [Cheng Pan] Always execute statement in async mode dc91d0d5 [Cheng Pan] Add new conf `kyuubi.session.engine.request.timeout` Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
b29b39b116
|
[KYUUBI #1323] Introduce EngineType to distinguish multiple engines
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1326 from yanghua/KYUUBI-1323. Closes #1323 aae6dcdd [yanghua] refactor enum to follow scala style cb5b67c4 [yanghua] addressed review suggestion and fixed test issue 5d73258a [yanghua] regenerated document 3824e838 [yanghua] [KYUUBI #1323] Introduce EngineType to distinguish multiple engines Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fu Chen
|
cc22888fa7
|
[KYUUBI #1320] Fix HA documents
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. the default value of `kyuubi.ha.zookeeper.namespace` is `kyuubi` 2. the beeline's variable `zooKeeperNamespace` is case-sensitive 3. how to configure kyuubi ha ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1320 from cfmcgrady/ha-bug. Closes #1320 16f56377 [Fu Chen] add ha configurations link 7acee980 [Fu Chen] update high_availability_guide.md bfcab300 [Fu Chen] update high_availability_guide.md 7d9f88cb [Fu Chen] Revert "add conf kyuubi.ha.enabled" d6874800 [Fu Chen] Revert "update docs" b31d9ca8 [Fu Chen] Revert "default false" 343bae44 [Fu Chen] default false 3a4934e8 [Fu Chen] update docs 7acf3d52 [Fu Chen] add conf kyuubi.ha.enabled 936d6e7d [Fu Chen] update high_availability_guide.md Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
fd17dd0ae4
|
[KYUUBI #1300] Detecting critical errors
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For critical errors at engine side, it is not handled properly. For example,when engine oom - server may not be able to get operation statuses because of no response of engine side. In this case, client only get a ambiguous `read timeout` as a final cause. - the oom hook of engine side might directly crash the engine, when the server is still trying to get operation status In this PR, - a config for retry to make the operation status updating process more robust - make the engine oom hook only de register itself to make it able to recover for some transient errors ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1312 from yaooqinn/1300. Closes #1300 a715ecca [Kent Yao] add comments a816b0f0 [Kent Yao] refine 3557c927 [Kent Yao] add comments 2ed8bfb4 [Kent Yao] refine aefd2a7f [Kent Yao] update doc f2ea7e4c [Kent Yao] restore SparkOperation 386e4eac [Kent Yao] [KYUUBI #1300] Detecting critical errors Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
xiebo03
|
046317c715
|
[KYUUBI #1286] Increase kyuubi.session.engine.initialize.timeout default to 3 min
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close https://github.com/apache/incubator-kyuubi/issues/1286 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1288 from MyLanPangzi/session.engine.initialize.timeout-3min. Closes #1286 c8b6a54f [xiebo03] modify doc 40d1fbec [xiebo03] increase kyuubi.session.engine.initialize.timeout default Authored-by: xiebo03 <xiebo03@qutoutiao.net> Signed-off-by: Cheng Pan <chengpan@apache.org> |
