e75166fe7a
139 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Fei Wang
|
ec719b82de
|
[KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Revert the dependency changes for kyuubi-hive-jdbc and kyuubi-hive-jdbc-shaded modules from #1842 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1854 from turboFei/slf4j-api. Closes #1769 dffe6c5c [Fei Wang] reserve for kyuubi-hive-beeline 20012434 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-beeline and kyuubi-hive-jdbc(shaded) modules Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
33d53db3ff
|
[KYUUBI #1813] Enable the internal secure authentication between Kyuubi server and engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Enable the internal secure access between Kyuubi server and engine. This pr use secret and cipher to encrypt a dynamic token identifier for authentication. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1813 from turboFei/engine_secure. Closes #1813 aa7ad0de [Fei Wang] run a simple query for engien security 34354d97 [Fei Wang] requre zk node must be defined for zk secret provider f84ce7c9 [Fei Wang] Require zk secret node non empty b545281d [Fei Wang] refactor d8d6dfd2 [Fei Wang] address comments 0e9273e1 [Fei Wang] refactor provider conf d3263c08 [Fei Wang] place holder e1266e32 [Fei Wang] reformat 9e252fea [Fei Wang] enhance prompt dbc9100c [Fei Wang] fix conf ff6b6dea [Fei Wang] refactor 4ea020ed [Fei Wang] refactor 20915902 [Fei Wang] refactor a7b4e41e [Fei Wang] add more ut: 1e1ceb27 [Fei Wang] use orignal cipher 8d8387b2 [Fei Wang] save ad777cb4 [Fei Wang] refactor b8997313 [Fei Wang] save b997095d [Fei Wang] refactor e05b47c5 [Fei Wang] move a322b784 [Fei Wang] refactor b14f7cbf [Fei Wang] Enable the internal secure authentication Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
53d59a02bf |
[KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This patch proposes to migrate from log4j1 to log4j2 in Kyuubi. Refer the spark patch: https://github.com/apache/spark/pull/34895 ### Does this PR introduce any user-facing change? Yes. Users may need to rewrite their log4j properties file for log4j2. As of version 2.4, log4j now supports configuration via properties files. Note that the property syntax is not the same as the syntax used in log4j 1, but during the migration I found the syntax is pretty close so the migration should be straightforward. ### _How was this patch tested?_ Passed all existing tests. Closes #1769 from turboFei/log4j2. Closes #1769 8613779c [Fei Wang] remove log4j dependencies from spark-sql-engine b2fe6dba [Fei Wang] Use String to present default log level 762e2d03 [Fei Wang] only remove org.apache.logging.log4j:log4j-slf4j-impl 8a912086 [Fei Wang] remove dependencies from spark-sql-engine 7e3a4980 [Fei Wang] address commments 051f49f5 [Fei Wang] address comments 85316a0b [Fei Wang] Keep compatible with log4j12 01d1a84e [Fei Wang] for log4j1 b9e17e1b [Fei Wang] refactor e24391ed [Fei Wang] revert log count 38803002 [Fei Wang] add log4j2.properties.template 4f0b22fc [Fei Wang] save 7ce84119 [Fei Wang] modify log level 1ea5ca53 [Fei Wang] add log4j to engine c4a86d4d [Fei Wang] use AbstractFilter 27b08b6a [Fei Wang] remove more 8cc15ae7 [Fei Wang] reformat c13ec29e [Fei Wang] save temporarily 33a38e2e [Fei Wang] exclude log4j12 from spark-sql 9129a64a [Fei Wang] refactor 5362b43d [Fei Wang] make it run at first 7f27f519 [Fei Wang] more 56f4f1ff [Fei Wang] fix logging a74b6d37 [Fei Wang] start appender dea964aa [Fei Wang] fix build erorr at first e20b7500 [Fei Wang] address comments 2ec02b4d [Fei Wang] fix LogDivertAppender dded1290 [Fei Wang] more c63e0008 [Fei Wang] add log4j2.properties Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Wang Zhen
|
916f81abaf
|
[KYUUBI #1775] Add kyuubi.server.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add `kyuubi.server.name` configuration. #1775 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate   - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1789 from wForget/KYUUBI-1775. Closes #1775 8b895044 [Wang Zhen] [KYUUBI-1775] Add kyuubi.server.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
b91854e33b
|
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR mainly focus on refactoring 1. Use Apache Thrift Naming style for thrift-based classes which is `TXyzAbc`, e.g. `TFrontendService` 2. Created `TFrontendService` and move the `TCLIService.Iface` APIs and other common fields in `TFrontendService`, later we will use it to implement the `KyuubiTHttpFrontendService` in the server module 3. introduce a new conf also for server FEs to be able to pick IP first ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1761 from yaooqinn/http1. Closes #1761 98b4afdd [Kent Yao] address comment 3138af29 [Kent Yao] [WIP] Thrift Http protocol support 1 a06026af [Kent Yao] [WIP] Thrift Http protocol support 1 efa899fa [Kent Yao] [WIP] Thrift Http protocol support 1 d1403e30 [Kent Yao] [WIP] Thrift Http protocol support 1 d114efd4 [Kent Yao] [WIP] Thrift Http protocol support 1 86e30c47 [Kent Yao] [WIP] Thrift Http protocol support 1 8117d12d [Kent Yao] [WIP] Thrift Http protocol support 1 62194323 [Kent Yao] [WIP] Thrift Http protocol support 1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
hongdongdong
|
11556034e3
|
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add TRINO engine type and TrinoProcessBuilder ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1695 from hddong/add-trino-process-builder. Closes #1510 394b853e [hongdongdong] fix 044ca8cc [hongdongdong] fix 568b1b8e [hongdongdong] use java options to pass args fbe7e60d [hongdongdong] [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
cb8721fef1
|
[KYUUBI #1629] Flink backend implementation
### _Why are the changes needed?_ This PR covers #1619. Overall, this PR contains the following changs, 1. change `build/dist` script to support flink sql engine 2. enhance `externals/flink-sql-engine/pom.xml` to support create a shaded jar 3. simplify `externals/kyuubi-flink-sql-engine/bin/flink-sql-engine.sh` 4. introduce `FlinkSQLEngine`(flink sql engine entrypoint) and `FlinkProcessBuilder`(kyuubi server launcher) 5. add ut in kyuubi server side After this PR, we can run the basic query e.g. `select now()` from beeline and get result, and Kyuubi Server can auto launch flink engine if there is no proper one. The Flink engine also supports other engine share levels defined in Kyuubi. The implementation based on Flink 1.14 codebase. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1629 from pan3793/flink-backend. Closes #1629 b7e5f0e3 [Cheng Pan] revert tgz name change a4496c37 [Cheng Pan] Fix reflection 3b4e86a1 [Cheng Pan] deps 68efa42c [Cheng Pan] log 8a9e37f3 [Cheng Pan] nit 10fb2bc3 [Cheng Pan] CI c1560fdb [Cheng Pan] nit 303e2f1e [Cheng Pan] Restore log conf d84720b1 [Cheng Pan] SessionContext b258d81a [Cheng Pan] cleanup 16edd528 [Cheng Pan] Cleanup 9ae54557 [Cheng Pan] Fix CI 25b6b57d [Cheng Pan] hadoop-client-api c12b5ca4 [Cheng Pan] Server UT pass 502d3f08 [Cheng Pan] pass dac4323b [yanghua] Laungh local flink engine container successfully Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang Zhen
|
be3a964f37
|
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add kyuubi.engine.pool.name configuration. #1554 ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1572 from wForget/KYUUBI-1554. Closes #1554 40850f5a [Wang Zhen] [KYUUBI-1554] Add kyuubi.engine.pool.name configuration Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
01da9b432e
|
[KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1556. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1562 from byyue/dev-1556. Closes #1556 8ceb244a [Brian Yue] Update the doc fdb73c4d [Brian Yue] Shorten kyuubi.session.engine.check.interval to one minute Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d805f49fe7
|
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1557 from yaooqinn/ttk. Closes #1557 8446180a [Kent Yao] [DOC] The TTL Of Kyuubi Engines Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
602bb23d36
|
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
### _Why are the changes needed?_ Compatible with lower version of hive driver. https://github.com/apache/incubator-kyuubi/issues/1174 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1553 from cxzl25/KYUUBI-1174-followup. Closes #1174 39cbdf4c [sychen] trigger test 908c4750 [sychen] gen settings f9e2eda0 [sychen] typo 25300d56 [sychen] Modify the default value to false Authored-by: sychen <sychen@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
2f46d4cc49
|
[KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE #1533 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1534 from zwangsheng/KYUUBI/1533. Closes #1533 803afeed [zwangsheng] fix 9fbc82e0 [zwangsheng] add doc Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
c30d580686
|
[KYUUBI #1527] [DOC] Improve High Availability Guide
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1527 from yaooqinn/hadoc. Closes #1527 249701b6 [Kent Yao] [DOC] Improve High Availability Guide 03244332 [Kent Yao] [DOC] Improve High Availability Guide 54c1cd9d [Kent Yao] [DOC] Improve High Availability Guide 6d8a0991 [Kent Yao] [DOC] Improve High Availability Guide 5b4dff3d [Kent Yao] [DOC] Improve High Availability Guide Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
97c3835738
|
[KYUUBI #1490] Introduce the basic framework for running scala
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce the basic framework for running scala, see #1490 for the detail ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` Beeline version 1.5.0-SNAPSHOT by Apache Kyuubi (Incubating) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:07.556 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.558 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:07.559 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:07.560 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:07.562 INFO scheduler.DAGScheduler: Asked to cancel job group 321dc15d-68d0-4f91-9216-1e08f09842df 2021-12-03 13:47:07.565 INFO operation.ExecuteStatement: Processing kent's query[321dc15d-68d0-4f91-9216-1e08f09842df]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.006 seconds 2021-12-03 13:47:07.565 ERROR operation.ExecuteStatement: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Query[08b8b6da-d434-4296-b613-2027e3518441] in ERROR_STATE 2021-12-03 13:47:07.569 INFO operation.ExecuteStatement: Processing kent's query[08b8b6da-d434-4296-b613-2027e3518441]: RUNNING_STATE -> ERROR_STATE, statement: spark.version, time taken: 0.009 seconds Error: Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'spark' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0) == SQL == spark.version ^^^ at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:75) at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:98) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.withLocalProperties(ExecuteStatement.scala:157) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:92) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:125) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> set kyuubi.operation.language=scala; 2021-12-03 13:47:11.982 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.983 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.984 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.985 INFO operation.ExecuteStatement: Spark application name: kyuubi_USER_SPARK_SQL_kent_default_61cff9fb-7035-4435-b509-80c1730876ed application ID: local-1638510289918 application web UI: http://10.242.189.214:65027 master: local[*] deploy mode: client version: 3.1.2 Start time: 2021-12-03T13:44:49.313 User: kent 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.language=scala 2021-12-03 13:47:11.995 INFO operation.ExecuteStatement: Execute in full collect mode 2021-12-03 13:47:12.006 INFO operation.ExecuteStatement: Processing kent's query[2b6ec68c-2ae9-4d93-a35b-9e2319df4d4f]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b] in FINISHED_STATE 2021-12-03 13:47:12.007 INFO operation.ExecuteStatement: Processing kent's query[e4e4dc55-4e05-460e-9d9d-86b2cf5c054b]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.language=scala, time taken: 0.022 seconds +----------------------------+--------+ | key | value | +----------------------------+--------+ | kyuubi.operation.language | scala | +----------------------------+--------+ 1 row selected (0.052 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.version; 2021-12-03 13:47:13.685 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.version 2021-12-03 13:47:15.541 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: PENDING_STATE -> RUNNING_STATE, statement: spark.version 2021-12-03 13:47:15.543 INFO operation.ExecuteStatement: Query[178ff72d-b870-44d4-99d8-08a0fa0e8efa] in FINISHED_STATE 2021-12-03 13:47:15.544 INFO operation.ExecuteStatement: Processing kent's query[178ff72d-b870-44d4-99d8-08a0fa0e8efa]: RUNNING_STATE -> FINISHED_STATE, statement: spark.version, time taken: 0.003 seconds +-----------------------+ | output | +-----------------------+ | res0: String = 3.1.2 | +-----------------------+ 1 row selected (1.871 seconds) 0: jdbc:hive2://10.242.189.214:10009/> spark.sql("select current_date()") . . . . . . . . . . . . . . . . . . .> ; 2021-12-03 13:47:36.512 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: INITIALIZED_STATE -> PENDING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.689 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: PENDING_STATE -> RUNNING_STATE, statement: spark.sql("select current_date()") 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Query[602c2d88-7e8a-4175-9b53-e34adfff007a] in FINISHED_STATE 2021-12-03 13:47:36.692 INFO operation.ExecuteStatement: Processing kent's query[602c2d88-7e8a-4175-9b53-e34adfff007a]: RUNNING_STATE -> FINISHED_STATE, statement: spark.sql("select current_date()"), time taken: 0.003 seconds +----------------------------------------------------+ | output | +----------------------------------------------------+ | res1: org.apache.spark.sql.DataFrame = [current_date(): date] | +----------------------------------------------------+ 1 row selected (0.187 seconds) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3); Error: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error: results += spark.range(1, 5, 2, 3) <console>:26: error: type mismatch; found : org.apache.spark.sql.Dataset[Long] required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] results += spark.range(1, 5, 2, 3) ^ at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.engine.spark.operation.ExecuteScala.runInternal(ExecuteScala.scala:70) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130) at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93) at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:62) at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:121) at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75) at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:118) at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:61) at org.apache.kyuubi.service.ThriftBinaryFrontendService.ExecuteStatement(ThriftBinaryFrontendService.scala:265) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) (state=,code=0) 0: jdbc:hive2://10.242.189.214:10009/> results += spark.range(1, 5, 2, 3).toDF; 2021-12-03 13:48:30.327 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: INITIALIZED_STATE -> PENDING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.700 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: PENDING_STATE -> RUNNING_STATE, statement: results += spark.range(1, 5, 2, 3).toDF 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Query[daccd163-978a-4d66-af85-af466086d1db] in FINISHED_STATE 2021-12-03 13:48:31.702 INFO operation.ExecuteStatement: Processing kent's query[daccd163-978a-4d66-af85-af466086d1db]: RUNNING_STATE -> FINISHED_STATE, statement: results += spark.range(1, 5, 2, 3).toDF, time taken: 0.003 seconds +-----+ | id | +-----+ | 1 | | 3 | +-----+ 2 rows selected (1.387 seconds) ``` ** Session level isolated **  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1491 from yaooqinn/scala. Closes #1490 af4d0a13 [Kent Yao] Merge branch 'master' into scala 2ebdf6a3 [Kent Yao] provided scala dep d58b2bfe [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala d256bfde [Kent Yao] [KYUUBI #1490] Introduce the basic framework for running scala 52090434 [Kent Yao] init Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fei Wang
|
4648b63a2c |
[KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1379 from turboFei/async_open_default. Closes #1379 a7c4c6a1 [Fei Wang] fix ut cc8d2730 [Fei Wang] Make SESSION_ENGINE_LAUNCH_ASYNC true by default Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Wang Zhen
|
cc40c87743
|
[KYUUBI #1472] allow the engine not to self-terminate.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Allow the engine not to self-terminate. For details: #1472. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1475 from wForget/KYUUBI-1472. Closes #1472 97412e54 [Wang Zhen] Merge remote-tracking branch 'remotes/origin/master' 8ff0cf72 [Wang Zhen] [KYUUBI-1472] allow the engine not to self-terminate. Use zero or negative to represent no limit. 7cfa03c1 [Wang Zhen] [KYUUBI-1472] allow the engine not to self-terminate. Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zwangsheng
|
96963ab293
|
[KYUUBI #1462] [DOC] Use relative paths instead of absolute paths
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Use relative paths instead of absolute paths, avoid jumping to the latest document page. 2. fix `setting` to `settings` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1462 from zwangsheng/doc/replace_html_to_md. Closes #1462 09c97004 [zwangsheng] remove 70379580 [zwangsheng] use uri replace url & fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zwangsheng
|
3968832815
|
[KYUUBI #1454] Add Doc About Deploy Kyuubi On Kubernetes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The Kyuubi documentation is missing about deploying Kyuubi on Kubernetes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1457 from zwangsheng/doc/kyuubi_on_k8s. Closes #1454 f916866f [zwangsheng] modify .html to .md 707368ce [zwangsheng] fix url & add todo 1a509781 [zwangsheng] Fix spelling a624a602 [zwangsheng] fix 4ca74164 [zwangsheng] rename & add Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
wForget
|
b7e94ff45c
|
[KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. For details: #1439. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1440 from wForget/KYUUBI-1439. Closes #1439 bbbf1f23 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. update settings.md da7779a7 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. Authored-by: wForget <643348094@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
116369ac84
|
[KYUUBI #1376] Implement MySQL Frontend based on Netty
### _Why are the changes needed?_ Close #1219. Implment MySQL text protocol, support basic query use mysql cli to run Spark SQL. Test: Add `MySQLSparkQuerySuite`. Change `OutputSchemaTPCDSSuite` from **Thrift binary protocol** to **MySQL protocol**. Limitations: 1) only support mysql_native_password authentication mechanism 2) only support MySQL text protocol 3) not support send back result which row larger than 16M 4) not support server-side prepared statement 5) not support overwrite engine_conf/session_conf via JDBC url parameters like Hive JDBC client 6) limited support MySQL metadata queries, thus some MySQL GUI client which send metadata queries implicitly may not work as expected 7) not support SSL ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1376 from pan3793/mysql-fe-impl. Closes #1376 96d3efb9 [Cheng Pan] Kyuubi MySQL FE Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Jagadesh Adireddi
|
4a9a853e69
|
[KYUUBI #1181] [BUG] Format file paths to specifications.
Issue: https://github.com/apache/incubator-kyuubi/issues/1181 ### _Why are the changes needed?_ Format file paths of local path and hdfs path to specifications Closes #1420 from jadireddi/1181-Format. Closes #1181 32aacc7f [Jagadesh Adireddi] update doc 679369ea [Jagadesh Adireddi] [KYUUBI #1181] [BUG] Format file paths to specifications. Authored-by: Jagadesh Adireddi <jadiredd@conviva.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
e5e49b5b1d
|
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1414 from turboFei/rebase-kyuubi_beeline. Closes #1414 4694bbca [fwang12] return engine launch handle anyway 20ca1400 [fwang12] address comments 2999c30a [fwang12] fix ut 4d3abf4f [fwang12] jars.target.dir to target 4659f1f6 [fwang12] add readme 7e74c5c8 [fwang12] integrate beeline and server 969af045 [fwang12] fix beeline shell 90395ade [fwang12] BeeLine version By Apache Kyuubi 4e63695e [fwang12] fix ut 026e41ef [fwang12] refactor properties a49677af [fwang12] refactor build and beeline script ca68eba1 [fwang12] use beeline mode property instead of static method 4a1932e7 [fwang12] cli -> beeline 624281e3 [fwang12] nit d7d27a09 [fwang12] refactor kyuubi cli assembly 9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete cc025ecf [fwang12] Rebase PR_1407 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
yanyu34946
|
93e9745a39
|
[KYUUBI #1293] Fix potential failure of engine discovery when mixed use subdomain
### _Why are the changes needed?_ Fix #1293 This PR is a rework of #1303, also related to #962 We have two options to fix this issue. This PR implements option 2. ``` [option 1] kyuubi.engine.share.level.subdomain: String -- default: "default" kyuubi.engine.pool.size: Int -- default: -1 val subdomain = if (kyuubi.engine.share.level.subdomain == "default" && kyuubi.engine.pool.size > 0) { s"engine-pool-[num]" } else { kyuubi.engine.share.level.subdomain } [option 2] kyuubi.engine.share.level.subdomain: Option[String] -- default: None kyuubi.engine.pool.size: Int -- default: -1 val subdomain = kyuubi.engine.share.level.subdomain match { case Some(value) => value case None if kyuubi.engine.pool.size > 0 => s"engine-pool-[num]" case None => "default" } ``` ``` [diff] [[case1]] kyuubi.engine.share.level.subdomain=default kyuubi.engine.pool.size=2 option 1 result: subdomain="engine-pool-[num]" option 2 result: subdomain="default" [[case2]] kyuubi.engine.share.level.subdomain=hello kyuubi.engine.pool.size=2 option 1 result: subdomain="hello" option 2 result: subdomain="hello" [[case3]] kyuubi.engine.share.level.subdomain= kyuubi.engine.pool.size=2 option 1 result: subdomain="engine-pool-[num]" option 2 result: subdomain="engine-pool-[num]" [[case4]] kyuubi.engine.share.level.subdomain= kyuubi.engine.pool.size= option 1 result: subdomain="default" option 2 result: subdomain="default" [[case5]] kyuubi.engine.share.level.subdomain="hello" kyuubi.engine.pool.size= option 1 result: subdomain="hello" option 2 result: subdomain="hello" ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1402 from pan3793/1293. Closes #1293 f6c4a7db [Cheng Pan] nit fd427bfb [yanyu34946] [KYUUBI #1293] Fix bootstrap potential failure when mixed use subdomain Lead-authored-by: yanyu34946 <875082356@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
bf9736e31b
|
[KYUUBI #1399] [DOCS] Add doc for engine share level
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> doc improvement ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  ---  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1399 from yaooqinn/sldoc. Closes #1399 d8bcce8a [Kent Yao] [DOCS] Add doc for engine share level 21e2548f [Kent Yao] [DOCS] Add doc for engine share level bba8dc9d [Kent Yao] [DOCS] Add doc for engine share level Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
1554032df2 |
[KYUUBI #1346] Support launch query engine asynchronously during opening session
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support launch query engine asynchronously during opening session, make it more user-friendly. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1346 from turboFei/kyuubi_hive_jdbc_ext. Closes #1346 513691d6 [fwang12] address comments 63b8f901 [fwang12] remove hanle to client 4590ad98 [fwang12] remove synchronized f3b136b4 [fwang12] add ut for open session with failure 2cae9afa [fwang12] address comments 35113d0f [fwang12] add engine session id to session event 34c27f9c [fwang12] refactor log b5f7e639 [fwang12] block b8b0cf9d [fwang12] to support aync client 66eb07f3 [fwang12] address comments 623c0511 [fwang12] volatile 282c9f07 [fwang12] refactor conf 5047d91d [fwang12] server session handle is not same with that of engine dc1cb8bc [fwang12] update docs b773fd9d [fwang12] refactor 5f657e44 [fwang12] add ut 9e56cbab [fwang12] exclude engine init engine 6abdb52c [fwang12] refactor d3e37a8c [fwang12] block until engine finished 085b324d [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
884de795c2
|
[KYUUBI #1361] Update available version of backported configurations to 1.3.2
### _Why are the changes needed?_ Sub task of #1361 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1373 from zhouyifan279/1361-4. Closes #1361 86b92e86 [zhouyifan279] [KYUUBI #1361] Update available version of backported configurations to 1.3.2 Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
1061d176c8
|
[KYUUBI #1358] Add KyuubiMySQLFrontendService stub
### _Why are the changes needed?_ Add KyuubiMySQLFrontendService stub, without Netty pipeline, part of #1334 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1358 from pan3793/mysql-fe-stub. Closes #1358 c091d955 [Cheng Pan] nit 61abb0f6 [Cheng Pan] Address comments 9ac70456 [Cheng Pan] Update conf 3d75acdf [Cheng Pan] nit 4a4d8a24 [Cheng Pan] Address comments 0136dd52 [Cheng Pan] nit 477474ff [Cheng Pan] KyuubiMySQLFrontendService Stub Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
32c3f5cb2e
|
[KYUUBI #1349] Add new conf kyuubi.frontend.protocols
### _Why are the changes needed?_ Adapt to multiple frontends protocol. This PR also marks REST API as experimental and exposes it to users. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1349 from pan3793/frontend-conf. Closes #1349 d113c4a6 [Cheng Pan] Simplify code by SAM 129c2c9a [Cheng Pan] Fix compile fa8f7f67 [Cheng Pan] Add new conf kyuubi.frontend.protocols Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhenjiaguo
|
0f1d4ec0e0
|
[KYUUBI #1347] [DOC] Fix miscellaneous doc typos
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Some spelling & grammar format fix. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1347 from zhenjiaguo/typo-fix. Closes #1347 5c20ae28 [zhenjiaguo] change serial to several db3fa969 [zhenjiaguo] typo fix Authored-by: zhenjiaguo <zhenjia_guo@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
5421b56440
|
[KYUUBI #1327] Fix socket timeout error when client sync execute statement cost time longer than engine.login.timeout
### _Why are the changes needed?_ Fix #1327 #### Analysis The typical error logs are ```log 2021-11-01 11:27:54.018 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:47 69 37 B3 13 38 48 DA 87 7A 8A B6 BD 22 FA 57, secret:C1 01 AE 0B 6F 5F 48 F1 9A F0 FD 84 E3 0F 2B 1E), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: INITIALIZED_STATE -> PENDING_STATE, statement: [...SQL] 2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: PENDING_STATE -> RUNNING_STATE, statement: [...SQL] 2021-11-01 11:28:09.034 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: RUNNING_STATE -> ERROR_STATE, statement: [...SQL], time taken: 15.015 seconds 2021-11-01 11:28:09.035 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: ERROR_STATE -> CLOSED_STATE, statement: [...SQL] 2021-11-01 11:28:09.035 WARN server.KyuubiThriftBinaryFrontendService: Error executing statement: org.apache.kyuubi.KyuubiSQLException: Error operating EXECUTE_STATEMENT: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out [...omit detail stacktrace here] ``` The key points here are: 1. client execute query in sync mode, Hive JDBC client use async mode since 2.1.0 [HIVE-6535](https://issues.apache.org/jira/browse/HIVE-6535) 2. query execute cost time great than `kyuubi.session.engine.login.timeout`, which default is `15s` Kyuubi server create Thrift clients use `kyuubi.session.engine.login.timeout` as both `socket_timeout` and `connect_timeout`, and there is no heartbeat/keepalive mechanism in Thrift Protocol layer, thus if engine does not send response to Kyuubi server in `socket_timeout`, the Thrift client(Kyuubi server) will fail with `SocketTimeoutException: Read timed out`. In sync mode, when user send a execute statement request to Kyuubi server, Kyuubi server forword the request to the engine, engine return nothing until the execution finished or any other error happend. In async mode, the query request passes in same way, but engine will return an `op_handle` immediately instead of waiting for query to finish, the client should use the `op_handle` to check query status periodically, in detail 1. Kyuubi server ask engine for query status, and keep the result in memory 2. User ask Kyuubi server for query status, Kyuubi server get result from memory #### Solution Option 1: change `socket_timeout` of Thrift clients created by Kyuubi server to infinite. This approach may cause another issue. if engine crash when executing the query, Kyuubi server will wait until the socket keepalive(controlled by OS) timeout, so the query status will always be `running` in Kyuubi server memory. Option 2: always run a query in async mode, simulate sync mode in Server side. This PR implement the option 2, also introduce a new conf `kyuubi.session.engine.request.timeout` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1328 from pan3793/timeout. Closes #1327 cc15e6d5 [Cheng Pan] Always execute statement in async mode dc91d0d5 [Cheng Pan] Add new conf `kyuubi.session.engine.request.timeout` Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
b29b39b116
|
[KYUUBI #1323] Introduce EngineType to distinguish multiple engines
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1326 from yanghua/KYUUBI-1323. Closes #1323 aae6dcdd [yanghua] refactor enum to follow scala style cb5b67c4 [yanghua] addressed review suggestion and fixed test issue 5d73258a [yanghua] regenerated document 3824e838 [yanghua] [KYUUBI #1323] Introduce EngineType to distinguish multiple engines Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fu Chen
|
cc22888fa7
|
[KYUUBI #1320] Fix HA documents
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. the default value of `kyuubi.ha.zookeeper.namespace` is `kyuubi` 2. the beeline's variable `zooKeeperNamespace` is case-sensitive 3. how to configure kyuubi ha ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1320 from cfmcgrady/ha-bug. Closes #1320 16f56377 [Fu Chen] add ha configurations link 7acee980 [Fu Chen] update high_availability_guide.md bfcab300 [Fu Chen] update high_availability_guide.md 7d9f88cb [Fu Chen] Revert "add conf kyuubi.ha.enabled" d6874800 [Fu Chen] Revert "update docs" b31d9ca8 [Fu Chen] Revert "default false" 343bae44 [Fu Chen] default false 3a4934e8 [Fu Chen] update docs 7acf3d52 [Fu Chen] add conf kyuubi.ha.enabled 936d6e7d [Fu Chen] update high_availability_guide.md Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
fd17dd0ae4
|
[KYUUBI #1300] Detecting critical errors
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For critical errors at engine side, it is not handled properly. For example,when engine oom - server may not be able to get operation statuses because of no response of engine side. In this case, client only get a ambiguous `read timeout` as a final cause. - the oom hook of engine side might directly crash the engine, when the server is still trying to get operation status In this PR, - a config for retry to make the operation status updating process more robust - make the engine oom hook only de register itself to make it able to recover for some transient errors ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1312 from yaooqinn/1300. Closes #1300 a715ecca [Kent Yao] add comments a816b0f0 [Kent Yao] refine 3557c927 [Kent Yao] add comments 2ed8bfb4 [Kent Yao] refine aefd2a7f [Kent Yao] update doc f2ea7e4c [Kent Yao] restore SparkOperation 386e4eac [Kent Yao] [KYUUBI #1300] Detecting critical errors Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
xiebo03
|
046317c715
|
[KYUUBI #1286] Increase kyuubi.session.engine.initialize.timeout default to 3 min
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close https://github.com/apache/incubator-kyuubi/issues/1286 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1288 from MyLanPangzi/session.engine.initialize.timeout-3min. Closes #1286 c8b6a54f [xiebo03] modify doc 40d1fbec [xiebo03] increase kyuubi.session.engine.initialize.timeout default Authored-by: xiebo03 <xiebo03@qutoutiao.net> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
24cf1bd720
|
[KYUUBI #1262] Support both KERBEROS and PLAIN authentication at the same time
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support both KERBEROS and PLAIN authentication at the same time. ### _How was this patch tested?_ Added UT & IT. I make integration testing on our dev cluster with KERBEROS and CUSTOM authentication. ``` kyuubi.authentication KERBEROS,CUSTOM kyuubi.authentication.custom.class=org.apache.kyuubi.ZeusCustom ``` For org.apache.kyuubi.ZeusCustom, it checks whether the user equals password. ``` package org.apache.kyuubi import javax.security.sasl.AuthenticationException import org.apache.kyuubi.service.authentication.PasswdAuthenticationProvider class ZeusCustom extends PasswdAuthenticationProvider with Logging { override def authenticate(user: String, password: String): Unit = { if (user == password) { info(s"Success log in of user: $user") } else { throw new AuthenticationException("Username or password is not valid!") } } } ``` 1. kerberos testing with user b_zeus  2. CUTOM authentication testing with user b_zeus  Note that: they share the same backend spark engine, because they are the same user. Closes #1266 from turboFei/multiple_auth_KYUUBI-1262. Closes #1262 71053aef [fwang12] adress nit 850d6b5d [fwang12] fix ut ea7db79f [fwang12] complete 11f409cb [fwang12] Update docs b1f83e55 [fwang12] add ut 8d137db9 [fwang12] make ldap password diff with custom d227aa74 [fwang12] fix ut d7cfaf4c [fwang12] only the first is valid 2e2283ba [fwang12] after all ee0e8bc0 [fwang12] make kerberoes enabled 4fc63081 [fwang12] refactor kerbereos helper 6691cc57 [fwang12] save cd813ecf [fwang12] refactor dd706740 [fwang12] retest f4038e93 [fwang12] fix code style 7b590a23 [fwang12] add ut e39e19e6 [fwang12] add it 7dc7c927 [fwang12] with password 8dadfd32 [fwang12] refactor ldap suite 8545a033 [fwang12] add ut 1aa30a5c [fwang12] refactor 8cc2ea66 [fwang12] fix ut 10f788ae [fwang12] before all 98f93640 [fwang12] revert sth bb75f8e9 [fwang12] save 314579f1 [fwang12] update default ac8b195f [fwang12] [KYUUBI #1262] Support multiple kinds of SASL authentication type Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
d6cefb1b9c
|
[KYUUBI #1188] Add Statement stats on Kyuubi Query Engine Page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add Statement stats on Kyuubi Query Engine Page  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1234 from timothy65535/ky-1188. Closes #1188 6e0ac523 [timothy65535] trigger rebuild 2b2b2a03 [timothy65535] update patch f0431359 [timothy65535] [KYUUBI #1188] Add Statement stats on Kyuubi Query Engine Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
936a532d2e
|
[KYUUBI #1269] Online documentation section in the readme is outdated
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/1269 ### _How was this patch tested?_ docs only Closes #1270 from ulysses-you/kyuubi#1269. Closes #1269 1cb99ef5 [ulysses-you] address comment 7a597e66 [ulysses-you] address comment 4b15c2c9 [ulysses-you] Online documentation section in the readme is outdated Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
wForget
|
0653dada40
|
[KYUUBI #1204] Enhance zookeeper authentication and acls
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Please see #1204 for details. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1213 from wForget/KYUUBI-1204. Closes #1204 fa88ffbe [wForget] [KYUUBI #1204] Change the default value of kyuubi.ha.zookeeper.auth.type to NONE 13cd7ecd [wForget] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into dev-1.3.0 3a84c0e1 [wForget] [KYUUBI #1204] fix tests and re-generate settings.md 92c331c7 [wForget] [KYUUBI #1204] Adjust the parameters of zookeeper authentication db016a03 [wForget] [KYUUBI #1204] Adjust the logic of obtaining the keytab file path 8adb812b [wForget] [KYUUBI #1204] Enhance zookeeper authentication and acls Authored-by: wForget <643348094@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
8fba917673
|
[KYUUBI #1248] Change kyuubi.engine.session.initialize.sql default value to Nil
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The previous default value of `kyuubi.engine.initialize.sql` and `kyuubi.engine.session.initialize.sql` both are `SHOW DATABASES`. The intention of `SHOW DATABASES`is to eagerly active HiveClient, it's quite redundant for executing many times. This PR proposes to change `kyuubi.engine.session.initialize.sql`'s default value to `Nil` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1248 from pan3793/sess-init. Closes #1248 ad2bdd7d [Cheng Pan] Update doc 0736857c [Cheng Pan] Change kyuubi.engine.session.initialize.sql default value to Nil Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
guoqing.yang
|
3e199ccdce
|
[KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant
feature: kyuubi.engine.share.level.subdomain can be more tolerant now end-user can configure kyuubi.engine.share.level.subdomain by zookeeper valid path I refer to the zookeeper doc http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#:~:text=ZooKeeper%20Basic%20Operations.-,The%20ZooKeeper%20Data%20Model,-ZooKeeper%20has%20a and source code: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/common/PathUtils.java#:~:text=public%20static%20void-,validatePath,-(String%20path)%20throws Closes #1232 from TyrealBM/feature/valid-subdomain-path. Closes #1209 8d00c260 [guoqing.yang] [KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant Authored-by: guoqing.yang <guoqing.yang@advance.ai> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
5d7a6056c6
|
[KYUUBI #1221] Revert "[KYUUBI #1176][FOLLOWUP] change configuration(ha.zookeeper.acl.engine.enabled) version"
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This reverts commit
|
||
|
timothy65535
|
badd5d516e
|
[KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1187 from timothy65535/ky-1159. Closes #1159 52daf139 [timothy65535] update conf md a9b50843 [timothy65535] improve patch 36676f2a [timothy65535] improve patch 892a6333 [timothy65535] [KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
bf629291bf
|
[KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The detail locates #1206 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1207 from yaooqinn/1206. Closes #1206 99e324a9 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level f41fd944 [Kent Yao] Merge branch 'master' into 1206 a7cb80da [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 6ecef063 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level d7ff7583 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 7b3b2f53 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
95c74cdefc |
[KYUUBI #1205] Support to show more engine submission failure log for issue tracking
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Customers might meet exception when launching a spark engine, especially for beginner, now kyuubi only show the last line of engine log and it is difficult to find the root cause in customer side. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1205 from turboFei/show_more_engine_log. Closes #1205 72002dd6 [fwang12] address comments a74fa1de [fwang12] update docs fd967371 [fwang12] refactor 434a093d [fwang12] update docs 880c8cc2 [fwang12] show more engine failure log for debug Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
sychen
|
464fdf456b
|
[KYUUBI #1176][FOLLOWUP] change configuration(ha.zookeeper.acl.engine.enabled) version
https://github.com/apache/incubator-kyuubi/pull/1177 ### _Why are the changes needed?_ change configuration(ha.zookeeper.acl.engine.enabled) version to `1.3.1` ### _How was this patch tested?_ - [] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [] Add screenshots for manual tests if appropriate - [] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1193 from cxzl25/KYUUBI-1176-followup. Closes #1176 72753ae5 [sychen] change config version Authored-by: sychen <sychen@trip.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sychen
|
6420a7eb04
|
[KYUUBI #1174] Service discovery supports kerberos
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1174 Now kyuubi servcie uses kerberos authentication and registers with the zookeeper to the node information, but the client cannot connect to the service. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1175 from cxzl25/KYUUBI-1174. Closes #1174 386c7d57 [sychen] add comment dc37a158 [sychen] respect user's setting 97401a96 [sychen] update settings.md fd8cceda [sychen] Service discovery supports kerberos Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
d332534325
|
[KYUUBI #1176] InvalidACL appears in the engine when zookeeper acl is turned on
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1176 When `kyuubi.ha.zookeeper.acl.enabled=true`, both service and engine will use zookeeper acl to create znode, but engine has no keytab information and cannot write information to zookeeper, throwing an exception. ```java Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi_USER/XXXX at org.apache.zookeeper.KeeperException.create(KeeperException.java:124) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740) ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1177 from cxzl25/KYUUBI-1176. Closes #1176 ecc08fa7 [sychen] fix engine acl 0b7cc2ec [sychen] fix InvalidACL Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
4f74aba50e |
[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support to config operation log root dir with kyuubi configuration both for server and engine sides. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1160 from turboFei/support_config_operationlog_dir. Closes #1160 ed424f46 [fwang12] fix ut 5edfd4ae [fwang12] refactor ut b7026b2f [fwang12] add ut 6ff4d891 [fwang12] save 30cd5952 [fwang12] fix ut f058619a [fwang12] fix ut 96eb020d [fwang12] refactor ae0ded96 [fwang12] set config value for testing 83d932f4 [fwang12] exclude 33e3e214 [fwang12] update 6f030cba [fwang12] fix 77ec9ca2 [fwang12] save 711ececf [fwang12] Fix code style cfa3bfac [fwang12] refactor b299f8cf [fwang12] complete ut 26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
timothy65535
|
22e6432e4d
|
[KYUUBI #1022] Add basic EngineStatusStore for events
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981 `EngineStatusStore` helps to push events to listener bus `EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides: - stores all elements, and sorted by startTimestamp. - cleanup the last elements when reach a certain threshold. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1023 from timothy65535/ky-1022. Closes #1022 b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
4f11b51673
|
[KYUUBI #1059] Add Plan Only Operations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement, PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them. This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite. This is similar to the `explain` command ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1060 from yaooqinn/1059. Closes #1059 e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
