97 Commits

Author	SHA1	Message	Date
fwang12	1b7762e5f9	[KYUUBI #457] Support configurable initialize sql statement for engine startup ...  [](https://github.com/yaooqinn/kyuubi/pull/462)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Currently, the engine uses hardcoded show databases to force initialization with catalog eagerly bootstrapping too. We can make it configurable with a comma separate query list and select current_database() might be used as default since it's more lightweight than show databases in real-world cases ### _How was this patch tested?_ Existing UT. Closes #462 from turboFei/KYUUBI-457. Closes #457 e7bcf52 [fwang12] add ut 7446cd5 [fwang12] [KYUUBI#457】Support configurable initialize sql statement for engine startup Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-01 01:31:58 +08:00
Kent Yao	c6585b4708	[KYUUBI #471] Separate Embedded Zookeeper server from ha ...  [](https://github.com/yaooqinn/kyuubi/pull/471)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one 2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage. 3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #471 from yaooqinn/qp. Closes #471 1c24d02 [Kent Yao] address comments 9666a40 [Kent Yao] refine f069fc0 [Kent Yao] refine 8bb2f3a [Kent Yao] refine c20593a [Kent Yao] refine 8fa9a40 [Kent Yao] Merge branch 'master' into qp 1e2e1a3 [Kent Yao] tmp 7d66d05 [Kent Yao] tmp Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-30 18:34:01 +08:00
ulysses-you	066075ac74	[KYUUBI #456] Can not cancel SQL job if the timeout value is small ...  [](https://github.com/yaooqinn/kyuubi/pull/461)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> * avoid condition race * avoid job leak ### _How was this patch tested?_ pass `org.apache.kyuubi.engine.spark.SparkEngineSuites` close #456 Closes #461 from ulysses-you/thread. 970ba7a [ulysses-you] fix 8786779 [ulysses-you] npe d82321e [ulysses-you] remove c28e671 [ulysses-you] interrupt execute statement 8dccf73 [ulysses-you] cleanup 29857e3 [ulysses-you] sync Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 23:50:24 +08:00
Kent Yao	392a10ce71	[KYUUBI #458] Mute Console Appender for CI ...  [](https://github.com/yaooqinn/kyuubi/pull/458)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Rely on the unit-tests.log is better for us to reduce the overhead of CIs ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #458 from yaooqinn/log. 7eb1521 [Kent Yao] logbetter Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 16:41:56 +08:00
ulysses-you	fecdba32a7	[KYUUBI #451] Support query auto timeout cancel on thriftserver ...  [](https://github.com/yaooqinn/kyuubi/pull/451)       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Manual cherry-pick some Spark patch into Kyuubi. 1. [Support query auto timeout cancel on thriftserver](https://github.com/apache/spark/pull/29933) 2. [Add config to control if cancel invoke interrupt task on thriftserver](https://github.com/apache/spark/pull/30481) In order to keep backward with early Spark version, we hard code the config key instead of refer to Spark SQLConf. Note that, the exists timeout of operator (`kyuubi.operation.idle.timeout`) is to cancel that client has no access with engine. That said if a query run a long time and the client is alive, the query would not be cancelled. Then the new added config `spark.sql.thriftServer.queryTimeout` can handle this case. ### _How was this patch tested?_ Add new test. Closes #451 from ulysses-you/query-timeout. 212f579 [ulysses-you] docs 9206538 [ulysses-you] empty flaky test ddab9bf [ulysses-you] flaty test 1da02a0 [ulysses-you] flaty test edfadf1 [ulysses-you] nit 3f9920b [ulysses-you] address comment 9492c48 [ulysses-you] correct timeout 5df997e [ulysses-you] nit 2124952 [ulysses-you] address comment 192fdcc [ulysses-you] fix tets d684af6 [ulysses-you] global config 1d1adda [ulysses-you] empty 967a63e [ulysses-you] correct import 128948e [ulysses-you] add session conf in session 144d51b [ulysses-you] fix a90248b [ulysses-you] unused import c90386f [ulysses-you] timeout move to operation manager d780965 [ulysses-you] update docs a5f7138 [ulysses-you] fix test f7c7308 [ulysses-you] config name 7f3fb3d [ulysses-you] change conf place 97a011e [ulysses-you] unnecessary change 0953a76 [ulysses-you] move test 38ac0c0 [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into query-timeout 71bea97 [ulysses-you] refector implementation 35ef6f9 [ulysses-you] update conf 0cad8e2 [ulysses-you] Support query auto timeout cancel on thriftserver Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 14:04:09 +08:00
Kent Yao	207a7df934	[KYUUBI #453] Refine ServiceDiscovery in Serverable API ...  [](https://github.com/yaooqinn/kyuubi/pull/453)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> A service discovery should be a default module of Severable API and shall be initialized after `FroundendService` for a resolved serviceUri. it's a reflection only fix to enhance this logic ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #453 from yaooqinn/refine. 077dca0 [Kent Yao] Refine ServiceDiscovery in Serverable API Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 12:18:37 +08:00
Kent Yao	9e7649296d	[KYUUBI #449] Add Basic Metrics Service framework ...  [](https://github.com/yaooqinn/kyuubi/pull/450)    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds a future-proofing Metrics Service framework close #499 add some reporters, like json file, jmx, etc.. add some basic metrics, like statement.error ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible WIP - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #450 from yaooqinn/449. ea21a43 [Kent Yao] add tests 6513d07 [Kent Yao] update codecov 7cfe318 [Kent Yao] update docs a8454d0 [Kent Yao] update docs 91c33f4 [Kent Yao] update deps 05b13fb [Kent Yao] fix dep issues babc739 [Kent Yao] Add Basic Metrics Service Platform acc716a [Kent Yao] Add Basic Metrics Service Platform Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-24 18:56:56 +08:00
ulysses-you	a09178b578	[KYUUBI #410] Failed to start Spark 3.1 with error msg 'Cannot modify the value of a Spark config' ...  [](https://github.com/yaooqinn/kyuubi/pull/440)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the exception if we pass some unnecessary config to Spark. It's hard to list the whole config which Spark disallowed, this patch just ingore the expcetion that Spark throw. ### _How was this patch tested?_ Pass the test `SessionSuite` Closes #440 from ulysses-you/fix-modify-spark-conf. 5ab858a [ulysses-you] msg 3ba3587 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-16 16:09:41 +08:00
谢波	9c13f73bd9	[KYUUBI #417] [MINOR] Remove all unused import ...  [](https://github.com/yaooqinn/kyuubi/pull/417)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> ### _Why are the changes needed?_ remove all unused import and replace deprecated class PersistentEphemeralNode with PersistentNode in ServiceDiscovery ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #417 from MyLanPangzi/master. 821e0d2 [hiscat] Merge branch 'master' into master db9cd3c [谢波] [MINOR]: revert ServiceDiscovery change and add scala maven plugin compiler args c56cd2b [谢波] [MINOR]: revert ProcBuilder indent ad5e24a [谢波] [MINOR]: revert maven.plugin.scalatest.exclude.tags properties 7c23e37 [谢波] [MINOR]: scala maven plugin config move to plugin management 6029b99 [谢波] [MINOR]: remove all unused import a305567 [谢波] [MINOR]: remove all unused import Lead-authored-by: 谢波 <xiebo1@yonghui.cn> Co-authored-by: hiscat <46845236+MyLanPangzi@users.noreply.github.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-03-15 20:05:08 +08:00
Honglun	e2fd967c58	[KYUUBI #397] Add test case for querying columns with dots ...  [](https://github.com/yaooqinn/kyuubi/pull/405)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add test case for querying columns with dots. closes #397 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #405 from Honglun/master. bd9d503 [Honglun] Add test case for querying columns with dots Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-08 20:13:54 +08:00
Kent Yao	5e2a1290c3	Prepare Kyuubi v1.2.0	2021-03-08 19:50:12 +08:00
ulysses-you	03b7112eee	[KYUUBI #403] FIx flaky test in ConnectionLevelSparkEngineSuite ...  [](https://github.com/yaooqinn/kyuubi/pull/403)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ `ignore` -> `test` Closes #403 from ulysses-you/flaky-test2. d8aa6c7 [ulysses-you] empty fd38e9c [ulysses-you] empty 2e5a7ee [ulysses-you] empty 444a526 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-08 11:22:19 +08:00
Kent Yao	352b4eb99e	[KYUUBI #400] Implement a simpler toHiveString to convert spark row ...  [](https://github.com/yaooqinn/kyuubi/pull/400)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Hive compatibility is essential! This PR tries to restore the previous behavior. 1. Not use Spark's internal class HiveResult to convert spark catalyst type-values to JDBC results, the interval APIs are not APIs and changed unexpectedly. It involves a lot of overhead. (For this reason, pan3793 and I propose a Casting way to solve it.) 2. Unfortunately, the casting way breaks things like #397. So this is a reverting PR. 3. I checked the spark `HiveResult` carefully, I noticed that the `nested` logic is not needed in Kyuubi as we already matched primitives ahead. 4. Also the date time formatter used in `HiveResult` is excessively packaged with massively complicated logic inside spark. It's not clear and simple enough to call. 5. (tiny breaking change)the trailing spaces of a timestamp in milliseconds may not be trimmed. But this is a correctness issue and acceptable. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #400 from yaooqinn/397. 388c86d [Kent Yao] nit bbf458b [Kent Yao] time stamp formatter e5a7436 [Kent Yao] more tests 83f3142 [Kent Yao] timezone 38f5a76 [Kent Yao] Merge branch 'master' into 397 563d69c [Kent Yao] mute flacky test 9d834b0 [Kent Yao] Merge branch 'master' into 397 e8e05b5 [Kent Yao] nit 2a2d47b [Kent Yao] Implement a simpler toHiveString to convert spark row Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-05 17:12:58 +08:00
Cheng Pan	4abc3ff6a0	[KYUUBI #398] Move to hadoop shaded client ...  [](https://github.com/yaooqinn/kyuubi/pull/399)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #398 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request manual test release workflow ``` ➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz ``` Closes #399 from pan3793/KYUUBI-398. a9294a1 [Cheng Pan] fix ut d1d816d [Cheng Pan] fix dist script 2e3bc20 [Cheng Pan] update release workflow and dist script 0428b1b [Cheng Pan] update travis.yml 4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 23:33:31 +08:00
Cheng Pan	8dd54a29b3	[KYUUBI #383] Support beeline --hivevar --hiveconf ...  [](https://github.com/yaooqinn/kyuubi/pull/394)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #383 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #394 from pan3793/KYUUBI-383. 2810b58 [Cheng Pan] [KYUUBI #383] remove regex match, add server side test cases 3d8b1e0 [Cheng Pan] [KYUUBI #383] Support beeline --hivevar --hiveconf Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 21:28:22 +08:00
Kent Yao	2e3315867b	[KYUUBI #298]Add Support for Apache Spark 3.1.1 ...  [](https://github.com/yaooqinn/kyuubi/pull/366)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved. In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7` TODOS: - Add Hadoop 3.2 support with some dependency issues - Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release - Using spark-3.1.1 as default - Restore HiveResult string? fix #298 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #366 from yaooqinn/spark3.1. 28ddf93 [Kent Yao] ga 37cb49e [Kent Yao] Merge branch 'master' into spark3.1 7b332ec [Kent Yao] typo b6b33d5 [Kent Yao] set up ga 00005c0 [Kent Yao] set up travis 2c09fe4 [Kent Yao] test fix 68c7edb [Kent Yao] deps 2c65e98 [Kent Yao] deps 236c7e9 [Kent Yao] nit 8c8a783 [Kent Yao] Merge branch 'master' into spark3.1 46c318a [Kent Yao] nit 32f6511 [Kent Yao] Merge branch 'master' into spark3.1 27c836b [Kent Yao] Merge branch 'master' into spark3.1 5574a8e [Kent Yao] ga d2ec2d5 [Kent Yao] [WIP]Spark3.1 49313cc [Kent Yao] [WIP]Travis ccb1a97 [Kent Yao] [WIP]Spark3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 13:30:05 +08:00
ulysses-you	aa878e4d48	[KYUUBI #386] Clean up discovery service before engine stop in connection share level ...  [](https://github.com/yaooqinn/kyuubi/pull/387)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time. ### _How was this patch tested?_ Add some new suites in spark engine moudle with zk emmbedded. Closes #387 from ulysses-you/issue-386. ccb1112 [ulysses-you] move method 1558dd9 [ulysses-you] test 6cd7e57 [ulysses-you] nit f8559d2 [ulysses-you] check level at engine discovery eedfaee [ulysses-you] split ServiceDiscovery to server and engine 8b20e6c [ulysses-you] timeout 0cf524c [ulysses-you] remove unnecessary test 41d36f9 [ulysses-you] move exception eaaa12d [ulysses-you] avoid stop twice b801c14 [ulysses-you] fix test db39960 [ulysses-you] add start check 2d1f6dd [ulysses-you] add state check e7374aa [ulysses-you] nit ddf383c [ulysses-you] fi ee088be [ulysses-you] nit 3c2013b [ulysses-you] nit 73b386f [ulysses-you] improve conf of test 2185f49 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 11:31:41 +08:00
ulysses-you	4eed2ee0a7	[KYUUBI #385] Improve pass config between kyuubi and spark engine ...  [](https://github.com/yaooqinn/kyuubi/pull/385)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make test more flexible and readable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #385 from ulysses-you/cleanup-test. c5d2034 [ulysses-you] readable fbc0f16 [ulysses-you] readable b1008b7 [ulysses-you] fix 37f5739 [ulysses-you] comment fdf52a7 [ulysses-you] add config method c58dfdb [ulysses-you] fix tes a1e8517 [ulysses-you] improve pass config 1378131 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-03 10:01:54 +08:00
ulysses-you	7f2a24bd9f	[KYUUBI #381] Release session if shared level is CONNECTION ...  [](https://github.com/yaooqinn/kyuubi/pull/382)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In CONNECTION level, it's better to release session if the connection is closed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #382 from ulysses-you/KYUUBI-381. cddba6c [ulysses-you] type a3fd691 [ulysses-you] increase engine init timeout 5030553 [ulysses-you] reload system properties 7a2e72f [ulysses-you] clean code 1034409 [ulysses-you] stopped d2a001a [ulysses-you] address comment 4eb62de [ulysses-you] 60s 64d0f16 [ulysses-you] fix test 4f10d34 [ulysses-you] sleep 62dc9f9 [ulysses-you] add new stop method 77be261 [ulysses-you] remove config fd9b73c [ulysses-you] add time sleep 209a59e [ulysses-you] fix test 88ed744 [ulysses-you] import 90015e9 [ulysses-you] simply 311fce3 [ulysses-you] fix test 7af3f72 [ulysses-you] fix test e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381 f77ddbe [ulysses-you] nit 2173222 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-02 11:44:32 +08:00
Kent Yao	0f62568171	[KYUUBI #379] Improve engine log capture ...  [](https://github.com/yaooqinn/kyuubi/pull/379)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead 2. update `error` immediately when detecting errors at the very first line ASAP 3. Using `Exception:` instead of `Exception` to search each line to filter better 4. add the engine log address in the exception message in case of error bootstraping ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #379 from yaooqinn/logsize. 7cf58da [Kent Yao] mute in console b58a3a2 [Kent Yao] update fbc4a0b [Kent Yao] update 0815f77 [Kent Yao] log 93f5f2f [Kent Yao] log Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 17:31:44 +08:00
Cheng Pan	2f8df2355d	[KYUUBI #377] build/dist support --spark-provided ...  [](https://github.com/yaooqinn/kyuubi/pull/378)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --spark-provided` ``` (kyuubi) ➜ kyuubi git:(master) ✗ tree dist dist ├── LICENSE ├── RELEASE ├── bin │   ├── kyuubi │   ├── kyuubi-logo │   └── load-kyuubi-env.sh ├── conf │   ├── kyuubi-defaults.conf.template │   ├── kyuubi-env.sh.template │   └── log4j.properties.template ├── externals │   └── engines │   └── spark │   └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar ├── jars │   ├── apacheds-i18n-2.0.0-M15.jar │   ├── apacheds-kerberos-codec-2.0.0-M15.jar │   ├── api-asn1-api-1.0.0-M20.jar │   ├── api-util-1.0.0-M20.jar │   ├── audience-annotations-0.5.0.jar │   ├── commons-beanutils-1.7.0.jar │   ├── commons-beanutils-core-1.8.0.jar │   ├── commons-codec-1.4.jar │   ├── commons-collections-3.2.2.jar │   ├── commons-configuration-1.6.jar │   ├── commons-digester-1.8.jar │   ├── commons-io-2.4.jar │   ├── commons-lang-2.6.jar │   ├── commons-lang3-3.10.jar │   ├── commons-math-2.2.jar │   ├── curator-client-2.7.1.jar │   ├── curator-framework-2.7.1.jar │   ├── curator-recipes-2.7.1.jar │   ├── curator-test-2.7.1.jar │   ├── guava-11.0.2.jar │   ├── hadoop-annotations-2.7.4.jar │   ├── hadoop-auth-2.7.4.jar │   ├── hadoop-common-2.7.4.jar │   ├── hive-service-rpc-2.3.7.jar │   ├── httpclient-4.5.6.jar │   ├── httpcore-4.4.12.jar │   ├── javassist-3.18.1-GA.jar │   ├── jcl-over-slf4j-1.7.30.jar │   ├── jsr305-3.0.2.jar │   ├── kyuubi-common-1.1.0-SNAPSHOT.jar │   ├── kyuubi-ha-1.1.0-SNAPSHOT.jar │   ├── kyuubi-main-1.1.0-SNAPSHOT.jar │   ├── libfb303-0.9.3.jar │   ├── libthrift-0.9.3.jar │   ├── log4j-1.2.17.jar │   ├── netty-3.7.0.Final.jar │   ├── scala-library-2.12.10.jar │   ├── slf4j-api-1.7.30.jar │   ├── slf4j-log4j12-1.7.30.jar │   ├── spotbugs-annotations-3.1.9.jar │   └── zookeeper-3.4.14.jar ├── logs ├── pid └── work ``` Bundle size ``` (kyuubi) ➜ kyuubi git:(KYUUBI-377) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz -rw-r--r-- 1 chengpan staff 20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #378 from pan3793/KYUUBI-377. 442569e [Cheng Pan] add profile spark-provided for skip download spark binary 0d8da24 [Cheng Pan] update doc 5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-02-26 15:00:42 +08:00
Kent Yao	968ed4f079	[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test ...  [](https://github.com/yaooqinn/kyuubi/pull/376)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #376 from yaooqinn/hive. 26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 00:28:17 +08:00
Kent Yao	ce7b122435	[KYUUBI #371] Avoid creating useless operation log for meta operation ...  [](https://github.com/yaooqinn/kyuubi/pull/371)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ 1. Operation log for metadata is useless for client-side 2. Operation log for metadata dumps nothing useful for the engine side too 3. Operation log will create files that bring overhead for meta operations and may fail operations ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #371 from yaooqinn/oplog. ca62e74 [Kent Yao] Avoid creating useless operation log for meta operation Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 20:47:48 +08:00
Cheng Pan	c659089bc2	[KYUUBI #360] Correct handle getNextRowSet with FETCH_PRIOR FETCH_FIRST ...  [](https://github.com/yaooqinn/kyuubi/pull/370)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #360 Ref: https://github.com/apache/spark/pull/30600 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #370 from pan3793/KYUUBI-360. e79b8cb [Cheng Pan] [KYUUBI #360] comments 0fae3db [Cheng Pan] fix import 3d1b2a6 [Cheng Pan] [KYUUBI #360] fix ut eda3e59 [Cheng Pan] [KYUUBI #360] fix import 16178d6 [Cheng Pan] [KYUUBI #360] ut 179404d [Cheng Pan] [KYUUBI #360] nit 455af6b [Cheng Pan] [KYUUBI #360] correct getNextRowSet with FETCH_PRIOR FETCH_FIRST 2307f1f [Cheng Pan] [KYUUBI #360] move ThriftUtils to kyuubi-common Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 18:19:02 +08:00
Kent Yao	4b13f1c551	[KYUUBI #367] Engine terminating checker should start after exposed ...  [](https://github.com/yaooqinn/kyuubi/pull/367)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating. In this PR, we move the checker to the end where the engine is exposed to the servers. Avoiding errors like: ``` org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #367 from yaooqinn/engine. f6a979e [Kent Yao] fix test 9cc561c [Kent Yao] address comment 3d15ab2 [Kent Yao] nit a85d2e0 [Kent Yao] Engine terminating checker should start after exposed Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 15:10:43 +08:00
Kent Yao	ee50890b00	[KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/358)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> close #310 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #358 from yaooqinn/310. 8cb30a4 [Kent Yao] sql wildcards to java regex 34d2c3a [Kent Yao] Merge branch 'master' into 310 d332be5 [Kent Yao] [KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-23 20:43:02 +08:00
Cheng Pan	23e1d61e47	[KYUUBI #350] Avoid using HiveResult ...  [](https://github.com/yaooqinn/kyuubi/pull/351)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #350 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #351 from pan3793/KYUUBI-350. 8b9ac46 [Cheng Pan] debug logs 21add73 [Cheng Pan] minor 46ceb54 [Cheng Pan] [KYUUBI-350] in favor cast string in executeStatment f4e3633 [Cheng Pan] [KYUUBI-350] Avoid using HiveResult in RowSet Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-12 14:00:39 +08:00
Kent Yao	a2818003e6	[KYUUBI #309] GetTables supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/346)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix #309 get tables through Spark DSv2 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #346 from yaooqinn/gettables. 16dbe4b [Kent Yao] nit e03b021 [Kent Yao] nit 7f3eee2 [Kent Yao] Merge branch 'master' into gettables 0180cc5 [Kent Yao] add a test 8498152 [Kent Yao] GetTables supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-07 13:45:53 +08:00
ulysses-you	fbf1e3506c	[KYUUBI #345] Catch Throwable instead of Exception ...  [](https://github.com/yaooqinn/kyuubi/pull/345)       [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We might get this error when running spark sql，so it's more reasonable to catch `Throwable`. ``` java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #345 from ulysses-you/throwable. b6ffacd [ulysses-you] throw Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-03 19:31:26 +08:00
Kent Yao	7deb2f7fd7	[KYUUBI #336] Fix a race in Kyuubi start and stop ...  [](https://github.com/yaooqinn/kyuubi/pull/336)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> there is something wrong with Travis these days due to some newly added test cases ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #336 from yaooqinn/travisagain. 0285f5a [Kent Yao] fix a race 3624889 [Kent Yao] fix a race 895b4d5 [Kent Yao] fix a race e253b9c [Kent Yao] fix a race 75d699a [Kent Yao] fix a race 7bde6b9 [Kent Yao] nit 6959e42 [Kent Yao] nit 062a151 [Kent Yao] nit ca8af10 [Kent Yao] kyuubi-common 6e4b8b6 [jhx1008] Try to restore travis Lead-authored-by: Kent Yao <yao@apache.org> Co-authored-by: jhx1008 <jhx1008@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 20:46:06 +08:00
Kent Yao	a4d507a9c6	[KYUUBI #337] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces with dot backtick ...  [](https://github.com/yaooqinn/kyuubi/pull/337)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> add support for '.' and '`' in a namespace part for instance, cases like https://github.com/yaooqinn/kyuubi/pull/337/files#diff-bcd0dfb958c3943a58a9705ac7053374c796449fefd6be7e9f014f9dd14b558fR94 produce the below catalog tree, we should treat `a.b`.c and a.`b.c` differently. ``` /var/folders/01/h81cs4sn3dq2dd_k4j6fhrmc0000gn/T/kyuubi-94c946f7-a64b-4980-9dd1-a63a573244dd ├── a │   └── b.c ├── a.b │   └── c ├── a.b.c ├── a.b`.c ├── db1 │   └── db2 │   └── db3 └── db4 ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #337 from yaooqinn/getschema3. fdb8b19 [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces 0091cfc [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces 5ccee06 [Kent Yao] [FOLLOWUP]GetSchemas supports DSv2 multipart namespaces Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-02 09:54:47 +08:00
Kent Yao	c9d44b482c	[KYUUBI #334] GetSchemas supports DSv2 multipart namespaces ...  [](https://github.com/yaooqinn/kyuubi/pull/334)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds support for using Spark DSv2 to get Schemas with multipart namespaces under multiple catalogs. The current works are all based on Apache Iceberg. TODOS: next step we will support get tables operation for multiple catalogs and namespaces. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #334 from yaooqinn/getschema2. 5994170 [Kent Yao] update 188258a [Kent Yao] GetSchemas supports DSv2 multipart namespaces Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-01 20:55:45 +08:00
Kent Yao	f186fcb15e	[KYUUBI #329] Support tests with Apache Iceberg in engine ...  [](https://github.com/yaooqinn/kyuubi/pull/329)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support tests with Apache Iceberg in engine - Improve debug experience for developers - Improve test coverage ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #329 from yaooqinn/iceberg2. 8bf6714 [Kent Yao] Reflect IcebergOperationSuit 329593a [Kent Yao] Reflect IcebergOperationSuit Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-31 02:18:18 +08:00
jhx1008	89e2ef743b	[KYUUBI #322] Override statement for better logging ...  [](https://github.com/yaooqinn/kyuubi/pull/322)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> ### _Which issue are you going to fix?_ <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #${ID} ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #322 from yaooqinn/stmt. 9cebdb7 [Kent Yao] nit 8618583 [jhx1008] Override statement for better logging Lead-authored-by: jhx1008 <jhx1008@gmail.com> Co-authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-30 14:06:36 +08:00
Kent Yao	fcea17bc83	[KYUUBI #318] [KYUUBI #311]GetSchemas supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/318)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> ### _Which issue are you going to fix?_ <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #311 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> GetSchemas supports DSv2 and keeps its backward compatibility we currently support only one part namespaces in this PR ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible add new tests - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #318 from yaooqinn/311. 1dec0e0 [Kent Yao] GetSchemas supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-29 22:44:01 +08:00
ulysses-you	3147d7c5b8	[KYUUBI #316] Nit remove some unnecessary blank ...  [](https://github.com/yaooqinn/kyuubi/pull/316)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> ### _Which issue are you going to fix?_ <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #316 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Nit. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #316 from ulysses-you/nit. e692b55 [ulysses-you] nit Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-01-29 16:23:40 +08:00
ulysses-you	2126565595	[KYUUBI #314] Add statement error msg in user log ...  [](https://github.com/yaooqinn/kyuubi/pull/314)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #314 <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Help to find problem - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Squashed commit of the following: commit e90840028848b1d307d11fd81b23f76f4a00ccf6 Author: ulysses-you <ulyssesyou18@gmail.com> Date: Fri Jan 29 11:20:03 2021 +0800 fix commit 4a2136aa59b35bbd7997f0a59c466830ec665b4f Author: ulysses-you <ulyssesyou18@gmail.com> Date: Fri Jan 29 11:15:29 2021 +0800 init	2021-01-29 15:23:04 +08:00
Kent Yao	efdd9fafc8	[KYUUBI #307]GetCatalogs supports DSv2 and keeps its backward compatibility (#307) ...  [](https://github.com/yaooqinn/kyuubi/pull/307)  [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> ### Please add issue ID here? <!-- replace ${issue ID} with the actual issue id --> Fixes #307 ### Why are the changes needed? <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> GetCatalogs supports DSv2 and keeps its backward compatibility ### Test Plan: - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request * GetCatalogs supports DSv2 * pr template * nit * nit * shim * add iceberg tests * nit	2021-01-29 10:27:48 +08:00
Kent Yao	eadca7589c	[KYUUBI #296] Add spark 2.4 profile and shade hive-sevice-rpc to engine jar to fix thrift build err (#296) ... * add spark 2.4.7 profile * nit * nit * scala versions for profile spark-2.4	2021-01-18 14:23:31 +08:00
Cheng Pan	6f8564ce59	[KYUUBI #280] Align Operation GET_SCHEMAS * behavior with Hive #292 ...  [](https://github.com/yaooqinn/kyuubi/pull/292)  [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> <!-- replace ${issue ID} with the actual issue id --> Fixes #280 closes #292 <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Before this PR, Kyuubi treat Operation GET_SCHEMAS * as an invalid Operation then cause HUE list databases failed, but HiveServer2 will return all databases on same request. - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request After PR:  Squashed commit of the following: commit f56fdc3fe35b4aa5a7b6f09cbbaa023ec713e2fe Author: Cheng Pan <379377944@qq.com> Date: Sun Jan 17 23:27:03 2021 +0800 [KYUUBI-280] code style and ut commit ff74dbaf3e6d5606e5a2171e6dcc3628aa631eea Author: Cheng Pan <379377944@qq.com> Date: Sun Jan 17 23:08:59 2021 +0800 [KYUUBI-280] remove comments and handle schema * in other place commit a8bcf98cd70e36903ece654e1c37d08e868b1c5b Author: Cheng Pan <379377944@qq.com> Date: Sun Jan 17 13:47:24 2021 +0800 [KYUUBI-280] Align Operation GET_SCHEMAS * behavior with Hive	2021-01-18 11:21:59 +08:00
Kent Yao	3a147f834b	Remove dulicated error msg for open sessions at engine side (#293)	2021-01-18 10:25:22 +08:00
Cheng Pan	0eeb99774f	Tune pom ... fixes #271 Squashed commit of the following: commit fe0f469b2068865398d6e5be1957fdf6c6f5eb87 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 22:28:43 2021 +0800 followup commit 527e32419ecb2ae584d1251cb345e3fc870965a9 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 22:21:29 2021 +0800 revert some stuffs commit 82f188972b8491c5d381ba076a53f84a2fbc1898 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 16:49:14 2021 +0800 remove unused dependencies commit c7eb27228d2749ab9100102467e94d3a6a98cf44 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 15:41:59 2021 +0800 minimize dependencies in kyuubi-spark-sql-engine commit da902b1b1e02c7e93810ca832f6c9e7109cf95b9 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 13:26:34 2021 +0800 exclude hive dependencies commit 55ae7f025a5ffcec402919e28b7b3af5987dfe16 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 01:08:09 2021 +0800 pin versions and excludes in <dependencyManagement> commit 9ef84a5f62afd66fb4b817d593ba1b13ff8348ed Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 14:12:04 2021 +0800 trigger kyuubi-spark-sql-engine package commit d492e3fdce64e6554fe0db1d810c6f4bea198baa Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 13:44:13 2021 +0800 trigger kyuubi-download commit 050281b8fc400f3ae5fa1d43d56da49846664ea4 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 15:30:58 2021 +0800 pin modules version in dependencyManagement commit 1655fc453ded4626ae5e12e7b17af6f10c30a0dd Author: Cheng Pan <379377944@qq.com> Date: Fri Jan 8 19:45:27 2021 +0800 tune modules in alphabetical order commit 7f3f4d987336d71d73eab059e93f3d008cfcd1a6 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 00:09:49 2021 +0800 change dependencies scope to test	2021-01-09 23:30:46 +08:00
Kent Yao	6292b51ec8	Support launch SparkProcessBuilder with keytab and principal (#269) ... * Support launch SparkProcessBuilder with keytab and principal * fix tests	2021-01-08 23:43:14 +08:00
Cheng Pan	90c120759e	Improve pom.xml ... fixes #268 Squashed commit of the following: commit d40d0831d578af23559ea2a2544ceebc47f6df50 Author: Cheng Pan <379377944@qq.com> Date: Fri Jan 8 18:30:51 2021 +0800 remove unused mockito commit 386e7e4eb1c93006e302c41ffbc04d0acfaf9a9b Author: Cheng Pan <379377944@qq.com> Date: Fri Jan 8 17:35:44 2021 +0800 extract versions to properties	2021-01-08 19:04:48 +08:00
Kent Yao	4c433fc59a	Add SparkSQLEngineListener	2021-01-03 02:09:25 +08:00
Kent Yao	ca1b452e9a	Prepare Kyuubi v1.1.0	2020-12-31 15:51:12 +08:00
Kent Yao	17fe83778e	ServiceDiscovery for kyuubi servver	2020-12-28 13:05:35 +08:00
Kent Yao	a0f900a629	Reformat ServiceDiscovery	2020-12-28 00:10:31 +08:00
Kent Yao	3ec0db6a40	Improve Session Share Level	2020-12-27 00:05:07 +08:00
Zen	7a79b16cdf	fix #259 engine scope ... Squashed commit of the following: commit bb598b101da43b145bd4c95976e8decd39a02210 Merge: 54e730c c4cf523 Author: zen <xinjingziranchan@gmail.com> Date: Thu Dec 24 10:53:42 2020 +0800 Merge remote-tracking branch 'origin/pr2-engine-scope' into pr2-engine-scope # Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala commit 54e730c5b26ffb147e5c1266bdd036460cc20d82 Merge: 62d1575 512fbc5 Author: zen <xinjingziranchan@gmail.com> Date: Thu Dec 24 10:51:21 2020 +0800 Merge branch 'master' into pr2-engine-scope # Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala commit c4cf523dc5bc409ea7ac742292d5e6e2d6421b8a Author: zen <xinjingziranchan@gmail.com> Date: Mon Dec 21 23:10:12 2020 +0800 merge master and fix the conflicts commit c53b166f1eb3e8a698f498e1d123c815cef5d665 Merge: 62d1575 e8bc44c Author: zen <xinjingziranchan@gmail.com> Date: Mon Dec 21 23:02:26 2020 +0800 Merge branch 'master' into pr2-engine-scope # Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala commit 62d15751ea6b213319501bcfac442d21d0cd5ca6 Author: zen <xinjingziranchan@gmail.com> Date: Thu Dec 10 15:40:48 2020 +0800 fix zkNamespace commit 7e3e432318ed931b1ad16c839c4a027e00bc6ef9 Author: zen <xinjingziranchan@gmail.com> Date: Thu Dec 10 14:34:34 2020 +0800 Support for configuring the session scope of SparkSQLEngine.	2020-12-24 19:02:40 +08:00