ffe41631ce
217 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
zhouyifan279
|
ffe41631ce
|
[KYUUBI #1383] Leverage Scalafmt to auto format scala code
### _Why are the changes needed?_ #1383 As IntelliJ IDEA's `Optimize Imports` result does not meet rules in `scalastyle-config.xml`, currently developers have to arrange package imports manually. In this PR, we introduce a `.scalafmt.conf` file to solve the problem. It works as follows: 1. Set scala code style formatter to `scalafmt`. 2. Execute `Reformat Code` 3. IDEA picks up configurations in `.scalafmt.conf` and format both code lines and imports. We also formatted all historical scala codes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1391 from zhouyifan279/1383. Closes #1383 e0696a9e [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code 18b8e229 [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Jagadesh Adireddi
|
15844dc373 |
[KYUUBI #1056] [BUG] Log error if no current spark sql engine is created.
Issue: https://github.com/apache/incubator-kyuubi/issues/1056 ### _Why are the changes needed?_ Log an error message if no current spark sql engine is created. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1423 from jadireddi/1056-no-currEng. Closes #1056 a8d0443d [Jagadesh Adireddi] refactor 565d41a4 [Jagadesh Adireddi] [KYUUBI #1056] [BUG] Log error if no current spark sql engine is created. Authored-by: Jagadesh Adireddi <jadiredd@conviva.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
ulysses-you
|
fda295865e |
[KYUUBI #1421] Improve beeline print
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We don't support isolation, so it's unnecessary to print isolation level. before this PR ``` Connected to: Spark SQL (version 1.4.0-SNAPSHOT) Driver: Kyuubi Project Hive JDBC Shaded Client (version 1.4.0-SNAPSHOT) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.4.0-SNAPSHOT by Apache Kyuubi ``` After: ``` Connected to: Apache Kyuubi (Incubating) (version 1.4.0-SNAPSHOT) Driver: Kyuubi Project Hive JDBC Shaded Client (version 1.4.0-SNAPSHOT) Beeline version 1.4.0-SNAPSHOT by Apache Kyuubi (Incubating) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1421 from ulysses-you/improve-log. Closes #1421 2e0e168e [ulysses-you] improve Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
yanghua
|
20ee845c0b
|
[KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1387 from yanghua/KYUUBI-1386. Closes #1386 29415904 [yanghua] [KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
cb9e55a9b4
|
[KYUUBI #1380] Fix kyuubi download duplicated plugins
### _Why are the changes needed?_ Fix maven pom.xml issue ``` (apache-kyuubi) ➜ apache-kyuubi git:(mysql-fe-impl) ✗ build/mvn clean install Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-download:pom:1.4.0-SNAPSHOT [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin com.googlecode.maven-download-plugin:download-maven-plugin line 62, column 21 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] ------------------------------------------------------------------------ ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request ``` (apache-kyuubi) ➜ apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided,flink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile' Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn [INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) kyuubi-download --- [INFO] compile in 0.0 s [INFO] No sources to compile [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release) kyuubi-download --- [INFO] maven-download-plugin:wget skipped [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release) kyuubi-download --- [INFO] maven-download-plugin:wget skipped [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.932 s [INFO] Finished at: 2021-11-15T10:41:05+08:00 [INFO] ------------------------------------------------------------------------ ``` ``` (apache-kyuubi) ➜ apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pflink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile' Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn [INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) kyuubi-download --- [INFO] compile in 0.0 s [INFO] No sources to compile [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release) kyuubi-download --- [INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/spark-3.1.2-bin-hadoop3.2.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release) kyuubi-download --- [INFO] maven-download-plugin:wget skipped [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.486 s [INFO] Finished at: 2021-11-15T10:42:22+08:00 [INFO] ------------------------------------------------------------------------ ``` ``` (apache-kyuubi) ➜ apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile' Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn [INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) kyuubi-download --- [INFO] compile in 0.0 s [INFO] No sources to compile [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release) kyuubi-download --- [INFO] maven-download-plugin:wget skipped [INFO] [INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release) kyuubi-download --- [INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/flink-1.12.5-bin-scala_2.12.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.114 s [INFO] Finished at: 2021-11-15T10:41:53+08:00 [INFO] ------------------------------------------------------------------------ ``` Closes #1380 from pan3793/download. Closes #1380 21c7cc22 [Cheng Pan] Fix kyuubi download duplicated plugins Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
4ac8b09b2f
|
[KYUUBI #1331] [TEST] Refactor test classes to match multi frontend protocol
### _Why are the changes needed?_ As Kyuubi supports REST frontend protocol now, and MySQL frontend protocol is coming soon, Thrift is not the only one, this PR refacotor the test classes to match the changes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1331 from pan3793/test. Closes #1331 67c65810 [Cheng Pan] nit f9168358 [Cheng Pan] [TEST] Refactor test classes to match multi frontend protocol Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
aad575ce3e
|
[KYUUBI #1329] Support download Flink binary package in kyuubi-download module
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1339 from yanghua/KYUUBI-1329. Closes #1329 895815fb [yanghua] address review suggestion eddf0605 [yanghua] [KYUUBI #1329] Support download Flink binary package in kyuubi-download module Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
fd17dd0ae4
|
[KYUUBI #1300] Detecting critical errors
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For critical errors at engine side, it is not handled properly. For example,when engine oom - server may not be able to get operation statuses because of no response of engine side. In this case, client only get a ambiguous `read timeout` as a final cause. - the oom hook of engine side might directly crash the engine, when the server is still trying to get operation status In this PR, - a config for retry to make the operation status updating process more robust - make the engine oom hook only de register itself to make it able to recover for some transient errors ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1312 from yaooqinn/1300. Closes #1300 a715ecca [Kent Yao] add comments a816b0f0 [Kent Yao] refine 3557c927 [Kent Yao] add comments 2ed8bfb4 [Kent Yao] refine aefd2a7f [Kent Yao] update doc f2ea7e4c [Kent Yao] restore SparkOperation 386e4eac [Kent Yao] [KYUUBI #1300] Detecting critical errors Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
0724f444c5
|
[KYUUBI #1310] Remove redundant store parameter
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The `store` is a member variable of `SparkSQLEngine`, we can get it from `SparkSQLEngine` directly. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1310 from timothy65535/patch. Closes #1310 1a97a1c7 [root] trigger rebuild eae0fe34 [timothy65535] Remove redundant parameter Lead-authored-by: timothy65535 <timothy65535@163.com> Co-authored-by: root <root@XTZJ-20211013DV.localdomain> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
6539731417
|
[KYUUBI #1287] [TEST] Refacotor test hierarchical
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Change the name to reflect functionalities, restore `SparkQueryTests` both on Engine&Server sides Rename `JDBCTestUtils` to `HiveJDBCTestHelper` Rename `BasicJDBCTests` to `SparkMetadataTests` Rename `HiveJDBCTests` to `HiveMetadataTests` Rename `BasicDeltaJDBCTests` to `DeltaMetadataTests` Rename `BasicHudiJDBCTests` to `HudiMetadataTests` Rename `BasicIcebergJDBCTests` to `IcebergMetadataTests` Rename `KyuubiOperationGroupSuite` to `KyuubiOperationPerGroupSuite` Rename sorts of `spark` prefixed fields in `HiveJDBCTestHelper` Use `UserGroupInformation.createUserForTesting` for `KyuubiOperationPerGroupSuite` Hierarchical ``` - HiveJDBCTestHelper - DeltaMetadataTests - [ENGINE] SparkDeltaOperationSuite - [SERVER] DeltaOperationSuite - HudiMetadataTests - [ENGINE] SparkHudiOperationSuite - [SERVER] HudiOperationSuite - IcebergMetadataTests - [ENGINE] SparkIcebergOperationSuite - [SERVER] IcebergOperationSuite - SparkMetadataTests - [SERVER] KyuubiJdbcDriverHASuite - HiveMetadataTests - [ENGINE] SparkOperationSuite mixin SparkQueryTests - [SERVER] KyuubiOperationHiveCatalogSuite - SparkQueryTests - [ENGINE] SparkOperationSuite mixin HiveMetadataTests - [SERVER] KubernetesJDBCTestsSuite - [SERVER] KyuubiOperationPerServerSuite - [SERVER] KyuubiOperationPerGroupSuite - [SERVER] KyuubiOperationPerUserSuite - [SERVER] KyuubiOperationYarnClusterSuite ... other Suites for specific cases ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1287 from pan3793/ut. Closes #1287 d9a410ca [Cheng Pan] Fix K8s tests 478a60dc [Cheng Pan] Fix scalastyle 91f2a27b [Cheng Pan] Rename BasicQueryTests to SparkQueryTests 40154223 [Cheng Pan] Rename JDBCTestHelper to HiveJDBCTestHelper efb99110 [Cheng Pan] Use UGI createUserForTesting d82b0635 [Cheng Pan] Reset UGI 11927101 [Cheng Pan] Refresh group mappings after update Hadoop conf b21f36e7 [Cheng Pan] Restore lazy val user 25008ad3 [Cheng Pan] Rename `KyuubiOperationGroupSuite` to `KyuubiOperationPerGroupSuite` d71c7e1b [Cheng Pan] Refacotor test hierarchical Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
14dfd14a93
|
[KYUUBI #1284] Add Kyuubi Query Sesstion Page
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. -->   ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1285 from timothy65535/1284. Closes #1284 13bffaaf [timothy65535] [KYUUBI #1284] Add Kyuubi Query Sesstion Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
7b3442a709
|
[KYUUBI #660] Add UDF session_user
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As we now fully support GROUP share level, the session user and the system user can be different, so we now need to add UDF session_user. Also, we fix the system_user to always return the `sparkUser` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1274 from yaooqinn/660. Closes #660 c8ed1f77 [Kent Yao] dead code 20c18640 [Kent Yao] nit 4ab1476c [Kent Yao] nit 081f32f2 [Kent Yao] nit e332ca78 [Kent Yao] Merge branch 'master' into 660 135b058f [Kent Yao] ci 646ddeec [Kent Yao] [KYUUBI #660] Add UDF session_user Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
wForget
|
ccb8117cff
|
[KYUUBI #1267] Modify comments for the golden file test suite and fix it
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Modify comments for the golden file test suite and fix it, see #1267 for details. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1272 from wForget/KYUUBI-1267. Closes #1267 f8c4c586 [wForget] [KYUUBI #1267] Make appName and appId to lazy 417c9db7 [wForget] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into KYUUBI-1267 512458be [wForget] [KYUUBI #1267] Modify comments for the golden file test suite and fix it Authored-by: wForget <643348094@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
fe8933c8ec |
[KYUUBI #1277] Add KDF engine_id
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> It is useful for client side. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1277 from turboFei/engine_id. Closes #1277 95526f09 [fwang12] remove redundant test 382c96dc [fwang12] test all the jdbc tests with yarn mode 01b2ea13 [fwang12] add udf engine_id Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
timothy65535
|
d6cefb1b9c
|
[KYUUBI #1188] Add Statement stats on Kyuubi Query Engine Page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add Statement stats on Kyuubi Query Engine Page  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1234 from timothy65535/ky-1188. Closes #1188 6e0ac523 [timothy65535] trigger rebuild 2b2b2a03 [timothy65535] update patch f0431359 [timothy65535] [KYUUBI #1188] Add Statement stats on Kyuubi Query Engine Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
589b23f578
|
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767 In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`. After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1257 from pan3793/pom. Closes #1257 1d871b6c [Cheng Pan] nit 874f30c1 [Cheng Pan] Address comments 00a32b25 [Cheng Pan] nit 445666ee [Cheng Pan] typo bd96097e [Cheng Pan] nit e85116cd [Cheng Pan] nit 98347c8c [Cheng Pan] Update doc 6615d7cc [Cheng Pan] Exclude HudiTest in cross test 087d3fde [Cheng Pan] Add spark 3.2 binary test ccff63c1 [Cheng Pan] update GA 23b81dcd [Cheng Pan] Address comments 3055c2a8 [Cheng Pan] nit 49a41e86 [Cheng Pan] nit 8d400798 [Cheng Pan] Reduce unnecessary maven profile ced8d987 [Cheng Pan] Reduce unnecessary maven profile Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
wForget
|
0653dada40
|
[KYUUBI #1204] Enhance zookeeper authentication and acls
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Please see #1204 for details. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1213 from wForget/KYUUBI-1204. Closes #1204 fa88ffbe [wForget] [KYUUBI #1204] Change the default value of kyuubi.ha.zookeeper.auth.type to NONE 13cd7ecd [wForget] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into dev-1.3.0 3a84c0e1 [wForget] [KYUUBI #1204] fix tests and re-generate settings.md 92c331c7 [wForget] [KYUUBI #1204] Adjust the parameters of zookeeper authentication db016a03 [wForget] [KYUUBI #1204] Adjust the logic of obtaining the keytab file path 8adb812b [wForget] [KYUUBI #1204] Enhance zookeeper authentication and acls Authored-by: wForget <643348094@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
3afcfe6626
|
[KYUUBI #1263] Kyuubi auxiliary UDF failed on Spark Yarn mode
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/1263 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1264 from ulysses-you/kyuubi-1263. Closes #1263 5685a409 [ulysses-you] id b65cfeff [ulysses-you] test 5a44f0ad [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
48f8d292d6
|
[KYUUBI #1228] [BUILD] Add maven profile spark-3.2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support Spark 3.2.0. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1228 from pan3793/32. Closes #1228 8f2dfd07 [Cheng Pan] Skip Interval test fcc4efc9 [Cheng Pan] Handle SPARK-35378 a7061650 [Cheng Pan] Add maven profile spark-3.2 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
eb5749d34b
|
[KYUUBI #1238] [BUILD] Add back Scala stuff to root POM
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This partial revert [KYUUBI #1203], recover codecov ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1238 from pan3793/mvn. Closes #1238 b3ba87b2 [Cheng Pan] remove unnecessary changes c124340a [Cheng Pan] nit 39c436be [Cheng Pan] pom 28546cbd [Cheng Pan] CI f22e9c91 [Cheng Pan] nit 72e9542d [Cheng Pan] Rebase master 7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
86b8e1ce62
|
[KYUUBI #1236] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The idea based on the analysis of https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1236 from pan3793/shade. Closes #1236 7a4e3ccc [Cheng Pan] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
2700de94c4
|
[KYUUBI #1229] Fix met error when debug KyuubiServer in local IDE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When debug KyuubiServer in IDEA, IDEA: Intellij IDEA 2021.2(Ultimate Edition) throw exception ``` /KyuubiHiveDriverSuite.scala:45:36 Class org.apache.hive.jdbc.HiveConnection not found - continuing with a stub. assert(connection.isInstanceOf[KyuubiConnection]) ``` ``` /KyuubiHiveDriverSuite.scala:47:34 Class org.apache.hive.jdbc.HiveDatabaseMetaData not found - continuing with a stub. assert(metaData.isInstanceOf[KyuubiDatabaseMetaData]) ``` For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/1229 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1231 from timothy65535/ky-1229. Closes #1229 873b3314 [timothy65535] [KYUUBI #1229] Fix met error when debug KyuubiServer in local IDE Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
16aaeafefe
|
[KYUUBI #1227] Remove kyuubi-spark-monitor dependency from kyuubi engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove dead module dependency. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1227 from timothy65535/master. Closes #1227 13dec05d [timothy65535] Remove kyuubi-spark-monitor dependency from kyuubi engine module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
556b5ad57e
|
[KYUUBI #1223] Remove dead module kyuubi-spark-monitor
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> clean dead code ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1223 from yaooqinn/mon. Closes #1223 3b7a9a57 [Kent Yao] Remove dead module kyuubi-spark-monitor b1e2239e [Kent Yao] Remove dead module kyuubi-spark-monitor Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
1f8fa86644
|
[KYUUBI #1152] [SUB-TASK][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Sub task of #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1152 from pan3793/ut-kyuubi-hive-jdbc. Closes #1152 2f508356 [Cheng Pan] Avoid hardcode Hive Jdbc class 5032d23d [Cheng Pan] [KYUUBI #1131][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
b1b7f25faf
|
[KYUUBI #1198] [FEATURE] Support incremental collection
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support incremental collection, [SPARK-25224](https://issues.apache.org/jira/browse/SPARK-25224) Introduce new conf: `kyuubi.operation.incremental.collect` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1198 from pan3793/inc-col. Closes #1198 946068e9 [Cheng Pan] Address comments 2798d0d8 [Cheng Pan] Correct conf doc 3720fd41 [Cheng Pan] Incremental collection Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
badd5d516e
|
[KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1187 from timothy65535/ky-1159. Closes #1159 52daf139 [timothy65535] update conf md a9b50843 [timothy65535] improve patch 36676f2a [timothy65535] improve patch 892a6333 [timothy65535] [KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
9e6eb086b1
|
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1203 from pan3793/pom. Closes #1203 8e6b8f9b [Cheng Pan] fix d31fd198 [Cheng Pan] fix 9da82843 [Cheng Pan] fix 6d150de9 [Cheng Pan] fix 9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhang1002
|
ca9ecd6e34
|
[KYUUBI #1066] Automatically create engine event log root dir when it does not exist
bugfix: #1066 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1183 from zhang1002/auto-create-log-root. Closes #1066 d686e33c [zhang1002] 1. change logRoot to engineLogRoot 2. add some judge for mkDir fb6987f5 [zhang1002] create log root automatically c319127e [张宇翔] test Lead-authored-by: zhang1002 <zhang1002@126.com> Co-authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
d44ccc3644
|
[KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine
### _Why are the changes needed?_ When communicating with NameNode HA, private tokens are created from original token:  In above image, "ha-hdfs:mycluster" is the original token. "ha-hdfs://192.168.56.101:8020" and "ha-hdfs://192.168.56.102:8020" are private tokens. `KyuubiHadoopUtils.getCredentialsInternal` was supposed to extract these private tokens at Kyuubi server side and send them to SQL engine. But in fact, SQL engine side private tokens are automatically created when adding original token, inside `org.apache.hadoop.security.Credentials#addToken`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate SequenceNumber of private tokens always equal to original token:   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1173 from zhouyifan279/#1008. Closes #1008 891dc2b6 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
4f74aba50e |
[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support to config operation log root dir with kyuubi configuration both for server and engine sides. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1160 from turboFei/support_config_operationlog_dir. Closes #1160 ed424f46 [fwang12] fix ut 5edfd4ae [fwang12] refactor ut b7026b2f [fwang12] add ut 6ff4d891 [fwang12] save 30cd5952 [fwang12] fix ut f058619a [fwang12] fix ut 96eb020d [fwang12] refactor ae0ded96 [fwang12] set config value for testing 83d932f4 [fwang12] exclude 33e3e214 [fwang12] update 6f030cba [fwang12] fix 77ec9ca2 [fwang12] save 711ececf [fwang12] Fix code style cfa3bfac [fwang12] refactor b299f8cf [fwang12] complete ut 26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Kent Yao
|
be0b9c1e69
|
[KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Make EngineLoggingService tolerant for creating if any error occurs, and then the main code goes. 2. Separate EngineLoggingService From SparkSQLEngine, and we will try to start EngineLoggingService before SparkSQLEngine to capture all the lifecycle statuses. On the other hand, add a shutdown hook that executed after SparkSQLEngine.stop for EngineLoggingService.stop 3. try-catch for every step of SparkSQLEngine bootstrapping for creating better diagnosis messages. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1171 from yaooqinn/1169. Closes #1169 3a661555 [Kent Yao] ci 733bd1ed [Kent Yao] address comments 36ba1661 [Kent Yao] [KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
413ba2eb24
|
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine
### _Why are the changes needed?_ Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`. ``` 21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds 21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before 21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` 21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds 21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1168 from zhouyifan279/#1009. Closes #1009 fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine 3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
fb6ce8fcc6
|
[KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> It's pre-step of #1152, otherwise will cause circular dependence. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1151 from pan3793/move-jdbc-test. Closes #1131 e65f9979 [Cheng Pan] nit f2e66535 [Cheng Pan] [KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
22e6432e4d
|
[KYUUBI #1022] Add basic EngineStatusStore for events
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981 `EngineStatusStore` helps to push events to listener bus `EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides: - stores all elements, and sorted by startTimestamp. - cleanup the last elements when reach a certain threshold. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1023 from timothy65535/ky-1022. Closes #1022 b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
103969a042
|
[KYUUBI #1067] Catch the real exception when the log directory is not exist
BUGFIX: #1067 fix: 2021-09-23 16:03:43.248 INFO HiveMetaStore.audit: ugi=zhangyu ip=unknown-ip-addr cmd=get_databases: * 2021-09-23 16:03:43.319 ERROR events.JsonEventLogger: Get log directory /Users/zhangyuxiang/Downloads/kyuubi_test failed java.io.FileNotFoundException: File /Users/zhangyuxiang/Downloads/kyuubi_test does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.kyuubi.events.JsonEventLogger.requireLogRootWritable(JsonEventLogger.scala:79) at org.apache.kyuubi.events.JsonEventLogger.initialize(JsonEventLogger.scala:93) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1136 from zhang1002/branch-1.3.0_checkRightException. Closes #1067 de16c63d [张宇翔] delete some ut 5a29bb80 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 21e856a3 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 873cc9c4 [张宇翔] Merge remote-tracking branch 'upstream/master' d1e7a019 [张宇翔] catch IOException when log directory is not exist bc442d8c [张宇翔] catch IOException when log directory is not exist 0d7418e9 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 41cff19a [张宇翔] Merge remote-tracking branch 'upstream/master' 74b36efb [张宇翔] catch IOException when log directory is not exist def8d1f6 [张宇翔] Merge remote-tracking branch 'upstream/master' 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
0ee8eff5a0
|
[KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> After this PR, the service named service discovery will be a child service of frontend service and it's optional. A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1144 from yaooqinn/1143. Closes #1143 f2beb721 [Kent Yao] fix test 128cf282 [Kent Yao] nit 37aaff9c [Kent Yao] lift FrontendService trait add335c4 [Kent Yao] nit 3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy 7e5a5cbb [Kent Yao] Merge branch 'master' into 1143 09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
ee8e5a12f1
|
[KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1142 from yanghua/KYUUBI-1137. Closes #1137 418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
d93a6c8b5b
|
[KYUUBI #1141] Remove redundant unit
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove redundant unit. `MB` ``` 2021-09-23 08:41:08.889 INFO spark.SparkSQLEngine: Spark application name: kyuubi_USER_root_6f67c12c-f362-4a07-87d0-7aa7e5f43171 application ID: spark-461363e9cc934d1986f7ae4191dc0410 application web UI: http://spark-740a727c11ce3da7-driver-svc.default.svc:44692 master: k8s://https://10.10.9.35:6443 version: 3.1.2 driver: [cpu: 1, mem: 1G] executor: [cpu: 2, mem: 2G MB, maxNum: 18] Start time: Thu Sep 23 08:40:39 UTC 2021 User: root (shared mode: USER) State: LATENT ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1141 from timothy65535/remove. Closes #1141 af1827e9 [timothy65535] Remove redundant unit Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
787ba9efe2
|
[KYUUBI #1134] [TEST] DataLake suites should clean up withKyuubiConf instead of extraConfigs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Keep consistent with ``` withKyuubiConf.foreach { case (k, v) => System.setProperty(k, v) kyuubiConf.set(k, v) } ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1134 from pan3793/t. Closes #1134 b8e0aa65 [Cheng Pan] [TEST] DataLake Suites should clean up withKyuubiConf instead of extraConfigs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
b6de06c5cc
|
[KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> **RUN QUERY** ``` set kyuubi.operation.plan.only.mode=ANALYZE; select 1; ``` **ERROR MSG** ``` 2021-09-16 15:19:57.278 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: INITIALIZED_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE 2021-09-16 15:19:57.293 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE, time taken: 0.015 seconds 2021-09-16 15:19:57.309 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-09-16 15:19:57.313 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1114 from timothy65535/ky-1113. Closes #1113 e8409d12 [timothy65535] [KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
4f11b51673
|
[KYUUBI #1059] Add Plan Only Operations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement, PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them. This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite. This is similar to the `explain` command ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1060 from yaooqinn/1059. Closes #1059 e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
6a01f65d3e
|
[KYUUBI #916] Send DelegationTokens to SQL engines
### _Why are the changes needed?_ This PR finishes issue #916 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1074 from zhouyifan279/KYUUBI#916. Closes #916 28176c7e [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines b6c931bd [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
张宇翔
|
354e3c0b5a
|
[KYUUBI #1068] event log can't upload to hdfs
bug: #1068 This version, event log don't support upload file to hdfs ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1075 from zhang1002/branch-1.3_supportLogUploadHdfs. Closes #1068 7b1483c5 [张宇翔] Modify some UT 27aa4596 [张宇翔] Modify some UT c8223914 [张宇翔] Modify some UT 8b3ef67b [张宇翔] Modify some UT fa06ead9 [张宇翔] Merge branch 'master' into branch-1.3_supportLogUploadHdfs 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 8801ece7 [张宇翔] Remove "toString" from ut 3288724b [张宇翔] Modify some UT 0bd7ecbd [张宇翔] Support the eventLog upload to hdfs 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
c080283671
|
[KYUUBI #1080] Move stringifyException to Utils
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Move stringifyException from KyuubiSQLException to Utils, make it more general. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1081 from timothy65535/ky-1080. Closes #1080 ce3015d5 [timothy65535] [KYUUBI #1080] Move stringifyException to Utils Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
8886513534
|
[KYUUBI #1057] [KYUUBI#1056] Output error log when currentEngine is None
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When createSpark in SparkSqlEngine get sparksession or execute initialize.sql failed, we cannot get any error info in logs. At this time, currentEngine is None. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1057 from hddong/engine-fix. Closes #1057 44ead3b3 [hongdongdong] fix 3462d36c [hongdongdong] [KYUUBI#1056] Output error log when currentEngine is None Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Brian Yue
|
033784c691
|
[KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To avoid performance issues in the topK scenario and close #1018 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1054 from byyue/feature-1018. Closes #1018 21de1fcb [Brian Yue] Set spark.sql.execution.topKSortFallbackThreshold to 10000 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
qiuliang
|
d83674248e
|
[KYUUBI #1034] Engine may deadlock when close operationLog
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1062 from qiuliang988/1034. Closes #1034 41d510f5 [qiuliang] add comments dbd7c770 [qiuliang] [KYUUBI #1034]Engine may deadlock when close operationLog Authored-by: qiuliang <qlcumt@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
59d9f06a46 |
[KYUUBI #1032][FOLLOWUP] Refine session hande in AbstractSession
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove session hande in `AbstractSession` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1046 from ulysses-you/kyuubi-1032-f. Closes #1032 1d7543a9 [ulysses-you] order 36a0a1cc [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
c1e5545c64 |
[KYUUBI #1043] Let spark history logger handle events asynchronously
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> > there seems to be no good way to solve the synchronization problem. For `EventLoggerType.SPARK` This looks simple, Please send a separate PR to solve this issue ```git - sc.eventLogger.foreach(_.onOtherEvent(kyuubiEvent)) + sc.listenerBus.post(kyuubiEvent) ``` _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/1023#issuecomment-913558016_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1044 from timothy65535/1043. Closes #1043 0ea1d4f7 [timothy65535] [KYUUBI #1043] Let spark history logger handle events asynchronously Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
