6509a6b925
1137 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
hongdongdong
|
6509a6b925
|
[KYUUBI #1047] Use deployment to run kyuubi server HA on k8s
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> kyuubi-pod.yaml start a single kyuubi server, use deployment to start multiple servers. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1048 from hddong/add-k8s-deployment. Closes #1047 661463b9 [hongdongdong] [KYUUBI#1047] Use deployment to run kyuubi server HA on k8s Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
simon
|
e3ba77ca94
|
[KYUUBI #1040] Add NoOpBootstrapIndex config for hudi table
Add create table options for DataLakeSuiteMixin ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1040 from simon824/master. Closes #1040 ea5db291 [simon] remove space 3d002afd [simon] fix "File line length exceeds 100 characters" 1939038e [simon] roll back d25d3721 [simon] roll back 41232e55 [simon] remove dataLakeSuiteMixin tableOptions 68d6e891 [simon] fix error commit b802603c [simon] add getTableOption f6d37426 [Simon] Merge branch 'apache:master' into master 8b3b4b85 [simon] add tableOptions Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
455bdac2bf |
[KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
### _Why are the changes needed?_ This PR finishes the work of task #1008 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1029 from zhouyifan279/KYUUBI#1008. Closes #1008 695cd955 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider da50aaef [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider bf7c628f [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider ed9739ab [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider 2512abf9 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider 9c88f721 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
eecf074bf1 |
[KYUUBI #1045] Clean up zorder .g4
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Clean up .g4 file. Remove some grammar we donn't need ### _How was this patch tested?_ Pass CI Closes #1045 from ulysses-you/clean-up-zorder. Closes #1045 cf1a5cc3 [ulysses-you] eight indention d6d7eb6e [ulysses-you] pom 8da9df92 [ulysses-you] clean up Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
41bed9f9b3 |
[KYUUBI #1036] New Bug Report Template based on YAML
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> See https://gh-community.github.io/issue-template-feedback/structured/ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1036 from yaooqinn/bugreport. Closes #1036 e3094ce1 [Kent Yao] Add bug-report based on yml a10522d7 [Kent Yao] Add bug-report based on yml a516d769 [Kent Yao] Add bug-report based on yml 0404788b [Kent Yao] Add bug-report based on yml 24506997 [Kent Yao] Add bug-report based on yml Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
10d8ab90dd |
[KYUUBI #1031] Implement api: /sessions/count
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1038 from yanghua/KYUUBI-1031. Closes #1031 36ad5b97 [yanghua] Addressed the review concerns 66d8b0fe [yanghua] [KYUUBI #1031] Implement api: /sessions/count Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
59d9f06a46 |
[KYUUBI #1032][FOLLOWUP] Refine session hande in AbstractSession
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove session hande in `AbstractSession` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1046 from ulysses-you/kyuubi-1032-f. Closes #1032 1d7543a9 [ulysses-you] order 36a0a1cc [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
347c932768 |
[KYUUBI #1032] Keep server session id and engien session id consistent
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/1032 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1033 from ulysses-you/engine-session-id. Closes #1032 5aa5f5f8 [ulysses-you] indention a97ae1b5 [ulysses-you] nit aef754b4 [ulysses-you] engine ref id 62399c40 [ulysses-you] cleanup 2bf13a63 [ulysses-you] nit 11e925dd [ulysses-you] engine side generate da3f51be [ulysses-you] nit 6136f66b [ulysses-you] keep server session id and engien session id consistent Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
c1e5545c64 |
[KYUUBI #1043] Let spark history logger handle events asynchronously
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> > there seems to be no good way to solve the synchronization problem. For `EventLoggerType.SPARK` This looks simple, Please send a separate PR to solve this issue ```git - sc.eventLogger.foreach(_.onOtherEvent(kyuubiEvent)) + sc.listenerBus.post(kyuubiEvent) ``` _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/1023#issuecomment-913558016_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1044 from timothy65535/1043. Closes #1043 0ea1d4f7 [timothy65535] [KYUUBI #1043] Let spark history logger handle events asynchronously Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
19072f9642 |
[KYUUBI #1042] [DEPS] Bump jersey 2.30
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Align with `spark-core` https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.1.2 https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.0.3 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1042 from pan3793/jersey. Closes #1042 f5a05cd0 [Cheng Pan] Update dependencyList abb517e2 [Cheng Pan] [DEPS] Bump jersey 2.30 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
4621842283 |
[KYUUBI #939][FOLLOWUP] Refine pom.xml
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1041 from pan3793/deps. Closes #939 29d566c8 [Cheng Pan] Address comments e18f2c4e [Cheng Pan] Refine pom.xml Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
f584953f7f
|
[KYUUBI #703][FOLLOWUP][DEPS][TEST] Reduce Hudi dependencies
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> HBase deps is optional for creating hudi table and query, but required for writing data to hudi table, as currently we only test create table and query for data lake formats, it's reasonable to remove the hbase deps to reduce the deps pulled from hudi. This PR also change all data lake test suites in spark engine from `hive` catalog to `in-memory` catalog. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1028 from pan3793/hudi. Closes #703 56c41a30 [Cheng Pan] Use in-memory catalog for DataLake test b582a865 [Cheng Pan] Revert "Default use in-memory in Spark engine test" e62c403a [Cheng Pan] Default use in-memory in Spark engine test e2b8e0fd [Cheng Pan] Remove unused hudi spark conf 069fbacc [Cheng Pan] Revert "HFileBootstrapIndex" ca1b20e1 [Cheng Pan] HFileBootstrapIndex da37b0fc [Cheng Pan] Remove log4j 41ce8526 [Cheng Pan] Align jackson version f56f5e89 [Cheng Pan] Remove HBase dependencies c2b76b54 [Cheng Pan] Exclude hudi-timeline-service 56641762 [Cheng Pan] nit e588c12f [Cheng Pan] [DEPS][TEST] Reduce Hudi dependencies Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
qphien
|
4cd83790ae |
[KYUUBI #939] Add Z-Order extensions to support optimize SQL
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ This PR adds Z-Order extension to support `OPTIMIZE` SQL in https://github.com/apache/incubator-kyuubi/issues/939. Currently this PR only supports optimize `HiveTableRelation`. We need to set `hive.exec.dynamic.partition` to `true` and `hive.exec.dynamic.partition.mode` to `nonstrict` when a partitioned table is optimized. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #990 from qphien/zorder. Closes #939 9190e315 [qphien] Add Z-Order extensions to support optimize SQL Authored-by: qphien <clxidian@126.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
121de7fbb0
|
[KYUUBI #1035] Add classification of SQL_Type: this PR is just for DML
Add classification of SQL_Type: this PR is just for DML ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1035 from zhang1002/branch-1.3_get-sql-type-for-dml. Closes #1035 eb6a96d8 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 8e586df1 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml ba0cd0a3 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' 18f35f2b [张宇翔] Add the classification of sql: DML 025bc3f8 [张宇翔] Add the classification of sql: DML 55ef6af7 [张宇翔] some modification f1f8b355 [张宇翔] change other to undefined 1052ae45 [张宇翔] Change some code standards 5e21dc62 [张宇翔] Change some code standards f531744d [张宇翔] Add dml test 3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql 93b5624a [张宇翔] Exclude license check for json d8187ced [张宇翔] Exclude license check for json e46bc86e [张宇翔] Add exception 3b358bf0 [张宇翔] Add licence 1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' ba8f99eb [张宇翔] Use extension to get simpleName c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 55b85849 [张宇翔] Update settings.md ecbd8000 [张宇翔] Repair the scalastyle violations. 76edd20d [张宇翔] Repair the scalastyle violations. d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' 65a90958 [张宇翔] Classification for sqlType: DDL a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
057bac1913
|
[KYUUBI #1024][FOLLOWUP] Fix source release folder name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1030 from pan3793/pkg. Closes #1024 4fad69f9 [Cheng Pan] [KYUUBI #1024][FOLLOWUP] Fix source release folder name Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
05fc1c0ba8
|
[KYUUBI #1027] [BUILD] Use relative path to generate sha512 for release tarballs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1025 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1027 from pan3793/sha. Closes #1027 3e384b80 [Cheng Pan] [BUILD] Use relative path to generate sha512 for release tarballs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
fdd51d55f8
|
[KYUUBI #1026] [BUILD] Rename tarball to apache-kyuubi
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1024 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1026 from pan3793/tarball. Closes #1026 bc515aab [Cheng Pan] [BUILD] Rename tarball to apache-kyuubi Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
simon
|
098e660a3d
|
[KYUUBI #703] support hudi-0.9.0
### _Why are the changes needed?_ Hudi has add Spark Sql Support in version 0.9.0. This PR add Hudi-0.9.0 dependency and Unit Testing #703 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #994 from simon824/master. Closes #703 a04b2e42 [simon] exclude jdk.tools dep 50e1ab01 [simon] fix jdk11 dep f33ebdf1 [simon] fix hudi dep conflicts ff2585c2 [simon] fix yarn dep conflicts 32dd1ea8 [simon] fix scala version conflicts b8a37401 [simon] add spark.sql.catalogImplementation 82f5422f [simon] fix spark3 dependency 2a6c497d [simon] fix spark3 dependency 4dbec8be [Simon] Merge branch 'apache:master' into master 3f180157 [simon] fix maven 73e48d0a [simon] add spark3 support maven dependency 3def658b [simon] fix missing spark-sql-engine hudi maven dependency 524132d3 [simon] bugfix e98998a6 [simon] fix pom error 59fc6669 [Simon] Merge branch 'apache:master' into master d9e17ebf [simon] fix conflict 2466ece3 [simon] add sparkSQL test on hudi-0.9.0 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
32111a3014
|
[KYUUBI #999] Build the basic framework for rest frontend service
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1003 from yanghua/KYUUBI-999. Closes #999 78fdf542 [yanghua] Changed rest frontend service port from 10009 to 10099 73929d5d [yanghua] [KYUUBI #999] Build the basic framework for rest frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
134da16f36
|
[KYUUBI #1019] [BUILD] Remove developers from pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> > I would recommend not doing this and in particular not not adding organisation as could be taken as misrepresenting who owns the project. People must act as individuals in the interest of the project when working on a project and not the company they are working for. > > Kind Regards, Justin ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1019 from pan3793/pom. Closes #1019 11369b7d [Cheng Pan] [BUILD] Remove developers from pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
8217f3d92f
|
[KYUUBI #1020] Fix incubating issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make it more cleaner. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1020 from ulysses-you/incubating. Closes #1020 5ae2d0d4 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
f8056f458b
|
[KYUUBI #1007] Implement delegation token renewal framework
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR finishes sub-task #1007 under umbrella issue #913. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1015 from zhouyifan279/KYUUBI#1007. Closes #1007 94263804 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework 6cea96d0 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework dbffcf15 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework 57100271 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
01bebe36e8 |
[KYUUBI #1011] Fallback some frontend service config option about thrift
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1016 from yanghua/KYUUBI-1011. Closes #1011 78107b76 [yanghua] Addressed review suggestion 651b5989 [yanghua] [KYUUBI #1011] Fallback some frontend service config option about thrift Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
7b289ae6ff
|
[KYUUBI #1002] Add classification of SQL_Type: this PR is just for DDL
Add classification of SQL_Type: this PR is just for DDL
Now we use the simple name to judge this operation belong to which classification of SQL_Type.
The matching rule store in #{sql-classification-default.json}.
The key is the simpleClassName of this analyzed logical plan, the value is the classification of this sql.
It will store in the #{sparksession.conf}, the configuration item is #{spark.sql.classification}.
Why we use the file to store the whole rules?
If we need to change the matching rule, we only need to revise this file.
When the kyuubi started, it will load this json file only once.
When user execute a sql, it will generate a logicalPlan and get its simple name.
By the rules, the function will return the classification that belong to this sql, then it will set this value into #{sparksession.conf}.
For this function, I use SparkSessionExtension to develop it.
### _Why are the changes needed?_
By these SQL_Type, we can do something below:
1. Reduce some log print:
For Auxiliary Statements(the sql type is others), we don't need to print operation log.
2. Optimizing some configuration item.
For example, in final stage, the conf is different between DML and DQL.
Through this configuration item, we can use different conf for them.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1002 from zhang1002/branch-1.3_get-sql-type-for-ddl.
Closes #1002
55ef6af7 [张宇翔] some modification
f1f8b355 [张宇翔] change other to undefined
1052ae45 [张宇翔] Change some code standards
5e21dc62 [张宇翔] Change some code standards
3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master'
c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql
93b5624a [张宇翔] Exclude license check for json
d8187ced [张宇翔] Exclude license check for json
e46bc86e [张宇翔] Add exception
3b358bf0 [张宇翔] Add licence
1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master'
ba8f99eb [张宇翔] Use extension to get simpleName
c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master'
55b85849 [张宇翔] Update settings.md
ecbd8000 [张宇翔] Repair the scalastyle violations.
76edd20d [张宇翔] Repair the scalastyle violations.
d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master'
65a90958 [张宇翔] Classification for sqlType: DDL
a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master'
b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master'
4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master'
8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master'
cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master'
0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master'
b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master'
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
|
||
|
Kent Yao
|
d375eb0cfc
|
[KYUUBI #1010] Rename engine.share.level.sub.domain to engine.share.level.subdomain
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `subdomain` is a widely used term in computer science ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1010 from yaooqinn/subdomain. Closes #1010 72ed19a3 [Kent Yao] address comments 4dcae414 [Kent Yao] fix test c3bb788d [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain b7762357 [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
5edc5066f2
|
[KYUUBI #1013] Enhance bootstrap shell script
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To make the error message more clear when `SPARK_HOME` is invalid. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1013 from pan3793/sh. Closes #1013 c9a573b1 [Cheng Pan] nit e1628640 [Cheng Pan] indent with 2 spaces 68a767f5 [Cheng Pan] Address comments 59d4f076 [Cheng Pan] Enhance load-kyuubi-env.sh Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
35f5c0dcdf
|
[KYUUBI #1014] [BUILD] Detected dependency list change only on JDK 8
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> There is a profile named `jdk11+` in `jersey-common` that will be activated on `jdk[11,)`, it will add a new dependency and cause the dependency list check to fail. ``` <profile> <id>jdk11+</id> <activation> <jdk>[11,)</jdk> </activation> <dependencies> <dependency> <groupId>com.sun.activation</groupId> <artifactId>jakarta.activation</artifactId> </dependency> </dependencies> ... </profile> ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1014 from pan3793/deps. Closes #1014 1a366dab [Cheng Pan] [BUILD] Detected dependency list change only on JDK 8 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
17de188193 |
[KYUUBI #1012] Add More Issue Templates
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> add qa/story/KBIP redirection on github issues ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate For verification, please go to https://github.com/yaooqinn/incubator-kyuubi/issues/new/choose - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1012 from yaooqinn/config. Closes #1012 07343c55 [Kent Yao] Add More Issue Templates Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Fu Chen
|
f43c78d216
|
[KYUUBI #1004] Creates the default zk address instead of falling back to frontend bind address
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As the Spark driver may launch from another node, when we start the Kyuubi server with an embedded zookeeper and bind the frontend address to `0.0.0.0`, the backend engine can't register correctly. This pr changes the default behavior of `kyuubi.zookeeper.embedded.client.port.address`, creates the default zookeeper address with local hostname instead of falling back to `kyuubi.frontend.bind.host`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1004 from cfmcgrady/fallback-address. Closes #1004 99f0a143 [Fu Chen] fix test 5e66c5f0 [Fu Chen] create default zk address instead of fall back to frontend bind address Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
866a63d7a7
|
[KYUUBI #1005] ServiceDiscoverySuite fails if hostname contains uppercase letter
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Executing test case "ServiceDiscoverySuite set up zookeeper auth" on a Linux host named "KYUUBI" fails. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1006 from zhouyifan279/KYUUBI#1005. Closes #1005 045c6b34 [zhouyifan279] [KYUUBI #1005] ServiceDiscoverySuite fails if hostname contains uppercase letter Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
b382ed0dd8 |
[KYUUBI #988][FOLLOW] Fix test failed since event type changed
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix test ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1001 from ulysses-you/kyuubi-988. Closes #988 378a75bb [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
hongdongdong
|
94d60ea505 |
[KYUUBI #1000] Use underscore instead hyphen in path
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Use underscore instead hyphen in path, friendly to sql. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1000 from hddong/fix-underscore. Closes #1000 8b3459a5 [hongdongdong] Use underscore instead hyphen in path Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
d49f8bf06d |
[KYUUBI #988] Add Kyuubi session event
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> closes https://github.com/apache/incubator-kyuubi/issues/988 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #989 from ulysses-you/kyuubi-session-event. Closes #988 baebf2eb [ulysses-you] address comment 283d574c [ulysses-you] fix c5d1d5f7 [ulysses-you] adddress comment 44cfd06f [ulysses-you] docs 43639993 [ulysses-you] simplify session 2618b764 [ulysses-you] config name a2d4608b [ulysses-you] docs 5adbe5c8 [ulysses-you] config ea5c70e1 [ulysses-you] nit eea73ed5 [ulysses-you] config cda6d38b [ulysses-you] test a754c76f [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
83833562d1
|
[KYUUBI #998] Update release doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #998 from pan3793/gpg. Closes #998 f32c561e [Cheng Pan] Update doc 07d5fe3d [Cheng Pan] Update release doc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
7141f2f13e
|
[KYUUBI #997] Fix build error when project path contains kyuubi-
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ The building will fail if the project name contains `kyuubi-`. For example: for our jenkins job, its name is `kyuubi-binary-build`, and the TMP dir won't be created because all the jar files are filtered because their absolute path contains `kyuubi-`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #997 from turboFei/fix_collect_license. Closes #997 eef47b79 [fwang12] follow up 45fe7834 [fwang12] replace tab with 2 spaces f6589403 [fwang12] refactor c0cddd96 [fwang12] fix build error Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
4ed5704590
|
[KYUUBI #983] Refactor the basic abstraction of frontend service
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #984 from yanghua/KYUUBI-983. Closes #983 db953b8a [yanghua] Rename KyuubiFrontendService to be KyuubiFrontendServices 8c3d7f2a [yanghua] [KYUUBI #983] Refactor the basic abstraction of frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
cd426798bf |
[KYUUBI #991][FOLLOWUP] Improve error msg and add SPARK_HOME in Examples
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> According [comments](https://github.com/apache/incubator-kyuubi/pull/991#issuecomment-906914618) 1. Improve error message. 2. add SPARK_HOME in Examples. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #993 from cfmcgrady/ky-991-followup. Closes #991 e7c25be6 [Fu Chen] update docs 945eeb73 [Fu Chen] review 98555196 [Fu Chen] improve error msg and add SPARK_HOME in Examples Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
acb4f29906
|
[KYUUBI #962] Support engine pool feature
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For detail, please go to _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/962#issuecomment-905116261_ ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request ``` ./bin/beeline -u 'jdbc:hive2://localhost:10009/;?kyuubi.engine.pool.enabled=true;kyuubi.engine.pool.size=3;' ``` Closes #986 from timothy65535/ky-962-v2. Closes #962 b79f8778 [timothy65535] fix test name 9a435607 [timothy65535] fix style abc746b8 [timothy65535] update tests 71b6933d [timothy65535] revert changes 018dee8b [timothy65535] revert changes e3492934 [timothy65535] revert changes 67fc8fd9 [timothy65535] improve threshold fa52acb2 [timothy65535] update pool threshold d8b5977d [timothy65535] remove dup get 7f39b8d7 [timothy65535] fix test error 4fdf8d76 [timothy65535] update thread pool size to 1 -> threshold 2477348a [timothy65535] rich engine pool tests 8eee427f [timothy65535] fix update error msg 104be5ea [timothy65535] fix error tests 8c2d1441 [timothy65535] fix doc ddd0eda9 [timothy65535] improve doc & tests 83d7c5cf [timothy65535] update doc 4b6c7ee3 [timothy65535] improve subdomain 169d9d06 [timothy65535] update todo 1424e565 [timothy65535] increase pool size range 3523cd6c [timothy65535] update todo 2cbeafb2 [timothy65535] remove enable option 424b5364 [timothy65535] revert ServiceDiscovery 6fc1c510 [timothy65535] remove random policy 9feef8f3 [timothy65535] [KYUUBI #962] Support engine pool feature Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
c59e54a675
|
[KYUUBI #992] Fix SPARK_EXTENSION_VERSION_ is seen as a variable
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix dist bug `$SPARK_EXTENSION_VERSION_` is empty `${SPARK_EXTENSION_VERSION}_` is `3-1_` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #992 from ulysses-you/fix-SPARK_EXTENSION_VERSION_. Closes #992 4379093e [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
9695f75889 |
[KYUUBI #991] [MINOR] improve the error message of SPARK package not found.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As `File.listFiles` may return null when spark package not found in folder `${PROJECT_ROOT_DIR}/kyuubi-server/externals/kyuubi-download/target`, the NPE will throw. Improve the error message. Before thie pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [9749221f-5b30-457b-b8e5-25affba25061] for fchen due to null at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:204) at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:204) at scala.collection.IndexedSeqOptimized.isEmpty(IndexedSeqOptimized.scala:30) at scala.collection.IndexedSeqOptimized.isEmpty$(IndexedSeqOptimized.scala:30) at scala.collection.mutable.ArrayOps$ofRef.isEmpty(ArrayOps.scala:198) at scala.collection.TraversableLike.headOption(TraversableLike.scala:608) at scala.collection.TraversableLike.headOption$(TraversableLike.scala:608) at scala.collection.mutable.ArrayOps$ofRef.headOption(ArrayOps.scala:198) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$1(SparkProcessBuilder.scala:50) at scala.Option.orElse(Option.scala:447) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:41) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` After this pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [5ea6a8b3-1727-4e0d-b542-992f84afcde8] for fchen due to SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$5(SparkProcessBuilder.scala:61) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:61) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #991 from cfmcgrady/improve-error-msg. Closes #991 1e8da683 [Fu Chen] imporove the error message Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
3afd20e03a
|
[KYUUBI #985] Add logCaptureThreadReleased flag
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `isInterrupted` is always false if some `InterruptedException` are throwed. So we should use a new flag to check if log capture thread is released. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #985 from ulysses-you/flaky-test. Closes #985 1fcc0a99 [ulysses-you] empty 463a996f [ulysses-you] empty d1f2ea54 [ulysses-you] eventually f9f37343 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
d704e4d49d |
[KYUUBI #922] Add OperationEvent for server to track operation state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Track operation state change and support output to custom system. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #968 from hddong/add-server-events. Closes #922 8be8f268 [hongdongdong] Add prefix for StatementEvent 9fdcc504 [hongdongdong] fix comment a0cdb35d [hongdongdong] fix test 4d340c67 [hongdongdong] add dependency 00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
d58a092245
|
[KYUUBI #976] [BUILD] Fix source release build
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support build, run unit tests, and create binary tarball from source release. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request The verify steps: 1. create a source release tarball ``` RELEASE_VERSION=1.4.0-SNAPSHOT build/release/create-package.sh source ``` 2. untar tarball ``` cd work/release tar -xzf kyuubi-1.4.0-SNAPSHOT-bin.tgz cd kyuubi-1.4.0-SNAPSHOT-bin ``` 3. compile and run unit test ``` build/mvn clean install ``` 4. create binary release tarball ``` build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1 ``` Closes #976 from pan3793/source-build. Closes #976 354968ae [Cheng Pan] Compress imgs 9c47b64f [Cheng Pan] Restore docs/imgs in source release 5d86ecc7 [Cheng Pan] Clean up imgs a3399279 [Cheng Pan] Support create binary from source release bb279863 [Cheng Pan] [BUILD] Fix source release ut failures Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3ebb50f1cb
|
[KYUUBI #978] [LICENSE] Fix the source license issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the license issue mentioned by WillemJiang ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request The final `NOTICE` for binary release is not `NOTICE-binary` in root path but is generated by `build/release/collect_licenses.sh`, which will extract all NOTICE content from the bundled jars. ``` Apache Kyuubi Copyright 2021 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (https://www.apache.org/). Apache Spark Copyright 2014 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Export Control Notice --------------------- This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <http://www.wassenaar.org/> for more information. The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: This software uses Apache Commons Crypto (https://commons.apache.org/proper/commons-crypto/) to support authentication, and encryption and decryption of data sent across the network between services. Metrics Copyright 2010-2013 Coda Hale and Yammer, Inc. This product includes software developed by Coda Hale and Yammer, Inc. This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, LongAdder), which was released with the following comments: Written by Doug Lea with assistance from members of JCP JSR-166 Expert Group and released to the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/ Curator Client Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Framework Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Recipes Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Hive Service RPC Copyright 2020 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). htrace-core4 Copyright 2016 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may be licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. Apache log4j Copyright 2007 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). ``` Closes #978 from pan3793/license. Closes #978 2a7eff44 [Cheng Pan] Update License d21825dd [Cheng Pan] [LICENSE] Fix the source license issue Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
cb70e062c3
|
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #980 from yanghua/KPIP-1. Closes #979 e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
odone
|
f1ae664910
|
[KYUUBI #961] Fixed: opHanle's operation may not be release
Align with `sessionManager.operationManager.closeOperation(operationHandle)` and remove first <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #975 from iodone/fix-opHanle-operation-not-release. Closes #961 bd798e0d [odone] [KYUUBI #961] fixed: opHanle's operation may not be release Authored-by: odone <odone.zhang@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
33be4516b6
|
[KYUUBI #973] [LICENSE] Fix license issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Remove the `scala.util.Using` - Remove redundant license declaration ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #973 from pan3793/license. Closes #973 b02206ea [Cheng Pan] Remove redundant license declaration 3a1401d9 [Cheng Pan] [LICENSE] Remove Scala Using Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
271ccc948f
|
[KYUUBI #972] [LICENSE] Exclude binary/doc files from source release
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #972 from pan3793/src-license. Closes #972 443cd05c [Cheng Pan] Exclude unused files from source release 52271827 [Cheng Pan] Exclude unused files from source release d475b511 [Cheng Pan] Exclude unused files from source release e76299bd [Cheng Pan] Exclude docs from source release 2e42bc8c [Cheng Pan] Exclude whole tpcds module from source release 93b41a8c [Cheng Pan] [LICENSE] Exclude binary files from source release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
2f82f33e9d
|
[KYUUBI #971] [LICENSE] Add DISCLAIMER to binary release
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #971 from pan3793/disclaimer. Closes #971 c0fe3821 [Cheng Pan] [LICENSE] Add DISCLAIMER to binary release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
ae4878349c
|
[KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Workaround to recover Minikube test until #937 is solved. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #969 from pan3793/747-ignore. Closes #659 e26153f6 [Cheng Pan] [KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |