065c98a259
145 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ulysses-you
|
065c98a259
|
[KYUUBI #897] Support stop Spark engine through web ui
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/897 ### _How was this patch tested?_ The following two images show the stop feature is enabled:   The following image show the stop feature is disabled:  Closes #898 from ulysses-you/kyuubi-897. Closes #897 61ad25c1 [ulysses-you] address comment b63997ec [ulysses-you] maven dbb36014 [ulysses-you] maven 3c91307c [ulysses-you] simplify 6e9e5bca [ulysses-you] pom 4e143657 [ulysses-you] config e579e227 [ulysses-you] Support stop Spark engine through web ui Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
4fecf13902 |
[KYUUBI #888] Add scala version suffix on scala modules
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Add scala version suffix on scala modules to match the name pattern. - Enable `kyuubi-extension-spark-3-1` in daily publish snapshot ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #888 from pan3793/scala. Closes #888 571eb9c3 [Cheng Pan] comments 16cb263a [Cheng Pan] remove hardcode scala version in pom 6090c798 [Cheng Pan] Fix shade 5178847f [Cheng Pan] Fixup a1547753 [Cheng Pan] Fix engine jar name 2cf4793e [Cheng Pan] fix cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot 1ca31de5 [Cheng Pan] Add scala version suffix on scala modules Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
f6d5069bb6
|
[KYUUBI #885] Add Event for Engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #885 from yaooqinn/engineevent. Closes #885 6a7ac341 [Kent Yao] refine 41baf0e7 [Kent Yao] refine 2c141a06 [Kent Yao] test history server d1b0823a [Kent Yao] test history server 3aae8417 [Kent Yao] dump the final engine status 257e50f5 [Kent Yao] Add Event for Engine f1f1f0e8 [Kent Yao] Add Event for Engine 85addc6e [Kent Yao] Add Event for Engine cece5888 [Kent Yao] Add Event for Engine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
6c3b722174
|
[KYUUBI #869] Add Kyuubi Query Engine for Spark UI with basic stats
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #869 from yaooqinn/ui. Closes #869 aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui 0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats 7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b963b75d58
|
[KYUUBI #857] Support init SQL for each session
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #857 from pan3793/seesion-init-sql. Closes #857 f6f59196 [Cheng Pan] remove unnecessary change 4960830a [Cheng Pan] run ENGINE_SESSION_INITIALIZE_SQL when engine startup 6e8100fa [Cheng Pan] address comments 08f4af62 [Cheng Pan] address comments 2bb1bb85 [Cheng Pan] fix import 880b38ec [Cheng Pan] session initialize sql will only execute once in single session mode 2a646890 [Cheng Pan] test session initialize sql will not execute in single session mode f03f3eff [Cheng Pan] update doc 6e4a72f2 [Cheng Pan] address comments 34285e25 [Cheng Pan] format 6d978025 [Cheng Pan] revert log4j conf change 881a89e0 [Cheng Pan] improve test b9ec18c9 [Cheng Pan] fix ut and add log b4f586d8 [Cheng Pan] Support init SQL for each session Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d5104ed277
|
[KYUUBI #868] Avoid calling a stopped SparkContext
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We shall avoid calling cancelJobGroup if the sc is already stopped. We shall stop the engine to stop serving before we stop the sc during the shutdown hook ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #868 from yaooqinn/shutdownhook. Closes #868 14810875 [Kent Yao] comment 972e2741 [Kent Yao] Avoid calling a stopped SparkContext Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
286e3fbcc1
|
[KYUUBI #862] Revise the unit test for "add kyuubiStatementInfo into queue"
### _Why are the changes needed?_ Revise the unit test for "add kyuubiStatementInfo into queue" ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #862 from zhang1002/update_statementinfo_ut. Closes #862 7e5f8b50 [张宇翔] Revise the unit test for "add kyuubiStatementInfo into queue" e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ce5fb97036
|
[KYUUBI #849] ASF Nexus SNAPSHOT publish
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> https://infra.apache.org/publishing-maven-artifacts.html https://issues.apache.org/jira/browse/INFRA-22114 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #849 from pan3793/asf. Closes #849 c609f6ba [Cheng Pan] update pom 31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent 9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Min Zhao
|
cd308f38ad
|
[KYUUBI #758] [KYUUBI 661] Add UDF system_user
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> add udf system_user. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #758 from zhaomin1423/661_add-system_user. Closes #758 70eba56a [Min Zhao] update system user name f5edc621 [Min Zhao] [KYUUBI 661] Add UDF system_user Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
e7e77f9277
|
[KYUUBI #814] Event Tracking: For jonInfo
### _Why are the changes needed?_ Store all jobInfos for each statement in mem. - First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds. - Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds. Some interfaces: - KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds. - KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on. - KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold. Test case: externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #814 from zhang1002/branch-1.2_add-jobinfo. Closes #814 7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException 0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle) bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener 741fcd0f [张宇翔] Add interval b61b9406 [张宇翔] Code changes cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code c1a1d732 [张宇翔] 1. Remove volatile for jobInfo 9547d49e [张宇翔] 1. Remove volatile for jobInfo 970c0002 [张宇翔] 1. Add volatile for jobInfo 62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations db1de381 [张宇翔] 1. Add some log 2. Add some annotations 8deb61f2 [张宇翔] change some test 87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test 86c49ebc [张宇翔] Merge master branch a0a99b3a [张宇翔] Merge master branch f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5d3b9afb [张宇翔] Code optimization db3a0b6f [张宇翔] Format changes 08d6d1fc [张宇翔] Modify some annotations f62f00e9 [张宇翔] Event Tracking: For job 85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo 3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing b4290bfa [张宇翔] Event tracking: For jobInfo fe1f7cce [张宇翔] Change directory structure 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml 71d33b9a [张宇翔] Add some comment 6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo 8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code f43b3c8f [张宇翔] merge master branch 55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor 6f0be547 [张宇翔] format change bbfba274 [张宇翔] Even tracking 5a62f586 [张宇翔] remove some unused conf b26345ac [张宇翔] Merge master branch and resolve conflict fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished d1676268 [张宇翔] event tracking: for SQLExecutionStart Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Min Zhao
|
3100e8c172
|
[KYUUBI #749] fix wrong hint in test for update doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix wrong hint in test for update doc ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #827 from zhaomin1423/kyuubi_749. Closes #749 a2b10247 [Min Zhao] fix wrong hint in test for update doc Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
98e44636fd
|
[KYUUBI #823] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #823 from pan3793/deps. Closes #823 51cdcb5d [Cheng Pan] restore thrift deps from spark-hive 148038a3 [Cheng Pan] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
vinoyang
|
d8f2be7dbc
|
[MINOR] Remove useless param comment (#806) | ||
|
Yuxiang Zhang
|
ff52b203e0
|
Event Tracking: For statement (#767)
Generate KStatement info. This object includes the following elements: 1. statement 2. statementId 3. appId 4. sessionId 5. executionId 6. physicalPlan 7. stateTime: contains each state and the time occurrence Those data was packaged in KStatement and we store those object in mem. You can get some summary data from this object. |
||
|
timothy65535
|
082b182757
|
[KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine (#780)
* [KYUUBI 770] Fix flaky test: Add config to control if cancel invoke interrupt task on engine |
||
|
ulysses-you
|
69d4197151 |
[KYUUBI #766] Use default zone id for start time
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Provide a more readable time for user. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #766 from ulysses-you/zoneid. Closes #766 55a4c4c1 [ulysses-you] zone id Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
82fd58f1a4
|
[KYUUBI #764] Add Customized Configuration Item: kyuubi.statement.id
For broadcast, Spark will introduce a new runId as SPARK_JOB_GROUP_ID. That cause we can't follow up this statement's whole life by SPARK_JOB_GROUP_ID. This configuration item will be set when this statement was submitted and it will not be changed. You can get the dependency relationship from job or stage to statement by this configuration item in this statement's whole life. Closes #764 from zhang1002/branch-1.2_set-customized-conf. Closes #764 f78f00ed [张宇翔] Format changes ee303e8f [张宇翔] Format changes 72e8a2ca [张宇翔] Format changes 7d135fbf [张宇翔] Add some Static Variable and clean kyuubi.statement.id in the finally block. 4364ef69 [张宇翔] Add Customized Configuration Item: kyuubi.statement.id Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
7385f01562
|
[KYUUBI #762] Add Kyuubi Hive JDBC Module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Aiming to make a better supported client for Kyuubi and Spark - Add catalog to getTables meta function for DataLakes (DONE) - Deploy to maven central (PENDING) ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #762 from yaooqinn/kyuubi-jdbc. Closes #762 5a881c7b [Kent Yao] address comments 6f01f8e0 [Kent Yao] Add Kyuubi Hive JDBC Module Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
张宇翔
|
64bcfe761e |
[KYUUBI #736] BugFix: resolve the problem that can't get jobEnd's metrics from SQLO…
resolve the problem that can't get jobEnd's metrics from SQLOperationListener description: Spark remove the listener before generating the jobEnd's data Closes #736 from zhang1002/branch-1.2_fix-listener-remove. Closes #736 a059a4dd [张宇翔] Remove SQLOperationListener when we close or cancle this operation. This change can ensure to clean the listener that event might be dropped by Spark then we can not receive this event which can cause the memory leak. d3f0143c [张宇翔] Merge branch 'master' into branch-1.2_fix-listener-remove e4871101 [张宇翔] format modification 37576209 [zhang1002] Update ExecuteStatement.scala f2574caf [zhang1002] Update ExecuteStatement.scala 0db4ddcd [张宇翔] remove some unused code 7131aeb0 [zhang1002] Update ExecuteStatement.scala 85cd847e [zhang1002] Update kyuubi-defaults.conf.template 822235b6 [张宇翔] Shutdown timeout cleaner as far as possible #735 c1a9bda7 [张宇翔] change some conf 8410a0c5 [张宇翔] merge master 12096102 [张宇翔] remove unused code 64c6486b [张宇翔] remove SQLOperationListener when the event: SparkListenerSQLExecutionEnd has occured 5f6b3cb8 [张宇翔] BugFix: resolve the problem that can't get jobEnd's metrics from SQLOperationListener Lead-authored-by: 张宇翔 <zhang1002@126.com> Co-authored-by: zhang1002 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
83b7f9aec3 |
[KYUUBI #763] add TPC-DS Tests for spark-monitor module
add TPC-DS Tests for spark-monitor module Closes #763 from zhang1002/branch-1.2_add-monitor-module_fix-tpc-tests. Closes #763 fb7c1bee [zhang1002] Update pom.xml 9a770f9a [zhang1002] Update pom.xml 58e74f8a [zhang1002] Update master.yml 349e1ec4 [张宇翔] merge master bff7fd53 [张宇翔] Merge branch 'branch-1.2_add-monitor-module' into branch-1.2_add-monitor-module_fix-tpc-tests 00e80fd3 [zhang1002] Update pom.xml 0e8d1553 [张宇翔] add module for tpc tests d39767a5 [张宇翔] change module's location 1d151459 [张宇翔] change module name d926d959 [张宇翔] change module name 9bf35e62 [zhang1002] Update pom.xml 34702432 [zhang1002] Update README.md 9a6a68cd [zhang1002] Update README.md 50e3e7d0 [张宇翔] remove dependency cf2cf0f8 [张宇翔] add kyuubi-monitor module Lead-authored-by: 张宇翔 <zhang1002@126.com> Co-authored-by: zhang1002 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
wForget
|
9654a936d2
|
[KYUUBI #751] Support single session mode
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #751 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #752 from wForget/dev. Closes #751 ae9dc873 [wForget] [KYUUBI #751] Support single session mode Authored-by: wForget <643348094@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
902b4b13a5
|
[KYUUBI #737] add monitor module
This module's feature: 1. get all SQL's metrics during runtime 2. monitor the sqlOperation 3. for auto-configuring Closes #737 from zhang1002/branch-1.2_add-monitor-module. Closes #737 00e80fd3 [zhang1002] Update pom.xml d39767a5 [张宇翔] change module's location 1d151459 [张宇翔] change module name d926d959 [张宇翔] change module name 9bf35e62 [zhang1002] Update pom.xml 34702432 [zhang1002] Update README.md 9a6a68cd [zhang1002] Update README.md 50e3e7d0 [张宇翔] remove dependency cf2cf0f8 [张宇翔] add kyuubi-monitor module Lead-authored-by: 张宇翔 <zhang1002@126.com> Co-authored-by: zhang1002 <zhang1002@126.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
cc7e239c20
|
[KYUUBI #735] Shutdown timeout cleaner as far as possible
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> If some `ExecuteStatement` are canceled before timeout in a short time, the timeout cleaners will hold the resource until timeout. This PR aims to release the timeout cleaner as far as possible. ### _How was this patch tested?_ Pass `org.apache.kyuubi.engine.spark.SparkEngineSuites` Closes #735 from ulysses-you/timeout-cleaner. Closes #735 92bc29a6 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
0eb9bdc376
|
[KYUUBI #722] [TEST] Use in-memory catalog in most test suites
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Followup #707, use in-memory catalog in most test suites to avoid derby lock to speed up test suites. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #722 from pan3793/catalog. Closes #722 fcc38af6 [Cheng Pan] fix ut 0c131629 [Cheng Pan] trait HiveJDBCTests extends BasicJDBCTests 19bb4237 [Cheng Pan] [TEST] Use in-memory catalog in most test suites Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ced29f5b8a |
[KYUUBI #707] Respect spark.sql.catalogImplementation in engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Respect `spark.sql.catalogImplementation` in engine. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #707 from pan3793/hive. Closes #707 d46af57b [Cheng Pan] address comments 8570ca89 [Cheng Pan] fixup 8bd4a256 [Cheng Pan] fixup c36659d6 [Cheng Pan] switch to spark.sql.catalogImplementation fb6ebf66 [Cheng Pan] Make engine enable hive optional Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Kent Yao
|
a2f1e22361 |
[KYUUBI #657] Add udf kyuubi_version
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ In this PR, I propose to add kyuubi_version as a user-defined function. This is also a good example to add other new functions that needed in the Kyuubi system. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #686 from yaooqinn/657. Closes #657 d30ac8f6 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version e1e585e9 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
e57a4346e2
|
[KYUUBI #669] Cleanup code and fix typos
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #669 from pan3793/cleanup. Closes #669 2cdc4a3 [Cheng Pan] Cleanup code and fix typos Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b567453462
|
[KYUUBI #643] [TEST] [DELTA] Refine CI and add test detla 1.0.0 with Spark 3.1
* [TEST] [DELTA] Test detla 1.0.0 with Spark 3.1 * seperate DataLakeTest * scalatest.exclude.tags * align gh action and travis * seperate tpcds in GA * travis * reduce redundant ci jobs * rename spark-3.2-snapshot to spark-master * rename tpcds suites * mvn --no-transfer-progress * address comments |
||
|
Kent Yao
|
bb9e8a50e5
|
Prepare Kyuubi v1.3.0 | ||
|
Kent Yao
|
b4a505971c |
[KYUUBI #627] Support multiple engines for non-connection share levels
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid. Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached. With multiple engines support, end-users are able to 1. create as many engines as they want to meet the concurrency. 2. leverage different queue/namespace resource with long-running mode 3. can give different resources or priorities to their own engines for different workloads 4. can perform fast failover by creating a new sub domain 5. etc. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #627 from yaooqinn/multipleengines. Closes #627 9f04bc3e [Kent Yao] fix tests d7af751e [Kent Yao] address comments be9c46d8 [Kent Yao] address comments ccce87d7 [Kent Yao] address comments d49116c7 [Kent Yao] Support multiple engines for non-connection share levels f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
0d8bbfad9e |
[KYUUBI #619] [TEST] Refactor data lake test suite hierarchical
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make `*SuiteMixin` resuable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #619 from pan3793/mixin. Closes #619 fa776869 [Cheng Pan] address comments 54ac9a19 [Cheng Pan] Revert "gather KyuubiConf" 92985964 [Cheng Pan] gather KyuubiConf 61492616 [Cheng Pan] extract DataLakeSuiteMixin 66b6ffa4 [Cheng Pan] [TEST] Refactor data lake test suite hierarchical Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Kent Yao
|
a28e62ad70
|
[KYUUBI #623] Improve OperationLog to deliver detail stats of sql statements
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `OperationLog` is used to divert SQL statement-specific logs to a temporary local file at the engine side and to be fetched to the client-side(e.g. beeline). It is an InheritableThreadLocal variable in which thread the `SparkSession` instance lays. Inside Spark, there are a lot of threads created via the java `Executors`, for which the `OperationLog` cannot be inherited, so the client-side lacks information on how their jobs are running. This is not user-friendly especially for queries that contain a lot of jobs. In this PR, we add a new SparkListener for SQL operation which diverts the SQL statement-specific logs by `spark.jobGroup.id`. The listener will dump Job/Stage and Task summaries to the `OperationLog` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request ```logtalk 2021-05-08 10:25:54.301 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 started with 4 stages, 1 active jobs running 2021-05-08 10:25:54.303 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 6 started with 5 tasks, 1 active stages running 2021-05-08 10:25:54.347 INFO kyuubi.SQLOperationListener: Finished stage: Stage(6, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 5; Took: 44 msec 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: task runtime:(count: 5, mean: 36.400000, stdev: 1.019804, max: 38.000000, min: 35.000000) 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 35.0 ms 35.0 ms 35.0 ms 36.0 ms 36.0 ms 37.0 ms 38.0 ms 38.0 ms 38.0 ms 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 5, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: fetch wait time:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: remote bytes read:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: task result size:(count: 5, mean: 1965.000000, stdev: 0.000000, max: 1965.000000, min: 1965.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 5, mean: 82.948068, stdev: 1.278517, max: 84.210526, min: 80.555556) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 81 % 81 % 81 % 83 % 83 % 84 % 84 % 84 % 84 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: other time pct: (count: 5, mean: 17.051932, stdev: 1.278517, max: 19.444444, min: 15.789474) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 16 % 16 % 16 % 16 % 17 % 17 % 19 % 19 % 19 % 2021-05-08 10:25:54.357 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 7 started with 300 tasks, 1 active stages running 2021-05-08 10:25:54.644 INFO kyuubi.SQLOperationListener: Finished stage: Stage(7, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 300; Took: 287 msec 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: task runtime:(count: 300, mean: 15.203333, stdev: 9.396559, max: 45.000000, min: 4.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 4.0 ms 5.0 ms 6.0 ms 8.0 ms 13.0 ms 18.0 ms 31.0 ms 38.0 ms 45.0 ms 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 300, mean: 56.030000, stdev: 0.298496, max: 59.000000, min: 56.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 59.0 B 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: fetch wait time:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: remote bytes read:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: task result size:(count: 300, mean: 2865.150000, stdev: 9.371633, max: 2906.000000, min: 2863.000000) 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 300, mean: 66.577940, stdev: 18.397778, max: 95.121951, min: 10.526316) 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 11 % 33 % 40 % 56 % 71 % 81 % 87 % 88 % 95 % 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: other time pct: (count: 300, mean: 33.422060, stdev: 18.397778, max: 89.473684, min: 4.878049) 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 5 % 12 % 13 % 19 % 29 % 44 % 60 % 67 % 89 % 2021-05-08 10:25:54.650 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 8 started with 1 tasks, 1 active stages running 2021-05-08 10:25:54.733 INFO kyuubi.SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 1; Took: 87 msec 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: task runtime:(count: 1, mean: 81.000000, stdev: 0.000000, max: 81.000000, min: 81.000000) 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 1, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: remote bytes read:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: task result size:(count: 1, mean: 3065.000000, stdev: 0.000000, max: 3065.000000, min: 3065.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 95.061728, stdev: 0.000000, max: 95.061728, min: 95.061728) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: other time pct: (count: 1, mean: 4.938272, stdev: 0.000000, max: 4.938272, min: 4.938272) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 2021-05-08 10:25:54.739 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 9 started with 200 tasks, 1 active stages running 2021-05-08 10:25:54.844 INFO kyuubi.SQLOperationListener: Finished stage: Stage(9, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 200; Took: 105 msec 2021-05-08 10:25:54.845 INFO scheduler.DAGScheduler: Job 3 finished: collect at ExecuteStatement.scala:82, took 0.547547 s 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: task runtime:(count: 200, mean: 15.100000, stdev: 11.464729, max: 48.000000, min: 4.000000) 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 4.0 ms 6.0 ms 7.0 ms 8.0 ms 10.0 ms 16.0 ms 40.0 ms 43.0 ms 48.0 ms 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: fetch wait time:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: remote bytes read:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: task result size:(count: 200, mean: 2326.135000, stdev: 26.673897, max: 2414.000000, min: 2311.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.4 KiB 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 200, mean: 2.971561, stdev: 8.036931, max: 47.727273, min: 0.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 10 % 18 % 48 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: other time pct: (count: 200, mean: 97.028439, stdev: 8.036931, max: 100.000000, min: 52.272727) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.849 INFO scheduler.StatsReportListener: 52 % 82 % 90 % 100 % 100 % 100 % 100 % 100 % 100 % 2021-05-08 10:25:54.849 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 succeeded, 0 active jobs running 2021-05-08 10:25:54.857 INFO codegen.CodeGenerator: Code generated in 7.116325 ms 2021-05-08 10:25:54.858 INFO operation.ExecuteStatement: Processing kentyao's query[3ac86686-f502-4f2d-9a98-d763e5b9548c]: RUNNING_STATE -> FINISHED_STATE, statement: select /*+ REPARTITION(200) */ count(1) from (select /*+ REPARTITION(300, a) */ a from values(1),(1),(1),(2),(3) t(a)), time taken: 0.702 seconds ``` Closes #623 from yaooqinn/log2. Closes #623 abeff64 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement fdf3b28 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement 783a594 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
34374f3b3e
|
[KYUUBI #613] [TEST] [DELTA] Add test for deltalake
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #613 from pan3793/delta. Closes #613 ccadedb [Cheng Pan] [TEST] [DELTA] Add test for deltalake Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
42fa7cfe49 |
[KYUUBI #544] Involve job max failures and deregister exception TTL
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve job max failures to limit the max job failures before deregistering the engine, involve deregister exception TTL for to restart counting when last failure has gone for one deregisterExceptionTTL. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #560 from turboFei/KYUUBI-544-exception-max-failures-new. Closes #544 951f269 [fwang12] [KYUUBI #544] Involve job max failures and deregister exception TTL Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
913d3f19d4
|
[KYUUBI #544] Combine deregister exception messages and stacktraces
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To simplify the usage and be more user friendly, combine `kyuubi.engine.deregister.exception.messages` and `kyuubi.engine.deregister.exception.stacktraces` to `kyuubi.engine.deregister.exception.messages`. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #565 from turboFei/KYUUBI-544-combine. Closes #544 ee63f60 [fwang12] [KYUUBI #544] Combine deregister exception messages and stacktraces Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
29e169b68b
|
[KYUUBI #544] Add UT for engine deregister exception and refactor package name for SparkSQLEngineListenerSuite
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add UT for engine deregister exception ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #558 from turboFei/KYUUBI-544-exception-test. Closes #544 7812d6e [fwang12] remove blank line 3b5252a [fwang12] find cause 6e09907 [fwang12] simplify ut dbafab2 [fwang12] log ve b062503 [fwang12] improvement 0793980 [fwang12] fix ut ac35be0 [fwang12] [KYUUBI #544][TEST] Add ut for engine deregister exception Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
5cc4fd83c1
|
[KYUUBI #561] Forbid set env:* in thrift connection configurations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This change only forbid the `set:env:*` from thrift connection configurations, to match hive behavior, forbid `set env:*` in SQL, we need to change it in Spark. #508 https://issues.apache.org/jira/browse/HIVE-6175 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #561 from pan3793/env. Closes #561 79b7e69 [Cheng Pan] update 00dfaf0 [Cheng Pan] Forbid set env:* in thrift connection configurations Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
7abcc9c5d1 |
[KYUUBI #544] Deregister engine when meeting specified exception stacktraces
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.stacktraces` to deregister engine when meeting specified exception stacktraces. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #557 from turboFei/KYUUBI-544-exception-msg-stack-rebase. Closes #544 beedb9c6 [fwang12] stopping -> deregistering 80373fca [fwang12] [KYUUBI #544] Deregister engine when meeting specified stacktrace Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
fwang12
|
7d72b31e81 |
[KYUUBI #544] Deregister engine when meeting specifies exception messages
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.messages` to deregister engine when meeting specified exception messages. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #556 from turboFei/KYUUBI-544-exception-msg-rebase. Closes #544 c1fed723 [fwang12] address comments 7cfba913 [fwang12] [KYUUBI #544] Deregister engine when meeting specified exception message Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
fwang12
|
3d3d004e2d |
[KYUUBI #544] Deregister engine when meeting specified exception class
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.classes` to deregister engine when job failed because of specified exception class ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #549 from turboFei/KYUUBI-544-exception-class. Closes #544 9ff83b80 [fwang12] use lazy val to avoid npe d80d4436 [fwang12] [KYUUBI #544] Add kyuubi.engine.deregister.exception.classes to deregister engine when job failed because of specified exception class Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
ulysses-you
|
cacc2b05b3 |
[KYUUBI #543] [TESTS] Add test for scheduler pool
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve test coverage. ### _How was this patch tested?_ Pass `org.apache.kyuubi.engine.spark.SchedulerPoolSuite`. Closes #543 from ulysses-you/add-scheduler-pool-test. Closes #543 f5563b1 [ulysses-you] trait 57938a1 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
c458d853c4 |
[KYUUBI #526] Support kyuubi.operation.scheduler.pool for queries
 [](https://github.com/yaooqinn/kyuubi/pull/528)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support scheduler pool which is native supported by Spark. ### _How was this patch tested?_ Pass CI + Manual tets The default pool name is `default`, aflter execute `set kyuubi.operation.scheduler.pool = default1;`, the pool changed.  Closes #528 from ulysses-you/support-spark-scheduler-mode. 1ba8e89 [ulysses-you] fux 746cb62 [ulysses-you] constraint 1b369a9 [ulysses-you] conf ec49c72 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
d47879e0bb
|
[KYUUBI #476][FOLLOWUP] Strip prefixes when ignoring and restricting configs
 [](https://github.com/yaooqinn/kyuubi/pull/510)    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. change the configs to seq[string] for simplification 2. bugfix for mismatched configs when ignoring and restricting because they are maybe prefixed with something like `set:hivevar:thekey=thevalue`, we should normalize it to `thekey=thevalue` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #510 from yaooqinn/476. Closes #476 84355de [Kent Yao] more test cc8c20e [Kent Yao] doc ee0b418 [Kent Yao] nit 7def6eb [Kent Yao] [KYUUBI #476][FOLLOWUP] Strip prefixes when ignoring and restricting configs Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
32b9ad0d6e |
[KYUUBI #304] Kyuubi service control client action implementation
 [](https://github.com/yaooqinn/kyuubi/pull/478)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Implement the create/list/get/delete action for Kyuubi-ctl ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #478 from turboFei/KYUUBI_CTL_304_CREATE. Closes #304 d81be9e8 [fwang12] refactor column name f4e18d1f [fwang12] address comments 94941079 [fwang12] use function t o generate expected output b07e55b6 [fwang12] align class name with ServiceControlCli 641f7021 [fwang12] refactor format 8098f96a [fwang12] refactor for delete action 489dad32 [fwang12] address comments 16f60218 [fwang12] add list/get/delete action and refer the zk nodes info 85c147ee [fwang12] [KYUUBI #304] Kyuubi service control client action- expose existing kyuubi server instances to another domain Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
ulysses-you
|
b0aeb26230 |
[KYUUBI #500] Speed up spark sql engine diagnostics
 [](https://github.com/yaooqinn/kyuubi/pull/500)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Mark it as `lazy val` to avoid rebuld the string text every time. ### _How was this patch tested?_ Pass exists CI Closes #500 from ulysses-you/speed-up-diagnostics. Closes #500 d4a61f1 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
9a8b2860f6 |
[KYUUBI #506] Shade curator dependencies to avoid impaction by external jars
 [](https://github.com/yaooqinn/kyuubi/pull/506)       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Shade curator dependencies to avoid impaction by external jars. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #506 from turboFei/shade_curator_new. Closes #506 4c6fdeaa [fwang12] Shade curator dependencies to avoid impaction by external jars Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
b7ac3cb48c |
Revert "[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars"
This reverts commit
|
||
|
fwang12
|
e058b87dd2
|
[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars
 [](https://github.com/yaooqinn/kyuubi/pull/488)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Shade curator dependency in kyuubi-ha module to avoid impaction by external jars. ### _How was this patch tested?_ Existing UT. - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #488 from turboFei/shade_curator. Closes #488 c88414c [fwang12] Use wildcards when selecting more than 6 elements a2dfbf6 [fwang12] add doc for install shaded jars 224cc38 [fwang12] remove transactions c7b9df3 [fwang12] address comments 81fc123 [fwang12] scala style 408c978 [fwang12] create kyuubi-shaded module to shade dependencies 6b15b45 [fwang12] shade hive service rpc b4b3b1a [fwang12] include curator-client 65086ed [fwang12] address comments a46f00f [fwang12] Shade curator dependencies to avoid impaction by external jars Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
1b7762e5f9
|
[KYUUBI #457] Support configurable initialize sql statement for engine startup
 [](https://github.com/yaooqinn/kyuubi/pull/462)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Currently, the engine uses hardcoded show databases to force initialization with catalog eagerly bootstrapping too. We can make it configurable with a comma separate query list and select current_database() might be used as default since it's more lightweight than show databases in real-world cases ### _How was this patch tested?_ Existing UT. Closes #462 from turboFei/KYUUBI-457. Closes #457 e7bcf52 [fwang12] add ut 7446cd5 [fwang12] [KYUUBI#457】Support configurable initialize sql statement for engine startup Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
c6585b4708
|
[KYUUBI #471] Separate Embedded Zookeeper server from ha
 [](https://github.com/yaooqinn/kyuubi/pull/471)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one 2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage. 3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #471 from yaooqinn/qp. Closes #471 1c24d02 [Kent Yao] address comments 9666a40 [Kent Yao] refine f069fc0 [Kent Yao] refine 8bb2f3a [Kent Yao] refine c20593a [Kent Yao] refine 8fa9a40 [Kent Yao] Merge branch 'master' into qp 1e2e1a3 [Kent Yao] tmp 7d66d05 [Kent Yao] tmp Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
