d20a92e5ae
914 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cheng Pan
|
d20a92e5ae
|
[KYUUBI #637] [METRICS] Remove unnecessary Guava Cache
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `MetricRegistry` hold counters in `ConcurrentHashMap`, it's unnecessary to use Guava Cache. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #637 from pan3793/metrics. Closes #637 aee29f1 [Cheng Pan] remove synchronized a8d607b [Cheng Pan] [METRICS] Remove unnecessary Guava Cache Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
ccb891d2ae |
[KYUUBI #638] Refine kyuubi extension config docs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Refine docs. ### _How was this patch tested?_ Not needed. Closes #638 from ulysses-you/refine-config-docs. Closes #638 e9f3f37c [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
42d57e9bd8 |
[KYUUBI #636] Add kyuubi-extension-spark_3.1 in release script
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make `kyuubi-extension-spark_3.1` module packaged during release. ### _How was this patch tested?_ Closes #636 from ulysses-you/release-script. Closes #636 363f74b5 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
e77ea13c6d |
[KYUUBI #635] Fix stage level config miss ReusedExchange
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Avoid the reused exchange cause the wrong stage config. ### _How was this patch tested?_ Add test. Closes #635 from ulysses-you/fix-stage-level-config-miss-reusedexchange. Closes #635 7d514a90 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
7f040fb2c2
|
[KYUUBI #634] Move kyuubi extension to its own dir when building
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make Kyuubi jars clear. ### _How was this patch tested?_ Manual test. After this PR, the packaged dir looks like:  Closes #634 from ulysses-you/extension-dir. Closes #634 2c5009a [ulysses-you] comment 35063ea [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into extension-dir f2b4fff [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b60d18aeb1
|
[KYUUBI #633] [FOLLOWUP #631] [BUILD] Fix kyuubi-extension-spark_3.1 module version
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #633 from pan3793/minor. Closes #633 108aca0 [Cheng Pan] [BUILD] Fix kyuubi-extension-spark_3.1 module version Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
43f40dcb8a
|
[KYUUBI #631] Add kyuubi-spark-extensions module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The added sql module structure looks like: ``` kyuubi | - dev | - kyuubi-extension-spark_3.1 ``` This PR mainly add 3 feature: * merging small files automatically (include dynamic partition insertion case) * insert shuffle node before Join to make AQE `OptimizeSkewedJoin` work * stage level config isolation in AQE Note that, the sql rule depend on the Apache Spark interface so we need make the sql module verion independence. Currently, this PR only supports the Spark 3.1.1. Due to the version issue, we need to check and deploy this extension manually currently. ### _How was this patch tested?_ Add new test. Closes #631 from ulysses-you/add-sql-module. Closes #631 2cf12f1 [ulysses-you] version cfbf72c [ulysses-you] address comment 7740ca6 [ulysses-you] module name 0f723eb [ulysses-you] workflow 45c23d8 [ulysses-you] line 80378f5 [ulysses-you] assembly 95528aa [ulysses-you] move module 5fe5d87 [ulysses-you] license 6578440 [ulysses-you] init work Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
bb9e8a50e5
|
Prepare Kyuubi v1.3.0 | ||
|
张宇翔
|
f1cd730b0a
|
[KYUUBI #632] solve the timeout problem when stop the kyuubiServer
### _Why are the changes needed?_ When you stop the KyuubiServiceDiscovery, you need to close node in zk and delete localNodePath at first. But when EmbeddedZookeeper is closed before KyuubiServiceDiscovery, the operation that stop kyuubiServer will timeout. Now use 'org.apache.hadoop.util.ShutdownHookManager' to close each service by different priority. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #632 from zhang1002/dev. Closes #632 ee21e68 [张宇翔] Change the relevant format f68fc7a [张宇翔] Merge branch 'master' into dev 128f244 [张宇翔] solve the timeout problem when stop the kyuubiServer 79b3d5a [张宇翔] ## solve the folder naming problem For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}. Lead-authored-by: 张宇翔 <zhang1002@126.com> Co-authored-by: 张宇翔 <zhangyuxiang03@corp.netease.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
0b4f4f242c
|
[KYUUBI #630] [METRICS] Support expose Prometheus metrics
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support expose Prometheus metrics ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #630 from pan3793/prometheus. Closes #630 fff68b0 [Cheng Pan] nit 47c19eb [Cheng Pan] simplify 203e776 [Cheng Pan] address comments 10a877c [Cheng Pan] synchronized 26eb26e [Cheng Pan] simplify random port 827427c [Cheng Pan] fix import 476ee3a [Cheng Pan] address comments 9787c77 [Cheng Pan] update ut c8e1b0e [Cheng Pan] update license 2387a97 [Cheng Pan] [METRICS] Support expose Prometheus metrics Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d2be0f34e8 |
[KYUUBI #629] Refine distributed lock to lock only engine bootstrap
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. refine the zk lock to lock only during engine bootstrapping, not wrap the part that only gets the address of engines. This can significantly reduce overhead. 2. add tests for the locking proccess. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #629 from yaooqinn/lock. Closes #629 a1b4e5f1 [Kent Yao] Refine distributed lock to lock only engine bootstrap a6738652 [Kent Yao] Refine distributed lock to lock only engine bootstrap ff238e6f [Kent Yao] Refine distributed lock to lock only engine bootstrap f3979fce [Kent Yao] Refine distributed lock to lock only engine bootstrap e8ca9954 [Kent Yao] Refine distributed lock to lock only engine bootstrap Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
张宇翔
|
0918492498
|
[KYUUBI #628] solve the folder naming problem
For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes #628 from zhang1002/dev.
Closes #628
79b3d5a [张宇翔] ## solve the folder naming problem For starting kyuubiServer, the script need to get 'SPARK_HOME' from the file: RELEASE. The rule is: spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}. But when we package project from source, the script named dist will create the directory about 'SPARK_HOME' by the rule: spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}. So it's the differences between two rules. This change let ${HADOOP_VERSION} as same as ${SPARK_HADOOP_VERSION}.
Lead-authored-by: 张宇翔 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhangyuxiang03@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
|
||
|
Kent Yao
|
b4a505971c |
[KYUUBI #627] Support multiple engines for non-connection share levels
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid. Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached. With multiple engines support, end-users are able to 1. create as many engines as they want to meet the concurrency. 2. leverage different queue/namespace resource with long-running mode 3. can give different resources or priorities to their own engines for different workloads 4. can perform fast failover by creating a new sub domain 5. etc. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #627 from yaooqinn/multipleengines. Closes #627 9f04bc3e [Kent Yao] fix tests d7af751e [Kent Yao] address comments be9c46d8 [Kent Yao] address comments ccce87d7 [Kent Yao] address comments d49116c7 [Kent Yao] Support multiple engines for non-connection share levels f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Honglun
|
d05c81b9a6
|
[KYUUBI #625]Kyuubi server-side support 'operation.idle. timeout'
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Now we only get 'operation.idle.timeout' from user request. Kyuubi server-side should support "operation.idle. timeout". ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #626 from Honglun/feature/operation-timeout. Closes #625 e56c11b [Honglun] Kyuubi server-side support 'operation.idle. timeout' Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
0d8bbfad9e |
[KYUUBI #619] [TEST] Refactor data lake test suite hierarchical
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make `*SuiteMixin` resuable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #619 from pan3793/mixin. Closes #619 fa776869 [Cheng Pan] address comments 54ac9a19 [Cheng Pan] Revert "gather KyuubiConf" 92985964 [Cheng Pan] gather KyuubiConf 61492616 [Cheng Pan] extract DataLakeSuiteMixin 66b6ffa4 [Cheng Pan] [TEST] Refactor data lake test suite hierarchical Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Honglun
|
41f0274ba4 |
[KYUUBI #624] Add lock while creating engine with non-CONNECTION mode
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We may create multiple SparkContext in non-CONNECTION ShareLevel. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #624 from Honglun/feature/lock. Closes #624 87445522 [Honglun] Add lock while creating engine with non-CONNECTION mode Authored-by: Honglun <283471011@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Kent Yao
|
a28e62ad70
|
[KYUUBI #623] Improve OperationLog to deliver detail stats of sql statements
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `OperationLog` is used to divert SQL statement-specific logs to a temporary local file at the engine side and to be fetched to the client-side(e.g. beeline). It is an InheritableThreadLocal variable in which thread the `SparkSession` instance lays. Inside Spark, there are a lot of threads created via the java `Executors`, for which the `OperationLog` cannot be inherited, so the client-side lacks information on how their jobs are running. This is not user-friendly especially for queries that contain a lot of jobs. In this PR, we add a new SparkListener for SQL operation which diverts the SQL statement-specific logs by `spark.jobGroup.id`. The listener will dump Job/Stage and Task summaries to the `OperationLog` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request ```logtalk 2021-05-08 10:25:54.301 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 started with 4 stages, 1 active jobs running 2021-05-08 10:25:54.303 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 6 started with 5 tasks, 1 active stages running 2021-05-08 10:25:54.347 INFO kyuubi.SQLOperationListener: Finished stage: Stage(6, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 5; Took: 44 msec 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: task runtime:(count: 5, mean: 36.400000, stdev: 1.019804, max: 38.000000, min: 35.000000) 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 35.0 ms 35.0 ms 35.0 ms 36.0 ms 36.0 ms 37.0 ms 38.0 ms 38.0 ms 38.0 ms 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 5, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: fetch wait time:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: remote bytes read:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: task result size:(count: 5, mean: 1965.000000, stdev: 0.000000, max: 1965.000000, min: 1965.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 5, mean: 82.948068, stdev: 1.278517, max: 84.210526, min: 80.555556) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 81 % 81 % 81 % 83 % 83 % 84 % 84 % 84 % 84 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: other time pct: (count: 5, mean: 17.051932, stdev: 1.278517, max: 19.444444, min: 15.789474) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 16 % 16 % 16 % 16 % 17 % 17 % 19 % 19 % 19 % 2021-05-08 10:25:54.357 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 7 started with 300 tasks, 1 active stages running 2021-05-08 10:25:54.644 INFO kyuubi.SQLOperationListener: Finished stage: Stage(7, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 300; Took: 287 msec 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: task runtime:(count: 300, mean: 15.203333, stdev: 9.396559, max: 45.000000, min: 4.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 4.0 ms 5.0 ms 6.0 ms 8.0 ms 13.0 ms 18.0 ms 31.0 ms 38.0 ms 45.0 ms 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 300, mean: 56.030000, stdev: 0.298496, max: 59.000000, min: 56.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 59.0 B 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: fetch wait time:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: remote bytes read:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: task result size:(count: 300, mean: 2865.150000, stdev: 9.371633, max: 2906.000000, min: 2863.000000) 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 300, mean: 66.577940, stdev: 18.397778, max: 95.121951, min: 10.526316) 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 11 % 33 % 40 % 56 % 71 % 81 % 87 % 88 % 95 % 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: other time pct: (count: 300, mean: 33.422060, stdev: 18.397778, max: 89.473684, min: 4.878049) 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 5 % 12 % 13 % 19 % 29 % 44 % 60 % 67 % 89 % 2021-05-08 10:25:54.650 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 8 started with 1 tasks, 1 active stages running 2021-05-08 10:25:54.733 INFO kyuubi.SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 1; Took: 87 msec 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: task runtime:(count: 1, mean: 81.000000, stdev: 0.000000, max: 81.000000, min: 81.000000) 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 1, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: remote bytes read:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: task result size:(count: 1, mean: 3065.000000, stdev: 0.000000, max: 3065.000000, min: 3065.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 95.061728, stdev: 0.000000, max: 95.061728, min: 95.061728) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: other time pct: (count: 1, mean: 4.938272, stdev: 0.000000, max: 4.938272, min: 4.938272) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 2021-05-08 10:25:54.739 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 9 started with 200 tasks, 1 active stages running 2021-05-08 10:25:54.844 INFO kyuubi.SQLOperationListener: Finished stage: Stage(9, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 200; Took: 105 msec 2021-05-08 10:25:54.845 INFO scheduler.DAGScheduler: Job 3 finished: collect at ExecuteStatement.scala:82, took 0.547547 s 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: task runtime:(count: 200, mean: 15.100000, stdev: 11.464729, max: 48.000000, min: 4.000000) 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 4.0 ms 6.0 ms 7.0 ms 8.0 ms 10.0 ms 16.0 ms 40.0 ms 43.0 ms 48.0 ms 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: fetch wait time:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: remote bytes read:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: task result size:(count: 200, mean: 2326.135000, stdev: 26.673897, max: 2414.000000, min: 2311.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.4 KiB 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 200, mean: 2.971561, stdev: 8.036931, max: 47.727273, min: 0.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 10 % 18 % 48 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: other time pct: (count: 200, mean: 97.028439, stdev: 8.036931, max: 100.000000, min: 52.272727) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.849 INFO scheduler.StatsReportListener: 52 % 82 % 90 % 100 % 100 % 100 % 100 % 100 % 100 % 2021-05-08 10:25:54.849 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 succeeded, 0 active jobs running 2021-05-08 10:25:54.857 INFO codegen.CodeGenerator: Code generated in 7.116325 ms 2021-05-08 10:25:54.858 INFO operation.ExecuteStatement: Processing kentyao's query[3ac86686-f502-4f2d-9a98-d763e5b9548c]: RUNNING_STATE -> FINISHED_STATE, statement: select /*+ REPARTITION(200) */ count(1) from (select /*+ REPARTITION(300, a) */ a from values(1),(1),(1),(2),(3) t(a)), time taken: 0.702 seconds ``` Closes #623 from yaooqinn/log2. Closes #623 abeff64 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement fdf3b28 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement 783a594 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
d3e2981f0f |
[KYUUBI #622] [NOTICE] Add NOTICE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add `NOTICE` file in this PRs. Have a look on other projects, I only see Apache Spark maintains `NOTICE-binary` in repo, most projects generate it on build process. For instance, Apache Flink use shell [`collect_license_files`](https://github.com/apache/flink/blob/release-1.13.0/tools/releasing/collect_license_files.sh) to collect LICENSE and NOTICE files. Apache Calcite use gradle plugin [vlsi-release-plugins](https://github.com/vlsi/vlsi-release-plugins) to manage the whole release process, see [build.gradle.kts](https://github.com/apache/calcite/blob/calcite-1.26.0/build.gradle.kts#L45). Apache Kafka write gradle scripts to gather LICENSE and NOTICE files, see [build.gradle](https://github.com/apache/kafka/blob/2.8.0/build.gradle). ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #622 from pan3793/notice. Closes #622 777ca699 [Cheng Pan] [NOTICE] Add NOTICE Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
2aea8d4867 |
[KYUUBI #620] [TEST] Close Source to avoid file leak
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #620 from pan3793/leak. Closes #620 7e63f623 [Cheng Pan] fix 8293b92c [Cheng Pan] eager consume 54bf713a [Cheng Pan] [TEST] Close Source to avoid file leak Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
iodone
|
998bf08197
|
[KYUUBI #615]setting the fetchType of TFetchResultsReq to 1 in getOperationLogRowSet
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #617 from iodone/master. Closes #615 fbbe515 [iodone] setting the fetchType of TFetchResultsReq to 1 in getOperationLogRowSet Authored-by: iodone <work@YaodongdeMacBook-Pro.local> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f742686357
|
[KYUUBI #614] [BUILD][DOCS] Bump sphinx
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Bump sphinx to latest version. close #612 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #614 from pan3793/doc. Closes #614 b1b2738 [Cheng Pan] [BUILD][DOCS] Bump sphinx Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
34374f3b3e
|
[KYUUBI #613] [TEST] [DELTA] Add test for deltalake
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #613 from pan3793/delta. Closes #613 ccadedb [Cheng Pan] [TEST] [DELTA] Add test for deltalake Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
20911ff73c |
[KYUUBI #611] [TPCDS][TEST] Introduce TPCDS 2.7 output schema check
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce TPCDS 2.7 output schema tests. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #611 from turboFei/tpcds_2_7_follow. Closes #611 6ecbbee [fwang12] Introduce tpcds 2.7 output schema tests Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
0e2263f759
|
[KYUUBI #606] Introduce ExtendedSQLTest tag to split build jobs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve ExtendedSQLTest tag to split build jobs ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #606 from turboFei/slow_tags. Closes #606 5f8464d [fwang12] do not use build/dist 9e813dc [fwang12] exclude extended sql test for default profile 4c5ea67 [fwang12] Involve ExtendedSQLTest to split build jobs Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
2af8b53dbf |
[KYUUBI #552] Sync build jobs between GA and travis
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Sync build jobs between GA and travis ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #604 from turboFei/sync_ga_travi. Closes #552 98f2e45a [fwang12] [KYUUBI #552]Sync build jobs between GA and travis Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Kent Yao
|
51a190ffb8
|
[KYUUBI #605] Improve the landscape for README
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #605 from yaooqinn/readme. Closes #605 63d8702 [Kent Yao] Update Readme c48d8e8 [Kent Yao] Update Readme 9a2e2d6 [Kent Yao] Update Readme Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
09c5f9123e |
[KYUUBI #601] [METRICS] Change metrics config keys
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Change the config keys of metrics to get a consistent name style, `metrics.{reporter_type}.{property}`. The change breaks the compatibility with master, but consider v1.2.0 is not out yet, so changes should be safe. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #601 from pan3793/metrics. Closes #601 89d1974f [Cheng Pan] address comments 361ae6e2 [Cheng Pan] revert default metrics reporter change 1c493e64 [Cheng Pan] rat exclude report.json fcdb4252 [Cheng Pan] exclude report.json 337e09ae [Cheng Pan] Default enable SFL4J metrics reporter 6b9f3963 [Cheng Pan] [METRICS] Change metrics config keys Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
fwang12
|
435e85f5d8 |
[KYUUBI #603] [TEST] Refactor test name and sort the tpcds queries
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Refactor the test name to show tpcds version information and sort the test queries by query number. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #603 from turboFei/tpcds_refactor. Closes #603 1c8daad5 [fwang12] add accident deleted q3 2f62b650 [fwang12] [TEST] Refactor test name and sort the tpcds queries Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
fwang12
|
540de12764 |
[KYUUBI #598] [TEST][TPCDS] Introduce schema check for queries-9
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(ninety plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #598 from turboFei/tpcds_9. Closes #598 fef260d [fwang12] [TEST][TPCDS] Introduce schema check for queries-9 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
e1fa5c62ed |
[KYUUBI #599] [TEST][TPCDS] Introduce schema check for queries-8
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(eighty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #599 from turboFei/tpcds_8. Closes #599 6cc4642 [fwang12] [TEST][TPCDS] Introduce schema check for queries-8 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
f318c0b8f8 |
[KYUUBI #597] [TEST][TPCDS] Introduce schema check for queries-7
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(seventy plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #597 from turboFei/tpcds_7. Closes #597 7d008ae [fwang12] [TEST][TPCDS] Introduce schema check for queries-7 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
c197be060b |
[KYUUBI #596] [TEST][TPCDS] Introduce schema check for queries-6
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(sixty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #596 from turboFei/tpcds_6. Closes #596 f6d45de [fwang12] [TEST][TPCDS] Introduce schema check for queries-6 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
9a80db57f5 |
[KYUUBI #595] [TEST][TPCDS] Introduce schema check for queries-5
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(fifty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #595 from turboFei/tpcds_5. Closes #595 7be2419 [fwang12] [TEST][TPCDS] Introduce schema check for queries-5 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
16e2673c72 |
[KYUUBI #594] [TEST][TPCDS] Introduce schema check for queries-4
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(forty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #594 from turboFei/tpcds_40+. Closes #594 c3590db [fwang12] [TEST][TPCDS] Introduce schema check for queries-4 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
fwang12
|
1618c382e8 |
[KYUUBI #593] [TEST][TPCDS] Introduce schema check for queries-3
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(thirty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #593 from turboFei/tpcds_30+. Closes #593 a796af1 [fwang12] [TEST][TPCDS] Introduce schema check for queries(thirty plus) Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
913c444696 |
[KYUUBI #600] [DOCS] Update building doc and tunes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #600 from pan3793/doc. Closes #600 e7ad0a3b [Cheng Pan] minor tune f11ae160 [Cheng Pan] [DOCS] Update building doc Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
325d59919b
|
[KYUUBI #592] [TEST][TPCDS] Introduce schema check for queries-2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for queries(twenty plus) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #592 from turboFei/tpcds_20+. Closes #592 5d0e417 [fwang12] [TEST][TPCDS] Introduce schema check for queries(twenty plus) Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
18208641dd |
[KYUUBI #590] [TPCDS][TEST] Fix TPCDS suite issue and introduce tpcds queries(ten plus)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix tpcds schema issue and covert all tests, whose file is named like q14a.sql. Introduce tpcds queries, whose index between ten and twenty. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #590 from turboFei/tpcds_10_20. Closes #590 d58c4e97 [fwang12] remove query 20 dbcd3379 [fwang12] [TPCDS][TEST] Fix tpcds suite issue and introduce q11-q20 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
fwang12
|
746736f4f3
|
[KYUUBI #581] [TEST][TPCDS] Introduce schema check for q11
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q11 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #581 from turboFei/tpcds_11. Closes #581 f6c38c9 [fwang12] [TEST][TPCDS] Introduce schema check for q11 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
879de5cbfe |
[KYUUBI #587] [TPCDS][TEST] Fix table schema issue and introduce q5
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix table schema issue and introduce q5 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #587 from turboFei/fix_tpcds. Closes #587 1f3f78a2 [fwang12] [TPCDS][TEST] Fix table schema issue and introduce q5 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
e1899bae61 |
[KYUUBI #580] [TEST][TPCDS] Introduce schema check for q9 and q10
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q9 and q10 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #580 from turboFei/tpcds_9_10. Closes #580 ae688ac [fwang12] [TEST][TPCDS] Introduce schema check for q9 and q10 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
8f3871e31f |
[KYUUBI #579] [TEST][TPCDS] Introduce schema check for q8
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q8 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #579 from turboFei/tpcds_8. Closes #579 e1b1473 [fwang12] [TEST][TPCDS] Introduce schema check for q8 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
0b0ffe63de |
[KYUUBI #578] [TEST][TPCDS] Introduce schema check for q7
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q7 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #578 from turboFei/tpcds_7. Closes #578 b4010d0 [fwang12] [TEST][TPCDS] Introduce schema check for q7 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
d8c1eaa56d
|
[KYUUBI #577] [TEST][TPCDS] Introduce schema check for q4
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q4 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #577 from turboFei/tpcds-4. Closes #577 bb8b1cf [fwang12] [TEST][TPCDS] Introduce schema check for q4 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
9e89df06da
|
[KYUUBI #576] [TEST][TPCDS] Introduce schema check for q3
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce schema check for q3 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #576 from turboFei/tpcds-3-5. Closes #576 a7a485c [fwang12] [TEST][TPCDS] Introduce schema check for q3 Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f311211aa3
|
[KYUUBI #574] [IDEA] Make IDEA support IssueNavigationLink
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #574 from pan3793/ide. Closes #574 eb48095 [Cheng Pan] typo 155c89b [Cheng Pan] [IDEA] Make IDEA support IssueNavigationLink Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b9c90f674f
|
[KYUUBI #575] [TEST] [DEPS] Pin scala-reflect to avoid potential conflicts
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> There is transitive dependency `scala-reflect` 2.12.13 from `scalatest` in test scope, we'd better to pin version align with `scala-library` to avoid potential conflicts. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #575 from pan3793/scala-reflect. Closes #575 b6019f7 [Cheng Pan] [TEST] [DEPS] Pin scala-reflect to avoid potential conflicts Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
68d8f046ac
|
[KYUUBI #573] [TEST] [TPCDS] Introduce schema check for q6
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #573 from pan3793/tpcds-6. Closes #573 e476dea [Cheng Pan] Add missing license header 98eb8c9 [Cheng Pan] [TEST] [TPCDS] Introduce schema check for q6 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
42fa7cfe49 |
[KYUUBI #544] Involve job max failures and deregister exception TTL
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve job max failures to limit the max job failures before deregistering the engine, involve deregister exception TTL for to restart counting when last failure has gone for one deregisterExceptionTTL. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #560 from turboFei/KYUUBI-544-exception-max-failures-new. Closes #544 951f269 [fwang12] [KYUUBI #544] Involve job max failures and deregister exception TTL Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
bd89d740e6 |
[KYUUBI #570] Complete doc for quick start with jdbc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Complete doc for quick start with jdbc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #571 from turboFei/quick_start_with_jdbc. Closes #570 342885d [fwang12] address comments 8ccf102 [fwang12] [KYUUBI #570] Complete doc for quick start with jdbc Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
