c59e54a675
1099 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ulysses-you
|
c59e54a675
|
[KYUUBI #992] Fix SPARK_EXTENSION_VERSION_ is seen as a variable
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix dist bug `$SPARK_EXTENSION_VERSION_` is empty `${SPARK_EXTENSION_VERSION}_` is `3-1_` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #992 from ulysses-you/fix-SPARK_EXTENSION_VERSION_. Closes #992 4379093e [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
9695f75889 |
[KYUUBI #991] [MINOR] improve the error message of SPARK package not found.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As `File.listFiles` may return null when spark package not found in folder `${PROJECT_ROOT_DIR}/kyuubi-server/externals/kyuubi-download/target`, the NPE will throw. Improve the error message. Before thie pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [9749221f-5b30-457b-b8e5-25affba25061] for fchen due to null at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:204) at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:204) at scala.collection.IndexedSeqOptimized.isEmpty(IndexedSeqOptimized.scala:30) at scala.collection.IndexedSeqOptimized.isEmpty$(IndexedSeqOptimized.scala:30) at scala.collection.mutable.ArrayOps$ofRef.isEmpty(ArrayOps.scala:198) at scala.collection.TraversableLike.headOption(TraversableLike.scala:608) at scala.collection.TraversableLike.headOption$(TraversableLike.scala:608) at scala.collection.mutable.ArrayOps$ofRef.headOption(ArrayOps.scala:198) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$1(SparkProcessBuilder.scala:50) at scala.Option.orElse(Option.scala:447) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:41) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` After this pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [5ea6a8b3-1727-4e0d-b542-992f84afcde8] for fchen due to SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$5(SparkProcessBuilder.scala:61) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:61) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #991 from cfmcgrady/improve-error-msg. Closes #991 1e8da683 [Fu Chen] imporove the error message Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
3afd20e03a
|
[KYUUBI #985] Add logCaptureThreadReleased flag
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `isInterrupted` is always false if some `InterruptedException` are throwed. So we should use a new flag to check if log capture thread is released. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #985 from ulysses-you/flaky-test. Closes #985 1fcc0a99 [ulysses-you] empty 463a996f [ulysses-you] empty d1f2ea54 [ulysses-you] eventually f9f37343 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
d704e4d49d |
[KYUUBI #922] Add OperationEvent for server to track operation state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Track operation state change and support output to custom system. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #968 from hddong/add-server-events. Closes #922 8be8f268 [hongdongdong] Add prefix for StatementEvent 9fdcc504 [hongdongdong] fix comment a0cdb35d [hongdongdong] fix test 4d340c67 [hongdongdong] add dependency 00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
d58a092245
|
[KYUUBI #976] [BUILD] Fix source release build
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support build, run unit tests, and create binary tarball from source release. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request The verify steps: 1. create a source release tarball ``` RELEASE_VERSION=1.4.0-SNAPSHOT build/release/create-package.sh source ``` 2. untar tarball ``` cd work/release tar -xzf kyuubi-1.4.0-SNAPSHOT-bin.tgz cd kyuubi-1.4.0-SNAPSHOT-bin ``` 3. compile and run unit test ``` build/mvn clean install ``` 4. create binary release tarball ``` build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1 ``` Closes #976 from pan3793/source-build. Closes #976 354968ae [Cheng Pan] Compress imgs 9c47b64f [Cheng Pan] Restore docs/imgs in source release 5d86ecc7 [Cheng Pan] Clean up imgs a3399279 [Cheng Pan] Support create binary from source release bb279863 [Cheng Pan] [BUILD] Fix source release ut failures Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3ebb50f1cb
|
[KYUUBI #978] [LICENSE] Fix the source license issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the license issue mentioned by WillemJiang ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request The final `NOTICE` for binary release is not `NOTICE-binary` in root path but is generated by `build/release/collect_licenses.sh`, which will extract all NOTICE content from the bundled jars. ``` Apache Kyuubi Copyright 2021 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (https://www.apache.org/). Apache Spark Copyright 2014 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Export Control Notice --------------------- This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <http://www.wassenaar.org/> for more information. The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: This software uses Apache Commons Crypto (https://commons.apache.org/proper/commons-crypto/) to support authentication, and encryption and decryption of data sent across the network between services. Metrics Copyright 2010-2013 Coda Hale and Yammer, Inc. This product includes software developed by Coda Hale and Yammer, Inc. This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, LongAdder), which was released with the following comments: Written by Doug Lea with assistance from members of JCP JSR-166 Expert Group and released to the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/ Curator Client Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Framework Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Recipes Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Hive Service RPC Copyright 2020 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). htrace-core4 Copyright 2016 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may be licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. Apache log4j Copyright 2007 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). ``` Closes #978 from pan3793/license. Closes #978 2a7eff44 [Cheng Pan] Update License d21825dd [Cheng Pan] [LICENSE] Fix the source license issue Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
cb70e062c3
|
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #980 from yanghua/KPIP-1. Closes #979 e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
odone
|
f1ae664910
|
[KYUUBI #961] Fixed: opHanle's operation may not be release
Align with `sessionManager.operationManager.closeOperation(operationHandle)` and remove first <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #975 from iodone/fix-opHanle-operation-not-release. Closes #961 bd798e0d [odone] [KYUUBI #961] fixed: opHanle's operation may not be release Authored-by: odone <odone.zhang@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
33be4516b6
|
[KYUUBI #973] [LICENSE] Fix license issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Remove the `scala.util.Using` - Remove redundant license declaration ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #973 from pan3793/license. Closes #973 b02206ea [Cheng Pan] Remove redundant license declaration 3a1401d9 [Cheng Pan] [LICENSE] Remove Scala Using Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
271ccc948f
|
[KYUUBI #972] [LICENSE] Exclude binary/doc files from source release
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #972 from pan3793/src-license. Closes #972 443cd05c [Cheng Pan] Exclude unused files from source release 52271827 [Cheng Pan] Exclude unused files from source release d475b511 [Cheng Pan] Exclude unused files from source release e76299bd [Cheng Pan] Exclude docs from source release 2e42bc8c [Cheng Pan] Exclude whole tpcds module from source release 93b41a8c [Cheng Pan] [LICENSE] Exclude binary files from source release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
2f82f33e9d
|
[KYUUBI #971] [LICENSE] Add DISCLAIMER to binary release
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #971 from pan3793/disclaimer. Closes #971 c0fe3821 [Cheng Pan] [LICENSE] Add DISCLAIMER to binary release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
ae4878349c
|
[KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Workaround to recover Minikube test until #937 is solved. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #969 from pan3793/747-ignore. Closes #659 e26153f6 [Cheng Pan] [KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
8bab5b8d6a
|
[KYUUBI #659] Add UDF engine_name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add UDF engine_name to show the runtime information about execution engine ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #747 from zhaomin1423/udf. Closes #659 8c3b3c16 [Min Zhao] Merge branch 'master' into udf f184a661 [Min Zhao] Merge branch 'master' into udf 9ea0c190 [Min Zhao] get appName from driver side 09f07523 [Min Zhao] get conf by SparkEnv.get.conf in the engine_name udf 6c000784 [Min Zhao] update the description of the engine_name udf 63ef2762 [Min Zhao] update class to org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite in functions.md db02884c [Min Zhao] [KYUUBI #659] Add UDF engine_name Lead-authored-by: Min Zhao <zhaomin1423@163.com> Co-authored-by: Min Zhao <49054376+zhaomin1423@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Baoqi Wu
|
484bb3c6bd
|
[KYUUBI #956] Fail to get table list if there is _ in schema's name
Just write some basic tests out in other projects. Since this toJavaRegex is not a static function, which is hard to write unit test if don't move it out as as static function in object. (But in that case, the change may be too large for this). ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #964 from Baoqi/bwu_fix_956. Closes #956 985a73b5 [Baoqi Wu] [KYUUBI #956] add a unit test for this cb111893 [Baoqi Wu] [KYUUBI #956] Fail to get table list if there is _ in schema's name Authored-by: Baoqi Wu <wubaoqi@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
4912542bda |
[KYUUBI #967] [TEST] Add log if pod status is unexpected
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Help debug failed test. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #967 from ulysses-you/k8s-log. Closes #967 6dfa1567 [ulysses-you] Add log if pod status is unexpected Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Fu Chen
|
c83c38b350 |
[KYUUBI #954] Refactor Dockerfile
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. As the binary distributions contain all things that we want, we don't need a pre-build spark as a docker base image, use java as the base image instead. 2. Build kyuubi from source code. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #955 from cfmcgrady/kyuubi-954. Closes #954 c1c4ed31 [Fu Chen] only compile module kyuubi-compile/kyuubi-integration-tests af96c7df [Fu Chen] update master.yaml 4c395c22 [Fu Chen] sleep for debug GA 770eef1f [Fu Chen] add local docker registry 749db9c4 [Fu Chen] build docker image feb86b01 [Fu Chen] fix Dockerfile arg bug 0a599749 [Fu Chen] add .dockerignore 49dc6e31 [Fu Chen] review dd1d73ce [Fu Chen] refactor Dockerfile Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
wangxuan0816
|
682ec1c582
|
[KYUUBI #963] Bump up maven-download-plugin to 1.6.6
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Bump up maven-download-plugin to 1.6.6 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #965 from wangxuan0816/kyuubi-963. Closes #963 291c367a [wangxuan0816] [KYUUBI #963] Bump up maven-download-plugin to 1.6.6 Authored-by: wangxuan0816 <1421615117@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
4aa93d3cbd
|
[KYUUBI #957] Reopen KyuubiServerSuite
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #960 from yanghua/KYUUBI-957. Closes #957 94cdddda [yanghua] [KYUUBI #957] Reopen KyuubiServerSuite Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
a263ee9cd2 |
[KYUUBI #959] Reduce EventLoggingService.onEvent time to one with compiled state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `setState` has already call `EventLoggingService.onEvent`, so we don't need to call `EventLoggingService.onEvent` twice with compiled state . ``` override def setState(newState: OperationState): Unit = { super.setState(newState) statementEvent.state = newState.toString statementEvent.stateTime = lastAccessTime EventLoggingService.onEvent(statementEvent) } ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #959 from ulysses-you/compile. Closes #959 dc517a65 [ulysses-you] reduce on event Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
32b18ba19c
|
[KYUUBI #933] Enhance the detection mechanism for engine startup
### 1. Description EngineRef use exists znode to determine whether engine started successfully or not(use the last znode), there are inconsistencies in this. So, take an idea to improve the detection mechanism by adding the started sessionid to the znone. `EngineRef` will check the sessionid in a loop after a engine started. ### 2. Here are two options **Option I, add sesssionid to the data of znode**  **Option II, add sessionid to the path of znode**  ### 3. Solution In order to be consistent with the design of server znode, prefer to choose _Option II_ . **Demo** ``` /kyuubi - serviceUri=bigdata:10009;version=1.3.0-SNAPSHOT;sequence=0000000000 /kyuubi_USER/test - serviceUri=bigdata:39869;version=1.3.0-SNAPSHOT;session=8178069f-0b22-4d06-b1c0-908094769397;sequence=0000000000] ``` Closes #935 from timothy65535/ky-933. Closes #933 ab5d6d80 [timothy65535] add issue id d21cb990 [timothy65535] compatible with back 568acf21 [timothy65535] fix error 3db8ba7f [timothy65535] move EngineRef get to Discovery 827f3780 [timothy65535] rename get to getEngineBySessionId 88229314 [timothy65535] update session id using internal ab06adf8 [timothy65535] improve EngineRef get logic 6a1cc39a [timothy65535] update session version c94b6e55 [timothy65535] rename sessionId to createSessionId 0cc6aae2 [timothy65535] update ha conf d24a152f [timothy65535] update ha conf 1a67c864 [timothy65535] fix initialize sql suite b3b502de [timothy65535] [KYUUBI #933] Enhance the detection mechanism for engine startup Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
636d60aa71
|
[KYUUBI #958] [TEST] Ensure two connections in user mode share the same engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Ensure changes like #935 do not break the rule for the semantics of engine sharing If and only if we want engines to be pooling, this test can be changed ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #958 from yaooqinn/same. Closes #958 e172cebf [Kent Yao] Ensure two connections in user mode share the same engine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
a76c344042
|
[KYUUBI #951] [LICENSE] Add license header on all docs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #951 from pan3793/license. Closes #951 4629eecd [Cheng Pan] Fix c45a0784 [Cheng Pan] nit b9a46b42 [Cheng Pan] pin license header at first line 80d1a71b [Cheng Pan] nit b2a46e4c [Cheng Pan] Update f6acaaf8 [Cheng Pan] minor ef99183f [Cheng Pan] Add license header on all docs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
6b23c1c1f5
|
[KYUUBI #952] Rename FrontendService and FrontendServiceSuite with a Thrift prefix
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #953 from yanghua/KYUUBI-952. Closes #952 a3029011 [yanghua] [KYUUBI 952] Rename FrontendService and FrontendServiceSuite with a Thrift prefix Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
960ac89776 |
[KYUUBI #936] Add date info for json event path
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add date info helps collect metrics across some days. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #936 from ulysses-you/event-date-partition. Closes #936 6d204084 [ulysses-you] time f142e052 [ulysses-you] test fd220112 [ulysses-you] statement 31f9f0fb [ulysses-you] empty 67acbd5e [ulysses-you] fix 4cb0ee91 [ulysses-you] fix 42159c08 [ulysses-you] partitions 3500d91c [ulysses-you] remove enent= 04c3dacb [ulysses-you] fix 5e7f19ec [ulysses-you] nit 83eaf529 [ulysses-you] date partition a2d6a38d [ulysses-you] start time 693e8ee9 [ulysses-you] Add date info for json event path Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
65e579271b
|
[KYUUBI #945] [DOCS] Update release guide
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #945 from pan3793/doc. Closes #945 a46af55e [Cheng Pan] [DOCS] Update release guide Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b88d9b29c8
|
[KYUUBI #948] [DEPS][TEST] Bump iceberg 0.12.0
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #943 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #948 from pan3793/iceberg. Closes #948 47a8a653 [Cheng Pan] Bump iceberg 0.12.0 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
11d1de2a0e
|
[KYUUBI #942] Support for adding internal attributes to ConfigBuilder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### Description `ConfigBuilder` currently only supports public configuration by default. In practice, there are also some configurations, which are internal configurations and do not need to be reserved for users. refer to https://github.com/apache/incubator-kyuubi/pull/935#issuecomment-900117832 ### Modification - Add `internal` to ConfigBuilder - Update `AllKyuubiConfiguration` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #949 from timothy65535/ky-942. Closes #942 5d4723b3 [timothy65535] update config b41461bb [timothy65535] [KYUUBI #942] Support for adding internal attributes to ConfigBuilder Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
d0c9c8ab25
|
[KYUUBI #903] Add event for statement
### _Why are the changes needed?_ - Through eventLog to write statementInfo into file ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #903 from zhang1002/branch-1.2_statementEvent. Closes #903 27cc7741 [张宇翔] 1. Add some introduce 2. change som UT 7cf73bde [张宇翔] Merge branch 'master' into branch-1.2_statementEvent b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' fc868609 [张宇翔] resolve conflicts 5dfcf868 [张宇翔] resolve conflicts b04adee8 [张宇翔] remove some unused code 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 2a4317a5 [张宇翔] remove some ut 0c474cc4 [张宇翔] Add statement event 4e05a395 [张宇翔] Add statement event 5f73e247 [张宇翔] Add statement event 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
c228ecc2ef
|
[KYUUBI #946] [TEST] Fix version test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #944 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #946 from pan3793/ver. Closes #946 81f69c8e [Cheng Pan] Fix version test Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
f21e118a44 |
[KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> After several benchmarks in the offline environment, kyuubi consumes a lot of CPU. ``` [bigdata]$ jps 3188 Main 6036 Jps 4189 KyuubiServer [bigdata]$ top top - 09:34:21 up 23 min, 1 user, load average: 10.67, 7.85, 4.98 Tasks: 320 total, 1 running, 319 sleeping, 0 stopped, 0 zombie %Cpu(s): 48.2 us, 23.9 sy, 0.0 ni, 27.4 id, 0.0 wa, 0.5 hi, 0.0 si, 0.0 st MiB Mem : 15806.0 total, 4738.4 free, 6085.0 used, 4982.5 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9306.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4189 bigdata 20 0 10.1g 311272 22484 S 633.3 1.9 48:12.21 java ``` ``` Threads: 53 total, 6 running, 47 sleeping, 0 stopped, 0 zombie %Cpu(s): 31.9 us, 22.8 sy, 0.0 ni, 44.8 id, 0.0 wa, 0.4 hi, 0.0 si, 0.0 st MiB Mem : 15806.0 total, 4770.5 free, 6052.9 used, 4982.5 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9338.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5222 bigdata 20 0 10.1g 311272 22484 R 99.3 1.9 10:00.91 process-logger- 5044 bigdata 20 0 10.1g 311272 22484 R 99.0 1.9 10:03.98 process-logger- 4730 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:13.24 process-logger- 4911 bigdata 20 0 10.1g 311272 22484 R 98.3 1.9 10:06.42 process-logger- 4523 bigdata 20 0 10.1g 311272 22484 R 98.0 1.9 10:41.55 process-logger- 5769 bigdata 20 0 10.1g 311272 22484 R 96.7 1.9 2:46.80 process-logger- ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #941 from timothy65535/ky-940. Closes #940 331b9423 [timothy65535] [KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
5fb143e50b
|
[KYUUBI #938] Disable Kyuubi version udf test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Kyuubi docker image has not updated to lastest version, the kubernetes integration tests failed in master branch. ### _How was this patch tested?_ Pass CI Closes #938 from ulysses-you/disable-udf. Closes #938 3260ed73 [ulysses-you] disable version test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
98f38ba517
|
[BUILD] Bump 1.4.0-SNAPSHOT | ||
|
Cheng Pan
|
396310670a
|
[KYUUBI #932] [DOC] Update collaborators page and correct terminology KPIP usage
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Partly fix #876 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #932 from pan3793/doc. Closes #932 43df4766 [Cheng Pan] switch to https 8d6e9761 [Cheng Pan] Correct terminology KPIP usage dd813af2 [Cheng Pan] Update collaborators Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
c0606713e7
|
[KYUUBI #874] [ASF] ASF Publish
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Add release script - Add release guide - Add license header in some files ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request The release script has not been tested yet. Closes #874 from pan3793/asf-pub. Closes #874 5e798a96 [Cheng Pan] LICENSE header 825f2a84 [Cheng Pan] Fix rat aecc3e6b [Cheng Pan] release doc a55503c5 [Cheng Pan] Kyuubi Release Guide 2f1a25a2 [Cheng Pan] Release Guide c5a40c24 [Cheng Pan] Release script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
9efa78b65e
|
[KYUUBI #930] Extract the zookeeper part from ServiceDiscovery
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ ### Description `ServiceDiscovery` contains `discovery` and `zookeeper` part, `ServiceDiscovery` should only has one responsibility.  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #931 from timothy65535/ky-930. Closes #930 29d48921 [timothy65535] Add ZooKeeperClientProvider 17fbe5a2 [timothy65535] [KYUUBI #930] Extract the zookeeper part from ServiceDiscovery Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
7e2d67ed61 |
[KYUUBI #926] Support add repartition for OptimizedCreateHiveTableAsSelectCommand
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We missed a ctas node `OptimizedCreateHiveTableAsSelectCommand`. This PR aims to support add repartition for `OptimizedCreateHiveTableAsSelectCommand`. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #926 from ulysses-you/cats. Closes #926 5ff28eed [ulysses-you] nit f530fb0d [ulysses-you] style 4bf0a1cf [ulysses-you] ctas Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
ad7eca1281 |
[KYUUBI #920] Add COMPILED State for ExecuteStatement af engine side
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> RUNNING -> COMPILED can be used to calculate to the duration of a statement being parsed, analyzed, and optimized ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #920 from yaooqinn/compiled. Closes #920 05c355d6 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side dfba0481 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
92a45e2679
|
[KYUUBI #919] [MISC] Add DISCLAIMER and misc changes on pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #919 from pan3793/asf-misc. Closes #919 336a4ea0 [Cheng Pan] Update 30ba74fb [Cheng Pan] Update developers db60eac4 [Cheng Pan] Update developers fcc9adc9 [Cheng Pan] Cleanup pom 70b6596a [Cheng Pan] Add DISCLAIMER Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
5526898f7b |
[KYUUBI #894] Enhance kyuubi-zookeeper module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### 1. Describe the feature Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host. From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project. - https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java - https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java ### 2. Describe the solution Improve - Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same - Replace `TestingServer` with `ZooKeeperServer` - Remove `curator-test` dependency ### 3. `deleteDataDirectoryOnClose` The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml` https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml ``` hadoop.security.dns.log-slow-lookups.enabled hadoop.security.dns.log-slow-lookups.threshold.ms dfs.client.block.write.replace-datanode-on-failure.policy ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #907 from timothy65535/ky-894. Closes #894 7d7fde9f [timothy65535] remove useless config ce06a0fd [timothy65535] limit travis build only master branch ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config a336544a [timothy65535] remove javassist 3da0bd55 [timothy65535] remove javassist 0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
6dd8dfb70c
|
[KYUUBI #914] Use isEmpty to trigger default init queries
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1 use isEmpty instead show for the init SQLs 2 add job desc for SQL tab of Spark UI 3 make it simple. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #914 from yaooqinn/isempty. Closes #914 70785da7 [Kent Yao] Use isEmpty to trigger default init queries 438009e4 [Kent Yao] Use isEmpty to trigger default init queries Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
f96256ced6
|
[KYUUBI #906] Add SessionEvent to track session status changes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #909 from yaooqinn/906. Closes #906 e202b70d [Kent Yao] nit 1271e841 [Kent Yao] fix test d696021f [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes bdd2b875 [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Yue Boyang
|
d49b47abaf
|
[KYUUBI #912] Fix broken links in the doc and PULL_REQUEST_TEMPLATE
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix broken links in the doc and PULL_REQUEST_TEMPLATE. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #912 from BoyangYue/testing. Closes #912 239ad681 [Yue Boyang] Fix broken links in the doc and PULL_REQUEST_TEMPLATE Authored-by: Yue Boyang <boyang.yue@student.unimelb.edu.au> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Yue Boyang
|
0f8c70b8f1
|
[KYUUBI #905] [MINOR] clear handleToTSessionHandle in KyuubiOperationManager
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To clear deprecated handleToTSessionHandle in KyuubiOperationManager. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #911 from BoyangYue/test-branch. Closes #905 3da8add4 [Yue Boyang] [KYUUBI #905] [MINOR] clear handleToTSessionHandle in KyuubiOperationManager Authored-by: Yue Boyang <boyang.yue@student.unimelb.edu.au> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
89d031264b
|
[KYUUBI #902] Wrong operation statistics on kyuubi page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When serveral sql statements finished, wrong statistics of operations are displayed on kyuubi page. ``` 1 session(s) are online, running 5 operations ```  **ROOT CAUSE** When run each statement, will run `getTypeInfo` internally. **SparkSQL Engine Log** ``` 2021-08-08 14:31:40.086 INFO log.OperationLog: Creating operation log file /tmp/hadoop-08fa33a4-0aee-4556-865e-bb9866f3fb3a/operation_logs/b9ba0414-4960-4e3a-b31c-717b9dcdd1e1/84e35aea-f84e-4da1-a827-97e818062d78 2021-08-08 14:31:40.087 INFO operation.GetTypeInfo: Processing hadoop's query[2714ab78-b9f6-4fd2-ac43-dc9d0ac52157]: INITIALIZED_STATE -> RUNNING_STATE, statement: GET_TYPE_INFO 2021-08-08 14:31:40.088 INFO operation.GetTypeInfo: Processing hadoop's query[2714ab78-b9f6-4fd2-ac43-dc9d0ac52157]: RUNNING_STATE -> FINISHED_STATE, statement: GET_TYPE_INFO, time taken: 0.001 seconds 2021-08-08 14:31:40.088 INFO monitor.KyuubiStatementMonitor: Add kyuubiStatementInfo into queue is [true], statementId is [84e35aea-f84e-4da1-a827-97e818062d78] 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: Processing hadoop's query[84e35aea-f84e-4da1-a827-97e818062d78]: INITIALIZED_STATE -> PENDING_STATE, statement: select 1 2021-08-08 14:31:40.089 INFO operation.ExecuteStatement: Processing hadoop's query[84e35aea-f84e-4da1-a827-97e818062d78]: PENDING_STATE -> RUNNING_STATE, statement: select 1 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #904 from timothy65535/ky-902. Closes #902 e5bce883 [timothy65535] [KYUUBI #902] Wrong operation statistics on kyuubi page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
52d10f1806
|
[KYUUBI #892] Make JsonEventLogger compatible with HDFS-like FileSystems
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> FSDataOutputStream is used for this ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #892 from yaooqinn/2021080417. Closes #892 e291a73c [Kent Yao] nit e6ebef1a [Kent Yao] Make JsonEventLogger compatible with HDFS-like FileSystems Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
c86767f336
|
[KYUUBI #901] Skip download module generate config docs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Imporve perf of generate config docs. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #901 from ulysses-you/enhance-comment. Closes #901 a86a9aa6 [ulysses-you] skip-download Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
065c98a259
|
[KYUUBI #897] Support stop Spark engine through web ui
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/897 ### _How was this patch tested?_ The following two images show the stop feature is enabled:   The following image show the stop feature is disabled:  Closes #898 from ulysses-you/kyuubi-897. Closes #897 61ad25c1 [ulysses-you] address comment b63997ec [ulysses-you] maven dbb36014 [ulysses-you] maven 3c91307c [ulysses-you] simplify 6e9e5bca [ulysses-you] pom 4e143657 [ulysses-you] config e579e227 [ulysses-you] Support stop Spark engine through web ui Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
20394398c0
|
[KYUUBI #889] [BUILD] Package release tarball
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ``` export RELEASE_VERSION=1.3.0-SNAPSHOT # keep same version with pom.xml export SKIP_GPG=false # optional, default false build/release/create-package.sh source build/release/create-package.sh binary ``` ``` ➜ apache-kyuubi git:(asf-pub) ls -lh work/release total 161256 -rw-r--r-- 1 chengpan staff 60M Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz -rw-r--r-- 1 chengpan staff 488B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.asc -rw-r--r-- 1 chengpan staff 212B Aug 4 15:12 kyuubi-1.3.0-SNAPSHOT-bin.tgz.sha512 -rw-r--r-- 1 chengpan staff 18M Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz -rw-r--r-- 1 chengpan staff 488B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.asc -rw-r--r-- 1 chengpan staff 215B Aug 4 15:10 kyuubi-1.3.0-SNAPSHOT-source.tgz.sha512 ``` Planning to add release script and release doc in following PRs. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #889 from pan3793/asf-package. Closes #889 0aef5d77 [Cheng Pan] Update docs 4b0be7da [Cheng Pan] [BUILD] Package release tarball Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
1a76d77b6f |
[KYUUBI #895] Enhance output uses the default spark conf
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When start the server, If not specified `SPARK_CONF_DIR`, it should output default. ``` KYUUBI_WORK_DIR_ROOT: /root/tmp/kyuubi/work SPARK_HOME: /root/package/spark-3.1.2-bin-hadoop2.7 SPARK_CONF_DIR: HADOOP_CONF_DIR: /root/package/hadoop-3.2.0/etc/hadoop ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #896 from timothy65535/ky-895. Closes #895 9a936a77 [timothy65535] [KYUUBI #895] Enhance output uses the default spark conf Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
34925fb3eb |
[KYUUBI #890] Fix minikube test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> change the k8s-it serviceaccount to kyuubi instead of default ### _How was this patch tested?_ Pass CI Closes #890 from ulysses-you/fix-minikube-test. Closes #890 5844c482 [ulysses-you] kyuubi 16192d6e [ulysses-you] test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
