413ba2eb24
185 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
zhouyifan279
|
413ba2eb24
|
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine
### _Why are the changes needed?_ Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`. ``` 21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds 21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before 21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` 21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds 21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1168 from zhouyifan279/#1009. Closes #1009 fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine 3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
fb6ce8fcc6
|
[KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> It's pre-step of #1152, otherwise will cause circular dependence. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1151 from pan3793/move-jdbc-test. Closes #1131 e65f9979 [Cheng Pan] nit f2e66535 [Cheng Pan] [KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
22e6432e4d
|
[KYUUBI #1022] Add basic EngineStatusStore for events
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981 `EngineStatusStore` helps to push events to listener bus `EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides: - stores all elements, and sorted by startTimestamp. - cleanup the last elements when reach a certain threshold. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1023 from timothy65535/ky-1022. Closes #1022 b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
103969a042
|
[KYUUBI #1067] Catch the real exception when the log directory is not exist
BUGFIX: #1067 fix: 2021-09-23 16:03:43.248 INFO HiveMetaStore.audit: ugi=zhangyu ip=unknown-ip-addr cmd=get_databases: * 2021-09-23 16:03:43.319 ERROR events.JsonEventLogger: Get log directory /Users/zhangyuxiang/Downloads/kyuubi_test failed java.io.FileNotFoundException: File /Users/zhangyuxiang/Downloads/kyuubi_test does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.kyuubi.events.JsonEventLogger.requireLogRootWritable(JsonEventLogger.scala:79) at org.apache.kyuubi.events.JsonEventLogger.initialize(JsonEventLogger.scala:93) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1136 from zhang1002/branch-1.3.0_checkRightException. Closes #1067 de16c63d [张宇翔] delete some ut 5a29bb80 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 21e856a3 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 873cc9c4 [张宇翔] Merge remote-tracking branch 'upstream/master' d1e7a019 [张宇翔] catch IOException when log directory is not exist bc442d8c [张宇翔] catch IOException when log directory is not exist 0d7418e9 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 41cff19a [张宇翔] Merge remote-tracking branch 'upstream/master' 74b36efb [张宇翔] catch IOException when log directory is not exist def8d1f6 [张宇翔] Merge remote-tracking branch 'upstream/master' 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
0ee8eff5a0
|
[KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> After this PR, the service named service discovery will be a child service of frontend service and it's optional. A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1144 from yaooqinn/1143. Closes #1143 f2beb721 [Kent Yao] fix test 128cf282 [Kent Yao] nit 37aaff9c [Kent Yao] lift FrontendService trait add335c4 [Kent Yao] nit 3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy 7e5a5cbb [Kent Yao] Merge branch 'master' into 1143 09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
ee8e5a12f1
|
[KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1142 from yanghua/KYUUBI-1137. Closes #1137 418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
d93a6c8b5b
|
[KYUUBI #1141] Remove redundant unit
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove redundant unit. `MB` ``` 2021-09-23 08:41:08.889 INFO spark.SparkSQLEngine: Spark application name: kyuubi_USER_root_6f67c12c-f362-4a07-87d0-7aa7e5f43171 application ID: spark-461363e9cc934d1986f7ae4191dc0410 application web UI: http://spark-740a727c11ce3da7-driver-svc.default.svc:44692 master: k8s://https://10.10.9.35:6443 version: 3.1.2 driver: [cpu: 1, mem: 1G] executor: [cpu: 2, mem: 2G MB, maxNum: 18] Start time: Thu Sep 23 08:40:39 UTC 2021 User: root (shared mode: USER) State: LATENT ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1141 from timothy65535/remove. Closes #1141 af1827e9 [timothy65535] Remove redundant unit Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
787ba9efe2
|
[KYUUBI #1134] [TEST] DataLake suites should clean up withKyuubiConf instead of extraConfigs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Keep consistent with ``` withKyuubiConf.foreach { case (k, v) => System.setProperty(k, v) kyuubiConf.set(k, v) } ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1134 from pan3793/t. Closes #1134 b8e0aa65 [Cheng Pan] [TEST] DataLake Suites should clean up withKyuubiConf instead of extraConfigs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
b6de06c5cc
|
[KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> **RUN QUERY** ``` set kyuubi.operation.plan.only.mode=ANALYZE; select 1; ``` **ERROR MSG** ``` 2021-09-16 15:19:57.278 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: INITIALIZED_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE 2021-09-16 15:19:57.293 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE, time taken: 0.015 seconds 2021-09-16 15:19:57.309 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-09-16 15:19:57.313 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1114 from timothy65535/ky-1113. Closes #1113 e8409d12 [timothy65535] [KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
4f11b51673
|
[KYUUBI #1059] Add Plan Only Operations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement, PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them. This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite. This is similar to the `explain` command ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1060 from yaooqinn/1059. Closes #1059 e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
6a01f65d3e
|
[KYUUBI #916] Send DelegationTokens to SQL engines
### _Why are the changes needed?_ This PR finishes issue #916 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1074 from zhouyifan279/KYUUBI#916. Closes #916 28176c7e [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines b6c931bd [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
张宇翔
|
354e3c0b5a
|
[KYUUBI #1068] event log can't upload to hdfs
bug: #1068 This version, event log don't support upload file to hdfs ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1075 from zhang1002/branch-1.3_supportLogUploadHdfs. Closes #1068 7b1483c5 [张宇翔] Modify some UT 27aa4596 [张宇翔] Modify some UT c8223914 [张宇翔] Modify some UT 8b3ef67b [张宇翔] Modify some UT fa06ead9 [张宇翔] Merge branch 'master' into branch-1.3_supportLogUploadHdfs 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 8801ece7 [张宇翔] Remove "toString" from ut 3288724b [张宇翔] Modify some UT 0bd7ecbd [张宇翔] Support the eventLog upload to hdfs 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
c080283671
|
[KYUUBI #1080] Move stringifyException to Utils
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Move stringifyException from KyuubiSQLException to Utils, make it more general. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1081 from timothy65535/ky-1080. Closes #1080 ce3015d5 [timothy65535] [KYUUBI #1080] Move stringifyException to Utils Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
8886513534
|
[KYUUBI #1057] [KYUUBI#1056] Output error log when currentEngine is None
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When createSpark in SparkSqlEngine get sparksession or execute initialize.sql failed, we cannot get any error info in logs. At this time, currentEngine is None. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1057 from hddong/engine-fix. Closes #1057 44ead3b3 [hongdongdong] fix 3462d36c [hongdongdong] [KYUUBI#1056] Output error log when currentEngine is None Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Brian Yue
|
033784c691
|
[KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To avoid performance issues in the topK scenario and close #1018 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1054 from byyue/feature-1018. Closes #1018 21de1fcb [Brian Yue] Set spark.sql.execution.topKSortFallbackThreshold to 10000 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
qiuliang
|
d83674248e
|
[KYUUBI #1034] Engine may deadlock when close operationLog
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1062 from qiuliang988/1034. Closes #1034 41d510f5 [qiuliang] add comments dbd7c770 [qiuliang] [KYUUBI #1034]Engine may deadlock when close operationLog Authored-by: qiuliang <qlcumt@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
59d9f06a46 |
[KYUUBI #1032][FOLLOWUP] Refine session hande in AbstractSession
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove session hande in `AbstractSession` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1046 from ulysses-you/kyuubi-1032-f. Closes #1032 1d7543a9 [ulysses-you] order 36a0a1cc [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
c1e5545c64 |
[KYUUBI #1043] Let spark history logger handle events asynchronously
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> > there seems to be no good way to solve the synchronization problem. For `EventLoggerType.SPARK` This looks simple, Please send a separate PR to solve this issue ```git - sc.eventLogger.foreach(_.onOtherEvent(kyuubiEvent)) + sc.listenerBus.post(kyuubiEvent) ``` _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/1023#issuecomment-913558016_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1044 from timothy65535/1043. Closes #1043 0ea1d4f7 [timothy65535] [KYUUBI #1043] Let spark history logger handle events asynchronously Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
f584953f7f
|
[KYUUBI #703][FOLLOWUP][DEPS][TEST] Reduce Hudi dependencies
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> HBase deps is optional for creating hudi table and query, but required for writing data to hudi table, as currently we only test create table and query for data lake formats, it's reasonable to remove the hbase deps to reduce the deps pulled from hudi. This PR also change all data lake test suites in spark engine from `hive` catalog to `in-memory` catalog. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1028 from pan3793/hudi. Closes #703 56c41a30 [Cheng Pan] Use in-memory catalog for DataLake test b582a865 [Cheng Pan] Revert "Default use in-memory in Spark engine test" e62c403a [Cheng Pan] Default use in-memory in Spark engine test e2b8e0fd [Cheng Pan] Remove unused hudi spark conf 069fbacc [Cheng Pan] Revert "HFileBootstrapIndex" ca1b20e1 [Cheng Pan] HFileBootstrapIndex da37b0fc [Cheng Pan] Remove log4j 41ce8526 [Cheng Pan] Align jackson version f56f5e89 [Cheng Pan] Remove HBase dependencies c2b76b54 [Cheng Pan] Exclude hudi-timeline-service 56641762 [Cheng Pan] nit e588c12f [Cheng Pan] [DEPS][TEST] Reduce Hudi dependencies Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
simon
|
098e660a3d
|
[KYUUBI #703] support hudi-0.9.0
### _Why are the changes needed?_ Hudi has add Spark Sql Support in version 0.9.0. This PR add Hudi-0.9.0 dependency and Unit Testing #703 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #994 from simon824/master. Closes #703 a04b2e42 [simon] exclude jdk.tools dep 50e1ab01 [simon] fix jdk11 dep f33ebdf1 [simon] fix hudi dep conflicts ff2585c2 [simon] fix yarn dep conflicts 32dd1ea8 [simon] fix scala version conflicts b8a37401 [simon] add spark.sql.catalogImplementation 82f5422f [simon] fix spark3 dependency 2a6c497d [simon] fix spark3 dependency 4dbec8be [Simon] Merge branch 'apache:master' into master 3f180157 [simon] fix maven 73e48d0a [simon] add spark3 support maven dependency 3def658b [simon] fix missing spark-sql-engine hudi maven dependency 524132d3 [simon] bugfix e98998a6 [simon] fix pom error 59fc6669 [Simon] Merge branch 'apache:master' into master d9e17ebf [simon] fix conflict 2466ece3 [simon] add sparkSQL test on hudi-0.9.0 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
01bebe36e8 |
[KYUUBI #1011] Fallback some frontend service config option about thrift
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1016 from yanghua/KYUUBI-1011. Closes #1011 78107b76 [yanghua] Addressed review suggestion 651b5989 [yanghua] [KYUUBI #1011] Fallback some frontend service config option about thrift Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
hongdongdong
|
94d60ea505 |
[KYUUBI #1000] Use underscore instead hyphen in path
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Use underscore instead hyphen in path, friendly to sql. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1000 from hddong/fix-underscore. Closes #1000 8b3459a5 [hongdongdong] Use underscore instead hyphen in path Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
4ed5704590
|
[KYUUBI #983] Refactor the basic abstraction of frontend service
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #984 from yanghua/KYUUBI-983. Closes #983 db953b8a [yanghua] Rename KyuubiFrontendService to be KyuubiFrontendServices 8c3d7f2a [yanghua] [KYUUBI #983] Refactor the basic abstraction of frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
d704e4d49d |
[KYUUBI #922] Add OperationEvent for server to track operation state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Track operation state change and support output to custom system. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #968 from hddong/add-server-events. Closes #922 8be8f268 [hongdongdong] Add prefix for StatementEvent 9fdcc504 [hongdongdong] fix comment a0cdb35d [hongdongdong] fix test 4d340c67 [hongdongdong] add dependency 00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
cb70e062c3
|
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #980 from yanghua/KPIP-1. Closes #979 e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Min Zhao
|
8bab5b8d6a
|
[KYUUBI #659] Add UDF engine_name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add UDF engine_name to show the runtime information about execution engine ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #747 from zhaomin1423/udf. Closes #659 8c3b3c16 [Min Zhao] Merge branch 'master' into udf f184a661 [Min Zhao] Merge branch 'master' into udf 9ea0c190 [Min Zhao] get appName from driver side 09f07523 [Min Zhao] get conf by SparkEnv.get.conf in the engine_name udf 6c000784 [Min Zhao] update the description of the engine_name udf 63ef2762 [Min Zhao] update class to org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite in functions.md db02884c [Min Zhao] [KYUUBI #659] Add UDF engine_name Lead-authored-by: Min Zhao <zhaomin1423@163.com> Co-authored-by: Min Zhao <49054376+zhaomin1423@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Baoqi Wu
|
484bb3c6bd
|
[KYUUBI #956] Fail to get table list if there is _ in schema's name
Just write some basic tests out in other projects. Since this toJavaRegex is not a static function, which is hard to write unit test if don't move it out as as static function in object. (But in that case, the change may be too large for this). ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #964 from Baoqi/bwu_fix_956. Closes #956 985a73b5 [Baoqi Wu] [KYUUBI #956] add a unit test for this cb111893 [Baoqi Wu] [KYUUBI #956] Fail to get table list if there is _ in schema's name Authored-by: Baoqi Wu <wubaoqi@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
a263ee9cd2 |
[KYUUBI #959] Reduce EventLoggingService.onEvent time to one with compiled state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `setState` has already call `EventLoggingService.onEvent`, so we don't need to call `EventLoggingService.onEvent` twice with compiled state . ``` override def setState(newState: OperationState): Unit = { super.setState(newState) statementEvent.state = newState.toString statementEvent.stateTime = lastAccessTime EventLoggingService.onEvent(statementEvent) } ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #959 from ulysses-you/compile. Closes #959 dc517a65 [ulysses-you] reduce on event Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
a76c344042
|
[KYUUBI #951] [LICENSE] Add license header on all docs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #951 from pan3793/license. Closes #951 4629eecd [Cheng Pan] Fix c45a0784 [Cheng Pan] nit b9a46b42 [Cheng Pan] pin license header at first line 80d1a71b [Cheng Pan] nit b2a46e4c [Cheng Pan] Update f6acaaf8 [Cheng Pan] minor ef99183f [Cheng Pan] Add license header on all docs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
960ac89776 |
[KYUUBI #936] Add date info for json event path
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add date info helps collect metrics across some days. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #936 from ulysses-you/event-date-partition. Closes #936 6d204084 [ulysses-you] time f142e052 [ulysses-you] test fd220112 [ulysses-you] statement 31f9f0fb [ulysses-you] empty 67acbd5e [ulysses-you] fix 4cb0ee91 [ulysses-you] fix 42159c08 [ulysses-you] partitions 3500d91c [ulysses-you] remove enent= 04c3dacb [ulysses-you] fix 5e7f19ec [ulysses-you] nit 83eaf529 [ulysses-you] date partition a2d6a38d [ulysses-you] start time 693e8ee9 [ulysses-you] Add date info for json event path Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
d0c9c8ab25
|
[KYUUBI #903] Add event for statement
### _Why are the changes needed?_ - Through eventLog to write statementInfo into file ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #903 from zhang1002/branch-1.2_statementEvent. Closes #903 27cc7741 [张宇翔] 1. Add some introduce 2. change som UT 7cf73bde [张宇翔] Merge branch 'master' into branch-1.2_statementEvent b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' fc868609 [张宇翔] resolve conflicts 5dfcf868 [张宇翔] resolve conflicts b04adee8 [张宇翔] remove some unused code 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 2a4317a5 [张宇翔] remove some ut 0c474cc4 [张宇翔] Add statement event 4e05a395 [张宇翔] Add statement event 5f73e247 [张宇翔] Add statement event 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
98f38ba517
|
[BUILD] Bump 1.4.0-SNAPSHOT | ||
|
timothy65535
|
9efa78b65e
|
[KYUUBI #930] Extract the zookeeper part from ServiceDiscovery
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ ### Description `ServiceDiscovery` contains `discovery` and `zookeeper` part, `ServiceDiscovery` should only has one responsibility.  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #931 from timothy65535/ky-930. Closes #930 29d48921 [timothy65535] Add ZooKeeperClientProvider 17fbe5a2 [timothy65535] [KYUUBI #930] Extract the zookeeper part from ServiceDiscovery Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
ad7eca1281 |
[KYUUBI #920] Add COMPILED State for ExecuteStatement af engine side
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> RUNNING -> COMPILED can be used to calculate to the duration of a statement being parsed, analyzed, and optimized ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #920 from yaooqinn/compiled. Closes #920 05c355d6 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side dfba0481 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
92a45e2679
|
[KYUUBI #919] [MISC] Add DISCLAIMER and misc changes on pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #919 from pan3793/asf-misc. Closes #919 336a4ea0 [Cheng Pan] Update 30ba74fb [Cheng Pan] Update developers db60eac4 [Cheng Pan] Update developers fcc9adc9 [Cheng Pan] Cleanup pom 70b6596a [Cheng Pan] Add DISCLAIMER Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
5526898f7b |
[KYUUBI #894] Enhance kyuubi-zookeeper module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### 1. Describe the feature Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host. From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project. - https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java - https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java ### 2. Describe the solution Improve - Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same - Replace `TestingServer` with `ZooKeeperServer` - Remove `curator-test` dependency ### 3. `deleteDataDirectoryOnClose` The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml` https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml ``` hadoop.security.dns.log-slow-lookups.enabled hadoop.security.dns.log-slow-lookups.threshold.ms dfs.client.block.write.replace-datanode-on-failure.policy ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #907 from timothy65535/ky-894. Closes #894 7d7fde9f [timothy65535] remove useless config ce06a0fd [timothy65535] limit travis build only master branch ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config a336544a [timothy65535] remove javassist 3da0bd55 [timothy65535] remove javassist 0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
6dd8dfb70c
|
[KYUUBI #914] Use isEmpty to trigger default init queries
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1 use isEmpty instead show for the init SQLs 2 add job desc for SQL tab of Spark UI 3 make it simple. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #914 from yaooqinn/isempty. Closes #914 70785da7 [Kent Yao] Use isEmpty to trigger default init queries 438009e4 [Kent Yao] Use isEmpty to trigger default init queries Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
f96256ced6
|
[KYUUBI #906] Add SessionEvent to track session status changes
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #909 from yaooqinn/906. Closes #906 e202b70d [Kent Yao] nit 1271e841 [Kent Yao] fix test d696021f [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes bdd2b875 [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
89d031264b
|
[KYUUBI #902] Wrong operation statistics on kyuubi page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When serveral sql statements finished, wrong statistics of operations are displayed on kyuubi page. ``` 1 session(s) are online, running 5 operations ```  **ROOT CAUSE** When run each statement, will run `getTypeInfo` internally. **SparkSQL Engine Log** ``` 2021-08-08 14:31:40.086 INFO log.OperationLog: Creating operation log file /tmp/hadoop-08fa33a4-0aee-4556-865e-bb9866f3fb3a/operation_logs/b9ba0414-4960-4e3a-b31c-717b9dcdd1e1/84e35aea-f84e-4da1-a827-97e818062d78 2021-08-08 14:31:40.087 INFO operation.GetTypeInfo: Processing hadoop's query[2714ab78-b9f6-4fd2-ac43-dc9d0ac52157]: INITIALIZED_STATE -> RUNNING_STATE, statement: GET_TYPE_INFO 2021-08-08 14:31:40.088 INFO operation.GetTypeInfo: Processing hadoop's query[2714ab78-b9f6-4fd2-ac43-dc9d0ac52157]: RUNNING_STATE -> FINISHED_STATE, statement: GET_TYPE_INFO, time taken: 0.001 seconds 2021-08-08 14:31:40.088 INFO monitor.KyuubiStatementMonitor: Add kyuubiStatementInfo into queue is [true], statementId is [84e35aea-f84e-4da1-a827-97e818062d78] 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: s statemendId 84e35aea-f84e-4da1-a827-97e818062d78, sessionId: b9ba0414-4960-4e3a-b31c-717b9dcdd1e1 2021-08-08 14:31:40.088 INFO operation.ExecuteStatement: Processing hadoop's query[84e35aea-f84e-4da1-a827-97e818062d78]: INITIALIZED_STATE -> PENDING_STATE, statement: select 1 2021-08-08 14:31:40.089 INFO operation.ExecuteStatement: Processing hadoop's query[84e35aea-f84e-4da1-a827-97e818062d78]: PENDING_STATE -> RUNNING_STATE, statement: select 1 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #904 from timothy65535/ky-902. Closes #902 e5bce883 [timothy65535] [KYUUBI #902] Wrong operation statistics on kyuubi page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
52d10f1806
|
[KYUUBI #892] Make JsonEventLogger compatible with HDFS-like FileSystems
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> FSDataOutputStream is used for this ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #892 from yaooqinn/2021080417. Closes #892 e291a73c [Kent Yao] nit e6ebef1a [Kent Yao] Make JsonEventLogger compatible with HDFS-like FileSystems Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
065c98a259
|
[KYUUBI #897] Support stop Spark engine through web ui
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/897 ### _How was this patch tested?_ The following two images show the stop feature is enabled:   The following image show the stop feature is disabled:  Closes #898 from ulysses-you/kyuubi-897. Closes #897 61ad25c1 [ulysses-you] address comment b63997ec [ulysses-you] maven dbb36014 [ulysses-you] maven 3c91307c [ulysses-you] simplify 6e9e5bca [ulysses-you] pom 4e143657 [ulysses-you] config e579e227 [ulysses-you] Support stop Spark engine through web ui Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
4fecf13902 |
[KYUUBI #888] Add scala version suffix on scala modules
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Add scala version suffix on scala modules to match the name pattern. - Enable `kyuubi-extension-spark-3-1` in daily publish snapshot ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #888 from pan3793/scala. Closes #888 571eb9c3 [Cheng Pan] comments 16cb263a [Cheng Pan] remove hardcode scala version in pom 6090c798 [Cheng Pan] Fix shade 5178847f [Cheng Pan] Fixup a1547753 [Cheng Pan] Fix engine jar name 2cf4793e [Cheng Pan] fix cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot 1ca31de5 [Cheng Pan] Add scala version suffix on scala modules Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
f6d5069bb6
|
[KYUUBI #885] Add Event for Engine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #885 from yaooqinn/engineevent. Closes #885 6a7ac341 [Kent Yao] refine 41baf0e7 [Kent Yao] refine 2c141a06 [Kent Yao] test history server d1b0823a [Kent Yao] test history server 3aae8417 [Kent Yao] dump the final engine status 257e50f5 [Kent Yao] Add Event for Engine f1f1f0e8 [Kent Yao] Add Event for Engine 85addc6e [Kent Yao] Add Event for Engine cece5888 [Kent Yao] Add Event for Engine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
6c3b722174
|
[KYUUBI #869] Add Kyuubi Query Engine for Spark UI with basic stats
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #869 from yaooqinn/ui. Closes #869 aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui 0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats 7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b963b75d58
|
[KYUUBI #857] Support init SQL for each session
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #857 from pan3793/seesion-init-sql. Closes #857 f6f59196 [Cheng Pan] remove unnecessary change 4960830a [Cheng Pan] run ENGINE_SESSION_INITIALIZE_SQL when engine startup 6e8100fa [Cheng Pan] address comments 08f4af62 [Cheng Pan] address comments 2bb1bb85 [Cheng Pan] fix import 880b38ec [Cheng Pan] session initialize sql will only execute once in single session mode 2a646890 [Cheng Pan] test session initialize sql will not execute in single session mode f03f3eff [Cheng Pan] update doc 6e4a72f2 [Cheng Pan] address comments 34285e25 [Cheng Pan] format 6d978025 [Cheng Pan] revert log4j conf change 881a89e0 [Cheng Pan] improve test b9ec18c9 [Cheng Pan] fix ut and add log b4f586d8 [Cheng Pan] Support init SQL for each session Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
d5104ed277
|
[KYUUBI #868] Avoid calling a stopped SparkContext
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We shall avoid calling cancelJobGroup if the sc is already stopped. We shall stop the engine to stop serving before we stop the sc during the shutdown hook ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #868 from yaooqinn/shutdownhook. Closes #868 14810875 [Kent Yao] comment 972e2741 [Kent Yao] Avoid calling a stopped SparkContext Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
286e3fbcc1
|
[KYUUBI #862] Revise the unit test for "add kyuubiStatementInfo into queue"
### _Why are the changes needed?_ Revise the unit test for "add kyuubiStatementInfo into queue" ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #862 from zhang1002/update_statementinfo_ut. Closes #862 7e5f8b50 [张宇翔] Revise the unit test for "add kyuubiStatementInfo into queue" e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ce5fb97036
|
[KYUUBI #849] ASF Nexus SNAPSHOT publish
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> https://infra.apache.org/publishing-maven-artifacts.html https://issues.apache.org/jira/browse/INFRA-22114 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #849 from pan3793/asf. Closes #849 c609f6ba [Cheng Pan] update pom 31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent 9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Min Zhao
|
cd308f38ad
|
[KYUUBI #758] [KYUUBI 661] Add UDF system_user
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> add udf system_user. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #758 from zhaomin1423/661_add-system_user. Closes #758 70eba56a [Min Zhao] update system user name f5edc621 [Min Zhao] [KYUUBI 661] Add UDF system_user Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
e7e77f9277
|
[KYUUBI #814] Event Tracking: For jonInfo
### _Why are the changes needed?_ Store all jobInfos for each statement in mem. - First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds. - Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds. Some interfaces: - KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds. - KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on. - KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold. Test case: externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #814 from zhang1002/branch-1.2_add-jobinfo. Closes #814 7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException 0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle) bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener 741fcd0f [张宇翔] Add interval b61b9406 [张宇翔] Code changes cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code c1a1d732 [张宇翔] 1. Remove volatile for jobInfo 9547d49e [张宇翔] 1. Remove volatile for jobInfo 970c0002 [张宇翔] 1. Add volatile for jobInfo 62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations db1de381 [张宇翔] 1. Add some log 2. Add some annotations 8deb61f2 [张宇翔] change some test 87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test 86c49ebc [张宇翔] Merge master branch a0a99b3a [张宇翔] Merge master branch f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5d3b9afb [张宇翔] Code optimization db3a0b6f [张宇翔] Format changes 08d6d1fc [张宇翔] Modify some annotations f62f00e9 [张宇翔] Event Tracking: For job 85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo 3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing b4290bfa [张宇翔] Event tracking: For jobInfo fe1f7cce [张宇翔] Change directory structure 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml 71d33b9a [张宇翔] Add some comment 6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo 8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code f43b3c8f [张宇翔] merge master branch 55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor 6f0be547 [张宇翔] format change bbfba274 [张宇翔] Even tracking 5a62f586 [张宇翔] remove some unused conf b26345ac [张宇翔] Merge master branch and resolve conflict fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished d1676268 [张宇翔] event tracking: for SQLExecutionStart Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
