16aaeafefe
1238 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
timothy65535
|
16aaeafefe
|
[KYUUBI #1227] Remove kyuubi-spark-monitor dependency from kyuubi engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove dead module dependency. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1227 from timothy65535/master. Closes #1227 13dec05d [timothy65535] Remove kyuubi-spark-monitor dependency from kyuubi engine module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
556b5ad57e
|
[KYUUBI #1223] Remove dead module kyuubi-spark-monitor
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> clean dead code ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1223 from yaooqinn/mon. Closes #1223 3b7a9a57 [Kent Yao] Remove dead module kyuubi-spark-monitor b1e2239e [Kent Yao] Remove dead module kyuubi-spark-monitor Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
5d7a6056c6
|
[KYUUBI #1221] Revert "[KYUUBI #1176][FOLLOWUP] change configuration(ha.zookeeper.acl.engine.enabled) version"
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This reverts commit
|
||
|
ulysses-you
|
36aab5b95f
|
[KYUUBI #1218] Z-order int and long should respect negative number
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Made a mistake at https://github.com/apache/incubator-kyuubi/pull/1192 that we can't reverse the short highest bit for int and int highest bit for long. Otherwise the negative number will get the wrong order. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1218 from ulysses-you/fix-zorder. Closes #1218 3f7f22d3 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
xiongyinke
|
cb886e9a1d
|
[KYUUBI #1217] [DOC] Z-order by and order by performance test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1217 from hzxiongyinke/zorder-by_and_order-by_performance_test. Closes #1217 c0232c68 [xiongyinke] format z-order-benchmark.md a7d71111 [xiongyinke] update zorder benchmark data 3bf5f81b [xiongyinke] update benchmark result secondary headlines and fix z-order test result; f5c9dfb5 [hzxiongyinke] Merge pull request #3 from apache/master 6f1892be [hzxiongyinke] Merge pull request #1 from apache/master Lead-authored-by: xiongyinke <1062376716@qq.com> Co-authored-by: hzxiongyinke <75288351+hzxiongyinke@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
b26ef2ba9e
|
[KYUUBI #1216] Skip zorder if only requires one column
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve the perf. If user only requires one column as zorde columns, it has no meaning to wrap a zorder expression. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1216 from ulysses-you/improve-zorder. Closes #1216 9975fccd [ulysses-you] skip zorder if only requires one column Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
1f8fa86644
|
[KYUUBI #1152] [SUB-TASK][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Sub task of #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1152 from pan3793/ut-kyuubi-hive-jdbc. Closes #1152 2f508356 [Cheng Pan] Avoid hardcode Hive Jdbc class 5032d23d [Cheng Pan] [KYUUBI #1131][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
b1b7f25faf
|
[KYUUBI #1198] [FEATURE] Support incremental collection
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support incremental collection, [SPARK-25224](https://issues.apache.org/jira/browse/SPARK-25224) Introduce new conf: `kyuubi.operation.incremental.collect` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1198 from pan3793/inc-col. Closes #1198 946068e9 [Cheng Pan] Address comments 2798d0d8 [Cheng Pan] Correct conf doc 3720fd41 [Cheng Pan] Incremental collection Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
badd5d516e
|
[KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1187 from timothy65535/ky-1159. Closes #1159 52daf139 [timothy65535] update conf md a9b50843 [timothy65535] improve patch 36676f2a [timothy65535] improve patch 892a6333 [timothy65535] [KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
simon
|
7a77b7da40
|
[KYUUBI #1039][FOLLOWUP] Application stop log redirect append to engineLog
### _Why are the changes needed?_ Now application stop log will overwrite log file `engineLog`, which cause engine logs are lost.  This pr change `overwrite` to `append`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1210 from simon824/redirect. Closes #1039 ae3e031d [simon] Redirect append to engineLog 2d399883 [simon] Merge remote-tracking branch 'upstream/master' 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Authored-by: simon <zhangshiming@cvte.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
simon
|
a55d5fbf79
|
[KYUUBI #1039] [TEST] Add kill application test
### Why are the changes needed? Add kill application test for #1119 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1194 from simon824/killTest. Closes #1039 36386726 [Simon] Merge branch 'apache:master' into killTest acb55492 [simon] init 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
bf629291bf
|
[KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The detail locates #1206 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1207 from yaooqinn/1206. Closes #1206 99e324a9 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level f41fd944 [Kent Yao] Merge branch 'master' into 1206 a7cb80da [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 6ecef063 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level d7ff7583 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 7b3b2f53 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
6160b9de30
|
[KYUUBI #1214] Refine sql extension and avoid setup data for unused test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Refine rule order - Speed up test ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1214 from ulysses-you/tests. Closes #1214 261271f0 [ulysses-you] test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
5230d09e1c
|
[KYUUBI #1211] Reopen test udf in JDBCTests
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve test coverage. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1211 from ulysses-you/minikube. Closes #1211 f02edeca [ulysses-you] test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
95c74cdefc |
[KYUUBI #1205] Support to show more engine submission failure log for issue tracking
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Customers might meet exception when launching a spark engine, especially for beginner, now kyuubi only show the last line of engine log and it is difficult to find the root cause in customer side. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1205 from turboFei/show_more_engine_log. Closes #1205 72002dd6 [fwang12] address comments a74fa1de [fwang12] update docs fd967371 [fwang12] refactor 434a093d [fwang12] update docs 880c8cc2 [fwang12] show more engine failure log for debug Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
h
|
fdff2b6240
|
[KYUUBI #1085][FOLLOWUP] Fix-Enforce maxOutputRows for aggregate with having statement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support `Union` case as below ``` SELECT * FROM t1 UNION [ALL] SELECT * FROM t2 ``` Support `Distinct` case as below ``` SELECT DISTINCT * FROM t1 ``` Fix The bug of watchdog with maxOutputRows happens in this situation as below Having and Sort ``` SELECT c1, COUNT(c2) AS cnt FROM t1 GROUP BY c1 HAVING cnt > 0 [ORDER BY c1, [c2 ...]] ``` It throws Exception as ``` org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate java.lang.ClassCastException: org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolvedAggregateFilter$1(Analyzer.scala:2451) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveFilterCondInAggregate(Analyzer.scala:2460) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveHaving(Analyzer.scala:2496) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2353) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$3(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:74) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$1(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:221) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp$(AnalysisHelper.scala:84) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2344) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216) at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) at scala.collection.immutable.List.foldLeft(List.scala:91) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205) at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173) at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.spark.sql.test.SQLTestUtilsBase.$anonfun$sql$1(SQLTestUtils.scala:231) at org.apache.spark.sql.KyuubiExtensionSuite.$anonfun$new$55(KyuubiExtensionSuite.scala:1331) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38) at ``` Reference related issue: https://issues.apache.org/jira/browse/SPARK-31519 Spark SQL Ananlyzer transform aggregate with having to ``` Filter +- Aggregate ``` Solution: 1. Skip the aggregate with havingCondition 2. Match Filter for Adding Limit ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-21 下午8 35 16" src="https://user-images.githubusercontent.com/635169/134171308-2842f0d4-acfa-4817-a03c-a7ef5e38df12.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1129 from i7xh/fixAggWithHavingInMaxOutput. Closes #1085 7577f4d3 [h] update 5955c89e [h] update 384b2333 [h] fix issue 5b0af156 [h] update 46327fc6 [h] Fix issue 6119a039 [h] fix issue a7b87dd7 [h] Fix issue 2570444e [h] BugFix: Aggregate with having statement Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
1b1b898123
|
[KYUUBI #1191] [SUB-TASK] KyuubiHiveDriver should implements Driver directly
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Before this change, `KyuubiHiveDriver extends HiveDriver`, even we don't expose `HiveDriver` to service loader, when `KyuubiHiveDriver` is loaded, the parent class `HiveDriver` will be loaded and initialized first, which is not expected. ``` static { try { DriverManager.registerDriver(new HiveDriver()); } catch (SQLException var1) { var1.printStackTrace(); } manifestAttributes = null; } ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1191 from pan3793/kyuubi-hive-jdbc. Closes #1191 e5276667 [Cheng Pan] KyuubiHiveDriver should implements Driver directly Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
9e6eb086b1
|
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1203 from pan3793/pom. Closes #1203 8e6b8f9b [Cheng Pan] fix d31fd198 [Cheng Pan] fix 9da82843 [Cheng Pan] fix 6d150de9 [Cheng Pan] fix 9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
simon
|
4f8f4d9c6b
|
[KYUUBI #1190] Enhance error log detection
### Why are the changes needed? When line2 is read, it does not contain `at` or `cause by`, so all of the following messages are discarded. #1190 ``` 2021-10-09 09:35:14.528 ERROR spark.SparkSQLEngine: Failed to instantiate SparkSession: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113) at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:225) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:137) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1200 from simon824/fixut. Closes #1190 d239df0e [simon] codestyle 83cb7d83 [simon] fixut 2d399883 [simon] Merge remote-tracking branch 'upstream/master' 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Authored-by: simon <zhangshiming@cvte.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
afb46ec152
|
[KYUUBI #1192] Refine zorder utils toByte
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix some issue: 1. int/long type miss update highest byte of byte array 2. short/int/long same value may have different byte array 3. the mask value of long is error. (1L << 31 instead of 1 << 31) 4. the default value of string should be Long.Max since we pad it to 8 bytes so we can keep the ordering of null value (null last) 5. boolean to byte doesn't need 4 bytes, 1 byte is enough ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1192 from ulysses-you/improve-zorder. Closes #1192 7a0cba28 [ulysses-you] byte 2271b427 [ulysses-you] refine b13d2652 [ulysses-you] refine zorder Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
95b884e9b7
|
[KYUUBI #1199] Refine docker image file
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> * Use apache/kyuubi docker hub repo instead of personal repo * change workflow env to ubuntu-20.04 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1199 from ulysses-you/refine-docker. Closes #1199 2ec73286 [ulysses-you] always d6ed71b5 [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
d177b0d7d6
|
[KYUUBI #937] Add push docker image workflow
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Every commit for master branch, we can build and push docker image automatically to docker hub. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1195 from ulysses-you/docker. Closes #937 8f8deca7 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sychen
|
464fdf456b
|
[KYUUBI #1176][FOLLOWUP] change configuration(ha.zookeeper.acl.engine.enabled) version
https://github.com/apache/incubator-kyuubi/pull/1177 ### _Why are the changes needed?_ change configuration(ha.zookeeper.acl.engine.enabled) version to `1.3.1` ### _How was this patch tested?_ - [] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [] Add screenshots for manual tests if appropriate - [] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1193 from cxzl25/KYUUBI-1176-followup. Closes #1176 72753ae5 [sychen] change config version Authored-by: sychen <sychen@trip.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhang1002
|
ca9ecd6e34
|
[KYUUBI #1066] Automatically create engine event log root dir when it does not exist
bugfix: #1066 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1183 from zhang1002/auto-create-log-root. Closes #1066 d686e33c [zhang1002] 1. change logRoot to engineLogRoot 2. add some judge for mkDir fb6987f5 [zhang1002] create log root automatically c319127e [张宇翔] test Lead-authored-by: zhang1002 <zhang1002@126.com> Co-authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
simon
|
2ae6acda65
|
[KYUUBI #1039] Kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED #1039 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1119 from simon824/killyarnjob. Closes #1039 7face4db [simon] get KYUUBI_HOME da8c3ec4 [Simon] Merge branch 'apache:master' into killyarnjob 95ae8f59 [simon] add appId 8a6ddcd1 [simon] kill application by script e0cd2af8 [simon] kill application by script d15d38c5 [simon] kill application by script 1fd1373b [simon] mv sparklauncher to sparkProcessBuilder 5e806426 [simon] fix codestyle a9cc4505 [simon] fix option No value Exception 9c7ca2f8 [simon] fix option No value Exception bbfe8e25 [simon] add set sparkHome 04f23c8a [simon] fix codestyle 75599233 [simon] add spark-launcher dep ef4b2706 [simon] implement by sparkLauncher 59c25b7c [simon] kill yarn application by restful api 49921a48 [simon] fix ut a31d8f6a [simon] fix ut 1abc6665 [simon] rename killApplication 43a95c1d [simon] Merge branch 'master' into killyarnjob 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 9eaeb16d [simon] fix ut 64ee1b11 [simon] code style 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' 139f3b79 [Simon] Merge branch 'apache:master' into killyarnjob 1a52401d [simon] #1039 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sychen
|
6420a7eb04
|
[KYUUBI #1174] Service discovery supports kerberos
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1174 Now kyuubi servcie uses kerberos authentication and registers with the zookeeper to the node information, but the client cannot connect to the service. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1175 from cxzl25/KYUUBI-1174. Closes #1174 386c7d57 [sychen] add comment dc37a158 [sychen] respect user's setting 97401a96 [sychen] update settings.md fd8cceda [sychen] Service discovery supports kerberos Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ac3e6c36ac
|
[KYUUBI #1149] [SUB-TASK][BUILD] Create shaded jar for kyuubi-hive-jdbc module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Create a shaded jar for `kyuubi-hive-jdbc` module, which relocate every classes except `org.apache.hadoop`, `org.slf4j`. Test passed on DataGrip w/o Kerberos. ``` ➜ kyuubi-hive-jdbc ll kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 12M Sep 24 16:46 kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar ``` ``` ➜ kyuubi-hive-jdbc tree -dL 5 . ├── META-INF │ └── services └── org └── apache ├── hadoop │ ├── fs │ ├── mapred │ └── security │ └── token └── kyuubi ├── jdbc │ └── hive └── shade ├── com └── org ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Pass #1152 Closes #1149 from pan3793/jdbc-shade. Closes #1149 448eaa6a [Cheng Pan] [BUILD] Create shaded jar for kyuubi-hive-jdbc module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
d44ccc3644
|
[KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine
### _Why are the changes needed?_ When communicating with NameNode HA, private tokens are created from original token:  In above image, "ha-hdfs:mycluster" is the original token. "ha-hdfs://192.168.56.101:8020" and "ha-hdfs://192.168.56.102:8020" are private tokens. `KyuubiHadoopUtils.getCredentialsInternal` was supposed to extract these private tokens at Kyuubi server side and send them to SQL engine. But in fact, SQL engine side private tokens are automatically created when adding original token, inside `org.apache.hadoop.security.Credentials#addToken`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate SequenceNumber of private tokens always equal to original token:   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1173 from zhouyifan279/#1008. Closes #1008 891dc2b6 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
d332534325
|
[KYUUBI #1176] InvalidACL appears in the engine when zookeeper acl is turned on
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1176 When `kyuubi.ha.zookeeper.acl.enabled=true`, both service and engine will use zookeeper acl to create znode, but engine has no keytab information and cannot write information to zookeeper, throwing an exception. ```java Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi_USER/XXXX at org.apache.zookeeper.KeeperException.create(KeeperException.java:124) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740) ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1177 from cxzl25/KYUUBI-1176. Closes #1176 ecc08fa7 [sychen] fix engine acl 0b7cc2ec [sychen] fix InvalidACL Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
d1e7f0cc20
|
[KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Correct typo. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1186 from pan3793/typo. Closes #1143 fd53e94a [Cheng Pan] [KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
0e68678f20
|
[KYUUBI #1184] Support ending with .* to restrict and ingore session config
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Now, restricted and ingored keys must be complete. We can support eding with '.*' to restricted a list of keys. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1185 from hddong/support-perfix-ignore. Closes #1184 dade3301 [hongdongdong] reset c58b0aa6 [hongdongdong] [KYUUBI#1184] Support ending with .* to restrict and ingore session config Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
hzxiongyinke
|
0ecf8fbc7e
|
[KYUUBI #939] z-order performance_test
### What is the purpose of the pull request pr for KYUUBI #939:Add Z-Order extensions to optimize table with zorder.Z-order is a technique that allows you to map multidimensional data to a single dimension. We did a performance test for this test ,we used aliyun Databricks Delta test case https://help.aliyun.com/document_detail/168137.html?spm=a2c4g.11186623.6.563.10d758ccclYtVb Prepare data for the three scenarios: 1. 10 billion data and 2 hundred files（parquet files）: for big file(1G) 2. 10 billion data and 1 thousand files（parquet files）: for medium file(200m) 3. one billion data and 10 hundred files（parquet files）: for smaller file(200k) test env： spark-3.1.2 hadoop-2.7.2 kyubbi-1.4.0 test step： Step1: create hive tables ```scala spark.sql(s"drop database if exists $dbName cascade") spark.sql(s"create database if not exists $dbName") spark.sql(s"use $dbName") spark.sql(s"create table $connRandomParquet (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"create table $connZorderOnlyIp (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"create table $connZorder (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"show tables").show(false) ``` Step2： prepare data for parquet table with three scenarios we use the following code ```scala def randomIPv4(r: Random) = Seq.fill(4)(r.nextInt(256)).mkString(".") def randomPort(r: Random) = r.nextInt(65536) def randomConnRecord(r: Random) = ConnRecord( src_ip = randomIPv4(r), src_port = randomPort(r), dst_ip = randomIPv4(r), dst_port = randomPort(r)) ``` Step3： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers . execute 'OPTIMIZE conn_zorder_only_ip ZORDER BY src_ip, dst_ip;' by kyuubi. Step4： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers . execute 'OPTIMIZE conn_zorder ZORDER BY src_ip, src_port, dst_ip, dst_port;' by kyuubi. --------------------- # benchmark result by querying the tables before and after optimization, we find that **10 billion data and 200 files and Query resource:200 core 600G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 1.2 G | 10,000,000,000 | 27.554 s | 0.0% | | conn_zorder_only_ip | 890 M | 43,170,600 | 2.459 s | 99.568% | | conn_zorder | 890 M | 54,841,302 | 3.185 s | 99.451% | **10 billion data and 2000 files and Query resource:200 core 600G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 234.8 M | 10,000,000,000 | 27.031 s | 0.0% | | conn_zorder_only_ip | 173.9 M | 43,170,600 | 2.668 s | 99.568% | | conn_zorder | 174.0 M | 54,841,302 | 3.207 s | 99.451% | **1 billion data and 10000 files and Query resource:10 core 40G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 2.7 M | 1,000,000,000 | 76.772 s | 0.0% | | conn_zorder_only_ip | 2.1 M | 406,572 | 3.963 s | 99.959% | | conn_zorder | 2.2 M | 387,942 | 3.621s | 99.961% | Closes #1178 from hzxiongyinke/zorder_performance_test. Closes #939 369a9b41 [hzxiongyinke] remove set spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension; 8c8ae458 [hzxiongyinke] add index z-order-benchmark 66bd20fd [hzxiongyinke] change tables to three scenarios cc80f4e7 [hzxiongyinke] add License 70c29daa [hzxiongyinke] z-order performance_test 6f1892be [hzxiongyinke] Merge pull request #1 from apache/master Lead-authored-by: hzxiongyinke <1062376716@qq.com> Co-authored-by: hzxiongyinke <75288351+hzxiongyinke@users.noreply.github.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
4f74aba50e |
[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support to config operation log root dir with kyuubi configuration both for server and engine sides. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1160 from turboFei/support_config_operationlog_dir. Closes #1160 ed424f46 [fwang12] fix ut 5edfd4ae [fwang12] refactor ut b7026b2f [fwang12] add ut 6ff4d891 [fwang12] save 30cd5952 [fwang12] fix ut f058619a [fwang12] fix ut 96eb020d [fwang12] refactor ae0ded96 [fwang12] set config value for testing 83d932f4 [fwang12] exclude 33e3e214 [fwang12] update 6f030cba [fwang12] fix 77ec9ca2 [fwang12] save 711ececf [fwang12] Fix code style cfa3bfac [fwang12] refactor b299f8cf [fwang12] complete ut 26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Kent Yao
|
be0b9c1e69
|
[KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Make EngineLoggingService tolerant for creating if any error occurs, and then the main code goes. 2. Separate EngineLoggingService From SparkSQLEngine, and we will try to start EngineLoggingService before SparkSQLEngine to capture all the lifecycle statuses. On the other hand, add a shutdown hook that executed after SparkSQLEngine.stop for EngineLoggingService.stop 3. try-catch for every step of SparkSQLEngine bootstrapping for creating better diagnosis messages. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1171 from yaooqinn/1169. Closes #1169 3a661555 [Kent Yao] ci 733bd1ed [Kent Yao] address comments 36ba1661 [Kent Yao] [KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
191d30e2ac
|
[KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Minor change, update `ZorderCoreBenchmark` doc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1172 from pan3793/t. Closes #1167 8372a3a7 [Cheng Pan] nit 1b6ee9da [Cheng Pan] typo bd1817a7 [Cheng Pan] [KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
413ba2eb24
|
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine
### _Why are the changes needed?_ Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`. ``` 21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds 21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before 21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` 21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds 21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1168 from zhouyifan279/#1009. Closes #1009 fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine 3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
d45421b37e
|
[KYUUBI #1008][FOLLOWUP] Can not obtain hadoop file system delegation token
### _Why are the changes needed?_
Kyuubi server throws exception when obtaining hadoop file system delegation token
```
21/09/27 19:53:07 WARN credentials.HadoopCredentialsManager: Failed to update tokens for morty, try again in 3000 ms
java.io.IOException: DestHost:destPort kdc.hadoop.com:9000 , LocalHost:localPort client.hadoop.com/192.168.56.1:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal n
ame
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:836)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:811)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:1081)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:703)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1845)
at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:66)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:64)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:105)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:57)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:884)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
at org.apache.hadoop.ipc.Client.call(Client.java:1452)
... 43 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
... 46 more
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1166 from zhouyifan279/#1008.
Closes #1008
43dbf605 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] Can not obtain hadoop fs delegation token
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
|
||
|
ulysses-you
|
627d271fc6
|
[KYUUBI #1167] Add zorder core benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add zorder core benchmark, so we can get the perf number of zorder. This PR adds two column type `int` and `long` benchmark: - 2 int cols - 3 int cols - 4 int cols - 2 long cols - 3 long cols - 4 long cols ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1167 from ulysses-you/benchmark. Closes #1167 27ecf51f [ulysses-you] Add zorder core benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
hongdongdong
|
55d5b109c0
|
[KYUUBI #1164] Grep WARN in dependency.sh
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When MVN dependency has WARN log, ./build/dependency.sh --replace will exit with execption: ``` awk: trying to access out of range field -1 input record number 1, file source line number 2 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1165 from hddong/fix-build-dependency. Closes #1164 1f2904dc [hongdongdong] [KYUUBI-1164] Grep WARN in dependency.sh Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
fb6ce8fcc6
|
[KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> It's pre-step of #1152, otherwise will cause circular dependence. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1151 from pan3793/move-jdbc-test. Closes #1131 e65f9979 [Cheng Pan] nit f2e66535 [Cheng Pan] [KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
114ec9a046
|
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS
### _Why are the changes needed?_ Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1162 from zhouyifan279/KYUUBI#1161. Closes #1009 f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
85a7c63418
|
[KYUUBI #1163] [BUILD] Disable parallelization in dist
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Seems that `maven-shade-plugin` has some issue with parallelization, #1149 stuck w/o this change. For refererence: https://github.com/apache/spark/pull/23061 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1163 from pan3793/dist. Closes #1163 7b322e5c [Cheng Pan] [BUILD] Disable parallelization in dist Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
d829bae526
|
[KYUUBI #939][FOLLOWUP][TEST] Add string column test for Z-Order
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Added string column test for zorder. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1157 from zhaomin1423/zorder_test. Closes #939 59543776 [Min Zhao] add string column test for zorder Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
c8d9d8be21
|
[KYUUBI #1155] [BUILD] Support passthrough env var CI into build container
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Continuous integration(aka. CI) services like GitHub Actions, Travis always provide an environment variable `CI` in runners, and we detect this variable to run some specific actions. In [KYUUBI #742], we add a `CI` env var detection in `build/mvn` to suppress noisy maven logs in CI scenes. This PR introduces a new build arg `CI` in `docker/Dockerfile`, which is intent to passthrough env var CI into the build container to suppress noisy maven logs in GitHub Action and Travis. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1155 from pan3793/ci. Closes #1155 0f66b37e [Cheng Pan] typo e1b9d35e [Cheng Pan] nit d78bb563 [Cheng Pan] update workflow 4209ae67 [Cheng Pan] Update comments ffc3a1e1 [Cheng Pan] nit 761c9e6c [Cheng Pan] Update usage 2f74b977 [Cheng Pan] Add build arg CI 509da2aa [Cheng Pan] [BUILD] Suppress maven log in Minikube Integration Test Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Brian Yue
|
683675be92
|
[KYUUBI #1146] Move KinitAuxiliaryService to kyuubi-server module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1146. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1156 from byyue/feature-1146. Closes #1146 6b677723 [Brian Yue] Move KinitAuxiliaryService to kyuubi-server module Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
21744a99bd
|
[KYUUBI #1158] [INFRA] Update ANNOUNCE mail template
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Update ANNOUNCE mail template to add more detailed descriptions of Kyuubi. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1158 from pan3793/ann. Closes #1158 46c19975 [Cheng Pan] typo a87321f5 [Cheng Pan] [INFRA] Update ANNOUNCE mail template Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
timothy65535
|
22e6432e4d
|
[KYUUBI #1022] Add basic EngineStatusStore for events
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981 `EngineStatusStore` helps to push events to listener bus `EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides: - stores all elements, and sorted by startTimestamp. - cleanup the last elements when reach a certain threshold. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1023 from timothy65535/ky-1022. Closes #1022 b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
3a1da695e6
|
[KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests
### _Why are the changes needed?_ - Kerberos login module outputs lots of debug level log records to stdout, which is annoying at most of the time. - MiniYARNCluster already outputs most of the log records to unit-tests.log. But some referred classes, such as `GuiceComponentProviderFactory` use java.util.logging.Logging and output log records to stdout. In this PR, we turn off kerberos logging by default and allow users to turn on it by setting java system property `sun.security.krb5.debug`. java.util.logging.Logging is redirected to slf4j using `SLF4JBridgeHandler`. **Note:** MiniDFSCluster outputs a few log records using `System.stdout`/`System.stderr`. ``` Formatting using clusterid: testClusterID isSaslEnabled:false Opened streaming server at /0.0.0.0:1025 Successfully obtained privileged resources (streaming port = ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=1025] ) (http listener port = 9864) Opened info server at /0.0.0.0:9864 ``` It should be acceptable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1150 from zhouyifan279/KYUUBI#1145. Closes #1145 5c91816d [zhouyifan279] [KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests 1a9f6345 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 9877085a [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 8b05cb02 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 435a4715 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
张宇翔
|
103969a042
|
[KYUUBI #1067] Catch the real exception when the log directory is not exist
BUGFIX: #1067 fix: 2021-09-23 16:03:43.248 INFO HiveMetaStore.audit: ugi=zhangyu ip=unknown-ip-addr cmd=get_databases: * 2021-09-23 16:03:43.319 ERROR events.JsonEventLogger: Get log directory /Users/zhangyuxiang/Downloads/kyuubi_test failed java.io.FileNotFoundException: File /Users/zhangyuxiang/Downloads/kyuubi_test does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.kyuubi.events.JsonEventLogger.requireLogRootWritable(JsonEventLogger.scala:79) at org.apache.kyuubi.events.JsonEventLogger.initialize(JsonEventLogger.scala:93) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1136 from zhang1002/branch-1.3.0_checkRightException. Closes #1067 de16c63d [张宇翔] delete some ut 5a29bb80 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 21e856a3 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 873cc9c4 [张宇翔] Merge remote-tracking branch 'upstream/master' d1e7a019 [张宇翔] catch IOException when log directory is not exist bc442d8c [张宇翔] catch IOException when log directory is not exist 0d7418e9 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 41cff19a [张宇翔] Merge remote-tracking branch 'upstream/master' 74b36efb [张宇翔] catch IOException when log directory is not exist def8d1f6 [张宇翔] Merge remote-tracking branch 'upstream/master' 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
eeb35f31dd
|
[KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR is based on discussion https://www.mail-archive.com/devkyuubi.apache.org/msg00977.html ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1147 from pan3793/jdbc-rename. Closes #1131 500edae6 [Cheng Pan] Add javadoc bd9f94c2 [Cheng Pan] Add deprecated KyuubiDriver test d9688d8b [Cheng Pan] [KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
