apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,227 Commits 23 Branches 0 Tags 65 MiB
bf629291bf
Commit Graph

1227 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kent Yao
bf629291bf
[KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

The detail locates #1206

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1207 from yaooqinn/1206.

Closes #1206

99e324a9 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
f41fd944 [Kent Yao] Merge branch 'master' into 1206
a7cb80da [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
6ecef063 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
d7ff7583 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
7b3b2f53 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-12 10:51:44 +08:00
ulysses-you
6160b9de30
[KYUUBI #1214] Refine sql extension and avoid setup data for unused test 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Refine rule order
- Speed up test

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1214 from ulysses-you/tests.

Closes #1214

261271f0 [ulysses-you] test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-12 10:50:03 +08:00
ulysses-you
5230d09e1c
[KYUUBI #1211] Reopen test udf in JDBCTests 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Improve test coverage.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1211 from ulysses-you/minikube.

Closes #1211

f02edeca [ulysses-you] test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-11 20:03:15 +08:00
fwang12
95c74cdefc [KYUUBI #1205] Support to show more engine submission failure log for issue tracking 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Customers might meet exception when launching a spark engine, especially for beginner, now kyuubi only show the last line of engine log and it is difficult to find the root cause in customer side.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1205 from turboFei/show_more_engine_log.

Closes #1205

72002dd6 [fwang12] address comments
a74fa1de [fwang12] update docs
fd967371 [fwang12] refactor
434a093d [fwang12] update docs
880c8cc2 [fwang12] show more engine failure log for debug

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-10-11 19:26:36 +08:00
h
fdff2b6240
[KYUUBI #1085][FOLLOWUP] Fix-Enforce maxOutputRows for aggregate with having statement 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support `Union` case as below
```
SELECT * FROM t1
UNION [ALL]
SELECT * FROM t2
```

Support `Distinct` case as below
```
SELECT DISTINCT * FROM t1
```

Fix The bug of watchdog with maxOutputRows happens in this situation as below

Having and Sort
```
SELECT c1, COUNT(c2) AS cnt
FROM t1
GROUP BY c1
HAVING cnt > 0
[ORDER BY c1, [c2 ...]]
```

It throws Exception as
```
org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate
java.lang.ClassCastException: org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolvedAggregateFilter$1(Analyzer.scala:2451)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveFilterCondInAggregate(Analyzer.scala:2460)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveHaving(Analyzer.scala:2496)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2353)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2345)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$3(AnalysisHelper.scala:90)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:74)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$1(AnalysisHelper.scala:90)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:221)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:86)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp$(AnalysisHelper.scala:84)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2345)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2344)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:91)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63)
	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at org.apache.spark.sql.test.SQLTestUtilsBase.$anonfun$sql$1(SQLTestUtils.scala:231)
	at org.apache.spark.sql.KyuubiExtensionSuite.$anonfun$new$55(KyuubiExtensionSuite.scala:1331)
	at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54)
	at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38)
	at
```
Reference related issue: https://issues.apache.org/jira/browse/SPARK-31519
Spark SQL Ananlyzer transform aggregate with having to
```
Filter
+- Aggregate
```

Solution:

1. Skip the aggregate with havingCondition
2. Match Filter for Adding Limit

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1440" alt="截屏2021-09-21 下午8 35 16" src="https://user-images.githubusercontent.com/635169/134171308-2842f0d4-acfa-4817-a03c-a7ef5e38df12.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1129 from i7xh/fixAggWithHavingInMaxOutput.

Closes #1085

7577f4d3 [h] update
5955c89e [h] update
384b2333 [h] fix issue
5b0af156 [h] update
46327fc6 [h] Fix issue
6119a039 [h] fix issue
a7b87dd7 [h] Fix issue
2570444e [h] BugFix: Aggregate with having statement

Authored-by: h <h@zhihu.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-11 09:49:33 +08:00
Cheng Pan
1b1b898123
[KYUUBI #1191] [SUB-TASK] KyuubiHiveDriver should implements Driver directly 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Before this change, `KyuubiHiveDriver extends HiveDriver`, even we don't expose `HiveDriver` to service loader, when `KyuubiHiveDriver` is loaded, the parent class `HiveDriver` will be loaded and initialized first, which is not expected.

```
static {
    try {
        DriverManager.registerDriver(new HiveDriver());
    } catch (SQLException var1) {
        var1.printStackTrace();
    }
    manifestAttributes = null;
}
```

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1191 from pan3793/kyuubi-hive-jdbc.

Closes #1191

e5276667 [Cheng Pan] KyuubiHiveDriver should implements Driver directly

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-09 20:01:25 +08:00
Cheng Pan
9e6eb086b1
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
#1131

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1203 from pan3793/pom.

Closes #1203

8e6b8f9b [Cheng Pan] fix
d31fd198 [Cheng Pan] fix
9da82843 [Cheng Pan] fix
6d150de9 [Cheng Pan] fix
9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-09 19:24:05 +08:00
simon
4f8f4d9c6b
[KYUUBI #1190] Enhance error log detection 
### Why are the changes needed?
When line2 is read, it does not contain `at` or `cause by`, so all of the following messages are discarded.

#1190

```
2021-10-09 09:35:14.528 ERROR spark.SparkSQLEngine: Failed to instantiate SparkSession: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:225)
	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:137)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1200 from simon824/fixut.

Closes #1190

d239df0e [simon] codestyle
83cb7d83 [simon] fixut
2d399883 [simon] Merge remote-tracking branch 'upstream/master'
3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master'
5b905dbd [simon] Merge remote-tracking branch 'upstream/master'

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-09 14:38:55 +08:00
ulysses-you
afb46ec152
[KYUUBI #1192] Refine zorder utils toByte 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix some issue:
1. int/long type miss update highest byte of byte array
2. short/int/long same value may have different byte array
3. the mask value of long is error. (1L << 31 instead of 1 << 31)
4. the default value of string should be Long.Max since we pad it to 8 bytes so we can keep the ordering of null value (null last)
5. boolean to byte doesn't need 4 bytes, 1 byte is enough

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1192 from ulysses-you/improve-zorder.

Closes #1192

7a0cba28 [ulysses-you] byte
2271b427 [ulysses-you] refine
b13d2652 [ulysses-you] refine zorder

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-09 14:23:24 +08:00
ulysses-you
95b884e9b7
[KYUUBI #1199] Refine docker image file 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
* Use apache/kyuubi docker hub repo instead of personal repo
* change workflow env to ubuntu-20.04

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1199 from ulysses-you/refine-docker.

Closes #1199

2ec73286 [ulysses-you] always
d6ed71b5 [ulysses-you] refine

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-09 14:22:35 +08:00
ulysses-you
d177b0d7d6
[KYUUBI #937] Add push docker image workflow 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Every commit for master branch, we can build and push docker image automatically to docker hub.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1195 from ulysses-you/docker.

Closes #937

8f8deca7 [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-09 10:03:55 +08:00
sychen
464fdf456b
[KYUUBI #1176][FOLLOWUP] change configuration(ha.zookeeper.acl.engine.enabled) version 
https://github.com/apache/incubator-kyuubi/pull/1177
### _Why are the changes needed?_
change configuration(ha.zookeeper.acl.engine.enabled) version to `1.3.1`

### _How was this patch tested?_
- [] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [] Add screenshots for manual tests if appropriate

- [] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1193 from cxzl25/KYUUBI-1176-followup.

Closes #1176

72753ae5 [sychen] change config version

Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-08 16:56:15 +08:00
zhang1002
ca9ecd6e34
[KYUUBI #1066] Automatically create engine event log root dir when it does not exist 
bugfix:
#1066

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1183 from zhang1002/auto-create-log-root.

Closes #1066

d686e33c [zhang1002] 1. change logRoot to engineLogRoot 2. add some judge for mkDir
fb6987f5 [zhang1002] create log root automatically
c319127e [张宇翔] test

Lead-authored-by: zhang1002 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-08 16:04:48 +08:00
simon
2ae6acda65
[KYUUBI #1039] Kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED 
kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
#1039

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1119 from simon824/killyarnjob.

Closes #1039

7face4db [simon] get KYUUBI_HOME
da8c3ec4 [Simon] Merge branch 'apache:master' into killyarnjob
95ae8f59 [simon] add appId
8a6ddcd1 [simon] kill application by script
e0cd2af8 [simon] kill application by script
d15d38c5 [simon] kill application by script
1fd1373b [simon] mv sparklauncher to sparkProcessBuilder
5e806426 [simon] fix codestyle
a9cc4505 [simon] fix option No value Exception
9c7ca2f8 [simon] fix option No value Exception
bbfe8e25 [simon] add set sparkHome
04f23c8a [simon] fix codestyle
75599233 [simon] add spark-launcher dep
ef4b2706 [simon] implement by sparkLauncher
59c25b7c [simon] kill yarn application by restful api
49921a48 [simon] fix ut
a31d8f6a [simon] fix ut
1abc6665 [simon] rename killApplication
43a95c1d [simon] Merge branch 'master' into killyarnjob
3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master'
9eaeb16d [simon] fix ut
64ee1b11 [simon] code style
5b905dbd [simon] Merge remote-tracking branch 'upstream/master'
139f3b79 [Simon] Merge branch 'apache:master' into killyarnjob
1a52401d [simon] #1039

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-10-08 16:01:08 +08:00
sychen
6420a7eb04
[KYUUBI #1174] Service discovery supports kerberos 
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/1174
Now kyuubi servcie uses kerberos authentication and registers with the zookeeper to the node information, but the client cannot connect to the service.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1175 from cxzl25/KYUUBI-1174.

Closes #1174

386c7d57 [sychen] add comment
dc37a158 [sychen] respect user's setting
97401a96 [sychen] update settings.md
fd8cceda [sychen] Service discovery supports kerberos

Authored-by: sychen <sychen@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-08 13:50:22 +08:00
Cheng Pan
ac3e6c36ac
[KYUUBI #1149] [SUB-TASK][BUILD] Create shaded jar for kyuubi-hive-jdbc module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Create a shaded jar for `kyuubi-hive-jdbc` module, which relocate every classes except `org.apache.hadoop`, `org.slf4j`.

Test passed on DataGrip w/o Kerberos.

```
➜  kyuubi-hive-jdbc ll kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    12M Sep 24 16:46 kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar
```
```
➜  kyuubi-hive-jdbc tree -dL 5
.
├── META-INF
│   └── services
└── org
    └── apache
        ├── hadoop
        │   ├── fs
        │   ├── mapred
        │   └── security
        │       └── token
        └── kyuubi
            ├── jdbc
            │   └── hive
            └── shade
                ├── com
                └── org
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/26535726/134647840-23bafa82-deb5-44cb-acf6-bda4df108f3c.png)

![image](https://user-images.githubusercontent.com/26535726/134647364-54cec223-f67f-434e-8501-02d3afc34382.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

- [x] Pass #1152

Closes #1149 from pan3793/jdbc-shade.

Closes #1149

448eaa6a [Cheng Pan] [BUILD] Create shaded jar for kyuubi-hive-jdbc module

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-08 10:35:14 +08:00
zhouyifan279
d44ccc3644
[KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine 
### _Why are the changes needed?_
When communicating with NameNode HA, private tokens are created from original token:
![image](https://user-images.githubusercontent.com/88070094/135081188-5311d161-eb74-4be7-bd6f-b595c512f575.png)
In above image, "ha-hdfs:mycluster" is the original token. "ha-hdfs://192.168.56.101:8020" and "ha-hdfs://192.168.56.102:8020" are private tokens.

`KyuubiHadoopUtils.getCredentialsInternal` was supposed to extract these private tokens at Kyuubi server side and send them to SQL engine.

But in fact, SQL engine side private tokens are automatically created when adding original token,
 inside `org.apache.hadoop.security.Credentials#addToken`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
SequenceNumber of private tokens always equal to original token:
![image](https://user-images.githubusercontent.com/88070094/135083412-5e5f3821-b4bc-4b20-a702-e2455c60044e.png)
![image](https://user-images.githubusercontent.com/88070094/135083421-654a0e62-9f85-47a0-b136-ee6ad7c32a05.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1173 from zhouyifan279/#1008.

Closes #1008

891dc2b6 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-08 10:32:51 +08:00
sychen
d332534325
[KYUUBI #1176] InvalidACL appears in the engine when zookeeper acl is turned on 
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/1176
When `kyuubi.ha.zookeeper.acl.enabled=true`, both service and engine will use zookeeper acl to create znode, but engine has no keytab information and cannot write information to zookeeper, throwing an exception.

```java
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi_USER/XXXX
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:124)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
	at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
```

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1177 from cxzl25/KYUUBI-1176.

Closes #1176

ecc08fa7 [sychen] fix engine acl
0b7cc2ec [sychen] fix InvalidACL

Authored-by: sychen <sychen@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-10-08 10:08:30 +08:00
Cheng Pan
d1e7f0cc20
[KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Correct typo.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1186 from pan3793/typo.

Closes #1143

fd53e94a [Cheng Pan] [KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-10-01 01:23:34 +08:00
hongdongdong
0e68678f20
[KYUUBI #1184] Support ending with .* to restrict and ingore session config 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Now, restricted and ingored keys must be complete. We can support eding with '.*' to restricted a list of keys.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1185 from hddong/support-perfix-ignore.

Closes #1184

dade3301 [hongdongdong] reset
c58b0aa6 [hongdongdong] [KYUUBI#1184] Support ending with .* to restrict and ingore session config

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-30 17:53:44 +08:00
hzxiongyinke
0ecf8fbc7e
[KYUUBI #939] z-order performance_test 
### What is the purpose of the pull request

pr for KYUUBI #939:Add Z-Order extensions to optimize table with zorder.Z-order is a technique that allows you to map multidimensional data to a single dimension. We did a performance test

for this test ,we used aliyun Databricks Delta test case
https://help.aliyun.com/document_detail/168137.html?spm=a2c4g.11186623.6.563.10d758ccclYtVb

Prepare data for the three scenarios:

1. 10 billion data and 2 hundred files（parquet files）: for big file(1G)
2. 10 billion data and 1 thousand files（parquet files）: for medium file(200m)
3. one billion data and 10 hundred files（parquet files）: for smaller file(200k)

test env：
spark-3.1.2
hadoop-2.7.2
kyubbi-1.4.0

test step：

Step1: create hive tables

```scala
spark.sql(s"drop database if exists $dbName cascade")
spark.sql(s"create database if not exists $dbName")
spark.sql(s"use $dbName")
spark.sql(s"create table $connRandomParquet (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet")
spark.sql(s"create table $connZorderOnlyIp (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet")
spark.sql(s"create table $connZorder (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet")
spark.sql(s"show tables").show(false)
```

Step2： prepare data for parquet table with three scenarios
we use the following code

```scala
def randomIPv4(r: Random) = Seq.fill(4)(r.nextInt(256)).mkString(".")
def randomPort(r: Random) = r.nextInt(65536)

def randomConnRecord(r: Random) = ConnRecord(
  src_ip = randomIPv4(r), src_port = randomPort(r),
  dst_ip = randomIPv4(r), dst_port = randomPort(r))
```

Step3： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers .
	execute  'OPTIMIZE conn_zorder_only_ip ZORDER BY src_ip, dst_ip;' by kyuubi.

Step4： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers .
	execute  'OPTIMIZE conn_zorder ZORDER BY src_ip, src_port, dst_ip, dst_port;' by kyuubi.

---------------------
# benchmark result
by querying the tables before and after optimization, we find that

**10 billion data and 200 files and Query resource:200 core 600G memory**

| Table               | Average File Size | Scan row count | Average query time | row count Skipping ratio |
| ------------------- | ----------------- | -------------- | ------------------ | ------------------------ |
| conn_random_parquet | 1.2 G             | 10,000,000,000 | 27.554 s           | 0.0%                     |
| conn_zorder_only_ip | 890 M             | 43,170,600     | 2.459 s            | 99.568%                  |
| conn_zorder         | 890 M             | 54,841,302     | 3.185 s            | 99.451%                  |

**10 billion data and 2000 files and Query resource:200 core 600G memory**

| Table               | Average File Size | Scan row count | Average query time | row count Skipping ratio |
| ------------------- | ----------------- | -------------- | ------------------ | ------------------------ |
| conn_random_parquet | 234.8 M           | 10,000,000,000 | 27.031 s           | 0.0%                     |
| conn_zorder_only_ip | 173.9 M           | 43,170,600     | 2.668 s            | 99.568%                  |
| conn_zorder         | 174.0 M           | 54,841,302     | 3.207 s            | 99.451%                  |

**1 billion data and 10000 files and Query resource:10 core 40G memory**

| Table               | Average File Size | Scan row count | Average query time | row count Skipping ratio |
| ------------------- | ----------------- | -------------- | ------------------ | ------------------------ |
| conn_random_parquet | 2.7 M             | 1,000,000,000  | 76.772 s           | 0.0%                     |
| conn_zorder_only_ip | 2.1 M             | 406,572        | 3.963 s            | 99.959%                  |
| conn_zorder         | 2.2 M             | 387,942        | 3.621s             | 99.961%                  |

Closes #1178 from hzxiongyinke/zorder_performance_test.

Closes #939

369a9b41 [hzxiongyinke] remove set spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension;
8c8ae458 [hzxiongyinke] add index z-order-benchmark
66bd20fd [hzxiongyinke] change tables to three scenarios
cc80f4e7 [hzxiongyinke] add License
70c29daa [hzxiongyinke] z-order performance_test
6f1892be [hzxiongyinke] Merge pull request #1 from apache/master

Lead-authored-by: hzxiongyinke <1062376716@qq.com>
Co-authored-by: hzxiongyinke <75288351+hzxiongyinke@users.noreply.github.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-29 17:51:37 +08:00
fwang12
4f74aba50e [KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support to config operation log root dir with kyuubi configuration both for server and engine sides.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1160 from turboFei/support_config_operationlog_dir.

Closes #1160

ed424f46 [fwang12] fix ut
5edfd4ae [fwang12] refactor ut
b7026b2f [fwang12] add ut
6ff4d891 [fwang12] save
30cd5952 [fwang12] fix ut
f058619a [fwang12] fix ut
96eb020d [fwang12] refactor
ae0ded96 [fwang12] set config value for testing
83d932f4 [fwang12] exclude
33e3e214 [fwang12] update
6f030cba [fwang12] fix
77ec9ca2 [fwang12] save
711ececf [fwang12] Fix code style
cfa3bfac [fwang12] refactor
b299f8cf [fwang12] complete ut
26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-09-29 00:43:23 +08:00
Kent Yao
be0b9c1e69
[KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. Make EngineLoggingService tolerant for creating if any error occurs, and then the main code goes.

2. Separate EngineLoggingService From SparkSQLEngine, and we will try to start EngineLoggingService before SparkSQLEngine to capture all the lifecycle statuses. On the other hand, add a shutdown hook that executed after SparkSQLEngine.stop for  EngineLoggingService.stop

3. try-catch for every step of SparkSQLEngine bootstrapping for  creating better diagnosis messages.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1171 from yaooqinn/1169.

Closes #1169

3a661555 [Kent Yao] ci
733bd1ed [Kent Yao] address comments
36ba1661 [Kent Yao] [KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-28 23:00:16 +08:00
Cheng Pan
191d30e2ac
[KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Minor change, update `ZorderCoreBenchmark` doc.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1172 from pan3793/t.

Closes #1167

8372a3a7 [Cheng Pan] nit
1b6ee9da [Cheng Pan] typo
bd1817a7 [Cheng Pan] [KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-28 11:10:44 +08:00
zhouyifan279
413ba2eb24
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine 
### _Why are the changes needed?_
Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`.

```
21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds
21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before
21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
```
21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds
21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1168 from zhouyifan279/#1009.

Closes #1009

fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine
3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-28 11:10:08 +08:00
zhouyifan279
d45421b37e
[KYUUBI #1008][FOLLOWUP] Can not obtain hadoop file system delegation token 
### _Why are the changes needed?_
Kyuubi server throws exception when obtaining hadoop file system delegation token

```
21/09/27 19:53:07 WARN credentials.HadoopCredentialsManager: Failed to update tokens for morty, try again in 3000 ms
java.io.IOException: DestHost:destPort kdc.hadoop.com:9000 , LocalHost:localPort client.hadoop.com/192.168.56.1:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal n
ame
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:836)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:811)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
        at org.apache.hadoop.ipc.Client.call(Client.java:1508)
        at org.apache.hadoop.ipc.Client.call(Client.java:1405)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:1081)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:703)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1845)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:66)
        at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:64)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:106)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:105)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:57)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:884)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
        at org.apache.hadoop.ipc.Client.call(Client.java:1452)
        ... 43 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
        at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327)
        at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
        ... 46 more
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/88070094/134904531-faac4be0-9fe4-4c4a-8ab8-70aa5ea5fb54.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1166 from zhouyifan279/#1008.

Closes #1008

43dbf605 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] Can not obtain hadoop fs delegation token

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-28 09:38:59 +08:00
ulysses-you
627d271fc6
[KYUUBI #1167] Add zorder core benchmark 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add zorder core benchmark, so we can get the perf number of zorder.

This PR adds two column type `int` and `long` benchmark:
- 2 int cols
- 3 int cols
- 4 int cols
- 2 long cols
- 3 long cols
- 4 long cols

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1167 from ulysses-you/benchmark.

Closes #1167

27ecf51f [ulysses-you] Add zorder core benchmark

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-28 09:24:19 +08:00
hongdongdong
55d5b109c0
[KYUUBI #1164] Grep WARN in dependency.sh 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
When MVN dependency has WARN log, ./build/dependency.sh --replace will exit with execption:
```
awk: trying to access out of range field -1
 input record number 1, file
 source line number 2
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1165 from hddong/fix-build-dependency.

Closes #1164

1f2904dc [hongdongdong] [KYUUBI-1164] Grep WARN in dependency.sh

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-27 18:30:20 +08:00
Cheng Pan
fb6ce8fcc6
[KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
It's pre-step of #1152, otherwise will cause circular dependence.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1151 from pan3793/move-jdbc-test.

Closes #1131

e65f9979 [Cheng Pan] nit
f2e66535 [Cheng Pan] [KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-27 12:37:26 +08:00
zhouyifan279
114ec9a046
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS 
### _Why are the changes needed?_
Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/88070094/134811093-264286cd-06cd-48cd-a9bd-8c8d700b31fa.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1162 from zhouyifan279/KYUUBI#1161.

Closes #1009

f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-27 12:35:28 +08:00
Cheng Pan
85a7c63418
[KYUUBI #1163] [BUILD] Disable parallelization in dist 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Seems that `maven-shade-plugin` has some issue with parallelization, #1149 stuck w/o this change.

For refererence: https://github.com/apache/spark/pull/23061

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1163 from pan3793/dist.

Closes #1163

7b322e5c [Cheng Pan] [BUILD] Disable parallelization in dist

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-27 11:32:39 +08:00
Min Zhao
d829bae526
[KYUUBI #939][FOLLOWUP][TEST] Add string column test for Z-Order 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Added string column test for zorder.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1157 from zhaomin1423/zorder_test.

Closes #939

59543776 [Min Zhao] add string column test for zorder

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-26 22:23:52 +08:00
Cheng Pan
c8d9d8be21
[KYUUBI #1155] [BUILD] Support passthrough env var CI into build container 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Continuous integration(aka. CI) services like GitHub Actions, Travis always provide an environment variable `CI` in runners, and we detect this variable to run some specific actions.

In [KYUUBI #742], we add a `CI` env var detection in `build/mvn` to suppress noisy maven logs in CI scenes.

This PR introduces a new build arg `CI` in `docker/Dockerfile`, which is intent to passthrough env var CI into the build container to suppress noisy maven logs in GitHub Action and Travis.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1155 from pan3793/ci.

Closes #1155

0f66b37e [Cheng Pan] typo
e1b9d35e [Cheng Pan] nit
d78bb563 [Cheng Pan] update workflow
4209ae67 [Cheng Pan] Update comments
ffc3a1e1 [Cheng Pan] nit
761c9e6c [Cheng Pan] Update usage
2f74b977 [Cheng Pan] Add build arg CI
509da2aa [Cheng Pan] [BUILD] Suppress maven log in Minikube Integration Test

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-26 18:04:34 +08:00
Brian Yue
683675be92
[KYUUBI #1146] Move KinitAuxiliaryService to kyuubi-server module 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1146.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1156 from byyue/feature-1146.

Closes #1146

6b677723 [Brian Yue] Move KinitAuxiliaryService to kyuubi-server module

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-26 17:57:25 +08:00
Cheng Pan
21744a99bd
[KYUUBI #1158] [INFRA] Update ANNOUNCE mail template 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Update ANNOUNCE mail template to add more detailed descriptions of Kyuubi.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1158 from pan3793/ann.

Closes #1158

46c19975 [Cheng Pan] typo
a87321f5 [Cheng Pan] [INFRA] Update ANNOUNCE mail template

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-26 17:55:04 +08:00
timothy65535
22e6432e4d
[KYUUBI #1022] Add basic EngineStatusStore for events 
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981

`EngineStatusStore` helps to push events to listener bus

`EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides:
- stores all elements, and sorted by startTimestamp.
- cleanup the last elements when reach a certain threshold.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1023 from timothy65535/ky-1022.

Closes #1022

b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-26 10:40:26 +08:00
zhouyifan279
3a1da695e6
[KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests 
### _Why are the changes needed?_

- Kerberos login module outputs lots of debug level log records to stdout, which is annoying at most of the time.
- MiniYARNCluster already outputs most of the log records to unit-tests.log. But some referred classes, such as `GuiceComponentProviderFactory` use java.util.logging.Logging and output log records to stdout.

In this PR, we turn off kerberos logging by default and allow users to turn on it by setting java system property `sun.security.krb5.debug`. java.util.logging.Logging is redirected to slf4j using `SLF4JBridgeHandler`.

**Note:** MiniDFSCluster outputs a few log records using `System.stdout`/`System.stderr`.
```
Formatting using clusterid: testClusterID
isSaslEnabled:false
Opened streaming server at /0.0.0.0:1025
Successfully obtained privileged resources (streaming port = ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=1025] ) (http listener port = 9864)
Opened info server at /0.0.0.0:9864
```
It should be acceptable.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1150 from zhouyifan279/KYUUBI#1145.

Closes #1145

5c91816d [zhouyifan279] [KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests
1a9f6345 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
9877085a [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
8b05cb02 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
435a4715 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-25 13:42:53 +08:00
张宇翔
103969a042
[KYUUBI #1067] Catch the real exception when the log directory is not exist 
BUGFIX:
#1067

fix:
2021-09-23 16:03:43.248 INFO HiveMetaStore.audit: ugi=zhangyu	ip=unknown-ip-addr	cmd=get_databases: *
2021-09-23 16:03:43.319 ERROR events.JsonEventLogger: Get log directory /Users/zhangyuxiang/Downloads/kyuubi_test failed
java.io.FileNotFoundException: File /Users/zhangyuxiang/Downloads/kyuubi_test does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
	at org.apache.kyuubi.events.JsonEventLogger.requireLogRootWritable(JsonEventLogger.scala:79)
	at org.apache.kyuubi.events.JsonEventLogger.initialize(JsonEventLogger.scala:93)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1136 from zhang1002/branch-1.3.0_checkRightException.

Closes #1067

de16c63d [张宇翔] delete some ut
5a29bb80 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException
21e856a3 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException
873cc9c4 [张宇翔] Merge remote-tracking branch 'upstream/master'
d1e7a019 [张宇翔] catch IOException when log directory is not exist
bc442d8c [张宇翔] catch IOException when log directory is not exist
0d7418e9 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException
41cff19a [张宇翔] Merge remote-tracking branch 'upstream/master'
74b36efb [张宇翔] catch IOException when log directory is not exist
def8d1f6 [张宇翔] Merge remote-tracking branch 'upstream/master'
8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master'
714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master'
af28520b [张宇翔] Merge remote-tracking branch 'upstream/master'
c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master'
abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master'
f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master'
112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master'
c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master'
ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master'
5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master'
8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master'
a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master'
b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master'
4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master'
8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master'
cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master'
0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master'
b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master'
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml

Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-24 20:24:20 +08:00
Cheng Pan
eeb35f31dd
[KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This PR is based on discussion https://www.mail-archive.com/devkyuubi.apache.org/msg00977.html

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1147 from pan3793/jdbc-rename.

Closes #1131

500edae6 [Cheng Pan] Add javadoc
bd9f94c2 [Cheng Pan] Add deprecated KyuubiDriver test
d9688d8b [Cheng Pan] [KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-24 13:36:19 +08:00
Kent Yao
0ee8eff5a0
[KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

After this PR, the service named service discovery will be a child service of frontend service and it's optional.

A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1144 from yaooqinn/1143.

Closes #1143

f2beb721 [Kent Yao] fix test
128cf282 [Kent Yao] nit
37aaff9c [Kent Yao] lift FrontendService trait
add335c4 [Kent Yao] nit
3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
7e5a5cbb [Kent Yao] Merge branch 'master' into 1143
09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-24 13:35:20 +08:00
Cheng Pan
1489d1dbf5
[KYUUBI #1107] [INFRA] Add release mail templates 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1107 from pan3793/template.

Closes #1107

659a2120 [Cheng Pan] Update ANNOUNCE template
971ce74a [Cheng Pan] Update ANNOUNCE template
13465519 [Cheng Pan] Update Links
6a5faf5c [Cheng Pan] typo
0f037c3f [Cheng Pan] Update rat excludes
d8e4cbb9 [Cheng Pan] Add license headers and remove incorrect words
8872a34a [Cheng Pan] Update template
d7f01643 [Cheng Pan] Add mail template
2a170071 [Cheng Pan] Add mail template

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-24 12:32:46 +08:00
ulysses-you
7a3545e748
[KYUUBI #1002][FOLLOWUP] Refine sql classification rule 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Follow up [KYUUBI #1002] [KYUUBI #1035] [KYUUBI #1037]

- correct the config key; we should use `spark.` as the prefix
- change the config to false by default
- correct the class loader; thread context is not safe, spark and hive often change it
- update the log and exception

following is the failed error msg:
```
Caused by: java.io.FileNotFoundException: file:/Users/cathy/Desktop/tmp/spark-3.1.2-bin-hadoop2.7/jars/kyuubi-extension-spark-3-1_2.12-1.4.0-SNAPSHOT.jar!/sql-classification-default.json (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2745)
	at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<init>(KyuubiGetSqlClassification.scala:51)
	at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<clinit>(KyuubiGetSqlClassification.scala)
	... 59 more

```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1140 from ulysses-you/refine.

Closes #1002

f39ef2cf [ulysses-you] address comment
5cb94459 [ulysses-you] nit
de9fd932 [ulysses-you] test
78e8c147 [ulysses-you] default
6f23c63e [ulysses-you] refine

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-24 12:02:30 +08:00
Cheng Pan
997d72a1cd
[KYUUBI #884][FOLLOWUP] Fix catalog in KyuubiDatabaseMetaData#getTables 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The behavior of `KyuubiDatabaseMetaData#getTables` changed by accident in #884[1], this PR restores the method's previous behavior and adds multi-catalog test case to verify it works as expected.

[1] [This](1a8b4ebaae (diff-c540737d1b293b909d07d50a05f67d35d274bc3163d723409135634a575e5e9dR68)) makes KyuubiDatabaseMetaData#getTables method make no sense, just invokes the superclass.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1133 from pan3793/jdbc.

Closes #884

f02327d9 [Cheng Pan] add provided scala-library
3702a08a [Cheng Pan] nit
18045a92 [Cheng Pan] Fix test jars scope
edd1ca8f [Cheng Pan] Address comments
3fecbf5b [Cheng Pan] Clean withKyuubiConf instead of extraConfigs
45e7bd19 [Cheng Pan] nit
f19def37 [Cheng Pan] [KYUUBI #884][FOLLOWUP] Fix catalog in KyuubiDatabaseMetaData#getTables

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-24 10:25:13 +08:00
yanghua
ee8e5a12f1
[KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1142 from yanghua/KYUUBI-1137.

Closes #1137

418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-09-23 20:04:58 +08:00
timothy65535
d93a6c8b5b
[KYUUBI #1141] Remove redundant unit 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Remove redundant unit. `MB`

```
2021-09-23 08:41:08.889 INFO spark.SparkSQLEngine:
    Spark application name: kyuubi_USER_root_6f67c12c-f362-4a07-87d0-7aa7e5f43171
          application ID:  spark-461363e9cc934d1986f7ae4191dc0410
          application web UI: http://spark-740a727c11ce3da7-driver-svc.default.svc:44692
          master: k8s://https://10.10.9.35:6443
          version: 3.1.2
          driver: [cpu: 1, mem: 1G]
          executor: [cpu: 2, mem: 2G MB, maxNum: 18]
    Start time: Thu Sep 23 08:40:39 UTC 2021

    User: root (shared mode: USER)
    State: LATENT
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1141 from timothy65535/remove.

Closes #1141

af1827e9 [timothy65535] Remove redundant unit

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-23 19:32:15 +08:00
Fu Chen
0be523f6fe [KYUUBI #1135] [SUB-TASK][KPIP-1] Register default ObjectMapper instead of manually serialize/deserialize JSON input argument. 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Register default `ObjectMapper` instead of manually serialize/deserialize JSON input argument.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1135 from cfmcgrady/objectmapper.

Closes #1135

580707cb [Fu Chen] fix style
48da4fe4 [Fu Chen] refactor unit test
067f11be [Fu Chen] fix style
9462d936 [Fu Chen] object mapper

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-23 13:39:39 +08:00
ulysses-you
b26563126f
[KYUUBI #1138] Revert "[KYUUBI #937] Push Kyuubi docker image" 
This reverts commit 0cd41e1eac.

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
```
2021-09-23T02:15:35.6185062Z error: denied: requested access to the resource is denied
2021-09-23T02:15:36.2512609Z ##[error]buildx failed with: error: denied: requested access to the resource is denied
```
push image failed, not sure why it happened.

### _How was this patch tested?_
Pass CI

Closes #1138 from ulysses-you/revert.

Closes #1138

Closes #937

b12b4b56 [ulysses-you] Revert "[KYUUBI #937] Push Kyuubi docker image"

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-23 10:53:51 +08:00
ulysses-you
0cd41e1eac
[KYUUBI #937] Push Kyuubi docker image 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Push docker image if we commit to master

### _How was this patch tested?_
test in forked repo.

Closes #1117 from ulysses-you/docker-image.

Closes #937

9d948806 [ulysses-you] push docker image

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-23 10:03:23 +08:00
Cheng Pan
787ba9efe2
[KYUUBI #1134] [TEST] DataLake suites should clean up withKyuubiConf instead of extraConfigs 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Keep consistent with
```
    withKyuubiConf.foreach { case (k, v) =>
      System.setProperty(k, v)
      kyuubiConf.set(k, v)
    }
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1134 from pan3793/t.

Closes #1134

b8e0aa65 [Cheng Pan] [TEST] DataLake Suites should clean up withKyuubiConf instead of extraConfigs

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-09-23 09:31:29 +08:00
Brian Yue
bd0989487d
[KYUUBI #1115] Add Affects Version/s area in bug report issue template 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1115.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1116 from byyue/feature-1115.

Closes #1115

d4513b22 [Brian Yue] Add Affects Version/s area in bug report issue template

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-09-22 17:55:06 +08:00