apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
902 Commits 23 Branches 0 Tags 65 MiB
b4a505971c
Commit Graph

902 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kent Yao
b4a505971c [KYUUBI #627] Support multiple engines for non-connection share levels 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR adds multiple engine support for non-connection share levels. A new configuration called  `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid.

Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached.

With multiple engines support, end-users are able to
1. create as many engines as they want to meet the concurrency.
2. leverage different queue/namespace resource with long-running mode
3. can give different resources or priorities to their own engines for different workloads
4. can perform fast failover by creating a new sub domain
5. etc.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #627 from yaooqinn/multipleengines.

Closes #627

9f04bc3e [Kent Yao] fix tests
d7af751e [Kent Yao] address comments
be9c46d8 [Kent Yao] address comments
ccce87d7 [Kent Yao] address comments
d49116c7 [Kent Yao] Support multiple engines for non-connection share levels
f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-05-11 10:14:23 +08:00
Honglun
d05c81b9a6
[KYUUBI #625]Kyuubi server-side support 'operation.idle. timeout' 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Now we only get 'operation.idle.timeout' from user request. Kyuubi server-side should support "operation.idle. timeout".

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #626 from Honglun/feature/operation-timeout.

Closes #625

e56c11b [Honglun] Kyuubi server-side support 'operation.idle. timeout'

Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-05-10 18:08:07 +08:00
Cheng Pan
0d8bbfad9e [KYUUBI #619] [TEST] Refactor data lake test suite hierarchical 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Make `*SuiteMixin` resuable.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #619 from pan3793/mixin.

Closes #619

fa776869 [Cheng Pan] address comments
54ac9a19 [Cheng Pan] Revert "gather KyuubiConf"
92985964 [Cheng Pan] gather KyuubiConf
61492616 [Cheng Pan] extract DataLakeSuiteMixin
66b6ffa4 [Cheng Pan] [TEST] Refactor data lake test suite hierarchical

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-05-09 11:03:14 +08:00
Honglun
41f0274ba4 [KYUUBI #624] Add lock while creating engine with non-CONNECTION mode 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

We may create multiple SparkContext in non-CONNECTION ShareLevel.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #624 from Honglun/feature/lock.

Closes #624

87445522 [Honglun] Add lock while creating engine with non-CONNECTION mode

Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-05-08 19:12:01 +08:00
Kent Yao
a28e62ad70
[KYUUBI #623] Improve OperationLog to deliver detail stats of sql statements 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

`OperationLog` is used to divert SQL statement-specific logs to a temporary local file at the engine side and to be fetched to the client-side(e.g. beeline). It is an InheritableThreadLocal variable in which thread the `SparkSession` instance lays. Inside Spark, there are a lot of threads created via the java `Executors`, for which the `OperationLog` cannot be inherited, so the client-side lacks information on how their jobs are running. This is not user-friendly especially for queries that contain a lot of jobs.

In this PR, we add a new SparkListener for SQL operation which diverts the SQL statement-specific logs by `spark.jobGroup.id`. The listener will dump Job/Stage and Task summaries to the `OperationLog`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

```logtalk
2021-05-08 10:25:54.301 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 started with 4 stages, 1 active jobs running
2021-05-08 10:25:54.303 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 6 started with 5 tasks, 1 active stages running
2021-05-08 10:25:54.347 INFO kyuubi.SQLOperationListener: Finished stage: Stage(6, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 5; Took: 44 msec
2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: task runtime:(count: 5, mean: 36.400000, stdev: 1.019804, max: 38.000000, min: 35.000000)
2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 	35.0 ms	35.0 ms	35.0 ms	36.0 ms	36.0 ms	37.0 ms	38.0 ms	38.0 ms	38.0 ms
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 5, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000)
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: fetch wait time:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: remote bytes read:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: task result size:(count: 5, mean: 1965.000000, stdev: 0.000000, max: 1965.000000, min: 1965.000000)
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	1965.0 B	1965.0 B	1965.0 B	1965.0 B	1965.0 B	1965.0 B	1965.0 B	1965.0 B	1965.0 B
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 5, mean: 82.948068, stdev: 1.278517, max: 84.210526, min: 80.555556)
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 	81 %	81 %	81 %	83 %	83 %	84 %	84 %	84 %	84 %
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: other time pct: (count: 5, mean: 17.051932, stdev: 1.278517, max: 19.444444, min: 15.789474)
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 	16 %	16 %	16 %	16 %	17 %	17 %	19 %	19 %	19 %
2021-05-08 10:25:54.357 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 7 started with 300 tasks, 1 active stages running
2021-05-08 10:25:54.644 INFO kyuubi.SQLOperationListener: Finished stage: Stage(7, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 300; Took: 287 msec
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: task runtime:(count: 300, mean: 15.203333, stdev: 9.396559, max: 45.000000, min: 4.000000)
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 	4.0 ms	5.0 ms	6.0 ms	8.0 ms	13.0 ms	18.0 ms	31.0 ms	38.0 ms	45.0 ms
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 300, mean: 56.030000, stdev: 0.298496, max: 59.000000, min: 56.000000)
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 	56.0 B	56.0 B	56.0 B	56.0 B	56.0 B	56.0 B	56.0 B	56.0 B	59.0 B
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: fetch wait time:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: remote bytes read:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B
2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: task result size:(count: 300, mean: 2865.150000, stdev: 9.371633, max: 2906.000000, min: 2863.000000)
2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB	2.8 KiB
2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 300, mean: 66.577940, stdev: 18.397778, max: 95.121951, min: 10.526316)
2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 	11 %	33 %	40 %	56 %	71 %	81 %	87 %	88 %	95 %
2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %
2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: other time pct: (count: 300, mean: 33.422060, stdev: 18.397778, max: 89.473684, min: 4.878049)
2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 	 5 %	12 %	13 %	19 %	29 %	44 %	60 %	67 %	89 %
2021-05-08 10:25:54.650 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 8 started with 1 tasks, 1 active stages running
2021-05-08 10:25:54.733 INFO kyuubi.SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 1; Took: 87 msec
2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: task runtime:(count: 1, mean: 81.000000, stdev: 0.000000, max: 81.000000, min: 81.000000)
2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 	81.0 ms	81.0 ms	81.0 ms	81.0 ms	81.0 ms	81.0 ms	81.0 ms	81.0 ms	81.0 ms
2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 1, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000)
2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B	59.0 B
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: remote bytes read:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: task result size:(count: 1, mean: 3065.000000, stdev: 0.000000, max: 3065.000000, min: 3065.000000)
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB	3.0 KiB
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 95.061728, stdev: 0.000000, max: 95.061728, min: 95.061728)
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 	95 %	95 %	95 %	95 %	95 %	95 %	95 %	95 %	95 %
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: other time pct: (count: 1, mean: 4.938272, stdev: 0.000000, max: 4.938272, min: 4.938272)
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 	 5 %	 5 %	 5 %	 5 %	 5 %	 5 %	 5 %	 5 %	 5 %
2021-05-08 10:25:54.739 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 9 started with 200 tasks, 1 active stages running
2021-05-08 10:25:54.844 INFO kyuubi.SQLOperationListener: Finished stage: Stage(9, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 200; Took: 105 msec
2021-05-08 10:25:54.845 INFO scheduler.DAGScheduler: Job 3 finished: collect at ExecuteStatement.scala:82, took 0.547547 s
2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: task runtime:(count: 200, mean: 15.100000, stdev: 11.464729, max: 48.000000, min: 4.000000)
2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 	4.0 ms	6.0 ms	7.0 ms	8.0 ms	10.0 ms	16.0 ms	40.0 ms	43.0 ms	48.0 ms
2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: fetch wait time:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms	0.0 ms
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: remote bytes read:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B	0.0 B
2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: task result size:(count: 200, mean: 2326.135000, stdev: 26.673897, max: 2414.000000, min: 2311.000000)
2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 	2.3 KiB	2.3 KiB	2.3 KiB	2.3 KiB	2.3 KiB	2.3 KiB	2.3 KiB	2.3 KiB	2.4 KiB
2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 200, mean: 2.971561, stdev: 8.036931, max: 47.727273, min: 0.000000)
2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	10 %	18 %	48 %
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %	 0 %
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: other time pct: (count: 200, mean: 97.028439, stdev: 8.036931, max: 100.000000, min: 52.272727)
2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
2021-05-08 10:25:54.849 INFO scheduler.StatsReportListener: 	52 %	82 %	90 %	100 %	100 %	100 %	100 %	100 %	100 %
2021-05-08 10:25:54.849 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 succeeded, 0 active jobs running
2021-05-08 10:25:54.857 INFO codegen.CodeGenerator: Code generated in 7.116325 ms
2021-05-08 10:25:54.858 INFO operation.ExecuteStatement: Processing kentyao's query[3ac86686-f502-4f2d-9a98-d763e5b9548c]: RUNNING_STATE -> FINISHED_STATE, statement: select /*+ REPARTITION(200) */ count(1) from (select  /*+ REPARTITION(300, a) */  a  from values(1),(1),(1),(2),(3) t(a)), time taken: 0.702 seconds
```

Closes #623 from yaooqinn/log2.

Closes #623

abeff64 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement
fdf3b28 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement
783a594 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-05-08 17:50:13 +08:00
Cheng Pan
d3e2981f0f [KYUUBI #622] [NOTICE] Add NOTICE 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add `NOTICE` file in this PRs.

Have a look on other projects, I only see Apache Spark maintains `NOTICE-binary` in repo, most projects generate it on build process.

For instance,

Apache Flink use shell [`collect_license_files`](https://github.com/apache/flink/blob/release-1.13.0/tools/releasing/collect_license_files.sh) to collect LICENSE and NOTICE files.
Apache Calcite use gradle plugin [vlsi-release-plugins](https://github.com/vlsi/vlsi-release-plugins)  to manage the whole release process, see [build.gradle.kts](https://github.com/apache/calcite/blob/calcite-1.26.0/build.gradle.kts#L45).
Apache Kafka write gradle scripts to gather LICENSE and NOTICE files, see [build.gradle](https://github.com/apache/kafka/blob/2.8.0/build.gradle).

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #622 from pan3793/notice.

Closes #622

777ca699 [Cheng Pan] [NOTICE] Add NOTICE

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-05-07 22:05:56 +08:00
Cheng Pan
2aea8d4867 [KYUUBI #620] [TEST] Close Source to avoid file leak 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #620 from pan3793/leak.

Closes #620

7e63f623 [Cheng Pan] fix
8293b92c [Cheng Pan] eager consume
54bf713a [Cheng Pan] [TEST] Close Source to avoid file leak

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-05-07 21:34:57 +08:00
iodone
998bf08197
[KYUUBI #615]setting the fetchType of TFetchResultsReq to 1 in getOperationLogRowSet 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #617 from iodone/master.

Closes #615

fbbe515 [iodone] setting the fetchType of TFetchResultsReq to 1 in getOperationLogRowSet

Authored-by: iodone <work@YaodongdeMacBook-Pro.local>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-30 16:30:15 +08:00
Cheng Pan
f742686357
[KYUUBI #614] [BUILD][DOCS] Bump sphinx 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Bump sphinx to latest version. close #612

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #614 from pan3793/doc.

Closes #614

b1b2738 [Cheng Pan] [BUILD][DOCS] Bump sphinx

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-28 20:16:26 +08:00
Cheng Pan
34374f3b3e
[KYUUBI #613] [TEST] [DELTA] Add test for deltalake 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #613 from pan3793/delta.

Closes #613

ccadedb [Cheng Pan] [TEST] [DELTA] Add test for deltalake

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-28 20:14:00 +08:00
fwang12
20911ff73c [KYUUBI #611] [TPCDS][TEST] Introduce TPCDS 2.7 output schema check 
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce TPCDS 2.7 output schema tests.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #611 from turboFei/tpcds_2_7_follow.

Closes #611

6ecbbee [fwang12] Introduce tpcds 2.7 output schema tests

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-27 09:23:53 +08:00
fwang12
0e2263f759
[KYUUBI #606] Introduce ExtendedSQLTest tag to split build jobs 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Involve ExtendedSQLTest tag to split build jobs

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #606 from turboFei/slow_tags.

Closes #606

5f8464d [fwang12] do not use build/dist
9e813dc [fwang12] exclude extended sql test for default profile
4c5ea67 [fwang12] Involve ExtendedSQLTest to split build jobs

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-26 20:15:46 +08:00
fwang12
2af8b53dbf [KYUUBI #552] Sync build jobs between GA and travis 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Sync build jobs between GA and travis
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #604 from turboFei/sync_ga_travi.

Closes #552

98f2e45a [fwang12] [KYUUBI #552]Sync build jobs between GA and travis

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-25 21:51:34 +08:00
Kent Yao
51a190ffb8
[KYUUBI #605] Improve the landscape for README 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #605 from yaooqinn/readme.

Closes #605

63d8702 [Kent Yao] Update Readme
c48d8e8 [Kent Yao] Update Readme
9a2e2d6 [Kent Yao] Update Readme

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-25 20:03:33 +08:00
Cheng Pan
09c5f9123e [KYUUBI #601] [METRICS] Change metrics config keys 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Change the config keys of metrics to get a consistent name style, `metrics.{reporter_type}.{property}`.

The change breaks the compatibility with master, but consider v1.2.0 is not out yet, so changes should be safe.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #601 from pan3793/metrics.

Closes #601

89d1974f [Cheng Pan] address comments
361ae6e2 [Cheng Pan] revert default metrics reporter change
1c493e64 [Cheng Pan] rat exclude report.json
fcdb4252 [Cheng Pan] exclude report.json
337e09ae [Cheng Pan] Default enable SFL4J metrics reporter
6b9f3963 [Cheng Pan] [METRICS] Change metrics config keys

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-04-25 18:30:06 +08:00
fwang12
435e85f5d8 [KYUUBI #603] [TEST] Refactor test name and sort the tpcds queries 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Refactor the test name to show tpcds version information and sort the test queries by query number.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #603 from turboFei/tpcds_refactor.

Closes #603

1c8daad5 [fwang12] add accident deleted q3
2f62b650 [fwang12] [TEST] Refactor test name and sort the tpcds queries

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-04-25 18:27:05 +08:00
fwang12
540de12764 [KYUUBI #598] [TEST][TPCDS] Introduce schema check for queries-9 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(ninety plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #598 from turboFei/tpcds_9.

Closes #598

fef260d [fwang12] [TEST][TPCDS] Introduce schema check for queries-9

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:33:11 +08:00
fwang12
e1fa5c62ed [KYUUBI #599] [TEST][TPCDS] Introduce schema check for queries-8 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(eighty plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #599 from turboFei/tpcds_8.

Closes #599

6cc4642 [fwang12] [TEST][TPCDS] Introduce schema check for queries-8

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:32:21 +08:00
fwang12
f318c0b8f8 [KYUUBI #597] [TEST][TPCDS] Introduce schema check for queries-7 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(seventy plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #597 from turboFei/tpcds_7.

Closes #597

7d008ae [fwang12] [TEST][TPCDS] Introduce schema check for queries-7

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:31:19 +08:00
fwang12
c197be060b [KYUUBI #596] [TEST][TPCDS] Introduce schema check for queries-6 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(sixty plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #596 from turboFei/tpcds_6.

Closes #596

f6d45de [fwang12] [TEST][TPCDS] Introduce schema check for queries-6

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:29:45 +08:00
fwang12
9a80db57f5 [KYUUBI #595] [TEST][TPCDS] Introduce schema check for queries-5 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(fifty plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #595 from turboFei/tpcds_5.

Closes #595

7be2419 [fwang12] [TEST][TPCDS] Introduce schema check for queries-5

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:28:09 +08:00
fwang12
16e2673c72 [KYUUBI #594] [TEST][TPCDS] Introduce schema check for queries-4 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(forty plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #594 from turboFei/tpcds_40+.

Closes #594

c3590db [fwang12] [TEST][TPCDS] Introduce schema check for queries-4

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:27:10 +08:00
fwang12
1618c382e8 [KYUUBI #593] [TEST][TPCDS] Introduce schema check for queries-3 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(thirty plus)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #593 from turboFei/tpcds_30+.

Closes #593

a796af1 [fwang12] [TEST][TPCDS] Introduce schema check for queries(thirty plus)

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-25 09:26:04 +08:00
Cheng Pan
913c444696 [KYUUBI #600] [DOCS] Update building doc and tunes 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #600 from pan3793/doc.

Closes #600

e7ad0a3b [Cheng Pan] minor tune
f11ae160 [Cheng Pan] [DOCS] Update building doc

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-25 06:58:58 +08:00
fwang12
325d59919b
[KYUUBI #592] [TEST][TPCDS] Introduce schema check for queries-2 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for queries(twenty plus)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #592 from turboFei/tpcds_20+.

Closes #592

5d0e417 [fwang12] [TEST][TPCDS] Introduce schema check for queries(twenty plus)

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-25 00:44:02 +08:00
fwang12
18208641dd [KYUUBI #590] [TPCDS][TEST] Fix TPCDS suite issue and introduce tpcds queries(ten plus) 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix tpcds schema issue and covert all tests, whose file is named like q14a.sql.

Introduce tpcds queries, whose index between ten and twenty.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #590 from turboFei/tpcds_10_20.

Closes #590

d58c4e97 [fwang12] remove query 20
dbcd3379 [fwang12] [TPCDS][TEST] Fix tpcds suite issue and introduce q11-q20

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 2021-04-25 00:07:59 +08:00
fwang12
746736f4f3
[KYUUBI #581] [TEST][TPCDS] Introduce schema check for q11 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for q11

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #581 from turboFei/tpcds_11.

Closes #581

f6c38c9 [fwang12] [TEST][TPCDS] Introduce schema check for q11

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 21:44:00 +08:00
fwang12
879de5cbfe [KYUUBI #587] [TPCDS][TEST] Fix table schema issue and introduce q5 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix table schema issue and introduce q5

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #587 from turboFei/fix_tpcds.

Closes #587

1f3f78a2 [fwang12] [TPCDS][TEST] Fix table schema issue and introduce q5

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-24 21:33:18 +08:00
fwang12
e1899bae61 [KYUUBI #580] [TEST][TPCDS] Introduce schema check for q9 and q10 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
 Introduce schema check for q9 and q10

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #580 from turboFei/tpcds_9_10.

Closes #580

ae688ac [fwang12] [TEST][TPCDS] Introduce schema check for q9 and q10

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-24 19:13:42 +08:00
fwang12
8f3871e31f [KYUUBI #579] [TEST][TPCDS] Introduce schema check for q8 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Introduce schema check for q8
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #579 from turboFei/tpcds_8.

Closes #579

e1b1473 [fwang12] [TEST][TPCDS] Introduce schema check for q8

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-24 19:12:56 +08:00
fwang12
0b0ffe63de [KYUUBI #578] [TEST][TPCDS] Introduce schema check for q7 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for q7

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #578 from turboFei/tpcds_7.

Closes #578

b4010d0 [fwang12] [TEST][TPCDS] Introduce schema check for q7

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-24 19:12:05 +08:00
fwang12
d8c1eaa56d
[KYUUBI #577] [TEST][TPCDS] Introduce schema check for q4 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for q4

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #577 from turboFei/tpcds-4.

Closes #577

bb8b1cf [fwang12] [TEST][TPCDS] Introduce schema check for q4

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 18:44:19 +08:00
fwang12
9e89df06da
[KYUUBI #576] [TEST][TPCDS] Introduce schema check for q3 
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce schema check for q3

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #576 from turboFei/tpcds-3-5.

Closes #576

a7a485c [fwang12] [TEST][TPCDS] Introduce schema check for q3

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 18:42:29 +08:00
Cheng Pan
f311211aa3
[KYUUBI #574] [IDEA] Make IDEA support IssueNavigationLink 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/26535726/115897874-7ed37d00-a48f-11eb-9f10-7990a1b9727e.png)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #574 from pan3793/ide.

Closes #574

eb48095 [Cheng Pan] typo
155c89b [Cheng Pan] [IDEA] Make IDEA support IssueNavigationLink

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 02:03:31 +08:00
Cheng Pan
b9c90f674f
[KYUUBI #575] [TEST] [DEPS] Pin scala-reflect to avoid potential conflicts 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
There is transitive dependency `scala-reflect` 2.12.13 from `scalatest` in test scope, we'd better to pin version align with `scala-library` to avoid potential conflicts.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #575 from pan3793/scala-reflect.

Closes #575

b6019f7 [Cheng Pan] [TEST] [DEPS] Pin scala-reflect to avoid potential conflicts

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 02:00:45 +08:00
Cheng Pan
68d8f046ac
[KYUUBI #573] [TEST] [TPCDS] Introduce schema check for q6 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #573 from pan3793/tpcds-6.

Closes #573

e476dea [Cheng Pan] Add missing license header
98eb8c9 [Cheng Pan] [TEST] [TPCDS] Introduce schema check for q6

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-24 01:58:01 +08:00
fwang12
42fa7cfe49 [KYUUBI #544] Involve job max failures and deregister exception TTL 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Involve job max failures to limit the max job failures before deregistering the engine, involve deregister exception TTL for to restart counting when last failure has gone for one deregisterExceptionTTL.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #560 from turboFei/KYUUBI-544-exception-max-failures-new.

Closes #544

951f269 [fwang12] [KYUUBI #544] Involve job max failures and deregister exception TTL

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-23 21:56:22 +08:00
fwang12
bd89d740e6 [KYUUBI #570] Complete doc for quick start with jdbc 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Complete doc for quick start with jdbc.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #571 from turboFei/quick_start_with_jdbc.

Closes #570

342885d [fwang12] address comments
8ccf102 [fwang12] [KYUUBI #570] Complete doc for quick start with jdbc

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 2021-04-23 14:53:48 +08:00
fwang12
f251a48d3a [KYUUBI #572] Complete the doc for integration test with kyuubi-spark-sql-engine 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Complete the doc for integration test with kyuubi-spark-sql-engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #572 from turboFei/test_engine.

Closes #572

ad615d9 [fwang12] Complete the doc for testing

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-23 13:16:18 +08:00
fwang12
8ea1ce39aa
[KYUUBI #568] Add TPCDS query output schema check for q2 
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add TPCDS query output schema check for q2
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #568 from turboFei/tpcds_2.

Closes #568

c2523ea [fwang12] Add TPCDS query output schema check for q2

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-23 01:01:24 +08:00
Cheng Pan
92aab7b536
[KYUUBI #569] [DOC] Update collaborator profiles 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #569 from pan3793/doc.

Closes #569

5da2a00 [Cheng Pan] [DOC] Update collaborator profiles

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-23 00:59:39 +08:00
ulysses
fd677f14eb
[KYUUBI #566] Add TPCDS query output schema check (#566) 
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Use TPCDS query to check if we have some behavior change.

Note that 
1. this PR only add `q1`.
2. this PR just to check the output schema. The output data check can be added in future.
3. this new added test file structure looks like
```
tpcds-1_4
  - q1
    - q1.sql
    - q1.output.schema 
    - q1.output.data (if we support this in future)
  - q2
  ......
```

### _How was this patch tested?_
Add new test
 2021-04-22 21:52:57 +08:00
fwang12
913d3f19d4
[KYUUBI #544] Combine deregister exception messages and stacktraces 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
To simplify the usage and be more user friendly, combine `kyuubi.engine.deregister.exception.messages` and `kyuubi.engine.deregister.exception.stacktraces` to `kyuubi.engine.deregister.exception.messages`.

### _How was this patch tested?_
Existing UT.

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #565 from turboFei/KYUUBI-544-combine.

Closes #544

ee63f60 [fwang12] [KYUUBI #544] Combine deregister exception messages and stacktraces

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-22 20:49:15 +08:00
Cheng Pan
ba26de688e [KYUUBI #563] [TEST] Bump scalatest 3.2.8 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Bump the latest version of scalatest.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #563 from pan3793/scalatest.

Closes #563

1b7f844 [Cheng Pan] [TEST] Bump scalatest 3.2.8

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-22 20:45:43 +08:00
Cheng Pan
59fa96e5ce [KYUUBI #564] [ICON] Add Kyuubi icon for Jetbrains Toolbox 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/26535726/115590945-2754d600-a304-11eb-9862-2fde8a4e3abf.png)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #564 from pan3793/icon.

Closes #564

23918ec [Cheng Pan] [ICON] Add Kyuubi icon for Jetbrains Toolbox

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-22 20:44:40 +08:00
fwang12
29e169b68b
[KYUUBI #544] Add UT for engine deregister exception and refactor package name for SparkSQLEngineListenerSuite 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add UT for engine deregister exception

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #558 from turboFei/KYUUBI-544-exception-test.

Closes #544

7812d6e [fwang12] remove blank line
3b5252a [fwang12] find cause
6e09907 [fwang12] simplify ut
dbafab2 [fwang12] log ve
b062503 [fwang12] improvement
0793980 [fwang12] fix ut
ac35be0 [fwang12] [KYUUBI #544][TEST] Add ut for engine deregister exception

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-21 23:38:05 +08:00
Cheng Pan
5cc4fd83c1
[KYUUBI #561] Forbid set env:* in thrift connection configurations 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This change only forbid the `set:env:*` from thrift connection configurations, to match hive behavior, forbid `set env:*` in SQL, we need to change it in Spark.
#508
https://issues.apache.org/jira/browse/HIVE-6175

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #561 from pan3793/env.

Closes #561

79b7e69 [Cheng Pan] update
00dfaf0 [Cheng Pan] Forbid set env:* in thrift connection configurations

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-21 23:36:51 +08:00
Cheng Pan
4edbb60238
[KYUUBI #562] [BUILD] Add maven profile mirror-cn 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Use `build/mvn -Pmirror-cn ...` to speed up downing spark bin tgz, it's more friendly for china mainland developers.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #562 from pan3793/mirror.

Closes #562

3c4cf7b [Cheng Pan] [BUILD] Add maven profile mirror-cn

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-04-21 23:31:58 +08:00
Cheng Pan
6431dcc25b [KYUUBI #554] [TEST] Remove deprecated FunSuite 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #554 from pan3793/scalatest.

Closes #554

4c94224 [Cheng Pan] update scalastyle conf
f29a3e6 [Cheng Pan] [TEST] Remove deprecated FunSuite

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-21 21:00:33 +08:00
Cheng Pan
c0aaeb362d [KYUUBI #555] [INFRA] Backport Using from Scala 2.13 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #553

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #555 from pan3793/using.

Closes #555

b934c66 [Cheng Pan] codestyle
d6e2643 [Cheng Pan] update code style
ecd0888 [Cheng Pan] [COMMON] Backport Using from Scala 2.13

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-04-21 13:36:54 +08:00