121 Commits

Author	SHA1	Message	Date
Cheng Pan	ced29f5b8a	[KYUUBI #707] Respect spark.sql.catalogImplementation in engine ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Respect `spark.sql.catalogImplementation` in engine. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #707 from pan3793/hive. Closes #707 d46af57b [Cheng Pan] address comments 8570ca89 [Cheng Pan] fixup 8bd4a256 [Cheng Pan] fixup c36659d6 [Cheng Pan] switch to spark.sql.catalogImplementation fb6ebf66 [Cheng Pan] Make engine enable hive optional Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-06-23 20:11:13 +08:00
Kent Yao	a2f1e22361	[KYUUBI #657] Add udf kyuubi_version ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ In this PR, I propose to add kyuubi_version as a user-defined function. This is also a good example to add other new functions that needed in the Kyuubi system. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #686 from yaooqinn/657. Closes #657 d30ac8f6 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version e1e585e9 [Kent Yao] [KYUUBI #657] Add udf kyuubi_version Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-06-12 10:16:01 +08:00
Cheng Pan	e57a4346e2	[KYUUBI #669] Cleanup code and fix typos ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #669 from pan3793/cleanup. Closes #669 2cdc4a3 [Cheng Pan] Cleanup code and fix typos Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-06-08 00:04:05 +08:00
Cheng Pan	b567453462	[KYUUBI #643] [TEST] [DELTA] Refine CI and add test detla 1.0.0 with Spark 3.1 ... * [TEST] [DELTA] Test detla 1.0.0 with Spark 3.1 * seperate DataLakeTest * scalatest.exclude.tags * align gh action and travis * seperate tpcds in GA * travis * reduce redundant ci jobs * rename spark-3.2-snapshot to spark-master * rename tpcds suites * mvn --no-transfer-progress * address comments	2021-05-26 09:57:33 +08:00
Kent Yao	bb9e8a50e5	Prepare Kyuubi v1.3.0	2021-05-13 13:58:20 +08:00
Kent Yao	b4a505971c	[KYUUBI #627] Support multiple engines for non-connection share levels ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid. Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached. With multiple engines support, end-users are able to 1. create as many engines as they want to meet the concurrency. 2. leverage different queue/namespace resource with long-running mode 3. can give different resources or priorities to their own engines for different workloads 4. can perform fast failover by creating a new sub domain 5. etc. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #627 from yaooqinn/multipleengines. Closes #627 9f04bc3e [Kent Yao] fix tests d7af751e [Kent Yao] address comments be9c46d8 [Kent Yao] address comments ccce87d7 [Kent Yao] address comments d49116c7 [Kent Yao] Support multiple engines for non-connection share levels f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-05-11 10:14:23 +08:00
Cheng Pan	0d8bbfad9e	[KYUUBI #619] [TEST] Refactor data lake test suite hierarchical ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make `*SuiteMixin` resuable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #619 from pan3793/mixin. Closes #619 fa776869 [Cheng Pan] address comments 54ac9a19 [Cheng Pan] Revert "gather KyuubiConf" 92985964 [Cheng Pan] gather KyuubiConf 61492616 [Cheng Pan] extract DataLakeSuiteMixin 66b6ffa4 [Cheng Pan] [TEST] Refactor data lake test suite hierarchical Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-05-09 11:03:14 +08:00
Kent Yao	a28e62ad70	[KYUUBI #623] Improve OperationLog to deliver detail stats of sql statements ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `OperationLog` is used to divert SQL statement-specific logs to a temporary local file at the engine side and to be fetched to the client-side(e.g. beeline). It is an InheritableThreadLocal variable in which thread the `SparkSession` instance lays. Inside Spark, there are a lot of threads created via the java `Executors`, for which the `OperationLog` cannot be inherited, so the client-side lacks information on how their jobs are running. This is not user-friendly especially for queries that contain a lot of jobs. In this PR, we add a new SparkListener for SQL operation which diverts the SQL statement-specific logs by `spark.jobGroup.id`. The listener will dump Job/Stage and Task summaries to the `OperationLog` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request ```logtalk 2021-05-08 10:25:54.301 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 started with 4 stages, 1 active jobs running 2021-05-08 10:25:54.303 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 6 started with 5 tasks, 1 active stages running 2021-05-08 10:25:54.347 INFO kyuubi.SQLOperationListener: Finished stage: Stage(6, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 5; Took: 44 msec 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: task runtime:(count: 5, mean: 36.400000, stdev: 1.019804, max: 38.000000, min: 35.000000) 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.347 INFO scheduler.StatsReportListener: 35.0 ms 35.0 ms 35.0 ms 36.0 ms 36.0 ms 37.0 ms 38.0 ms 38.0 ms 38.0 ms 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 5, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: fetch wait time:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.348 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: remote bytes read:(count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: task result size:(count: 5, mean: 1965.000000, stdev: 0.000000, max: 1965.000000, min: 1965.000000) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 1965.0 B 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 5, mean: 82.948068, stdev: 1.278517, max: 84.210526, min: 80.555556) 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.349 INFO scheduler.StatsReportListener: 81 % 81 % 81 % 83 % 83 % 84 % 84 % 84 % 84 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 5, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: other time pct: (count: 5, mean: 17.051932, stdev: 1.278517, max: 19.444444, min: 15.789474) 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.350 INFO scheduler.StatsReportListener: 16 % 16 % 16 % 16 % 17 % 17 % 19 % 19 % 19 % 2021-05-08 10:25:54.357 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 7 started with 300 tasks, 1 active stages running 2021-05-08 10:25:54.644 INFO kyuubi.SQLOperationListener: Finished stage: Stage(7, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 300; Took: 287 msec 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: task runtime:(count: 300, mean: 15.203333, stdev: 9.396559, max: 45.000000, min: 4.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 4.0 ms 5.0 ms 6.0 ms 8.0 ms 13.0 ms 18.0 ms 31.0 ms 38.0 ms 45.0 ms 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 300, mean: 56.030000, stdev: 0.298496, max: 59.000000, min: 56.000000) 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.645 INFO scheduler.StatsReportListener: 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 56.0 B 59.0 B 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: fetch wait time:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: remote bytes read:(count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.646 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: task result size:(count: 300, mean: 2865.150000, stdev: 9.371633, max: 2906.000000, min: 2863.000000) 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.647 INFO scheduler.StatsReportListener: 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2.8 KiB 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 300, mean: 66.577940, stdev: 18.397778, max: 95.121951, min: 10.526316) 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.648 INFO scheduler.StatsReportListener: 11 % 33 % 40 % 56 % 71 % 81 % 87 % 88 % 95 % 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 300, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.649 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: other time pct: (count: 300, mean: 33.422060, stdev: 18.397778, max: 89.473684, min: 4.878049) 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.650 INFO scheduler.StatsReportListener: 5 % 12 % 13 % 19 % 29 % 44 % 60 % 67 % 89 % 2021-05-08 10:25:54.650 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 8 started with 1 tasks, 1 active stages running 2021-05-08 10:25:54.733 INFO kyuubi.SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 1; Took: 87 msec 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: task runtime:(count: 1, mean: 81.000000, stdev: 0.000000, max: 81.000000, min: 81.000000) 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.733 INFO scheduler.StatsReportListener: 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 81.0 ms 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 1, mean: 59.000000, stdev: 0.000000, max: 59.000000, min: 59.000000) 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.734 INFO scheduler.StatsReportListener: 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 59.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: remote bytes read:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: task result size:(count: 1, mean: 3065.000000, stdev: 0.000000, max: 3065.000000, min: 3065.000000) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 3.0 KiB 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 95.061728, stdev: 0.000000, max: 95.061728, min: 95.061728) 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.735 INFO scheduler.StatsReportListener: 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 95 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: other time pct: (count: 1, mean: 4.938272, stdev: 0.000000, max: 4.938272, min: 4.938272) 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.736 INFO scheduler.StatsReportListener: 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 5 % 2021-05-08 10:25:54.739 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Stage 9 started with 200 tasks, 1 active stages running 2021-05-08 10:25:54.844 INFO kyuubi.SQLOperationListener: Finished stage: Stage(9, 0); Name: 'collect at ExecuteStatement.scala:82'; Status: succeeded; numTasks: 200; Took: 105 msec 2021-05-08 10:25:54.845 INFO scheduler.DAGScheduler: Job 3 finished: collect at ExecuteStatement.scala:82, took 0.547547 s 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: task runtime:(count: 200, mean: 15.100000, stdev: 11.464729, max: 48.000000, min: 4.000000) 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: 4.0 ms 6.0 ms 7.0 ms 8.0 ms 10.0 ms 16.0 ms 40.0 ms 43.0 ms 48.0 ms 2021-05-08 10:25:54.845 INFO scheduler.StatsReportListener: shuffle bytes written:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: fetch wait time:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: remote bytes read:(count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.846 INFO scheduler.StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: task result size:(count: 200, mean: 2326.135000, stdev: 26.673897, max: 2414.000000, min: 2311.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.3 KiB 2.4 KiB 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: executor (non-fetch) time pct: (count: 200, mean: 2.971561, stdev: 8.036931, max: 47.727273, min: 0.000000) 2021-05-08 10:25:54.847 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 10 % 18 % 48 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: fetch wait time pct: (count: 200, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: other time pct: (count: 200, mean: 97.028439, stdev: 8.036931, max: 100.000000, min: 52.272727) 2021-05-08 10:25:54.848 INFO scheduler.StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100% 2021-05-08 10:25:54.849 INFO scheduler.StatsReportListener: 52 % 82 % 90 % 100 % 100 % 100 % 100 % 100 % 100 % 2021-05-08 10:25:54.849 INFO kyuubi.SQLOperationListener: Query [3ac86686-f502-4f2d-9a98-d763e5b9548c]: Job 3 succeeded, 0 active jobs running 2021-05-08 10:25:54.857 INFO codegen.CodeGenerator: Code generated in 7.116325 ms 2021-05-08 10:25:54.858 INFO operation.ExecuteStatement: Processing kentyao's query[3ac86686-f502-4f2d-9a98-d763e5b9548c]: RUNNING_STATE -> FINISHED_STATE, statement: select /*+ REPARTITION(200) */ count(1) from (select /*+ REPARTITION(300, a) */ a from values(1),(1),(1),(2),(3) t(a)), time taken: 0.702 seconds ``` Closes #623 from yaooqinn/log2. Closes #623 abeff64 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement fdf3b28 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement 783a594 [Kent Yao] Improve OperationLog to deliver detail stats of sql statement Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-05-08 17:50:13 +08:00
Cheng Pan	34374f3b3e	[KYUUBI #613] [TEST] [DELTA] Add test for deltalake ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #613 from pan3793/delta. Closes #613 ccadedb [Cheng Pan] [TEST] [DELTA] Add test for deltalake Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-28 20:14:00 +08:00
fwang12	42fa7cfe49	[KYUUBI #544] Involve job max failures and deregister exception TTL ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve job max failures to limit the max job failures before deregistering the engine, involve deregister exception TTL for to restart counting when last failure has gone for one deregisterExceptionTTL. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #560 from turboFei/KYUUBI-544-exception-max-failures-new. Closes #544 951f269 [fwang12] [KYUUBI #544] Involve job max failures and deregister exception TTL Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com>	2021-04-23 21:56:22 +08:00
fwang12	913d3f19d4	[KYUUBI #544] Combine deregister exception messages and stacktraces ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To simplify the usage and be more user friendly, combine `kyuubi.engine.deregister.exception.messages` and `kyuubi.engine.deregister.exception.stacktraces` to `kyuubi.engine.deregister.exception.messages`. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #565 from turboFei/KYUUBI-544-combine. Closes #544 ee63f60 [fwang12] [KYUUBI #544] Combine deregister exception messages and stacktraces Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-22 20:49:15 +08:00
fwang12	29e169b68b	[KYUUBI #544] Add UT for engine deregister exception and refactor package name for SparkSQLEngineListenerSuite ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add UT for engine deregister exception ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #558 from turboFei/KYUUBI-544-exception-test. Closes #544 7812d6e [fwang12] remove blank line 3b5252a [fwang12] find cause 6e09907 [fwang12] simplify ut dbafab2 [fwang12] log ve b062503 [fwang12] improvement 0793980 [fwang12] fix ut ac35be0 [fwang12] [KYUUBI #544][TEST] Add ut for engine deregister exception Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-21 23:38:05 +08:00
Cheng Pan	5cc4fd83c1	[KYUUBI #561] Forbid set env:* in thrift connection configurations ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This change only forbid the `set:env:*` from thrift connection configurations, to match hive behavior, forbid `set env:*` in SQL, we need to change it in Spark. #508 https://issues.apache.org/jira/browse/HIVE-6175 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #561 from pan3793/env. Closes #561 79b7e69 [Cheng Pan] update 00dfaf0 [Cheng Pan] Forbid set env:* in thrift connection configurations Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-21 23:36:51 +08:00
fwang12	7abcc9c5d1	[KYUUBI #544] Deregister engine when meeting specified exception stacktraces ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.stacktraces` to deregister engine when meeting specified exception stacktraces. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #557 from turboFei/KYUUBI-544-exception-msg-stack-rebase. Closes #544 beedb9c6 [fwang12] stopping -> deregistering 80373fca [fwang12] [KYUUBI #544] Deregister engine when meeting specified stacktrace Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-04-21 13:35:06 +08:00
fwang12	7d72b31e81	[KYUUBI #544] Deregister engine when meeting specifies exception messages ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.messages` to deregister engine when meeting specified exception messages. ### _How was this patch tested?_ Existing UT. - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #556 from turboFei/KYUUBI-544-exception-msg-rebase. Closes #544 c1fed723 [fwang12] address comments 7cfba913 [fwang12] [KYUUBI #544] Deregister engine when meeting specified exception message Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-04-21 11:01:36 +08:00
fwang12	3d3d004e2d	[KYUUBI #544] Deregister engine when meeting specified exception class ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Involve `kyuubi.engine.deregister.exception.classes` to deregister engine when job failed because of specified exception class ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #549 from turboFei/KYUUBI-544-exception-class. Closes #544 9ff83b80 [fwang12] use lazy val to avoid npe d80d4436 [fwang12] [KYUUBI #544] Add kyuubi.engine.deregister.exception.classes to deregister engine when job failed because of specified exception class Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-04-21 09:36:58 +08:00
ulysses-you	cacc2b05b3	[KYUUBI #543] [TESTS] Add test for scheduler pool ... <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve test coverage. ### _How was this patch tested?_ Pass `org.apache.kyuubi.engine.spark.SchedulerPoolSuite`. Closes #543 from ulysses-you/add-scheduler-pool-test. Closes #543 f5563b1 [ulysses-you] trait 57938a1 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-04-19 17:09:55 +08:00
ulysses-you	c458d853c4	[KYUUBI #526] Support kyuubi.operation.scheduler.pool for queries ...  [](https://github.com/yaooqinn/kyuubi/pull/528)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support scheduler pool which is native supported by Spark. ### _How was this patch tested?_ Pass CI + Manual tets The default pool name is `default`, aflter execute `set kyuubi.operation.scheduler.pool = default1;`, the pool changed.  Closes #528 from ulysses-you/support-spark-scheduler-mode. 1ba8e89 [ulysses-you] fux 746cb62 [ulysses-you] constraint 1b369a9 [ulysses-you] conf ec49c72 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-04-15 10:27:26 +08:00
Kent Yao	d47879e0bb	[KYUUBI #476][FOLLOWUP] Strip prefixes when ignoring and restricting configs ...  [](https://github.com/yaooqinn/kyuubi/pull/510)    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. change the configs to seq[string] for simplification 2. bugfix for mismatched configs when ignoring and restricting because they are maybe prefixed with something like `set:hivevar:thekey=thevalue`, we should normalize it to `thekey=thevalue` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #510 from yaooqinn/476. Closes #476 84355de [Kent Yao] more test cc8c20e [Kent Yao] doc ee0b418 [Kent Yao] nit 7def6eb [Kent Yao] [KYUUBI #476][FOLLOWUP] Strip prefixes when ignoring and restricting configs Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-12 15:51:28 +08:00
fwang12	32b9ad0d6e	[KYUUBI #304] Kyuubi service control client action implementation ...  [](https://github.com/yaooqinn/kyuubi/pull/478)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Implement the create/list/get/delete action for Kyuubi-ctl ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #478 from turboFei/KYUUBI_CTL_304_CREATE. Closes #304 d81be9e8 [fwang12] refactor column name f4e18d1f [fwang12] address comments 94941079 [fwang12] use function t o generate expected output b07e55b6 [fwang12] align class name with ServiceControlCli 641f7021 [fwang12] refactor format 8098f96a [fwang12] refactor for delete action 489dad32 [fwang12] address comments 16f60218 [fwang12] add list/get/delete action and refer the zk nodes info 85c147ee [fwang12] [KYUUBI #304] Kyuubi service control client action- expose existing kyuubi server instances to another domain Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-04-11 21:47:34 +08:00
ulysses-you	b0aeb26230	[KYUUBI #500] Speed up spark sql engine diagnostics ...  [](https://github.com/yaooqinn/kyuubi/pull/500)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Mark it as `lazy val` to avoid rebuld the string text every time. ### _How was this patch tested?_ Pass exists CI Closes #500 from ulysses-you/speed-up-diagnostics. Closes #500 d4a61f1 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: fwang12 <fwang12@ebay.com>	2021-04-11 09:37:42 +08:00
fwang12	9a8b2860f6	[KYUUBI #506] Shade curator dependencies to avoid impaction by external jars ...  [](https://github.com/yaooqinn/kyuubi/pull/506)       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Shade curator dependencies to avoid impaction by external jars. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #506 from turboFei/shade_curator_new. Closes #506 4c6fdeaa [fwang12] Shade curator dependencies to avoid impaction by external jars Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-04-10 11:07:13 +08:00
Cheng Pan	b7ac3cb48c	Revert "[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars" ... This reverts commit e058b87dd2.	2021-04-10 00:16:04 +08:00
fwang12	e058b87dd2	[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars ...  [](https://github.com/yaooqinn/kyuubi/pull/488)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Shade curator dependency in kyuubi-ha module to avoid impaction by external jars. ### _How was this patch tested?_ Existing UT. - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #488 from turboFei/shade_curator. Closes #488 c88414c [fwang12] Use wildcards when selecting more than 6 elements a2dfbf6 [fwang12] add doc for install shaded jars 224cc38 [fwang12] remove transactions c7b9df3 [fwang12] address comments 81fc123 [fwang12] scala style 408c978 [fwang12] create kyuubi-shaded module to shade dependencies 6b15b45 [fwang12] shade hive service rpc b4b3b1a [fwang12] include curator-client 65086ed [fwang12] address comments a46f00f [fwang12] Shade curator dependencies to avoid impaction by external jars Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-09 11:29:24 +08:00
fwang12	1b7762e5f9	[KYUUBI #457] Support configurable initialize sql statement for engine startup ...  [](https://github.com/yaooqinn/kyuubi/pull/462)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Currently, the engine uses hardcoded show databases to force initialization with catalog eagerly bootstrapping too. We can make it configurable with a comma separate query list and select current_database() might be used as default since it's more lightweight than show databases in real-world cases ### _How was this patch tested?_ Existing UT. Closes #462 from turboFei/KYUUBI-457. Closes #457 e7bcf52 [fwang12] add ut 7446cd5 [fwang12] [KYUUBI#457】Support configurable initialize sql statement for engine startup Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-04-01 01:31:58 +08:00
Kent Yao	c6585b4708	[KYUUBI #471] Separate Embedded Zookeeper server from ha ...  [](https://github.com/yaooqinn/kyuubi/pull/471)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one 2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage. 3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #471 from yaooqinn/qp. Closes #471 1c24d02 [Kent Yao] address comments 9666a40 [Kent Yao] refine f069fc0 [Kent Yao] refine 8bb2f3a [Kent Yao] refine c20593a [Kent Yao] refine 8fa9a40 [Kent Yao] Merge branch 'master' into qp 1e2e1a3 [Kent Yao] tmp 7d66d05 [Kent Yao] tmp Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-30 18:34:01 +08:00
ulysses-you	066075ac74	[KYUUBI #456] Can not cancel SQL job if the timeout value is small ...  [](https://github.com/yaooqinn/kyuubi/pull/461)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> * avoid condition race * avoid job leak ### _How was this patch tested?_ pass `org.apache.kyuubi.engine.spark.SparkEngineSuites` close #456 Closes #461 from ulysses-you/thread. 970ba7a [ulysses-you] fix 8786779 [ulysses-you] npe d82321e [ulysses-you] remove c28e671 [ulysses-you] interrupt execute statement 8dccf73 [ulysses-you] cleanup 29857e3 [ulysses-you] sync Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 23:50:24 +08:00
Kent Yao	392a10ce71	[KYUUBI #458] Mute Console Appender for CI ...  [](https://github.com/yaooqinn/kyuubi/pull/458)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Rely on the unit-tests.log is better for us to reduce the overhead of CIs ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #458 from yaooqinn/log. 7eb1521 [Kent Yao] logbetter Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 16:41:56 +08:00
ulysses-you	fecdba32a7	[KYUUBI #451] Support query auto timeout cancel on thriftserver ...  [](https://github.com/yaooqinn/kyuubi/pull/451)       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Manual cherry-pick some Spark patch into Kyuubi. 1. [Support query auto timeout cancel on thriftserver](https://github.com/apache/spark/pull/29933) 2. [Add config to control if cancel invoke interrupt task on thriftserver](https://github.com/apache/spark/pull/30481) In order to keep backward with early Spark version, we hard code the config key instead of refer to Spark SQLConf. Note that, the exists timeout of operator (`kyuubi.operation.idle.timeout`) is to cancel that client has no access with engine. That said if a query run a long time and the client is alive, the query would not be cancelled. Then the new added config `spark.sql.thriftServer.queryTimeout` can handle this case. ### _How was this patch tested?_ Add new test. Closes #451 from ulysses-you/query-timeout. 212f579 [ulysses-you] docs 9206538 [ulysses-you] empty flaky test ddab9bf [ulysses-you] flaty test 1da02a0 [ulysses-you] flaty test edfadf1 [ulysses-you] nit 3f9920b [ulysses-you] address comment 9492c48 [ulysses-you] correct timeout 5df997e [ulysses-you] nit 2124952 [ulysses-you] address comment 192fdcc [ulysses-you] fix tets d684af6 [ulysses-you] global config 1d1adda [ulysses-you] empty 967a63e [ulysses-you] correct import 128948e [ulysses-you] add session conf in session 144d51b [ulysses-you] fix a90248b [ulysses-you] unused import c90386f [ulysses-you] timeout move to operation manager d780965 [ulysses-you] update docs a5f7138 [ulysses-you] fix test f7c7308 [ulysses-you] config name 7f3fb3d [ulysses-you] change conf place 97a011e [ulysses-you] unnecessary change 0953a76 [ulysses-you] move test 38ac0c0 [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into query-timeout 71bea97 [ulysses-you] refector implementation 35ef6f9 [ulysses-you] update conf 0cad8e2 [ulysses-you] Support query auto timeout cancel on thriftserver Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 14:04:09 +08:00
Kent Yao	207a7df934	[KYUUBI #453] Refine ServiceDiscovery in Serverable API ...  [](https://github.com/yaooqinn/kyuubi/pull/453)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> A service discovery should be a default module of Severable API and shall be initialized after `FroundendService` for a resolved serviceUri. it's a reflection only fix to enhance this logic ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #453 from yaooqinn/refine. 077dca0 [Kent Yao] Refine ServiceDiscovery in Serverable API Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-26 12:18:37 +08:00
Kent Yao	9e7649296d	[KYUUBI #449] Add Basic Metrics Service framework ...  [](https://github.com/yaooqinn/kyuubi/pull/450)    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds a future-proofing Metrics Service framework close #499 add some reporters, like json file, jmx, etc.. add some basic metrics, like statement.error ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible WIP - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #450 from yaooqinn/449. ea21a43 [Kent Yao] add tests 6513d07 [Kent Yao] update codecov 7cfe318 [Kent Yao] update docs a8454d0 [Kent Yao] update docs 91c33f4 [Kent Yao] update deps 05b13fb [Kent Yao] fix dep issues babc739 [Kent Yao] Add Basic Metrics Service Platform acc716a [Kent Yao] Add Basic Metrics Service Platform Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-24 18:56:56 +08:00
ulysses-you	a09178b578	[KYUUBI #410] Failed to start Spark 3.1 with error msg 'Cannot modify the value of a Spark config' ...  [](https://github.com/yaooqinn/kyuubi/pull/440)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the exception if we pass some unnecessary config to Spark. It's hard to list the whole config which Spark disallowed, this patch just ingore the expcetion that Spark throw. ### _How was this patch tested?_ Pass the test `SessionSuite` Closes #440 from ulysses-you/fix-modify-spark-conf. 5ab858a [ulysses-you] msg 3ba3587 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-16 16:09:41 +08:00
谢波	9c13f73bd9	[KYUUBI #417] [MINOR] Remove all unused import ...  [](https://github.com/yaooqinn/kyuubi/pull/417)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> ### _Why are the changes needed?_ remove all unused import and replace deprecated class PersistentEphemeralNode with PersistentNode in ServiceDiscovery ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #417 from MyLanPangzi/master. 821e0d2 [hiscat] Merge branch 'master' into master db9cd3c [谢波] [MINOR]: revert ServiceDiscovery change and add scala maven plugin compiler args c56cd2b [谢波] [MINOR]: revert ProcBuilder indent ad5e24a [谢波] [MINOR]: revert maven.plugin.scalatest.exclude.tags properties 7c23e37 [谢波] [MINOR]: scala maven plugin config move to plugin management 6029b99 [谢波] [MINOR]: remove all unused import a305567 [谢波] [MINOR]: remove all unused import Lead-authored-by: 谢波 <xiebo1@yonghui.cn> Co-authored-by: hiscat <46845236+MyLanPangzi@users.noreply.github.com> Signed-off-by: Cheng Pan <379377944@qq.com>	2021-03-15 20:05:08 +08:00
Honglun	e2fd967c58	[KYUUBI #397] Add test case for querying columns with dots ...  [](https://github.com/yaooqinn/kyuubi/pull/405)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add test case for querying columns with dots. closes #397 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #405 from Honglun/master. bd9d503 [Honglun] Add test case for querying columns with dots Authored-by: Honglun <283471011@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-08 20:13:54 +08:00
Kent Yao	5e2a1290c3	Prepare Kyuubi v1.2.0	2021-03-08 19:50:12 +08:00
ulysses-you	03b7112eee	[KYUUBI #403] FIx flaky test in ConnectionLevelSparkEngineSuite ...  [](https://github.com/yaooqinn/kyuubi/pull/403)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ `ignore` -> `test` Closes #403 from ulysses-you/flaky-test2. d8aa6c7 [ulysses-you] empty fd38e9c [ulysses-you] empty 2e5a7ee [ulysses-you] empty 444a526 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-08 11:22:19 +08:00
Kent Yao	352b4eb99e	[KYUUBI #400] Implement a simpler toHiveString to convert spark row ...  [](https://github.com/yaooqinn/kyuubi/pull/400)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Hive compatibility is essential! This PR tries to restore the previous behavior. 1. Not use Spark's internal class HiveResult to convert spark catalyst type-values to JDBC results, the interval APIs are not APIs and changed unexpectedly. It involves a lot of overhead. (For this reason, pan3793 and I propose a Casting way to solve it.) 2. Unfortunately, the casting way breaks things like #397. So this is a reverting PR. 3. I checked the spark `HiveResult` carefully, I noticed that the `nested` logic is not needed in Kyuubi as we already matched primitives ahead. 4. Also the date time formatter used in `HiveResult` is excessively packaged with massively complicated logic inside spark. It's not clear and simple enough to call. 5. (tiny breaking change)the trailing spaces of a timestamp in milliseconds may not be trimmed. But this is a correctness issue and acceptable. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #400 from yaooqinn/397. 388c86d [Kent Yao] nit bbf458b [Kent Yao] time stamp formatter e5a7436 [Kent Yao] more tests 83f3142 [Kent Yao] timezone 38f5a76 [Kent Yao] Merge branch 'master' into 397 563d69c [Kent Yao] mute flacky test 9d834b0 [Kent Yao] Merge branch 'master' into 397 e8e05b5 [Kent Yao] nit 2a2d47b [Kent Yao] Implement a simpler toHiveString to convert spark row Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-05 17:12:58 +08:00
Cheng Pan	4abc3ff6a0	[KYUUBI #398] Move to hadoop shaded client ...  [](https://github.com/yaooqinn/kyuubi/pull/399)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #398 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request manual test release workflow ``` ➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz -rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz -rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz ``` Closes #399 from pan3793/KYUUBI-398. a9294a1 [Cheng Pan] fix ut d1d816d [Cheng Pan] fix dist script 2e3bc20 [Cheng Pan] update release workflow and dist script 0428b1b [Cheng Pan] update travis.yml 4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 23:33:31 +08:00
Cheng Pan	8dd54a29b3	[KYUUBI #383] Support beeline --hivevar --hiveconf ...  [](https://github.com/yaooqinn/kyuubi/pull/394)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #383 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #394 from pan3793/KYUUBI-383. 2810b58 [Cheng Pan] [KYUUBI #383] remove regex match, add server side test cases 3d8b1e0 [Cheng Pan] [KYUUBI #383] Support beeline --hivevar --hiveconf Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 21:28:22 +08:00
Kent Yao	2e3315867b	[KYUUBI #298]Add Support for Apache Spark 3.1.1 ...  [](https://github.com/yaooqinn/kyuubi/pull/366)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved. In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7` TODOS: - Add Hadoop 3.2 support with some dependency issues - Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release - Using spark-3.1.1 as default - Restore HiveResult string? fix #298 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #366 from yaooqinn/spark3.1. 28ddf93 [Kent Yao] ga 37cb49e [Kent Yao] Merge branch 'master' into spark3.1 7b332ec [Kent Yao] typo b6b33d5 [Kent Yao] set up ga 00005c0 [Kent Yao] set up travis 2c09fe4 [Kent Yao] test fix 68c7edb [Kent Yao] deps 2c65e98 [Kent Yao] deps 236c7e9 [Kent Yao] nit 8c8a783 [Kent Yao] Merge branch 'master' into spark3.1 46c318a [Kent Yao] nit 32f6511 [Kent Yao] Merge branch 'master' into spark3.1 27c836b [Kent Yao] Merge branch 'master' into spark3.1 5574a8e [Kent Yao] ga d2ec2d5 [Kent Yao] [WIP]Spark3.1 49313cc [Kent Yao] [WIP]Travis ccb1a97 [Kent Yao] [WIP]Spark3.1 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 13:30:05 +08:00
ulysses-you	aa878e4d48	[KYUUBI #386] Clean up discovery service before engine stop in connection share level ...  [](https://github.com/yaooqinn/kyuubi/pull/387)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time. ### _How was this patch tested?_ Add some new suites in spark engine moudle with zk emmbedded. Closes #387 from ulysses-you/issue-386. ccb1112 [ulysses-you] move method 1558dd9 [ulysses-you] test 6cd7e57 [ulysses-you] nit f8559d2 [ulysses-you] check level at engine discovery eedfaee [ulysses-you] split ServiceDiscovery to server and engine 8b20e6c [ulysses-you] timeout 0cf524c [ulysses-you] remove unnecessary test 41d36f9 [ulysses-you] move exception eaaa12d [ulysses-you] avoid stop twice b801c14 [ulysses-you] fix test db39960 [ulysses-you] add start check 2d1f6dd [ulysses-you] add state check e7374aa [ulysses-you] nit ddf383c [ulysses-you] fi ee088be [ulysses-you] nit 3c2013b [ulysses-you] nit 73b386f [ulysses-you] improve conf of test 2185f49 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-03-04 11:31:41 +08:00
ulysses-you	4eed2ee0a7	[KYUUBI #385] Improve pass config between kyuubi and spark engine ...  [](https://github.com/yaooqinn/kyuubi/pull/385)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make test more flexible and readable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #385 from ulysses-you/cleanup-test. c5d2034 [ulysses-you] readable fbc0f16 [ulysses-you] readable b1008b7 [ulysses-you] fix 37f5739 [ulysses-you] comment fdf52a7 [ulysses-you] add config method c58dfdb [ulysses-you] fix tes a1e8517 [ulysses-you] improve pass config 1378131 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-03 10:01:54 +08:00
ulysses-you	7f2a24bd9f	[KYUUBI #381] Release session if shared level is CONNECTION ...  [](https://github.com/yaooqinn/kyuubi/pull/382)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> In CONNECTION level, it's better to release session if the connection is closed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #382 from ulysses-you/KYUUBI-381. cddba6c [ulysses-you] type a3fd691 [ulysses-you] increase engine init timeout 5030553 [ulysses-you] reload system properties 7a2e72f [ulysses-you] clean code 1034409 [ulysses-you] stopped d2a001a [ulysses-you] address comment 4eb62de [ulysses-you] 60s 64d0f16 [ulysses-you] fix test 4f10d34 [ulysses-you] sleep 62dc9f9 [ulysses-you] add new stop method 77be261 [ulysses-you] remove config fd9b73c [ulysses-you] add time sleep 209a59e [ulysses-you] fix test 88ed744 [ulysses-you] import 90015e9 [ulysses-you] simply 311fce3 [ulysses-you] fix test 7af3f72 [ulysses-you] fix test e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381 f77ddbe [ulysses-you] nit 2173222 [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-03-02 11:44:32 +08:00
Kent Yao	0f62568171	[KYUUBI #379] Improve engine log capture ...  [](https://github.com/yaooqinn/kyuubi/pull/379)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead 2. update `error` immediately when detecting errors at the very first line ASAP 3. Using `Exception:` instead of `Exception` to search each line to filter better 4. add the engine log address in the exception message in case of error bootstraping ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #379 from yaooqinn/logsize. 7cf58da [Kent Yao] mute in console b58a3a2 [Kent Yao] update fbc4a0b [Kent Yao] update 0815f77 [Kent Yao] log 93f5f2f [Kent Yao] log Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 17:31:44 +08:00
Cheng Pan	2f8df2355d	[KYUUBI #377] build/dist support --spark-provided ...  [](https://github.com/yaooqinn/kyuubi/pull/378)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request - [x] Manual test `build/dist --spark-provided` ``` (kyuubi) ➜ kyuubi git:(master) ✗ tree dist dist ├── LICENSE ├── RELEASE ├── bin │   ├── kyuubi │   ├── kyuubi-logo │   └── load-kyuubi-env.sh ├── conf │   ├── kyuubi-defaults.conf.template │   ├── kyuubi-env.sh.template │   └── log4j.properties.template ├── externals │   └── engines │   └── spark │   └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar ├── jars │   ├── apacheds-i18n-2.0.0-M15.jar │   ├── apacheds-kerberos-codec-2.0.0-M15.jar │   ├── api-asn1-api-1.0.0-M20.jar │   ├── api-util-1.0.0-M20.jar │   ├── audience-annotations-0.5.0.jar │   ├── commons-beanutils-1.7.0.jar │   ├── commons-beanutils-core-1.8.0.jar │   ├── commons-codec-1.4.jar │   ├── commons-collections-3.2.2.jar │   ├── commons-configuration-1.6.jar │   ├── commons-digester-1.8.jar │   ├── commons-io-2.4.jar │   ├── commons-lang-2.6.jar │   ├── commons-lang3-3.10.jar │   ├── commons-math-2.2.jar │   ├── curator-client-2.7.1.jar │   ├── curator-framework-2.7.1.jar │   ├── curator-recipes-2.7.1.jar │   ├── curator-test-2.7.1.jar │   ├── guava-11.0.2.jar │   ├── hadoop-annotations-2.7.4.jar │   ├── hadoop-auth-2.7.4.jar │   ├── hadoop-common-2.7.4.jar │   ├── hive-service-rpc-2.3.7.jar │   ├── httpclient-4.5.6.jar │   ├── httpcore-4.4.12.jar │   ├── javassist-3.18.1-GA.jar │   ├── jcl-over-slf4j-1.7.30.jar │   ├── jsr305-3.0.2.jar │   ├── kyuubi-common-1.1.0-SNAPSHOT.jar │   ├── kyuubi-ha-1.1.0-SNAPSHOT.jar │   ├── kyuubi-main-1.1.0-SNAPSHOT.jar │   ├── libfb303-0.9.3.jar │   ├── libthrift-0.9.3.jar │   ├── log4j-1.2.17.jar │   ├── netty-3.7.0.Final.jar │   ├── scala-library-2.12.10.jar │   ├── slf4j-api-1.7.30.jar │   ├── slf4j-log4j12-1.7.30.jar │   ├── spotbugs-annotations-3.1.9.jar │   └── zookeeper-3.4.14.jar ├── logs ├── pid └── work ``` Bundle size ``` (kyuubi) ➜ kyuubi git:(KYUUBI-377) ll | grep tar.gz -rw-r--r-- 1 chengpan staff 230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz -rw-r--r-- 1 chengpan staff 20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz ``` Closes #378 from pan3793/KYUUBI-377. 442569e [Cheng Pan] add profile spark-provided for skip download spark binary 0d8da24 [Cheng Pan] update doc 5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>	2021-02-26 15:00:42 +08:00
Kent Yao	968ed4f079	[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test ...  [](https://github.com/yaooqinn/kyuubi/pull/376)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #376 from yaooqinn/hive. 26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-26 00:28:17 +08:00
Kent Yao	ce7b122435	[KYUUBI #371] Avoid creating useless operation log for meta operation ...  [](https://github.com/yaooqinn/kyuubi/pull/371)     [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ 1. Operation log for metadata is useless for client-side 2. Operation log for metadata dumps nothing useful for the engine side too 3. Operation log will create files that bring overhead for meta operations and may fail operations ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #371 from yaooqinn/oplog. ca62e74 [Kent Yao] Avoid creating useless operation log for meta operation Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 20:47:48 +08:00
Cheng Pan	c659089bc2	[KYUUBI #360] Correct handle getNextRowSet with FETCH_PRIOR FETCH_FIRST ...  [](https://github.com/yaooqinn/kyuubi/pull/370)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #360 Ref: https://github.com/apache/spark/pull/30600 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #370 from pan3793/KYUUBI-360. e79b8cb [Cheng Pan] [KYUUBI #360] comments 0fae3db [Cheng Pan] fix import 3d1b2a6 [Cheng Pan] [KYUUBI #360] fix ut eda3e59 [Cheng Pan] [KYUUBI #360] fix import 16178d6 [Cheng Pan] [KYUUBI #360] ut 179404d [Cheng Pan] [KYUUBI #360] nit 455af6b [Cheng Pan] [KYUUBI #360] correct getNextRowSet with FETCH_PRIOR FETCH_FIRST 2307f1f [Cheng Pan] [KYUUBI #360] move ThriftUtils to kyuubi-common Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 18:19:02 +08:00
Kent Yao	4b13f1c551	[KYUUBI #367] Engine terminating checker should start after exposed ...  [](https://github.com/yaooqinn/kyuubi/pull/367)      [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating. In this PR, we move the checker to the end where the engine is exposed to the servers. Avoiding errors like: ``` org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50) at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43) at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #367 from yaooqinn/engine. f6a979e [Kent Yao] fix test 9cc561c [Kent Yao] address comment 3d15ab2 [Kent Yao] nit a85d2e0 [Kent Yao] Engine terminating checker should start after exposed Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-25 15:10:43 +08:00
Kent Yao	ee50890b00	[KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility ...  [](https://github.com/yaooqinn/kyuubi/pull/358)    [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> close #310 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #358 from yaooqinn/310. 8cb30a4 [Kent Yao] sql wildcards to java regex 34d2c3a [Kent Yao] Merge branch 'master' into 310 d332be5 [Kent Yao] [KYUUBI #310] GetColumns supports DSv2 and keeps its backward compatibility Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>	2021-02-23 20:43:02 +08:00