<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid.
Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached.
With multiple engines support, end-users are able to
1. create as many engines as they want to meet the concurrency.
2. leverage different queue/namespace resource with long-running mode
3. can give different resources or priorities to their own engines for different workloads
4. can perform fast failover by creating a new sub domain
5. etc.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#627 from yaooqinn/multipleengines.
Closes#627
9f04bc3e [Kent Yao] fix tests
d7af751e [Kent Yao] address comments
be9c46d8 [Kent Yao] address comments
ccce87d7 [Kent Yao] address comments
d49116c7 [Kent Yao] Support multiple engines for non-connection share levels
f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Change the config keys of metrics to get a consistent name style, `metrics.{reporter_type}.{property}`.
The change breaks the compatibility with master, but consider v1.2.0 is not out yet, so changes should be safe.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#601 from pan3793/metrics.
Closes#601
89d1974f [Cheng Pan] address comments
361ae6e2 [Cheng Pan] revert default metrics reporter change
1c493e64 [Cheng Pan] rat exclude report.json
fcdb4252 [Cheng Pan] exclude report.json
337e09ae [Cheng Pan] Default enable SFL4J metrics reporter
6b9f3963 [Cheng Pan] [METRICS] Change metrics config keys
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Involve job max failures to limit the max job failures before deregistering the engine, involve deregister exception TTL for to restart counting when last failure has gone for one deregisterExceptionTTL.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#560 from turboFei/KYUUBI-544-exception-max-failures-new.
Closes#544
951f269 [fwang12] [KYUUBI #544] Involve job max failures and deregister exception TTL
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
To simplify the usage and be more user friendly, combine `kyuubi.engine.deregister.exception.messages` and `kyuubi.engine.deregister.exception.stacktraces` to `kyuubi.engine.deregister.exception.messages`.
### _How was this patch tested?_
Existing UT.
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#565 from turboFei/KYUUBI-544-combine.
Closes#544
ee63f60 [fwang12] [KYUUBI #544] Combine deregister exception messages and stacktraces
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Involve `kyuubi.engine.deregister.exception.stacktraces` to deregister engine when meeting specified exception stacktraces.
### _How was this patch tested?_
Existing UT.
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#557 from turboFei/KYUUBI-544-exception-msg-stack-rebase.
Closes#544
beedb9c6 [fwang12] stopping -> deregistering
80373fca [fwang12] [KYUUBI #544] Deregister engine when meeting specified stacktrace
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Involve `kyuubi.engine.deregister.exception.messages` to deregister engine when meeting specified exception messages.
### _How was this patch tested?_
Existing UT.
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#556 from turboFei/KYUUBI-544-exception-msg-rebase.
Closes#544
c1fed723 [fwang12] address comments
7cfba913 [fwang12] [KYUUBI #544] Deregister engine when meeting specified exception message
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Involve `kyuubi.engine.deregister.exception.classes` to deregister engine when job failed because of specified exception class
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#549 from turboFei/KYUUBI-544-exception-class.
Closes#544
9ff83b80 [fwang12] use lazy val to avoid npe
d80d4436 [fwang12] [KYUUBI #544] Add kyuubi.engine.deregister.exception.classes to deregister engine when job failed because of specified exception class
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Refactor the key words after repo renamed.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
 [](https://github.com/yaooqinn/kyuubi/pull/451)       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Manual cherry-pick some Spark patch into Kyuubi.
1. [Support query auto timeout cancel on thriftserver](https://github.com/apache/spark/pull/29933)
2. [Add config to control if cancel invoke interrupt task on thriftserver](https://github.com/apache/spark/pull/30481)
In order to keep backward with early Spark version, we hard code the config key instead of refer to Spark SQLConf.
Note that, the exists timeout of operator (`kyuubi.operation.idle.timeout`) is to cancel that client has no access with engine. That said if a query run a long time and the client is alive, the query would not be cancelled. Then the new added config `spark.sql.thriftServer.queryTimeout` can handle this case.
### _How was this patch tested?_
Add new test.
Closes#451 from ulysses-you/query-timeout.
212f579 [ulysses-you] docs
9206538 [ulysses-you] empty flaky test
ddab9bf [ulysses-you] flaty test
1da02a0 [ulysses-you] flaty test
edfadf1 [ulysses-you] nit
3f9920b [ulysses-you] address comment
9492c48 [ulysses-you] correct timeout
5df997e [ulysses-you] nit
2124952 [ulysses-you] address comment
192fdcc [ulysses-you] fix tets
d684af6 [ulysses-you] global config
1d1adda [ulysses-you] empty
967a63e [ulysses-you] correct import
128948e [ulysses-you] add session conf in session
144d51b [ulysses-you] fix
a90248b [ulysses-you] unused import
c90386f [ulysses-you] timeout move to operation manager
d780965 [ulysses-you] update docs
a5f7138 [ulysses-you] fix test
f7c7308 [ulysses-you] config name
7f3fb3d [ulysses-you] change conf place
97a011e [ulysses-you] unnecessary change
0953a76 [ulysses-you] move test
38ac0c0 [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into query-timeout
71bea97 [ulysses-you] refector implementation
35ef6f9 [ulysses-you] update conf
0cad8e2 [ulysses-you] Support query auto timeout cancel on thriftserver
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/367)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating.
In this PR, we move the checker to the end where the engine is exposed to the servers.
Avoiding errors like:
```
org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#367 from yaooqinn/engine.
f6a979e [Kent Yao] fix test
9cc561c [Kent Yao] address comment
3d15ab2 [Kent Yao] nit
a85d2e0 [Kent Yao] Engine terminating checker should start after exposed
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
fixes#284
We will create and write a log during submit a new session (like spark-submit) but we don't delete or overwrite it. This PR aims to add a time config to decide if we need to overwrite a session submit log.
The log path template is `KYUUBI_WORD_DIR/user/$engine-log.$index`. We will increase the `index` automatically if a user create some session concurrently.
After this PR. The log path template is `KYUUBI_WORD_DIR/user/$engine.log.$index`.
Squashed commit of the following:
commit 7794406098df0a3129ccd2ab04a445d2155bef8e
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 19:52:10 2021 +0800
fix
commit 13fd3bbb577c8bdad0ef73851262344d2870fd4d
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 19:02:36 2021 +0800
fix docs
commit 82c192bc6bccf749f7d7232ed4f03f3c0a56c65f
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 18:04:50 2021 +0800
fix
commit 134ea4389b0b9fddc777c79b1a9f43fc59ec3f7e
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 17:20:55 2021 +0800
docs
commit 921ac11b27c028c508e834b88f9f9d9e17eb104a
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 17:16:09 2021 +0800
fix config
commit bfcbd9633a18e9a4cf0b5b2308855ad8af7f7efd
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:36:00 2021 +0800
prefix
commit bceb8c59cbd7d7757b2c30058f845fa1e3c7ee9d
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:14:20 2021 +0800
doc
commit 4dd556333c7e6e8181f595293bb3877473c00ee7
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:11:00 2021 +0800
init
