https://github.com/apache/incubator-kyuubi/pull/1177
### _Why are the changes needed?_
change configuration(ha.zookeeper.acl.engine.enabled) version to `1.3.1`
### _How was this patch tested?_
- [] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [] Add screenshots for manual tests if appropriate
- [] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1193 from cxzl25/KYUUBI-1176-followup.
Closes#1176
72753ae5 [sychen] change config version
Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
(cherry picked from commit 464fdf456b)
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
https://github.com/apache/incubator-kyuubi/issues/1176
When `kyuubi.ha.zookeeper.acl.enabled=true`, both service and engine will use zookeeper acl to create znode, but engine has no keytab information and cannot write information to zookeeper, throwing an exception.
```java
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi_USER/XXXX
at org.apache.zookeeper.KeeperException.create(KeeperException.java:124)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1177 from cxzl25/KYUUBI-1176.
Closes#1176
ecc08fa7 [sychen] fix engine acl
0b7cc2ec [sychen] fix InvalidACL
Authored-by: sychen <sychen@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit d332534325)
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Executing test case "ServiceDiscoverySuite set up zookeeper auth" on a Linux host named "KYUUBI" fails.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1006 from zhouyifan279/KYUUBI#1005.
Closes#1005
045c6b34 [zhouyifan279] [KYUUBI #1005] ServiceDiscoverySuite fails if hostname contains uppercase letter
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 866a63d7a7)
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
### Description
`ServiceDiscovery` contains `discovery` and `zookeeper` part, `ServiceDiscovery` should only has one responsibility.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#931 from timothy65535/ky-930.
Closes#930
29d48921 [timothy65535] Add ZooKeeperClientProvider
17fbe5a2 [timothy65535] [KYUUBI #930] Extract the zookeeper part from ServiceDiscovery
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The abstract `ServiceDiscovery` class has two subclass (KyuubiServiceDiscovery and EngineServiceDiscovery), but only abstract class name is always printed in the log.
```
2021-07-01 00:47:30.046 INFO kyuubi.SparkSQLEngineListener: Received ApplicationEnd Message from Spark at INITIALIZED, stopping
2021-07-01 00:47:30.047 INFO spark.SparkSQLEngine: Service: [ServiceDiscovery] is stopping.
2021-07-01 00:47:30.051 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
```
2021-07-01 01:53:09.124 INFO zookeeper.ClientCnxn: Socket error occurred: localhost/127.0.0.1:2181: Connection refused
2021-07-01 01:53:09.128 INFO client.EngineServiceDiscovery: Service[EngineServiceDiscovery] is initialized.
2021-07-01 01:53:09.128 INFO spark.SparkSQLEngine: Service[SparkSQLEngine] is initialized.
2021-07-01 01:54:13.417 INFO kyuubi.SparkSQLEngineListener: Received ApplicationEnd Message from Spark at INITIALIZED, stopping
2021-07-01 01:54:13.418 INFO spark.SparkSQLEngine: Service: [EngineServiceDiscovery] is stopping.
2021-07-01 01:54:13.420 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
```
Closes#733 from timothy65535/work.
Closes#732
31d8b04b [timothy65535] remove redundant constructor
2fa8e7b2 [timothy65535] [KYUUBI #732] Fix always print the name of abstract ServiceDiscovery class
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. refine the zk lock to lock only during engine bootstrapping, not wrap the part that only gets the address of engines. This can significantly reduce overhead.
2. add tests for the locking proccess.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#629 from yaooqinn/lock.
Closes#629
a1b4e5f1 [Kent Yao] Refine distributed lock to lock only engine bootstrap
a6738652 [Kent Yao] Refine distributed lock to lock only engine bootstrap
ff238e6f [Kent Yao] Refine distributed lock to lock only engine bootstrap
f3979fce [Kent Yao] Refine distributed lock to lock only engine bootstrap
e8ca9954 [Kent Yao] Refine distributed lock to lock only engine bootstrap
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds multiple engine support for non-connection share levels. A new configuration called `kyuubi.engine.share.level.sub.domain` is added for this functionality. For simplification, the configuration only accepts [1, 10]-length case insensitive string, only '_' and alphabets are valid.
Unlike connection mode, user mode supports the "long-running" feature across different JDBC connections. But the current implementation only is able to create one engine for a user. End-users don't want to put all eggs into one basket. It's not only less robust and flexible for a static engine to handle different workloads, and also hard to leverage failover when there is a zombied engine being cached.
With multiple engines support, end-users are able to
1. create as many engines as they want to meet the concurrency.
2. leverage different queue/namespace resource with long-running mode
3. can give different resources or priorities to their own engines for different workloads
4. can perform fast failover by creating a new sub domain
5. etc.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#627 from yaooqinn/multipleengines.
Closes#627
9f04bc3e [Kent Yao] fix tests
d7af751e [Kent Yao] address comments
be9c46d8 [Kent Yao] address comments
ccce87d7 [Kent Yao] address comments
d49116c7 [Kent Yao] Support multiple engines for non-connection share levels
f1d8bc17 [Kent Yao] Support multiple engines for non-connection share levels
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We may create multiple SparkContext in non-CONNECTION ShareLevel.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#624 from Honglun/feature/lock.
Closes#624
87445522 [Honglun] Add lock while creating engine with non-CONNECTION mode
Authored-by: Honglun <283471011@qq.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
 [](https://github.com/yaooqinn/kyuubi/pull/387)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time.
### _How was this patch tested?_
Add some new suites in spark engine moudle with zk emmbedded.
Closes#387 from ulysses-you/issue-386.
ccb1112 [ulysses-you] move method
1558dd9 [ulysses-you] test
6cd7e57 [ulysses-you] nit
f8559d2 [ulysses-you] check level at engine discovery
eedfaee [ulysses-you] split ServiceDiscovery to server and engine
8b20e6c [ulysses-you] timeout
0cf524c [ulysses-you] remove unnecessary test
41d36f9 [ulysses-you] move exception
eaaa12d [ulysses-you] avoid stop twice
b801c14 [ulysses-you] fix test
db39960 [ulysses-you] add start check
2d1f6dd [ulysses-you] add state check
e7374aa [ulysses-you] nit
ddf383c [ulysses-you] fi
ee088be [ulysses-you] nit
3c2013b [ulysses-you] nit
73b386f [ulysses-you] improve conf of test
2185f49 [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
