# 🔍 Description
## Issue References 🔗
This pull request fixes#6216
## Describe Your Solution 🔧
Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
SessionLimiterSuite.test("test session limiter with ip deny list")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6217 from wForget/KYUUBI-6216.
Closes#6216
e0a058c1d [wforget] update ctl
f66426eda [wforget] typo
5c5308ad8 [wforget] typo
959242e9b [wforget] Support to deny some client ips to make connection
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
# 🔍 Description
## Issue References 🔗
It was reported that https://github.com/beltran/gohive can not parse the Kyuubi-registered Zk endpoint, because it relies on the znode name's `serverUri` key, while Hive JDBC driver does not care about the znode name but the znode data
gohive's behavior:
5bd0599248/hive.go (L160-L168)
Kyuubi Hive JDBC driver behavior:
<img width="1043" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/06740985-82ef-4006-b7d6-de0b5b2c0c7c">
## Describe Your Solution 🔧
Simply change the zonde name's key from `serviceUri` to `serverUri` to keep it compatible with Hive behavior.
I suppose it won't break the usage of old Kyuubi Hive JDBC Driver / Kyuubi Beeline / Hive JDBC Driver / Hive Beeline.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested with Apache Hive 2.3.9 Beeline, works fine.
```
➜ ~ beeline -u 'jdbc:hive2://localhost:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi'
Connecting to jdbc:hive2://localhost:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi
Connected to: Spark SQL (version 3.3.3)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://localhost:2181/> select kyuubi_version();
...
23/12/08 16:40:23 INFO ExecuteStatement:
Spark application name: kyuubi_CONNECTION_SPARK_SQL_anonymous_5b8bf1dc-dbc8-4623-8aaa-b43dba435f83
application ID: local-1702024801170
application web UI: http://10.221.99.150:32835
master: local[*]
deploy mode: client
version: 3.3.3
Start time: 2023-12-08T16:40:00.220
User: anonymous
23/12/08 16:40:23 INFO ExecuteStatement: Execute in full collect mode
...
+-------------------+
| kyuubi_version() |
+-------------------+
| 1.9.0-SNAPSHOT |
+-------------------+
1 row selected (0.815 seconds)
0: jdbc:hive2://localhost:2181/>
```
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5833 from pan3793/gohive.
Closes#5833
0a3910426 [Cheng Pan] Rename service register endpoint key from serviceUri to serverUri
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
TL;DR there are some issues with shading Thrift RPC classes during the engine packaging phase, see details in the PR description of https://github.com/apache/kyuubi-shaded/pull/20.
## Describe Your Solution 🔧
This PR aims to migrate from vanilla `hive-service-rpc`, `libfb303`, `libthrift` to `kyuubi-relocated-hive-service-rpc` introduced in https://github.com/apache/kyuubi-shaded/pull/20, the detailed works are:
- replace imported deps in `pom.xml` and rename the package prefix in all modules, except for
- `kyuubi-server` there are a few places use vanilla thrift classes to access HMS to get token
- `kyuubi-hive-sql-engine` Hive method invocation
- update relocations rules in modules that creates shaded jar
- introduce `HiveRpcUtils` in `kyuubi-hive-sql-engine` module for object conversion.
As part of the whole change, this PR upgrades from the Kyuubi Shaded 0.1.0 to 0.2.0, which changes the jars name. see https://kyuubi.apache.org/shaded-release/0.2.0.html
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Pass all Hive UT with Hive 3.1.3, and IT with Hive 3.1.3 and 2.3.9 (also tested with 2.1.1-cdh6.3.2)
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5783 from pan3793/rpc-shaded.
Closes#5783
b45d4deaa [Cheng Pan] remove staging repo
890076a20 [Cheng Pan] Kyuubi Shaded 0.2.0 RC0
071945d45 [Cheng Pan] Rebase
199794ed9 [Cheng Pan] fix
fc128b170 [Cheng Pan] fix
26d313896 [Cheng Pan] fix
632984c92 [Cheng Pan] fix
428305589 [Cheng Pan] fix
6301e28fd [Cheng Pan] fix
955cdb33b [Cheng Pan] Switch to kyuubi-shaded-hive-service-rpc
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Infer the batch user from session or metadata, user do not need to specify the proxy user anymore.
This pr also align the behavior of BatchesResource with that of SessionsResource and OperationsResource(no proxy user parameter).
For Kyuubi Batch, Session and Operation, these resources have the user attiribute.
So we only need to check whether the authentication user has the permission to access the resource.
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [ ] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5717 from turboFei/hive_server2_proxy_user.
Closes#5717
70ad7e76d [fwang12] comment
c721a751a [fwang12] ignore
da92bd5a1 [fwang12] fix ut
9a197d005 [fwang12] doc
c8ed5f9cf [fwang12] ut
cef9e329c [fwang12] do not use proxy user
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.
This PR to fix the performance issue on our side.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5400 from ITzhangqiang/KYUUBI_5365.
Closes#5365
dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4994
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5157 from lsm1/branch-kyuubi_4994.
Closes#4994
02e7eb5fa [senmiaoliu] use abbr a
6d001e519 [senmiaoliu] replace 'allengine' with 'all'
f9c548299 [senmiaoliu] fix style
df64e773e [senmiaoliu] fix style
6a7d40e63 [senmiaoliu] remove allenginecommand
6306dd8a2 [senmiaoliu] support list all engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
Due to #5078, we marked kyuubi_instance in metadata can be null.
We should append kyuubi_instance after we check it's non-null.
Otherwise we may face the following error when we use v2 submit job
```
2023-09-14 19:05:43 [INFO] [main] org.apache.kyuubi.client.RetryableRestClient#74 - Current connect server uri http://null/api/v1
2023-09-14 19:05:43 [ERROR] [main] org.apache.kyuubi.client.RestClient#189 - Error:
java.net.UnknownHostException: null: Name or service not known
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5297 from zwangsheng/client/resetclient_get_null.
Closes#5297
2b4b56020 [zwangsheng] [CLIENT] Get null from metadata.kyuubiInstance
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Improvement: https://github.com/apache/kyuubi/issues/4540
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5182 from zhuyaogai/issue-4540.
Closes#4540
73652ae6d [Fantasy-Jay] Merge remote-tracking branch 'origin/issue-4540' into issue-4540
ae7a3b5bb [Fantasy-Jay] Support to deny some user to make connection.
46ee2f158 [Fantasy-Jay] Support to limit some user to make connection.
3d7e5e534 [Fantasy-Jay] Support to limit some user to make connection.
1f173ffe4 [Fantasy-Jay] Merge branch 'master' into issue-4540
1d3a3d47f [Fantasy-Jay] Support to limit some user to make connection.
e4a1b3eb3 [Fantasy-Jay] Support to limit some user to make connection.
1c120a992 [Fantasy-Jay] Support to limit some user to make connection.
6b657aa99 [Fantasy-Jay] Support to limit some user to make connection.
506c4751b [Fantasy-Jay] Support to limit some user to make connection.
f99072ef5 [Fantasy-Jay] Support to limit some user to make connection.
Authored-by: Fantasy-Jay <13631435453@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- introducing `scala-2.13` for development use only
- adapt the common modules compilable on Scala 2.13, including
- kyuubi-common
- kyuubi-ctl
- kyuubi-ha
- use separate compile options for Scala 2.13 , skipping deprecated scala compiler option `-Yno-adapted-args`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5183 from bowenliang123/scala213-common.
Closes#5183
1e65a5d03 [Bowen Liang] Align scala 2.13 version with Spark to 2.13.8
0003c8bed [liangbowen] duplicate scala-maven-plugin configs in profile
8b448a5f0 [liangbowen] 2.13.11
36d2284c4 [liangbowen] comment
28b515a0c [liangbowen] comment
865c5e55b [liangbowen] update
52e381200 [liangbowen] ensure kyuubi-ctl module compilable on scala 2.13
70dbc7ae2 [liangbowen] ensure kyuubi-ha module compilable on scala 2.13
84c1e4b4d [liangbowen] ensure kyuubi-common module compilable on scala 2.13
e116d77f5 [liangbowen] skip deprecated compiler option "-Yno-adapted-args" in 2.13
26c27a2ee [liangbowen] introduce new profile scala-2.13
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Use `Iterable` instead of `Seq` in kyuubi-ctl commands for Scala compatibility, as
1. in Scala 2.13, the `scala.Seq` is now an alias for `scala.collection.immutable.Seq` (instead of `scala.collection.Seq`)
2. in Scala 2.13, `scala.collection.mutable.ListBuffer` (or Buffers) does not extend `scala.collection.immutable.Seq` according to [Scala collection migration guide](https://docs.scala-lang.org/overviews/core/collections-migration-213.html)
3. in both Scala 2.12 and 2.13, `ListBuffer` (or Buffers) extends `scala.collection.mutable.Iterable`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5179 from bowenliang123/ctlcmd-iter.
Closes#5179
3265e3b99 [liangbowen] Use Iterable instead of Seq in kyuubi-ctl commands
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to filter batch with batch name filter condition.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5063 from turboFei/batch_name.
Closes#5063
63915a56d [fwang12] ut
2548815a2 [fwang12] ut
34a9229b0 [fwang12] style
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This is a followup of #4843
To support load kubernetes conf during runtime, so that we can support more kuberntes contexts without restarting the kyuubi server.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4996 from turboFei/refresh_kubernetes_conf.
Closes#4996
807fb92e4 [fwang12] comments
d42d25af7 [fwang12] from conf
809a7d3df [fwang12] refresh
22743f9e5 [fwang12] save
dedebbe71 [fwang12] api
d6f58cfc7 [fwang12] refresh kubernetes config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- To improve Scala code with corrections, simplification, scala style, redundancy cleaning-up. No feature changes introduced.
Corrections:
- Class doesn't correspond to file name (SparkListenerExtensionTest)
- Correct package name in ResultSetUtil and PySparkTests
Improvements:
- 'var' could be a 'val'
- GetOrElse(null) to orNull
Cleanup & Simplification:
- Redundant cast inspection
- Redundant collection conversion
- Simplify boolean expression
- Redundant new on case class
- Redundant return
- Unnecessary parentheses
- Unnecessary partial function
- Simplifiable empty check
- Anonymous function convertible to a method value
Scala Style:
- Constructing range for seq indices
- Get and getOrElse to getOrElse
- Convert expression to Single Abstract Method (SAM)
- Scala unnecessary semicolon inspection
- Map and getOrElse(false) to exists
- Map and flatten to flatMap
- Null initializer can be replaced by _
- scaladoc link to method
Other Improvements:
- Replace map and getOrElse(true) with forall
- Unit return type in the argument of map
- Size to length on arrays and strings
- Type check can be pattern matching
- Java mutator method accessed as parameterless
- Procedure syntax in method definition
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4959 from bowenliang123/scala-Improve.
Closes#4959
2d36ff351 [liangbowen] code improvement for Scala
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Support to list server with kyuubi-admin/kyuubi rest client.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- UT for AdminRestAPI
- UT for AminCtlArgument
- UT for AdminCtl
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4889 from turboFei/list_server.
Closes#4889
bfd13fbde [fwang12] nit
0a0131552 [fwang12] list server
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17.
There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical.
With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested
zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).
We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4852 from pan3793/shaded-zk.
Closes#4852
d960cc945 [Cheng Pan] remove staging repo
1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Usually, the host url to create a batch is the load balancer uri.
To reduce the internal rest redirection when fetching log, we support to call the kyuubi instance that created the batch directly.
It is better to add original host urls when building the rest client to kyuubi instance, in case that there is firewall to the kyuubi instance directly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4657 from turboFei/direct_connect.
Closes#4657
084fdfb49 [fwang12] host urls
689ff8f8c [fwang12] rest client
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The CONNECTION share level engines always use a UUID as the subdomain in registering path, kyuubi-ctl's wrong subdomain calculation causes a failure on listing engines.
```
./kyuubi-ctl list engine --zk-quorum xxx --namespace kyuubi --user xxx
```
```
2023-03-10 13:53:32.939 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2023-03-10 13:53:32.945 ERROR org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Failed to get service node info
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kyuubi_1.7.0_CONNECTION_SPARK_SQL/xxx/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1659) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230) ~[curator-framework-2.12.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[curator-framework-2.12.0.jar:?]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.12.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216) ~[curator-framework-2.12.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[curator-framework-2.12.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[curator-framework-2.12.0.jar:?]
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.getServiceNodesInfo(ZookeeperDiscoveryClient.scala:214) ~[kyuubi-ha_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.util.CtlUtils$.getServiceNodes(CtlUtils.scala:63) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.util.CtlUtils$.$anonfun$listZkServerNodes$1(CtlUtils.scala:86) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.util.CtlUtils$.$anonfun$listZkServerNodes$1$adapted(CtlUtils.scala:80) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.util.CtlUtils$.listZkServerNodes(CtlUtils.scala:80) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cmd.list.ListCommand.doRun(ListCommand.scala:32) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cmd.list.ListCommand.doRun(ListCommand.scala:24) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
2023-03-10 13:53:32.949 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2023-03-10 13:53:32.952 INFO org.apache.zookeeper.ZooKeeper: Session: 0x500011010ecc76e closed
2023-03-10 13:53:32.953 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x500011010ecc76e
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
buid model kyuubi-ctl and deploy to my environment and run : kyuubi-ctl list engine -zk xxx:2181 -n kyuubi -u xxx
get correct result.
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4492 from Kiss736921/fix_list_engine_error.
Closes#4492
29b87ed57 [Alex] change param name engine to engineNode
9e87b0853 [Alex] optimize get cmd architecture and complete delete engine for all share level
20544c296 [Cheng Pan] ctl should handle server and engine nodes seperately
9000129df [Alex] fix list engine no node exception
Lead-authored-by: Alex <zoulimin@kanzhun.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4438
Follow up of #4390
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4463 from turboFei/render_batch.
Closes#4438
b5ae211bc [fwang12] render batch info
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support to refresh the unlimited users for session limiter, so that we can unblock some customers without restart the kyuubi server.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4360 from turboFei/limier_whitelist.
Closes#4360
c846148bd [fwang12] typo
839e71365 [fwang12] nit
904bc9fbc [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Remove duplicate buildConf methods in a series configs , change to import and use `KyuubiConf.buildConf` directly
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4239 from bowenliang123/duplicate-buildconf.
Closes#4239
04495e69 [liangbowen] remove duplciate buildConf methods
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As Kyuubi graduated as top level project, the setting page will be more often requested and should be increasingly reliable and readable with less grammar and spelling mistakes.
This PR is to
- correct mistakes in grammar, spelling, abbreviation and terminology
- with no config name or essential meanings changed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4161 from bowenliang123/conf-grammar.
Closes#4161
038edfbea [liangbowen] nit
1ec073a4b [liangbowen] to JSON
4f5259a32 [liangbowen] to Prometheus
523855008 [liangbowen] to K8s
fc7a3a81e [liangbowen] to AUTO-GENERATED
da64f54fa [liangbowen] update
d54f9a528 [liangbowen] fix `comma separated` to `comma-separated`
f1d7cc1f1 [liangbowen] update
d84208844 [liangbowen] update
1b75f011c [liangbowen] correction of grammar and spelling mistakes
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To address comment https://github.com/apache/kyuubi/pull/4119#discussion_r1064237556
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4129 from turboFei/4119_followup.
Closes#4119
30626e5d6 [fwang12] appSubmissionTime to appStartTime
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Return the batch app submission time, it is useful to provide the app elapse time.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4119 from turboFei/open_time.
Closes#4119
9c7ddefa2 [fwang12] rebase
1c3c7e4bb [fwang12] save
bf5177a9e [fwang12] ut
ae1c0be27 [fwang12] get from recovery
48b3a1e11 [fwang12] refactor
f8778e673 [fwang12] engine open time
1989aa4f9 [fwang12] add
8884ee5a6 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3982 .
Introduce feature of refresh user defaults config (as `___${user}___.*` which starts with three continuous underscores "___") from config file via `kyuubi-admin` cli and `refresh/user_defaults_conf` Rest API.
1. add `refreshUserDefaultsConf` methond in KyuubiServer to read user defautls configs from property file and apply config changes to server's KyuubiConf
3. add `refresh/user_defaults_conf` api to AdminRestApi calling `refreshUserDefaultsConf` of KyuubiServer
3. add config type `userDefautls` in kyuubi-admin cli refresh command
This feature will
- help to apply user defaults conf without restarting server or losing connections
- load latest config for engine launch, e.g. spark related config `spark.*`
It won't
- affect the components already started and using the clone of server conf
- affect configs for launched engine instance
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3983 from bowenliang123/3982-reload-server-conf.
Closes#3983
a8fb0bf2 [liangbowen] fix typo
b9b80f56 [liangbowen] update
b47a1541 [liangbowen] minor
786cb2a0 [liangbowen] add logging for statistics
0860e3a7 [liangbowen] fix: loadFileDefaults in refreshUserDefaultsConf
8dbbbcb8 [liangbowen] fix typo
619acd2e [liangbowen] import
e405dc8e [liangbowen] fix user defaults key filtering by adding `getAllUserDefaults` to `KyuubiConf`
ac407bd5 [liangbowen] rename config refresh option to `userDefaultsConf` and extracted to RefreshConfigCommandConfigType
c65398b2 [liangbowen] fix redundant loadFileDefaults in refreshUserDefaultsConf
1b046feb [liangbowen] update comments
e5dd5dbb [liangbowen] typo
c0a358ba [liangbowen] change to refresh users' config. rename cli command to `refresh config userDefaults`
bf5448e3 [liangbowen] support reload server config from config file 1. add reloadServerConf in KyuubiServer to read config and put all to config, 2. add "refresh/server_conf" api to AdminRestApi, 3. add config type "serverConf" in kyuubi-admin cli
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
9342a29284/kyuubi-ctl/src/main/scala/org/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala (L134-L141)
We need to reset the operation log before fetching new operation log, otherwise, we might meet corner case.
- The last operation log is not empty
- we meet log not found exception
- we can not terminate the loop because the `shouldFinishLog` is always false
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4067 from turboFei/check_status.
Closes#4067
b61e14f6 [fwang12] style
5787ad97 [fwang12] do not fetch again
6dc479d1 [fwang12] refactor
37a91909 [fwang12] reset the log before fetch log
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To fix https://github.com/apache/incubator-kyuubi/issues/3973, we found similar problem during upgrade from 1.5 to 1.6
### _How was this patch tested?_
Manual test on internal cluster, UT will be added later on
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3997 from CavemanIV/kyuubi-3973.
Closes#3973
7dae77cf [John Caveman] fix validation for ut
96adf586 [John Caveman] use conf.get
0b0df721 [John Caveman] style fix
f0f167fb [John Caveman] replace for-conf-key-lambda to option chain for a more consistent coding style
6a308d5c [John Caveman] revert ut, command only read conf from kyuubi-defaults directly
ff999221 [John Caveman] fix kyuubi-ctl list fall back to HA_ZK_QUORUM in kyuubi-default.conf when HA_ADDRESSES is not set
Authored-by: John Caveman <selnteer@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
With yarn-cluster mode, if `spark.yarn.submit.waitAppCompletion` is false and `kyuubi-ctl` wait completion is true, we need still print some simple batch report to prevent misunderstand.
The batch report format refer spark-submit log.
```
Application report for <ApplicationID> (state: RUNNING)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3949 from turboFei/ctl_log.
Closes#3949
8fbba0f8 [fwang12] keep same format with app state
ea5597fa [fwang12] print appId
d6d839c1 [fwang12] Print the simple batch report if no log fetch
bb908b1b [fwang12] log batch state if there is no log
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Optimize discovery makePath api.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3917 from hddong/optimize-ds-path-api.
Closes#3917
a152373e [hongdongdong] Update
9dc37929 [hongdongdong] Optimize discovery makePath api
Authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: hongdongdong <hongdd@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose the engine id when registering the service.
So that we can get more details about the engine when using `list engine` command.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3684 from turboFei/extra_info.
Closes#3684
afbcfdef3 [Fei Wang] refactor
513dd7234 [Fei Wang] split 2
ad3a6ac4f [Fei Wang] comments
cc51c5954 [Fei Wang] fix ut
32e31618a [Fei Wang] check engine id
202bbdb68 [Fei Wang] refactor to details
c5c153b8d [Fei Wang] render extra info
d2642cc8a [Fei Wang] save
1da67c258 [Fei Wang] dto
724ac3af4 [Fei Wang] save
c16c15988 [Fei Wang] expose engine id
e91a61e5a [Fei Wang] save
e38bb0b93 [Fei Wang] save
b1c158ba8 [Fei Wang] parse extra info
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Using `unbounded()` for `--conf`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3746 from turboFei/ctl_config.
Closes#3746
1cc58605c [Fei Wang] use unbounded
41c0d1a34 [Fei Wang] Revert "Support to multiple config items"
1d5afb7ea [Fei Wang] Revert "fix style"
c1d8ff8d1 [Fei Wang] fix style
67411367c [Fei Wang] Support to multiple config items
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Set the config version to 1.6.1.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3734 from turboFei/typo_version.
Closes#3720
4a552048 [Fei Wang] [KYUUBI #3720][FOLLOWUP] Using correct config version for CTL_BATCH_LOG_ON_FAILURE_TIMEOUT
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
If the batch failed, we shall try to show the remaining logs with timeout.
### _How was this patch tested?_
Existing UT.
Closes#3720 from turboFei/wait_log_fetched.
Closes#3720
a5d0b37f [Fei Wang] comments
220be4ee [Fei Wang] in same thread
90ada13c [Fei Wang] revert
e3ab5ec8 [Fei Wang] address comments
e94a348c [Fei Wang] refactor
c901e4b6 [Fei Wang] prevent NPE
9ed76685 [Fei Wang] save
1f3560bd [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3721 from lightning-L/kyuubi-3713.
Closes#3713
70959512 [Tianlin Liao] [KYUUBI #3713] kyuubi-ctl list current sessions
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3679
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3680 from lightning-L/admin-ctl.
Closes#3679
9c748f073 [Tianlin Liao] [KYUUBI #3679] Admin command line supports delete/list engine operation
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix NPE if batch request/args/config is not specified.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3571 from turboFei/npe_batch.
Closes#3571
682de4f4 [Fei Wang] fix npe if value is null
f5b94d98 [Fei Wang] ut
68ee3101 [Fei Wang] fix npe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix npe issue:
```
Exception in thread "main" java.lang.NullPointerException
at com.jakewharton.fliptables.FlipTable.<init>(FlipTable.java:42)
at com.jakewharton.fliptables.FlipTable.of(FlipTable.java:20)
at org.apache.kyuubi.ctl.util.Tabulator$.formatTextTable(Tabulator.scala:37)
at org.apache.kyuubi.ctl.util.Tabulator$.format(Tabulator.scala:25)
at org.apache.kyuubi.ctl.util.Render$.renderBatchListInfo(Render.scala:39)
at org.apache.kyuubi.ctl.cmd.list.ListBatchCommand.$anonfun$render$1(ListBatchCommand.scala:58)
at org.apache.kyuubi.ctl.cmd.Command.info(Command.scala:86)
at org.apache.kyuubi.ctl.cmd.list.ListBatchCommand.render(ListBatchCommand.scala:58)
at org.apache.kyuubi.ctl.cmd.list.ListBatchCommand.render(ListBatchCommand.scala:26)
at org.apache.kyuubi.ctl.cmd.Command.$anonfun$run$1(Command.scala:47)
at org.apache.kyuubi.ctl.cmd.Command.$anonfun$run$1$adapted(Command.scala:47)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47)
at org.apache.kyuubi.ctl.ControlCli.doAction(ControlCli.scala:45)
at org.apache.kyuubi.ctl.ControlCli$$anon$1.doAction(ControlCli.scala:78)
at org.apache.kyuubi.ctl.ControlCli$.main(ControlCli.scala:86)
at org.apache.kyuubi.ctl.ControlCli.main(ControlCli.scala)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3158 from turboFei/batch_npe.
Closes#3158
fa9b53ec [Fei Wang] refactor
eed61515 [Fei Wang] nit
b54bff84 [Fei Wang] fix npe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
BatchUtil class is helpful for programming way, move it into batch sdk.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3153 from turboFei/batch_utils.
Closes#3153
6e7988a2 [Fei Wang] add comments
fe7f9dfe [Fei Wang] Move BatchUtil to kyuubi-rest-client
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
1. The version options should be in zkOptions.
2. Add docs for kyuubi-adminctl.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3066 from turboFei/admin_ctl_docs.
Closes#3045
983d133f [Fei Wang] comments
65675ad7 [Fei Wang] add more
b5b772d5 [Fei Wang] comments
222b409c [Fei Wang] add docs
20c50c00 [Fei Wang] kyuubi-adminctl -> kyuubi-admin
078044e0 [Fei Wang] refactor
4e89b52d [Fei Wang] remove unused
c2d1dc70 [Fei Wang] add admin ctl docs
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3045
Introduce kyuubi-adminctl for admin control.
```
Usage: kyuubi-adminctl [refresh] [options]
-v, --version <value> Using the compiled KYUUBI_VERSION default, change it if the active service is running in another.
-b, --verbose Print additional debug output.
--hostUrl <value> Host url for rest api.
--authSchema <value> Auth schema for rest api, valid values are basic, spnego.
--username <value> Username for basic authentication.
--password <value> Password for basic authentication.
--spnegoHost <value> Spnego host for spnego authentication.
--conf <value> Kyuubi config property pair, formatted key=value.
Command: refresh [config] <args>...
Refresh the resource.
Command: refresh config [<configType>]
Refresh the config with specified type.
<configType> The valid config type can be one of the following: hadoopConf.
-h, --help Show help message and exit.
```
Now support `kyuubi-adminctl refresh config hadoopConf` to reload hadoop conf.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3057 from turboFei/kyuubi_admin_ctl.
Closes#3045
b6b46f3a [Fei Wang] refactor the info render
d3b600f2 [Fei Wang] add toString
234ae84c [Fei Wang] fix ut
7db818d0 [Fei Wang] save
e9e50a0f [Fei Wang] add ut
849edad8 [Fei Wang] add ut
c4d9b784 [Fei Wang] fix ut
1c8d7c9b [Fei Wang] add kyuubi-adminctl
8b1921ae [Fei Wang] save
155f57af [Fei Wang] saev
2ca8b665 [Fei Wang] save
5ad40cea [Fei Wang] save
bb2e1178 [Fei Wang] add refresh config
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3037 from pan3793/jul.
Closes#3037
fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
