### _Why are the changes needed?_
Followup #4498
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5354 from turboFei/list_sessions.
Closes#4498
5a3020ac2 [fwang12] list users sessions
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
close#4994
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5157 from lsm1/branch-kyuubi_4994.
Closes#4994
02e7eb5fa [senmiaoliu] use abbr a
6d001e519 [senmiaoliu] replace 'allengine' with 'all'
f9c548299 [senmiaoliu] fix style
df64e773e [senmiaoliu] fix style
6a7d40e63 [senmiaoliu] remove allenginecommand
6306dd8a2 [senmiaoliu] support list all engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
Improvement: https://github.com/apache/kyuubi/issues/4540
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5182 from zhuyaogai/issue-4540.
Closes#4540
73652ae6d [Fantasy-Jay] Merge remote-tracking branch 'origin/issue-4540' into issue-4540
ae7a3b5bb [Fantasy-Jay] Support to deny some user to make connection.
46ee2f158 [Fantasy-Jay] Support to limit some user to make connection.
3d7e5e534 [Fantasy-Jay] Support to limit some user to make connection.
1f173ffe4 [Fantasy-Jay] Merge branch 'master' into issue-4540
1d3a3d47f [Fantasy-Jay] Support to limit some user to make connection.
e4a1b3eb3 [Fantasy-Jay] Support to limit some user to make connection.
1c120a992 [Fantasy-Jay] Support to limit some user to make connection.
6b657aa99 [Fantasy-Jay] Support to limit some user to make connection.
506c4751b [Fantasy-Jay] Support to limit some user to make connection.
f99072ef5 [Fantasy-Jay] Support to limit some user to make connection.
Authored-by: Fantasy-Jay <13631435453@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove duplicated assignment for the same variable in adjacent lines in `FastHiveDecimalImpl`
- replace redundant `putAll` with collection initialization in `BatchRestApi`
- use `try-with-resources` statement with the reader and avoid declaring two variables in the same line of code in `KyuubiCommands`
- fix `warning: Tag 'return:' is not recognised` compilation warning in `KyuubiGetSqlClassification:L53`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5117 from bowenliang123/fastsignum.
Closes#5117
595b5747d [liangbowen] simplify
be530fac4 [liangbowen] fix warning: Tag '@return:' is not recognised compilation warning in KyuubiGetSqlClassification:L53
249706905 [liangbowen] use try-with-resources in KyuubiCommands
a54a97fdd [liangbowen] remove redundant addAll call to collection initialization
cc76d5d0f [liangbowen] remove repeated assignment
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
#### How is it done today?
The current procedure of Batch Job API, called V1
##### CREATE batch job procedure in Batch V1
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Create Batch Job
Server ->> Server : Create Batch Operator
Server ->> Metastore : Persist Job metadata (PENDING)
Server ->> Server : Put Batch Operator into Execution thread pool
Server ->> Client : Batch Job Info
Server ->> RM : Submit Applicition (in Execution thread pool)
loop Application Check
Server ->> RM : Query Application Status
Server ->> Metastore : Update Batch Status
end
```
##### GET batch job info procedure in Batch V1
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Query Batch Job Info
alt KyuubiInstance matched
Server ->> Client : Batch Job Info
else
Server ->> Server : Forward Request to expected KyuubiInstance
end
```
<!--
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Fetch Batch Job logs
alt KyuubiInstance matched
Server ->> Client : Batch Job logs
else
Server ->> Server : Forward Request to expected KyuubiInstance
end
Client ->> Server : Close Batch Job
alt KyuubiInstance matched
Server ->> RM : Close the Application
Server ->> Metastore : Update Batch Status
Server ->> Client : Closed Batch Job Info
else
Server ->> Server : Forward Request to expected KyuubiInstance
end
```
-->
#### What is new in your approach?
This PR proposes a new way for batch job submission, called V2
##### CREATE batch job procedure in Batch V2
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Create Batch Job
Server ->> Metastore : Persist Job metadata (INITIALIZED)
Server ->> Client : Batch Job Info
loop Forever in dedicated thread pool
Server ->> Metastore : Pick up and lock INITIALIZED job
Server ->> RM : Submit Application
Server ->> RM : Query Application Status
Server ->> Metastore : Update Batch Status
end
```
##### GET batch job info procedure in Batch V2
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Query Batch Job Info
Server ->> Metastore : Query Batch Job Info
Server ->> Client : Batch Job Info
```
<!--
```mermaid
sequenceDiagram
participant Client
participant Server
participant Metastore
participant RM
Client ->> Server : Fetch Batch Job logs
alt KyuubiInstance matched
Server ->> Client : Batch Job logs
else
Server ->> Server : Forward Request to expected KyuubiInstance
end
Client ->> Server : Close Batch Job
alt KyuubiInstance matched
Server ->> RM : Close the Application
Server ->> Metastore : Update Batch Status
Server ->> Client : Closed Batch Job Info
else
Server ->> Server : Forward Request to expected KyuubiInstance
end
```
-->
#### What are the limits of current practice, and why do you think it will be successful?
Pros:
1. The CREATE request becomes light and returns faster. In V1, we have struggled with whether the response should wait for the engine to be submitted to RM, and how to report the un-submitted job status to the client; in V2, the CREATE request just simply inserts a new record into metastore and returns w/ INITIALIZED state.
2. In common practice, Kyuubi server cluster is deployed behind the load balancer, and the load balancer does not know the real load of each Kyuubi server, suppose it uses Random/RoundRobbin/IPHash policies to forward requests, the existing Batch V1 implementation may cause some Kyuubi servers in high load but others' load are low, because it always uses the requested Kyuubi server to do batch submission; in V2, the Kyuubi server is easy to know the load of itself, e.g. measure by CPU/memory usage, or active batch sessions, and then decides to pick up new batch jobs or not. Besides, when all Kyuubi servers overload, the V1 cannot benefit immediately even if the admin scale up the cluster size.
3. In V1, the metrics are almost independent in each Kyuubi server; in V2, it's easy to expose global metrics of batch jobs when using sharable storage as metastore backend, e.g. we can easily get how many batches are queued in metastore, and how many batches are managed by each Kyuubi server, by querying the metastore backend directly or metrics exposed by each Kyuubi server.
Cons:
1. V1 assumes Kyuubi server tolerant long time outage of metastore, V2 forcibly depends on the availability of metastore. But we can move the existing forwarding logic and async retry logic to the implementation of `Metastore` to overcome this regression.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4790 from pan3793/batch-v2.
Closes#4790
860698ad6 [Cheng Pan] BATCH_IMPL_VERSION
b9c68aa2f [Cheng Pan] kyuubi.batch.impl.version
17e4f199a [Cheng Pan] submitter.threads=100
7c0bdb0c1 [Cheng Pan] Initial implement Batch v2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to filter batch with batch name filter condition.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5063 from turboFei/batch_name.
Closes#5063
63915a56d [fwang12] ut
2548815a2 [fwang12] ut
34a9229b0 [fwang12] style
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
To prevent conflicts in case that other dependency also has a resource named `version.properties`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5054 from turboFei/version_conflicts.
Closes#5054
fe1333dbe [fwang12] prevent version properties conflicts
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This is a followup of #4843
To support load kubernetes conf during runtime, so that we can support more kuberntes contexts without restarting the kyuubi server.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4996 from turboFei/refresh_kubernetes_conf.
Closes#4996
807fb92e4 [fwang12] comments
d42d25af7 [fwang12] from conf
809a7d3df [fwang12] refresh
22743f9e5 [fwang12] save
dedebbe71 [fwang12] api
d6f58cfc7 [fwang12] refresh kubernetes config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4920
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4921 from lightning-L/kyuubi-4920.
Closes#4920
d395f8a8a [Tianlin Liao] remove KyuubiEvent
3a6730f9b [Tianlin Liao] [KYUUBI #4920] add OperationRestApi
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- apply the usage of `ReflectUtils` and `Dyn*` to the modules of engines and plugins (eg. Spark engine, Authz plugin, lineage plugin, beeline)
- remove similar redundant methods for calling reflected methods or getting field values
- unified reflection helper methods with type casting support, as `getField[T]` for getting field values from `getFields`, `invokeAs[T]` for invoking methods in `getMethods`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4879 from bowenliang123/reflect-use.
Closes#4879
c685fb67d [liangbowen] bug fix for "Cannot bind static field options" when executing "bin/beeline"
fc1fdf1de [liangbowen] import
59c3dd032 [liangbowen] comment
c435c131d [liangbowen] reflect util usage
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4893 from turboFei/minor_dto.
Closes#4893
aeb5a25a9 [fwang12] prevent null:
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support to list server with kyuubi-admin/kyuubi rest client.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- UT for AdminRestAPI
- UT for AminCtlArgument
- UT for AdminCtl
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4889 from turboFei/list_server.
Closes#4889
bfd13fbde [fwang12] nit
0a0131552 [fwang12] list server
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Return operation metrics and fix typo.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4890 from turboFei/op_data_metrics.
Closes#4890
ef2ed36eb [fwang12] return metrics and fix bug
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Add List Kyuubi Server Api for `AdminResource`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4670 from zwangsheng/KYUUBI_3653.
Closes#3653
b91a6c617 [zwangsheng] fxi
4271d0fd0 [zwangsheng] fix comments
e14f8cd55 [zwangsheng] [KYUUBI #3653][REST] AdminResource add list server api
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Usually, the host url to create a batch is the load balancer uri.
To reduce the internal rest redirection when fetching log, we support to call the kyuubi instance that created the batch directly.
It is better to add original host urls when building the rest client to kyuubi instance, in case that there is firewall to the kyuubi instance directly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4657 from turboFei/direct_connect.
Closes#4657
084fdfb49 [fwang12] host urls
689ff8f8c [fwang12] rest client
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As title.
With this pr, customer can execute SCALA code with `confOverlay`.
```
kyuubi.operation.language=SCALA
```
execute PYTHON code with
```
kyuubi.operation.language=PYTHON
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4639 from turboFei/init_scala.
Closes#4639
cdf828f9a [fwang12] add ut
f4f2bc883 [fwang12] doc
2fe8a1659 [fwang12] save
f840cb4d9 [fwang12] conf overlay
eb49537ea [fwang12] conf overlay
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#4611
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4612 from lightning-L/kyuubi-4602.
Closes#4611
9ad95c6ad [Tianlin Liao] fix
0b3792e3e [Tianlin Liao] fix
7db90d570 [Tianlin Liao] [KYUUBI #4611] support all sessions api in SessionRestApi
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
as title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4521 from turboFei/rest_docs.
Closes#3739
589237ca6 [fwang12] save
8e7d9bec6 [fwang12] final
35b67ba1e [fwang12] doc
40163311f [fwang12] save
5711d3d72 [fwang12] fix
bab07902f [fwang12] fix ut
d58c2753a [fwang12] save
989eaffd5 [fwang12] remove
2960602bc [fwang12] remove outdate docs
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As title.
BTW, for KyuubiSessionEvents, you can get kyuubiInstance from serverIP.
```
* param serverIP A unique Kyuubi server id, e.g. kyuubi server ip address and port,
* it is useful if has multi-instance Kyuubi Server
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4518 from turboFei/instance_follow.
Closes#4518
2ad4c1646 [fwang12] event
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
It is helpful, because now kyuubi gateway does not support multiple nodes ha.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4516 from turboFei/kyuubi_instance.
Closes#4516
a679dc524 [fwang12] return kyuubi instance
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4506 from turboFei/session_data_type.
Closes#4506
9617ee68c [fwang12] Session type
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- To fix missing package `org.apache.commons.logging` used by `org.apache.httpcomponents:httpclient`in `kyuubi-rest-client`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4489 from bowenliang123/httpclient-commonlogging.
Closes#4489
ccaff9b62 [liangbowen] Include `jcl-over-slf4j` dependency
740e56158 [liangbowen] Revert "skip excluding commons-logging in rest-client"
4d9ad5dcf [liangbowen] skip excluding commons-logging in rest-client
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4476 from lightning-L/kyuubi-4472.
Closes#4472
329b40c81 [Tianlin Liao] minor fix
72eaa05de [Tianlin Liao] [KYUUBI #4472] add session/operation methods in AdminRestApi
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- add dto for operation data, which is programming friendly
- show exception for session data
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4468 from turboFei/session_op_dto.
Closes#4439
fa905e70c [fwang12] fix ut
5c1c7c845 [fwang12] save
2d20215a0 [fwang12] comments
46cd2384e [fwang12] saev
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to allow the user to provide a batch id on submitting a batch job. If the batch id already existed in metastore, Kyuubi ignores this submission and just returns the existing one, w/ a marker in response, this could avoid duplicated batch job submission.
Talking about the implementation, the key things are
How does the user set the custom batch id?
- User can optionally set the `kyuubi.batch.id` in `conf: Map[String, String]`, and the value must be a UUID, for Java users, it can be generated by `UUID.randomUUID().toString()`
How does the Kyuubi Server detect the duplication?
- It's simple in single Kyuubi Server instance case, Kyuubi just needs to look up the metastore before creating a batch job
- In HA mode, suppose the user requests to create the batch jobs w/ the same batch id concurrently, multiple Kyuubi Servers may process the request and try to insert to metastore DB at the same time, but only the first insertion success, others will fail w/ "duplicated key", Kyuubi Server needs to catch this error and return the existing batch job information instead of creating a new one.
How does the user know if the returned batch job is new created or duplicated?
- a new field `batchInfo: Map[String, String]` is added to the response, and for duplicated batch job, `"kyuubi.batch.duplicated": "true"` will be contained.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4390 from pan3793/batch-id.
Closes#4390
b6917babf [Cheng Pan] move constant to rest client
79ef1b5d8 [Cheng Pan] flaky test
f82228506 [Cheng Pan] it
88bdfa50a [Cheng Pan] ut
fd8bc222a [Cheng Pan] ut
c820f5e43 [Cheng Pan] Support user provided batch id on batch job submission
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#4216
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4370 from turboFei/client_version.
Closes#4216
efd934c6b [fwang12] save
cf8acd54a [fwang12] version properties
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support to refresh the unlimited users for session limiter, so that we can unblock some customers without restart the kyuubi server.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4360 from turboFei/limier_whitelist.
Closes#4360
c846148bd [fwang12] typo
839e71365 [fwang12] nit
904bc9fbc [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
It can help to know the backend pressure if the exec pool is full.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4344 from turboFei/wait_queue.
Closes#4344
161e3808a [fwang12] nit
6d122e238 [fwang12] save
55a4b499d [fwang12] version
668ff8bfe [fwang12] save
9f56b98a8 [fwang12] save
a401771ec [fwang12] wait
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
according to `io.trino.jdbc.ColumnInfo`, there are some type requring signature parameter.
- varchar(n)
- char(n)
- decimal(precision, scale)
It failed with trino jdbc now
<img width="613" alt="image" src="https://user-images.githubusercontent.com/12025282/218707052-a2e9dc91-0333-483c-bc0a-96baec213578.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4328 from ulysses-you/fix-signature.
Closes#4328
aede5cec [ulysses-you] nit
ae1a7968 [ulysses-you] fix test
8ecdb346 [ulysses-you] Make Trino jdbc driver work
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
[NoHttpResponseException](https://hc.apache.org/httpclient-legacy/exception-handling.html)
> In some circumstances, usually when under heavy load, the web server may be able to receive requests but unable to process them. A lack of sufficient resources like worker threads is a good example. This may cause the server to drop the connection to the client without giving any response. HttpClient throws NoHttpResponseException when it encounters such a condition. In most cases it is safe to retry a method that failed with NoHttpResponseException.
In case of Kyuubi Server is overloaded and does not have enough resources to allocate threads to handle the corresponding request, the request will be dropped and no response will be returned, Kyuubi Rest Client should catch this exception and retry.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4334 from zwangsheng/rest_client/retry_with_no_http_response.
Closes#4334
d526e4ab [zwangsheng] Remove Unrelated Code
708f2496 [zwangsheng] [REST] Retry should catch NoHttpResponse Exception and retry
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- REST client support for creating batch with uploading resource file (https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches-with-uploading-resource) introduced in #4144
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4221 from bowenliang123/restclient-batch-upload.
Closes#4221
9ec83766 [liangbowen] add MultiPart class to avoid exposing httpclient classes to RestClient interface
83051843 [liangbowen] fix content type
056edb84 [liangbowen] nit
74ad5642 [liangbowen] style
68dcdb38 [liangbowen] update dependencyList
a274d392 [liangbowen] client for creating batch with uploading resource file
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The current default socket timeout is set to 3s, which is too short, when Kyuubi Server is busy, e.g. during GC, the client will fail w/ socket read timeout after 3s if no bytes are returned in the socket.
This PR proposes to enlarge the REST client default timeout
- connect timeout from 3s to 30s
- socket timeout from 3s to 2min
- attempt interval keep 3s
They keep consistent w/ `org.apache.kyuubi.ctl.CtlConf`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4224 from pan3793/rest-timeout.
Closes#4224
0db2a3788 [Cheng Pan] nit
01fa50942 [Cheng Pan] Enlarge REST client default timeout
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
```
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "appStartTime" (class org.apache.kyuubi.client.api.v1.dto.Batch), not marked as ignorable (13 known properties: "appDiagnostic", "createTime", "kyuubiInstance", "user", "appUrl", "appState", "state", "batchType", "cluster", "id", "appId", "endTime", "name"])
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4208 from turboFei/ignore_unknown.
Closes#4119
0e81ac7ae [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The output of building and testing is now brought with repeated unuseful warnings from `javac`.
This PR reduces over 500 unhelpful warning message lines from `-Xlint` rules in `javac` compilation output in a single run, by eliminating :
- 74 of `deprecation`, by adding `SuppressWarning` to 6 Java classes
- 6 of `rawtypes`, by adding `SuppressWarning` to methods of Java class
- 4 of `unchecked`, by adding `SuppressWarning` to methods of Java class
- 1 of `fallthrough`, by adding `SuppressWarning` to method of Java class
- 1 of `cast`, by removal of redundant type casting
- repeated `processing` for, by adding `-processing` to disable annotation processor warning for annotations from testing framework :
```
- Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.scalatest.TagAnnotation
Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.junit.Test,/org.junit.Before,/org.junit.After
```
As 1 [fallthrough] warning message remain, which will be reported in followup issues or PR to fix missing breaks for branches of `switch-case` in Java.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4166 from bowenliang123/java-warnings.
Closes#4166
ec4cfc42 [liangbowen] add @SuppressWarnings("fallthrough") to KyuubiConnection#waitLaunchEngineToComplete
c9a59522 [liangbowen] skip deprecation warnings for SQLResultSet and KyuubiBaseResultSet
cab8c358 [liangbowen] remove redundant casting
f2ce8963 [liangbowen] style
bf6c5936 [liangbowen] add `-processing` to javac Xlint option for eliminating the missing annotation processer warnings
01a89a8c [liangbowen] eliminate warning outputs in java by adding @SuppressWarnings
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
We should put the authorization info in request header, user,password,ip in SessionOpenRequest are unnecessary.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3739 from hddong/rm-unused-params.
Closes#3739
7fac61853 [hongdongdong] update
8f1f4ae01 [hongdongdong] added on migration guide
95962b472 [hongdongdong] fix
63606350b [hongdongdong] [REST] Remove unused parameters in SessionOpenRequest
Authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
To address comment https://github.com/apache/kyuubi/pull/4119#discussion_r1064237556
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4129 from turboFei/4119_followup.
Closes#4119
30626e5d6 [fwang12] appSubmissionTime to appStartTime
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Return the batch app submission time, it is useful to provide the app elapse time.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4119 from turboFei/open_time.
Closes#4119
9c7ddefa2 [fwang12] rebase
1c3c7e4bb [fwang12] save
bf5177a9e [fwang12] ut
ae1c0be27 [fwang12] get from recovery
48b3a1e11 [fwang12] refactor
f8778e673 [fwang12] engine open time
1989aa4f9 [fwang12] add
8884ee5a6 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3982 .
Introduce feature of refresh user defaults config (as `___${user}___.*` which starts with three continuous underscores "___") from config file via `kyuubi-admin` cli and `refresh/user_defaults_conf` Rest API.
1. add `refreshUserDefaultsConf` methond in KyuubiServer to read user defautls configs from property file and apply config changes to server's KyuubiConf
3. add `refresh/user_defaults_conf` api to AdminRestApi calling `refreshUserDefaultsConf` of KyuubiServer
3. add config type `userDefautls` in kyuubi-admin cli refresh command
This feature will
- help to apply user defaults conf without restarting server or losing connections
- load latest config for engine launch, e.g. spark related config `spark.*`
It won't
- affect the components already started and using the clone of server conf
- affect configs for launched engine instance
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3983 from bowenliang123/3982-reload-server-conf.
Closes#3983
a8fb0bf2 [liangbowen] fix typo
b9b80f56 [liangbowen] update
b47a1541 [liangbowen] minor
786cb2a0 [liangbowen] add logging for statistics
0860e3a7 [liangbowen] fix: loadFileDefaults in refreshUserDefaultsConf
8dbbbcb8 [liangbowen] fix typo
619acd2e [liangbowen] import
e405dc8e [liangbowen] fix user defaults key filtering by adding `getAllUserDefaults` to `KyuubiConf`
ac407bd5 [liangbowen] rename config refresh option to `userDefaultsConf` and extracted to RefreshConfigCommandConfigType
c65398b2 [liangbowen] fix redundant loadFileDefaults in refreshUserDefaultsConf
1b046feb [liangbowen] update comments
e5dd5dbb [liangbowen] typo
c0a358ba [liangbowen] change to refresh users' config. rename cli command to `refresh config userDefaults`
bf5448e3 [liangbowen] support reload server config from config file 1. add reloadServerConf in KyuubiServer to read config and put all to config, 2. add "refresh/server_conf" api to AdminRestApi, 3. add config type "serverConf" in kyuubi-admin cli
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Because we do not support multiple node HA now for SessionsResource.
It is difficult to use if load balancer is used for kyuubi gateway.
In this pr, we return the actual kyuubi instance so that user can call the instance directly for follow operations.
It should be a temporary workaround to support multiple kyuubi instances use case.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3876 from turboFei/session_instance.
Closes#3876
b123effb [fwang12] Return kyuubi instance when opening interactive session via SessionsResource
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose the engine id when registering the service.
So that we can get more details about the engine when using `list engine` command.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3684 from turboFei/extra_info.
Closes#3684
afbcfdef3 [Fei Wang] refactor
513dd7234 [Fei Wang] split 2
ad3a6ac4f [Fei Wang] comments
cc51c5954 [Fei Wang] fix ut
32e31618a [Fei Wang] check engine id
202bbdb68 [Fei Wang] refactor to details
c5c153b8d [Fei Wang] render extra info
d2642cc8a [Fei Wang] save
1da67c258 [Fei Wang] dto
724ac3af4 [Fei Wang] save
c16c15988 [Fei Wang] expose engine id
e91a61e5a [Fei Wang] save
e38bb0b93 [Fei Wang] save
b1c158ba8 [Fei Wang] parse extra info
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3721 from lightning-L/kyuubi-3713.
Closes#3713
70959512 [Tianlin Liao] [KYUUBI #3713] kyuubi-ctl list current sessions
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3679
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3680 from lightning-L/admin-ctl.
Closes#3679
9c748f073 [Tianlin Liao] [KYUUBI #3679] Admin command line supports delete/list engine operation
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3629 from lightning-L/list-engine.
Closes#3626
f63faa53 [Tianlin Liao] fix test case
3d9af3a1 [Tianlin Liao] do not need LIstEnginesResponse
718c115f [Tianlin Liao] refactor
defc9f52 [Tianlin Liao] refactor getUserName and getEngineSpace method
e5015508 [Tianlin Liao] typo and grammer fix
84256f68 [Tianlin Liao] [KYUUBI #3626] AdminResources rest api support to list current engines
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
BatchUtil class is helpful for programming way, move it into batch sdk.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3153 from turboFei/batch_utils.
Closes#3153
6e7988a2 [Fei Wang] add comments
fe7f9dfe [Fei Wang] Move BatchUtil to kyuubi-rest-client
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
