# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Preparing v1.11.0-SNAPSHOT after branch-1.10 cut
```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6769 from bowenliang123/bump-1.11.
Closes#6769
6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
The server module does not consume the `spark-sql-engine` classes, but it supposes that
`spark-sql-engine`'s jar existed on the target folder, so here we declare it as a dependency
to make sure that Maven always processes `spark-sql-engine` module before the server module.
IntelliJ IDEA 2024.1 fixed the IDEA-93855, thus the relocated classes inside the
`spark-sql-engine`'s shaded jar are visible in the server module in IDEA, for example,
`org.apache.kyuubi.shaded.spark.connect.proto.ExecutePlanRequest`, which silently breaks
the IDEA code analysis and jumping capabilities.
## Describe Your Solution 🔧
Changing the dependency type from `jar`(default value) to `pom` seems to be a workaround.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
<img width="593" alt="image" src="https://github.com/user-attachments/assets/cc53709a-5f9d-4452-a24f-0c84e2342191">
#### Behavior Without This Pull Request ⚰️
<img width="1511" alt="image" src="https://github.com/user-attachments/assets/1c547c6d-a603-4c9d-92b4-8e2059b35fac">
#### Behavior With This Pull Request 🎉
<img width="1509" alt="image" src="https://github.com/user-attachments/assets/55f9e6e5-b9ea-4959-9142-ab7db21ab9b1">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6708 from pan3793/IDEA-93855.
Closes#6708
a73fd4235 [Cheng Pan] fix
e4a8d36e7 [Cheng Pan] nit
8c58d2ddc [Cheng Pan] IDEA-93855
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Improve data lake dependency management by extracting the following Maven properties:
- `delta.artifact`
- `hudi.artifact`
- `iceberg.artifact`
- `paimon.artifact`
It often takes a while for the downstream data lakes to support the new Spark versions, extracting those properties makes it easy to override in the new profile on the Kyuubi project's `pom.xml` to workaround before data lakes jars are available.
One use case is a19bb7c18e
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6427 from pan3793/datalake-dep.
Closes#6427
74a9300e0 [Cheng Pan] Improve datalake dependency management
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
The `kyuubi-util-scala_2.12-<version>-tests.jar` accidentally leaked to the compile scope but should be in the test scope.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Run `build/dist` and check `dist/jars`
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6439 from pan3793/util-scala-test.
Closes#6439
0576248f5 [Cheng Pan] fix
2bf2408f5 [Cheng Pan] fix
f7151dfc6 [Cheng Pan] kyuubi-util-scala test jar leaked to compile scope
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
We can not manage Jersey dependencies globally because SPARK-47118 (4.0.0) upgrades Jersey from 2 to 3, using an unmatched Jersey version in modules like Spark SQL engine would fail Spark to start thus abort UT.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested with Spark 4.0
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6415 from pan3793/server-jersey.
Closes#6415
804d6367b [Cheng Pan] nit
fcb6ee573 [Cheng Pan] manage jersey in kyuubi-server module
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11)
Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6392 from pan3793/servlet.
Closes#6392
27d412599 [Cheng Pan] fix
9f1e72272 [Cheng Pan] other spark modules
f4545dc76 [Cheng Pan] fix
313826fa7 [Cheng Pan] exclude
7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6294
## Describe Your Solution 🔧
The netty jars come from `grpc-netty`, `arrow-memory-netty`, use `netty-bom` and `grpc-bom` to simplify the dependency management.
Run `./build/dependency.sh --replace`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6310 from PorterZhang2021/issue-6294.
Closes#6294
327030170 [Cheng Pan] simplify netty dependency management
cdb435d01 [PorterZhang2021] [# 6294] Prune unused Netty libraries
e0676ed5a [PorterZhang2021] Finished Prune unused Netty libraries [#6294]
Lead-authored-by: PorterZhang2021 <PorterZhang2021@outlook.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
They have stopped patching the JDK 1.5 jars that Hadoop uses (see [HADOOP-18540](https://issues.apache.org/jira/browse/HADOOP-18540)).
The new artifacts have similar names - but the names are like bcprov-jdk18on as opposed to bcprov-jdk15on.
CVE-2023-33201 is an example of a security issue that seems only to be fixed in the JDK 1.8 artifacts (ie no JDK 1.5 jar has the fix).
https://www.bouncycastle.org/releasenotes.html#r1rv77 latest current release but the CVE was fixed in 1.74.
To be clear, Kyuubi only uses BouncyCastle for testing, the CVE does not affect Kyuubi distribution.
## Describe Your Solution 🔧
Bump BouncyCastle from 1.67 to 1.77, and change the artifactId from `*-jdk15on` to `*jdk18on`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6177 from pan3793/bouncycastle.
Closes#6177
8595b98c1 [Cheng Pan] Bump BouncyCastle from 1.67 to 1.77
b9e7123f6 [Cheng Pan] Bump bouncycastle from 1.67 to 1.77
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request removes unused dependency commons-lang from kyuubi-server pom.xml.
## Describe Your Solution 🔧
Removes unused dependency commons-lang from kyuubi-server pom.xml.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6161 from zhouyifan279/commons-lang.
Closes#6145
8d50b66ca [zhouyifan279] [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request uses HiveMetaStore Docker for testing to remove HiveMetaStore test scope deps.
## Describe Your Solution 🔧
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6145 from zhouyifan279/secured-hms-container.
Closes#6145
f77429481 [zhouyifan279] Use kyuubi-relocated-hive-metastore-client
2ed1f0f2d [Cheng Pan] image
635fc2e51 [zhouyifan279] Use HiveMetaStore Docker for testing
Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
Kyuubi Shaded 0.3 introduces a light kyuubi-relocated-hive-metastore-client, for refreshing
delegation token, this PR aims to migrate from the vanilla HMS client to this light shaded HMS client, then we can get rid of Hive dependencies, especially the vulnerable thrift 0.9, from the Kyuubi server.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6118 from zhouyifan279/relocated-hms-client.
Closes#6118
4d7e50915 [Cheng Pan] comment
845e39f87 [Cheng Pan] notice
b4a58a614 [Cheng Pan] Update NOTICE
f4bfa9310 [zhouyifan279] Use kyuubi-relocated-hive-metastore-client
af17be1fe [zhouyifan279] Use kyuubi-relocated-hive-metastore-client
Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request removes support of Derby for Kyuubi metastore.
## Describe Your Solution 🔧
Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore.
Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal.
Also, the Derby we used suffers CVE-2022-46337
> Mitigation:
>
> Users should upgrade to Java 21 and Derby 10.17.1.0.
>
> Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Some tests are modified accordingly. Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6099 from pan3793/remove-derby.
Closes#6099
7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5674
## Describe Your Solution 🔧
Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
`org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite`
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6027 from tigrulya-exe/feature/5674-postgresql-metadata-backend.
Closes#5674
111658002 [Tigran Manasyan] Fix metastore db type option comment
44a22bfcf [Tigran Manasyan] Fix dependency list and metastore db type option comment
b638c8942 [Tigran Manasyan] Add PostgreSQL as backend database for kyuubi metadata
Authored-by: Tigran Manasyan <t.manasyan@arenadata.io>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
It's a regular dependency upgrading task.
## Describe Your Solution 🔧
Bump gRPC from 1.53.0 to 1.60.1, a new module [grpc-util](https://mvnrepository.com/artifact/io.grpc/grpc-util) was added in 1.58.0, LICENSE files updated accordingly.
https://github.com/grpc/grpc-java/releases
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#5973 from pan3793/grpc.
Closes#5973
b47c7f295 [Cheng Pan] nit
dd0e966f0 [Cheng Pan] Bump gRPC from 1.53.0 to 1.60.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
TL;DR there are some issues with shading Thrift RPC classes during the engine packaging phase, see details in the PR description of https://github.com/apache/kyuubi-shaded/pull/20.
## Describe Your Solution 🔧
This PR aims to migrate from vanilla `hive-service-rpc`, `libfb303`, `libthrift` to `kyuubi-relocated-hive-service-rpc` introduced in https://github.com/apache/kyuubi-shaded/pull/20, the detailed works are:
- replace imported deps in `pom.xml` and rename the package prefix in all modules, except for
- `kyuubi-server` there are a few places use vanilla thrift classes to access HMS to get token
- `kyuubi-hive-sql-engine` Hive method invocation
- update relocations rules in modules that creates shaded jar
- introduce `HiveRpcUtils` in `kyuubi-hive-sql-engine` module for object conversion.
As part of the whole change, this PR upgrades from the Kyuubi Shaded 0.1.0 to 0.2.0, which changes the jars name. see https://kyuubi.apache.org/shaded-release/0.2.0.html
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Pass all Hive UT with Hive 3.1.3, and IT with Hive 3.1.3 and 2.3.9 (also tested with 2.1.1-cdh6.3.2)
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5783 from pan3793/rpc-shaded.
Closes#5783
b45d4deaa [Cheng Pan] remove staging repo
890076a20 [Cheng Pan] Kyuubi Shaded 0.2.0 RC0
071945d45 [Cheng Pan] Rebase
199794ed9 [Cheng Pan] fix
fc128b170 [Cheng Pan] fix
26d313896 [Cheng Pan] fix
632984c92 [Cheng Pan] fix
428305589 [Cheng Pan] fix
6301e28fd [Cheng Pan] fix
955cdb33b [Cheng Pan] Switch to kyuubi-shaded-hive-service-rpc
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close https://github.com/apache/kyuubi/issues/5484
Kyuubi provides a basic new Web UI which is built on top of Vue3, we can remove the legacy dummy Web UI in 1.8.
The new UI hosts at `http://<host>:<port>/ui/` and the legacy UI hosts at `http://<host>:<port>/`, we should
1. Remove the legacy UI routing from Jetty
2. Remove all files related to legacy UI
3. Redirect `http://<host>:<port>/` to `http://<host>:<port>/ui/`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
building with the command `build/dist --web-ui`, then `cd dist` and perform `bin/kyuubi run`
access http://0.0.0.0:10099 could correctly redirect to http://0.0.0.0:10099/ui/
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1e8a67f6-e4db-415e-8a47-dd7c41b487cf">
swagger is render correctly too.
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1cb4ba31-9965-4468-b7c3-b0319ba959e6">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5516 from pan3793/5484.
Closes#5484
9d58ef758 [Cheng Pan] address comment and fix test
6d4c0981b [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close [#5449](https://github.com/apache/kyuubi/issues/5449).
Unlike the initial preview release, Delta Spark 3.0.0 is now built on top of Apache Spark™ 3.5.
Delta Spark maven artifact has been renamed from delta-core to delta-spark.
https://github.com/delta-io/delta/releases/tag/v3.0.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5450 from zml1206/5449.
Closes#5449
a7969ed6a [zml1206] bump Delta Lake 3.0.0
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi Server Proxy Engine UI
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4795 from zwangsheng/KYUUBI_3420.
Closes#3420
079dc1c60 [zwangsheng] fix frontend unit test case
6e71b4518 [Cheng Pan] fix
cf7ca5145 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala
9a91d62a0 [Cheng Pan] polish
a5dcfae18 [zwangsheng] fix
5d4a8c239 [zwangsheng] Rebase
71d22fc9a [zwangsheng] fix
3b0152f33 [zwangsheng] [KYUUBI #3420][UI] Proxy Engnie UI
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- introduce new event logger type `KAFKA`
- send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle
- use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4733 from bowenliang123/kafka-logger.
Closes#4733
b5220d234 [liangbowen] introduce kafka server event logger
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR aims to remove Hudi integration tests from the Kyuubi project.
Actually, there is no obvious benefit to running Hudi tests w/ Kyuubi, since the real work happens on the compute engine and Hudi integration. Besides, Hudi's horrible dependency management brings significant maintenance efforts to the Kyuubi community.
This change only affects tests, does not affect any functionality.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4744 from pan3793/remove-hudi.
Closes#4744
ea99f747e [Cheng Pan] Remove Hudi integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4325
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4417 from yehere/kyuubi-4325.
Closes#4325
7b2864b53 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,update the dependency file, run './build/dependency.sh --replace'
749b1c15c [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
33ea9ba2b [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
568418a21 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Add test case for DEALLOCATE PREPARE
358a8e3b8 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Support DEALLOCATE PREPARE
7d4a32402 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
91392add6 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
63bf8c462 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
f5b7fb786 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
b0476a79d [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
1a8f147a0 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
Authored-by: yehere <867171931@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
```
Usage:
+----------------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--web-ui] [--flink-provided] [--hive-provided] |
| [--spark-provided] [--mvn <maven_executable>] <maven build options> |
+----------------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
web-ui: - whether to include web ui
flink-provided: - whether to make a package without Flink binary
hive-provided: - whether to make a package without Hive binary
spark-provided: - whether to make a package without Spark binary
mvn: - external maven executable location
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Create binary artifacts using `build/dist --tgz --web-ui` and run, then open `http://0.0.0.0:10099/ui`
<img width="1401" alt="image" src="https://user-images.githubusercontent.com/26535726/220753103-ce801f12-f394-4ece-92a4-1902e93c62c7.png">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4397 from pan3793/webui-build.
Closes#4397
97901d63e [Cheng Pan] doc
37d5e2ad3 [Cheng Pan] mirror-cdn
c5751dd5b [Cheng Pan] remove unused dep
d308defb7 [Cheng Pan] nit
9abca4705 [Cheng Pan] nit
c1d184afd [Cheng Pan] nit
7091d5bf5 [Cheng Pan] regex
f0ac16b3c [Cheng Pan] [BUILD] `build/dist` support --web-ui
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3935
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4232 from iodone/kyuubi-3935.
Closes#3935
936ea1f8 [odone] address
e7bd01a1 [odone] support trino client connect kyuubi trino server
9ea8b6af [odone] [WIP] trion request/response implementation
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to close#4106 .
1. add `POST /batches`API in `BatchesResource` of REST API, which consumes `multipart/form-data` media type to support uploading `resourcefile`. And a `batchRequest` form data part in the JSON format string of `BatchRequest`, as in the required request body of `POST /batches`
2. the uploaded `resourceFile` is saved to a temp local file which will be cleaned up after job execution at the end of `submitAndMonitorBatchJob`
3. the local temp copy of `resourceFile` will be used as `resource` in `BatchJobSubmission`, eg. as <application-jar> for spark-submit
Todos in follow-up:
1. add a related description in Rest API doc
4. add `multipart` media type support to `RestClient` and implement `createBatchWithUploadingResource` in `BatchRestApi`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4144 from bowenliang123/rest-batch-upload.
Closes#4106
f6723a02 [Bowen Liang] Merge branch 'master' into rest-batch-upload
0dd67245 [liangbowen] correct dependencyList
6365a0cd [liangbowen] introducing resource file upload support in batch creation
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Close https://github.com/apache/incubator-kyuubi/issues/3926, we intend to introduce the parser module based on antlr4 in the Apache kyuubi server. Through this module, we can achieve:
1. Expand kyuubi capabilities, for example, implement `SHOW KYUUBI_SESSIONS` through SQL.
2. Sql translation for mysql front-end protocol.
3. Sql translation for trino front-end protocol.
This issue is the first step of the parser module, which introduces the parser module and implements `SHOW KYUUBI_SESSION`, and then improves the Apache kyuubi parser module based on this initial pr.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3944 from Yikf/parser-init.
Closes#3926
4463b298 [Yikf] [KYUUBI #3926] Introduce antlr4 to parse query statement
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3905
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3977 from iodone/kyuubi-3905.
Closes#3977Closes#3905
4e4a22b1 [odone] trino server init and run
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
1. `mvn antrun:runbuild-info -pl kyuubi-common`
2. debug engine suite
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3847 from cxzl25/profile_jdbc_debug.
Closes#3847
e1c32067 [Cheng Pan] Merge branch 'master' into profile_jdbc_debug
f41d3cd9 [sychen] use jdbc-shaded
25b54b18 [sychen] use jdbc-shaded
dd23f864 [sychen] use kyuubi-hive-jdbc as default
79b0ffcb [sychen] add jdbc profile
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to reduce the source file size.
This PR removes the swagger UI resource file in the folder `kyuubi-server/src/main/resources/org/apache/kyuubi/ui/swagger` which was introduced by [PR-2061](https://github.com/apache/incubator-kyuubi/pull/2061) , and now we rely on Maven dependency `org.webjars::swagger-ui::4.9.1` to package these resource files, we also add a new REST endpoint `/swagger-static/` for downloading these resource files
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3710 from cfmcgrady/reduce-src-size.
Closes#3710
c83bd313 [Fu Chen] update comment
441129b5 [Fu Chen] update LICENSE-binary
14eac59b [Fu Chen] update dev/dependencyList
7971c388 [Fu Chen] bump swagger-ui to 4.9.1
455aa206 [Fu Chen] reduce source file size
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-0.14.0
Notable changes:
- Support Spark 3.3
- Unified package name
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3050 from pan3793/iceberg.
Closes#3050
3e700a11 [Cheng Pan] Fix ut caused by SPARK-37931
039c4e2d [Cheng Pan] Enable Iceberg test w/ Spark 3.3
53cac396 [Cheng Pan] Bump Apache Iceberg 0.14.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add etcd discovery client for HA
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2767 from hddong/etcd-support.
Closes#2644
6fa37578 [hongdongdong] Rebase and remove from rat-excludes
fc23a2b4 [hongdongdong] upgrade grpc-1.47.0
61c77d27 [hongdongdong] Fix tests
28abd13e [hongdongdong] fix comments
9e81a6a3 [hongdongdong] Rename HA_ZK_ENGINE_REF_ID to HA_ENGINE_REF_ID
aa2b4260 [hongdongdong] [KYUUBI #2644][WIP] Add etcd discovery client for HA
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3037 from pan3793/jul.
Closes#3037
fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR moves engine integration tests except to spark engine from `kyuubi-server` module to dedicated IT modules e.g. `kyuubi-hive-it`, then we can spilt different engine test into jobs to speed up CI tes
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
before
<img width="836" alt="image" src="https://user-images.githubusercontent.com/26535726/173265839-5d0f1ce6-e133-4d60-9a95-c95aa16f817f.png">
after
<img width="825" alt="image" src="https://user-images.githubusercontent.com/26535726/173265903-fb01e910-8f78-4091-91da-dca25be4357d.png">
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2827 from pan3793/hive-it.
Closes#2827
df3798f1 [Cheng Pan] Remove Java 11 test for Hive
e1645338 [Cheng Pan] nit
66524226 [Cheng Pan] nit
3aef2cef [Cheng Pan] review
500c9cff [Cheng Pan] MVN_OPT
3b4ad142 [Cheng Pan] nit
daa6e0ab [Cheng Pan] review
3383d4c1 [Cheng Pan] [BUILD][TEST] Decouple integration tests from kyuubi-server
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#2879
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2883 from zhouyifan279/2879.
Closes#2883
bbd6c2e7 [zhouyifan279] [Bug] java.lang.NoClassDefFoundError: org/apache/hadoop/hive/common/ValidWriteIdList in HiveDelegationTokenProvider#initialize
d5046f36 [zhouyifan279] [Bug] java.lang.NoClassDefFoundError: org/apache/hadoop/hive/common/ValidWriteIdList in HiveDelegationTokenProvider#initialize
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Different from spark engine, all Flink integration tests has been moved into `kyuubi-flink-it`, then it can be removed from `kyuubi-server` dependencies.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2825 from pan3793/flink-it.
Closes#2825
1c066137 [Cheng Pan] Remove kyuubi-flink-sql-engine from kyuubi-server dependencies
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2674
Rewrite the rest DTO classes in java, which is the preparation for kyuubi rest sdk #2643
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2668 from turboFei/client_do.
Closes#2668
e2bf779c [Fei Wang] revert by reformat
a23f65f6 [Fei Wang] address comments
6648ef90 [Fei Wang] regen the equals & hashCode without accepct subclasses as parameter to equals()
840866c9 [Fei Wang] add equals & hashCode for dto classes
1cc9afaa [Fei Wang] return empty collections
2d49db50 [Fei Wang] return empty batch info map by defaults
1bb399c0 [Fei Wang] only reserve slf4j-api
c6621ed8 [Fei Wang] fix ut
e15ea9f0 [Fei Wang] refactor
ef6dc16c [Fei Wang] add dependes into assembly
ffacf68d [Fei Wang] rewrite the dto classes in java
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
