Commit Graph

140 Commits

Author SHA1 Message Date
Cheng Pan
6ee77dc0d3
Revert "[KYUUBI #6528] Bump Netty 4.1.108.Final"
This reverts commit a6bedc02d0.
2024-07-16 19:24:49 +08:00
avishnus
9c7fec5cb2 [KYUUBI #6529] Upgrade kubernetes-client from 6.8.1 to 6.13.1
This pull request fixes #

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

---

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6529 from avishnus/kubernetes.

Closes #6529

2a82df557 [Cheng Pan] Update dev/dependencyList
a4855467c [Cheng Pan] Update pom.xml
1ef0c40aa [Cheng Pan] Update dev/dependencyList
4b45b0038 [Cheng Pan] Update dev/dependencyList
d0c5174e8 [Cheng Pan] Update dev/dependencyList
bc2f6b8eb [Cheng Pan] Update dev/dependencyList
f76a7f487 [avishnus] Upgrading kubernetes-client

Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 7db5948d22a32a0c9de035b2e5bc024d3c16f429)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-07-15 13:32:45 +08:00
madlnu
a6bedc02d0
[KYUUBI #6528] Bump Netty 4.1.108.Final
# 🔍 Description
Upgrade the Netty version to fix the CVEs

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

GHAs

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6528 from Madhukar525722/netty_upgrade.

Closes #6528

6c083c1fa [madlnu] Bump Netty 4.1.108.Final

Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 80f7496dde)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-07-06 00:09:57 +08:00
avishnus
3790831011
[KYUUBI #6515] Bumping up dropwizard metrics to 4.2.26
# 🔍 Description
## Issue References 🔗

This pull request fixes #6515

## Describe Your Solution 🔧

Bumped up dropwizard metrics from 4.2.23 to 4.2.26

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6527 from avishnus/metrics.

Closes #6515

253036ca3 [avishnus] Bumping up dropwizard metrics to 4.2.26

Authored-by: avishnus <avishnus@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 84f2cf3c5a)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-07-04 22:38:47 +08:00
Cheng Pan
460e553002
[KYUUBI #6458] Remove commons-logging from binary release
# 🔍 Description

[`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues.

The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6458 from pan3793/commons-logging.

Closes #6458

114ec766a [Cheng Pan] fix
79d4121a1 [Cheng Pan] fix
6633e83ee [Cheng Pan] fix
21127ed0b [Cheng Pan] always perform install on dep check
98b13dfcf [Cheng Pan] Remove commons-logging from binary release

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 017d8ccd7e)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-06-11 11:32:44 +08:00
Cheng Pan
20d639d313
[KYUUBI #6457] Strip dependencies from REST client
# 🔍 Description

This PR removes two dependencies from the `kyuubi-rest-client` module
- `commons-collections` - has CVE Cx78f40514-81ff and is only used in one place, just rewrite to remove the dependency
- `javax.servlet-api` - only used for UT, correct the scope from `compile` to `test`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6457 from pan3793/rest-client-dep.

Closes #6457

c24af70ac [Cheng Pan] dep list
80969d0d3 [Cheng Pan] nit
950d4b730 [Cheng Pan] fix
169281903 [Cheng Pan] fix import
c068ba10f [Cheng Pan] nit
d46653d53 [Cheng Pan] fix
ca7831732 [Cheng Pan] Strip dependencies from REST client

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit f7e8d69ac0)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-06-07 17:17:17 +08:00
Cheng Pan
3c3a8dd4e6
[KYUUBI #6419] Bump Jetty 9.4.54.v20240208
# 🔍 Description

Regular dependency upgrade.

- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.53.v20231009
- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.54.v20240208

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6419 from pan3793/jetty-9.4.54.

Closes #6419

a9530da3a [Cheng Pan] Bump Jetty 9.4.54.v20240208

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 6888874808)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-05-24 18:23:42 +08:00
Cheng Pan
6b3f3d25e8
[KYUUBI #6418] Bump Jersey to 2.40
# 🔍 Description

This is a regular dependency upgrade.

https://github.com/eclipse-ee4j/jersey/releases/tag/2.40

Though 2.40 is not the latest version of the Jersey 2 serial, using a version adopted by Spark 3.5 is not harmful.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6418 from pan3793/jersey-240.

Closes #6418

e925acdbd [Cheng Pan] update dependencyList
a853dc289 [Cheng Pan] Bump Jersey to 2.40

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit d40f9155b0)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-05-24 18:21:47 +08:00
Cheng Pan
c36e535703
[KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
# 🔍 Description

This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11)

Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6392 from pan3793/servlet.

Closes #6392

27d412599 [Cheng Pan] fix
9f1e72272 [Cheng Pan] other spark modules
f4545dc76 [Cheng Pan] fix
313826fa7 [Cheng Pan] exclude
7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 6bdf2bdaf8)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-05-20 21:09:50 +08:00
PorterZhang2021
88952c568e
[KYUUBI #6294] Simplify Netty and gRPC dependency management
# 🔍 Description
## Issue References 🔗

This pull request fixes #6294

## Describe Your Solution 🔧

The netty jars come from `grpc-netty`, `arrow-memory-netty`, use `netty-bom` and `grpc-bom` to simplify the dependency management.

Run `./build/dependency.sh --replace`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6310 from PorterZhang2021/issue-6294.

Closes #6294

327030170 [Cheng Pan] simplify netty dependency management
cdb435d01 [PorterZhang2021] [# 6294] Prune unused Netty libraries
e0676ed5a [PorterZhang2021] Finished Prune unused Netty libraries [#6294]

Lead-authored-by: PorterZhang2021 <PorterZhang2021@outlook.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 962de72681)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-04-17 15:41:16 +08:00
liuxiao
4ec70f2c5d
[KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
# 🔍 Description
## Issue References 🔗

This pull request fixes #6271

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6274 from liuxiaocs7/issue-6271.

Closes #6271

36215436d [liuxiao] Upgrade dep list
13c41d150 [liuxiao] Upgrade kafka-clients from 3.5.1 to 3.5.2

Authored-by: liuxiao <liuxiao2103@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 6910d755ce)
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-04-07 23:52:11 +08:00
zhouyifan279
bb54334fa7 [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang
# 🔍 Description
## Issue References 🔗

This pull request removes unused dependency commons-lang from kyuubi-server pom.xml.

## Describe Your Solution 🔧

Removes unused dependency commons-lang from kyuubi-server pom.xml.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6161 from zhouyifan279/commons-lang.

Closes #6145

8d50b66ca [zhouyifan279] [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-03-12 00:49:23 +08:00
zhouyifan279
6fc7552ef0
[KYUUBI #6118] Migrate to shaded HMS client for getting delegation token on server
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Kyuubi Shaded 0.3 introduces a light kyuubi-relocated-hive-metastore-client, for refreshing
delegation token, this PR aims to migrate from the vanilla HMS client to this light shaded HMS client, then we can get rid of Hive dependencies, especially the vulnerable thrift 0.9, from the Kyuubi server.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6118 from zhouyifan279/relocated-hms-client.

Closes #6118

4d7e50915 [Cheng Pan] comment
845e39f87 [Cheng Pan] notice
b4a58a614 [Cheng Pan] Update NOTICE
f4bfa9310 [zhouyifan279] Use kyuubi-relocated-hive-metastore-client
af17be1fe [zhouyifan279] Use kyuubi-relocated-hive-metastore-client

Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-03-08 13:07:11 +08:00
Cheng Pan
182f4349a7
[KYUUBI #6117] Bump Kyuubi Shaded 0.3.0
# 🔍 Description
## Issue References 🔗

This PR upgrades Kyuubi Shaded to 0.3.0, the major changes are:

- upgrade Thrift from 0.9.3-1 to 0.16.0
- upgrade JLine used by Zookeeper CLI from 0.9.94 to 2.14.6
- split kyuubi-relocated-thrift from kyuubi-relocated-hive-service-rpc
- introduce a light kyuubi-relocated-hive-metastore-client for getting DelegationToken

Due to THRIFT-5297 (fixed in 0.14.0) removes requestTimeout and beBackoffSlotLength, the following Kyuubi configurations are removed too.

- kyuubi.frontend.login.timeout
- kyuubi.frontend.thrift.login.timeout
- kyuubi.frontend.backoff.slot.length
- kyuubi.frontend.thrift.backoff.slot.length

Previously, Zookeeper 3.4 depends on JLine 0.9.94 while Zookeeper 3.6 and Hive Beeline depends on JLine 2.14.6, we pulls different versions of JLine into the binary distribution. Now we align them to a single JLine 2.14.6.

## Describe Your Solution 🔧

Upgrade dependencies and change code to adapt to the breaking changes, also upgrade migration guide to mention the removed configurations.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6117 from pan3793/shaded-0.3.

Closes #6117

5639c9fd8 [Cheng Pan] nit
d281cdb4b [Cheng Pan] fix
184e1b95b [Cheng Pan] fix
1628337c8 [Cheng Pan] remove unused conf
24db2d5ad [Cheng Pan] try fix
1e995bb34 [Cheng Pan] nit
1d8e9bce9 [Cheng Pan] dep list
7f33624e8 [Cheng Pan] Bump Kyuubi Shaded 0.3.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-03-08 10:19:30 +08:00
Deepali Gupta
8026e72b3c
[KYUUBI #6101] Bump Jackson from 2.15.0 to 2.15.4
# 🔍 Description
## Issue References 🔗

This pull request fixes https://github.com/apache/kyuubi/issues/6101

## Describe Your Solution 🔧

This updates the jackson version from 2.15.0 to 2.15.4

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6110 from Deepali1211/upgrade-jackson.

Closes #6101

5da3a2527 [Deepali Gupta] Upgrade jackson to 2.15.4
2666a1eb0 [Deepali Gupta] Upgrade jackson to 2.15.4

Authored-by: Deepali Gupta <Deepali.gupta@ibm.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-02-29 14:55:11 +08:00
Cheng Pan
d007d7af10
[KYUUBI #6099] Remove support of Derby for Kyuubi metastore
# 🔍 Description
## Issue References 🔗

This pull request removes support of Derby for Kyuubi metastore.

## Describe Your Solution 🔧

Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore.

Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal.

Also, the Derby we used suffers CVE-2022-46337

> Mitigation:
>
> Users should upgrade to Java 21 and Derby 10.17.1.0.
>
> Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Some tests are modified accordingly. Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6099 from pan3793/remove-derby.

Closes #6099

7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-02-27 19:15:41 +08:00
Cheng Pan
0689b37234
[KYUUBI #6085] Bump gRPC from 1.60.1 to 1.60.2
# 🔍 Description
## Issue References 🔗

A routine work, keep deps up-to-date.

## Describe Your Solution 🔧

Bump gRPC from 1.60.1 to 1.60.2

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6085 from pan3793/grpc-1-60-2.

Closes #6085

68f1cae53 [Cheng Pan] fix
3cdf9bd1b [Cheng Pan] Bump gRPC from 1.60.1 to 1.60.2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-02-26 15:19:43 +08:00
Tigran Manasyan
35a910a43e
[KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
# 🔍 Description
## Issue References 🔗

This pull request fixes #5674

## Describe Your Solution 🔧

Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests
`org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite`

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6027 from tigrulya-exe/feature/5674-postgresql-metadata-backend.

Closes #5674

111658002 [Tigran Manasyan] Fix metastore db type option comment
44a22bfcf [Tigran Manasyan] Fix dependency list and metastore db type option comment
b638c8942 [Tigran Manasyan] Add PostgreSQL as backend database for kyuubi metadata

Authored-by: Tigran Manasyan <t.manasyan@arenadata.io>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-30 13:15:35 +08:00
Cheng Pan
87ed400562
[KYUUBI #5975] Bump trino-client 411
# 🔍 Description
## Issue References 🔗

This is a regular dependency upgrading,

## Describe Your Solution 🔧

Upgrade `trino-client` from 363 to 411. 411 is the latest version which uses okhttp 3.x, hence it does not have kotlin runtime dependencies.

This PR also updates the docs, especially the Trino cluster version requirement.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5975 from pan3793/trino-411.

Closes #5975

2b57df34d [Cheng Pan] fix
c498a5bb3 [Cheng Pan] fix
21948ca4f [Cheng Pan] Fix compile
e4f1397cc [Cheng Pan] license
66583ca16 [Cheng Pan] Bump trino-client 411

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-15 22:25:41 +08:00
Cheng Pan
35e988d182
[KYUUBI #5974] Bump jetcd 0.7.7
# 🔍 Description
## Issue References 🔗

It's regular dependency upgrading, and jetcd 0.7.7 may be the latest version that supports Java 8.

## Describe Your Solution 🔧

Upgrading jetcd to 0.7.7, and this upgrading involves the transitive dep `failsafe` major upgrades, the group is changed from `net.jodah` to `dev.failsafe`

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5974 from pan3793/jetcd-version.

Closes #5974

7c1d815a4 [Cheng Pan] relocation
5eca6cdb4 [Cheng Pan] Bump jetcd 0.7.7

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-15 17:32:15 +08:00
Cheng Pan
f35761b50f
[KYUUBI #5973] Bump gRPC from 1.53.0 to 1.60.1
# 🔍 Description
## Issue References 🔗

It's a regular dependency upgrading task.

## Describe Your Solution 🔧

Bump gRPC from 1.53.0 to 1.60.1, a new module [grpc-util](https://mvnrepository.com/artifact/io.grpc/grpc-util) was added in 1.58.0, LICENSE files updated accordingly.

https://github.com/grpc/grpc-java/releases

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5973 from pan3793/grpc.

Closes #5973

b47c7f295 [Cheng Pan] nit
dd0e966f0 [Cheng Pan] Bump gRPC from 1.53.0 to 1.60.1

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-15 14:26:15 +08:00
Bowen Liang
e2d9a8efb6
[KYUUBI #5879] Bump dropwizard metrics from 4.2.8 to 4.2.23
# 🔍 Description
## Issue References 🔗

As described.

## Describe Your Solution 🔧

- Currently used Dropwiard Metrics v4.2.8 was released on Feb 3, 2022
- Dropwiard Metrics v4.2.23 released on Dec 6, 2023, release notes: https://github.com/dropwizard/metrics/releases/tag/v4.2.23

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
No behaviour changes.

#### Behavior With This Pull Request 🎉
No behaviour changes.

#### Related Unit Tests
No behaviour changes.

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5879 from bowenliang123/metrics-4.2.23.

Closes #5879

08c3469f4 [Bowen Liang] bump dropwizard to 4.2.23

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-20 14:58:30 +08:00
Bowen Liang
c2e90a9e9e [KYUUBI #5395] Bump netty from 4.1.93.Final to 4.1.100.Final
Bumps [io.netty:netty-codec-http2](https://github.com/netty/netty) from 4.1.93.Final to 4.1.100.Final.
<details>
<summary>Commits</summary>
<ul>
<li><a href="58df783eb4"><code>58df783</code></a> [maven-release-plugin] prepare release netty-4.1.100.Final</li>
<li><a href="58f75f665a"><code>58f75f6</code></a> Merge pull request from GHSA-xpw8-rcwv-8f8p</li>
<li><a href="491144865a"><code>4911448</code></a> Do not fail when compressing empty HttpContent (<a href="https://redirect.github.com/netty/netty/issues/13655">#13655</a>)</li>
<li><a href="caca5e5a1e"><code>caca5e5</code></a> When read PoolSubpage's variant fields, it should lock on PoolSubpage's head ...</li>
<li><a href="d97f2a5606"><code>d97f2a5</code></a> Update checkout action to latest version (<a href="https://redirect.github.com/netty/netty/issues/13649">#13649</a>)</li>
<li><a href="275341f01c"><code>275341f</code></a> Fix issue with unrecognized JVM option while running with Java 11 (<a href="https://redirect.github.com/netty/netty/issues/13648">#13648</a>)</li>
<li><a href="5db037beed"><code>5db037b</code></a> Speedup max direct memory estimation via Unsafe (<a href="https://redirect.github.com/netty/netty/issues/13643">#13643</a>)</li>
<li><a href="ce5c78cec1"><code>ce5c78c</code></a> Update actions to the latest version (<a href="https://redirect.github.com/netty/netty/issues/13644">#13644</a>)</li>
<li><a href="d7a8169f1b"><code>d7a8169</code></a> [maven-release-plugin] prepare for next development iteration</li>
<li><a href="8e3fe28425"><code>8e3fe28</code></a> [maven-release-plugin] prepare release netty-4.1.99.Final</li>
<li>Additional commits viewable in <a href="https://github.com/netty/netty/compare/netty-4.1.93.Final...netty-4.1.100.Final">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=io.netty:netty-codec-http2&package-manager=maven&previous-version=4.1.93.Final&new-version=4.1.100.Final)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).

</details>

> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.

Closes #5395 from dependabot[bot]/dependabot/maven/io.netty-netty-codec-http2-4.1.100.Final.

Closes #5395

51028b289 [Cheng Pan] remove obsolete comments
a8f916c0f [Bowen Liang] update dependencyList
6813ccb4e [dependabot[bot]] Bump io.netty:netty-codec-http2 from 4.1.93.Final to 4.1.100.Final

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 19:40:56 +08:00
liangbowen
ab4dfa3688 [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1
### _Why are the changes needed?_

- Bump Kafka client version from 3.4.0 to 3.5.1
- Bump Docker image for Kafka 3 to 3.5.0 (equivalent to Confluent Platform 7.5.x)
- https://docs.confluent.io/platform/current/installation/versions-interoperability.html#cp-and-apache-ak-compatibility

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5486 from bowenliang123/bump-kafka.

Closes #5486

017ea860d [Bowen Liang] update dependencyList
20ad988ed [liangbowen] bump kafka client version 3.5.1

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-25 20:15:30 +08:00
Cheng Pan
75428bb979
[KYUUBI #5484] Remove legacy Web UI
### _Why are the changes needed?_

Close https://github.com/apache/kyuubi/issues/5484

Kyuubi provides a basic new Web UI which is built on top of Vue3, we can remove the legacy dummy Web UI in 1.8.

The new UI hosts at `http://<host>:<port>/ui/` and the legacy UI hosts at `http://<host>:<port>/`, we should

1. Remove the legacy UI routing from Jetty
2. Remove all files related to legacy UI
3. Redirect `http://<host>:<port>/` to `http://<host>:<port>/ui/`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

building with the command `build/dist --web-ui`, then `cd dist` and perform `bin/kyuubi run`

access http://0.0.0.0:10099 could correctly redirect to http://0.0.0.0:10099/ui/

<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1e8a67f6-e4db-415e-8a47-dd7c41b487cf">

swagger is render correctly too.
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1cb4ba31-9965-4468-b7c3-b0319ba959e6">

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5516 from pan3793/5484.

Closes #5484

9d58ef758 [Cheng Pan] address comment and fix test
6d4c0981b [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2023-10-25 13:36:00 +08:00
pengqli
9b93e874a0 [KYUUBI #5293] upgrade snakeyaml from 1.33 to 2.2
### _Why are the changes needed?_

upgrade  snakeyaml from 1.33 to 2.2 reducing direct CVE vulnerabilities, see (https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes)
[CVE-2022-1471](https://nvd.nist.gov/vuln/detail/CVE-2022-1471)
SnakeYaml's Constructor() class does not restrict types which can be instantiated during deserialization. Deserializing yaml content provided by an attacker can lead to remote code execution. We recommend using SnakeYaml's SafeConsturctor when parsing untrusted content to restrict deserialization. We recommend upgrading to version 2.0 and beyond.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5293 from dev-lpq/snakeyaml_critical.

Closes #5293

5b2412d8e [pengqli] upgrade  snakeyaml from 1.33 to 2.2

Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-15 11:32:09 +00:00
Sidhant Nagpal
5abc2625f1
[KYUUBI #5234] Upgrade Jetty version from 9.4.51.v20230217 to 9.4.52.v20230823
Fixes #5234

Closes #5239 from sidhant-nagpal/fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823.

Closes #5234

cbd4b4990 [Sidhant Nagpal] Merge branch 'master' of https://github.com/sidhant-nagpal/kyuubi into fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823
608a510a9 [Sidhant Nagpal] Merge branch 'master' of https://github.com/sidhant-nagpal/kyuubi into fix#5234/jetty-upgraded-from-9.4.51.v20230217-to-9.4.52.v20230823
d2ac35e08 [Sidhant Nagpal] Revert "#5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823"
f15af280d [Sidhant Nagpal] #5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823
2498302b4 [Sidhant Nagpal] #5234 Upgrade Jetty from 9.4.51.v20230217 to 9.4.52.v20230823

Authored-by: Sidhant Nagpal <sidnagpal2000@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-04 21:46:34 +08:00
Bowen Liang
be93d89744 [KYUUBI #5241] Bump Commons Lang3 from 3.12.0 to 3.13.0
### _Why are the changes needed?_

- Apache Commons Lang 3.13.0 release note: https://commons.apache.org/proper/commons-lang/changes-report.html#a3.13.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5241 from bowenliang123/commonslang3-3.13.

Closes #5241

99e5ffbd9 [Bowen Liang] update dependencyList
f2bf9c554 [Bowen Liang] bump commons lang3 to 3.13.0

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-09-04 16:22:04 +08:00
fwang12
b1eef09dce [KYUUBI #5205] [BUILD][k8s] Upgrade kubernetes-client to 6.8.1
### _Why are the changes needed?_

This PR aims to upgrade kubernetes-client to 6.8.1.
https://github.com/fabric8io/kubernetes-client/releases/tag/v6.8.0
https://github.com/fabric8io/kubernetes-client/releases/tag/v6.8.1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5205 from turboFei/k8s_client_version_upgrade.

Closes #5205

3d8d06ca3 [fwang12] Upgrade kubernetes client version to 6.8.1

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-08-28 19:46:08 +08:00
Cheng Pan
ea16606596
[KYUUBI #5116] Bump Hadoop 3.3.6
### _Why are the changes needed?_

https://hadoop.apache.org/release/3.3.6.html

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5116 from pan3793/hadoop-3.3.6.

Closes #5116

c3717e7fb [Cheng Pan] Bump Hadoop 3.3.6

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-01 17:07:43 +08:00
fwang12
1ea9f8df98
[KYUUBI #5059] [BUILD] Upgrade kubernetes client version to 6.7.2
### _Why are the changes needed?_

Now the kubernetes client version in apache/spark is 6.7.2(https://github.com/apache/spark/pull/41490), I think it is better to align with that.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5059 from turboFei/k8s_client_version.

Closes #5059

79955d17a [fwang12] deps
4a68cc820 [fwang12] bump

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-17 18:09:11 +08:00
dependabot[bot]
59838015ce
Bump grpc-protobuf from 1.48.0 to 1.53.0 (#5024) 2023-07-07 10:16:30 +00:00
liangbowen
9855802505 [KYUUBI #5001] Bump Scala from 2.12.17 to 2.12.18
### _Why are the changes needed?_

- Bump Scala to 2.12.18
  - release note: https://github.com/scala/scala/releases/tag/v2.12.18
- Bump silencer-plugin from 1.7.10 to 1.7.13 for compatibility

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5001 from bowenliang123/scala-2.12.18.

Closes #5001

e4a5b9b0d [liangbowen] update dependencyList
e847bbced [liangbowen] Bump silencer-plugin from 1.7.10 to 1.7.13
daceece66 [liangbowen] Bump scala from 2.12.17 to 2.12.18

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-06-27 18:23:34 +08:00
Cheng Pan
c8905bfb89 [KYUUBI #4992] Bump Netty 4.1.93.Final
### _Why are the changes needed?_

Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final

Currently, we can not upgrade to 4.1.94.Final or above because of https://github.com/netty/netty/pull/13408

```
java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
	at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
	at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #4992 from pan3793/netty.

Closes #4992

9dd8f9ad0 [Cheng Pan] nit
e6953144e [Cheng Pan] Bump Netty 4.1.93.Final

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-06-26 13:28:02 +08:00
Cheng Pan
06a915aff4 [KYUUBI #4950] Migrate Kyuubi embedded database from Derby to SQLite
### _Why are the changes needed?_

Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.

SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.

SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4950 from pan3793/sqlite.

Closes #4950

738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-06-12 21:01:51 +08:00
liangbowen
a9fcf3f280 [KYUUBI #4943] Bump Guava from 31.1 to 32.0.1
### _Why are the changes needed?_

- Guava release note:
  - 32.0.0: https://github.com/google/guava/releases/tag/v32.0.0
  - 32.0.1: https://github.com/google/guava/releases/tag/v32.0.1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4943 from bowenliang123/guava-32.0.1.

Closes #4943

b8cfe62a1 [liangbowen] update dependencyList
6c2dcef4d [liangbowen] bump guava from 31.1 to 32.0.1

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-06-09 10:19:15 +08:00
zwangsheng
4cd00a8777
[KYUUBI #3420][UI] Kyuubi Server Proxy Engine UI
### _Why are the changes needed?_

Kyuubi Server Proxy Engine UI

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![截屏2023-06-06 10 35 54](https://github.com/apache/kyuubi/assets/52876270/ecbc33aa-11dd-418f-bfef-19aad9e7ea39)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4795 from zwangsheng/KYUUBI_3420.

Closes #3420

079dc1c60 [zwangsheng] fix frontend unit test case
6e71b4518 [Cheng Pan] fix
cf7ca5145 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala
9a91d62a0 [Cheng Pan] polish
a5dcfae18 [zwangsheng] fix
5d4a8c239 [zwangsheng] Rebase
71d22fc9a [zwangsheng] fix
3b0152f33 [zwangsheng] [KYUUBI #3420][UI] Proxy Engnie UI

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-06-06 15:17:40 +08:00
Cheng Pan
208ab3af62
[KYUUBI #4852] Switch to Kyuubi Shaded Zookeeper
### _Why are the changes needed?_

This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17.

There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical.

With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested

zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).

We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4852 from pan3793/shaded-zk.

Closes #4852

d960cc945 [Cheng Pan] remove staging repo
1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-21 20:49:00 +08:00
Fu Chen
ad97b03579
[KYUUBI #4827] Bump Apache Arrow from 11.0.0 to 12.0.0
### _Why are the changes needed?_

https://arrow.apache.org/release/12.0.0.html

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4827 from cfmcgrady/arrow-12.0.0.

Closes #4827

324a8ddbf [Fu Chen] bump arrow from 11.0.0 to 12.0.0

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-11 22:35:52 +08:00
Fu Chen
4c37a882d2 [KYUUBI #4824] Bump Jackson from 2.14.2 to 2.15.0
### _Why are the changes needed?_

spark bump jackson from 2.14.2 to 2.15.0 in https://github.com/apache/spark/pull/40933

to fix

https://github.com/apache/kyuubi/actions/runs/4943800010/jobs/8838642303

```
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (localhost executor driver): java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/StreamReadConstraints
	at org.apache.spark.sql.catalyst.json.JSONOptions.buildJsonFactory(JSONOptions.scala:195)
	at org.apache.spark.sql.catalyst.json.JsonInferSchema.$anonfun$infer$1(JsonInferSchema.scala:83)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1514)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.StreamReadConstraints
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 16 more
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4824 from cfmcgrady/jackson-2.15.0.

Closes #4824

7a8a3de89 [Fu Chen] update dev/dependencyList
2d01b4b9d [Fu Chen] bump jackson

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-05-11 21:05:09 +08:00
liangbowen
4e0562df8f [KYUUBI #4808] Bump Jetty from 9.4.50 to 9.4.51
### _Why are the changes needed?_

- to fix 2 Dependabot alerts
  1. https://github.com/apache/kyuubi/security/dependabot/19
  2. https://github.com/apache/kyuubi/security/dependabot/18

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4808 from bowenliang123/jetty-9.4.51.

Closes #4808

84789292b [liangbowen] update dependencyList
d1ebdc68a [liangbowen] Revert "update dependencyList"
e4e26b7b5 [liangbowen] update dependencyList
6cfb37dbb [liangbowen] bump jetty to 9.4.51

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: bowenliang <bowenliang@apache.org>
2023-05-09 08:44:00 +08:00
liangbowen
d73ec64b37 [KYUUBI #4733] Introduce Kafka event logger for server events
### _Why are the changes needed?_

- introduce new event logger type `KAFKA`
- send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle
- use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4733 from bowenliang123/kafka-logger.

Closes #4733

b5220d234 [liangbowen] introduce kafka server event logger

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-05-08 22:45:52 +08:00
liangbowen
0b72a6151c [KYUUBI #4772] Bump Jersey from 2.39 to 2.39.1
### _Why are the changes needed?_

- Jersey 2.39.1 release note: https://github.com/eclipse-ee4j/jersey/releases/tag/2.39.1
    - https://github.com/eclipse-ee4j/jersey/pull/5282

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4772 from bowenliang123/jersey-2.39.1.

Closes #4772

356753565 [liangbowen] update dependencyList
e3d3f11bd [liangbowen] bump jersey from 2.39 to 2.39.1

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-04-25 22:22:16 +08:00
Cheng Pan
a9d3e11926
[KYUUBI #4652] Upgrade Hadoop from 3.3.4 to 3.3.5
### _Why are the changes needed?_

Upgrade Hadoop from 3.3.4 to 3.3.5, close #4652

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4689 from huapan123456/master.

Closes #4652

810af9848 [Cheng Pan] fix
380786e9f [Cheng Pan] nit
7c5f18ef9 [Cheng Pan] nit
2d308943b [Cheng Pan] nit
76869d822 [Cheng Pan] handle HDFS-16591
36142c787 [huapan] fix(KYUUBI #4652): the position of JaasConfiguration has changed on hadoop-clint-api-3.3.5
1b5de6dfe [huapan] feat(KYUUBI #4652): upgrade hadoop-version from 3.3.4 to 3.3.5
f9d34c716 [huapan] feat(KYUUBI #4652): upgrade hadoop-client-api from 3.3.4 to 3.3.5

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: huapan <huapan@52tt.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-04-13 13:06:29 +08:00
yehere
ca93a4e5c2
[KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
### _Why are the changes needed?_

close #4325

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4417 from yehere/kyuubi-4325.

Closes #4325

7b2864b53 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,update the dependency file, run './build/dependency.sh --replace'
749b1c15c [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
33ea9ba2b [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
568418a21 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Add test case for DEALLOCATE PREPARE
358a8e3b8 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Support DEALLOCATE PREPARE
7d4a32402 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
91392add6 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
63bf8c462 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
f5b7fb786 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
b0476a79d [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
1a8f147a0 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc

Authored-by: yehere <867171931@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-30 20:37:55 +08:00
zwangsheng
1bc05e5e45
[KYUUBI #4453][Improvement][K8S] Bump Kubernetes Client Version to 6.4.1
### _Why are the changes needed?_

Close #4453

kubernetes client [compare](https://github.com/fabric8io/kubernetes-client/compare/v5.12.1...v6.4.1)

version | K8s 1.26.0 | K8s 1.25.3 | K8s 1.24.7 | K8s 1.23.13 | K8s 1.22.1 | K8s 1.21.1 | K8s 1.20.2 | K8s 1.19.1 | K8s 1.18.0 | K8s 1.17.0 | K8s 1.16.0 | K8s 1.15.3 | K8s 1.14.2 | K8s 1.12.0 | K8s 1.11.0 | K8s 1.10.0 | K8s 1.9.0
--|-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
kubernetes-client 6.4.1 |   | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -
kubernetes-client 5.12.1 |   |   |   | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

- [x] [Run ci test]

Closes #4456 from zwangsheng/bump/kubernetes-client-6.4.1.

Closes #4453

39039f0f9 [zwangsheng] [KYUUBI #4453] Remove dup dependence
4a5b27ce8 [zwangsheng] [KYUUBI #4453] IT test with same logic
95a292f99 [zwangsheng] [KYUUBI #4453] Fix Dependency
e4bf0107e [zwangsheng] [KYUUBI #4453] Init Bump

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-07 14:02:26 +08:00
liangbowen
cc4ec5a5d5 [KYUUBI #4448] Bump Netty from 4.1.87.Final to 4.1.89.Final
### _Why are the changes needed?_

- Netty 4.1.89.Final: https://netty.io/news/2023/02/13/4-1-89-Final.html
- Netty 4.1.88.Final: https://netty.io/news/2023/02/12/4-1-88-Final.html

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4448 from bowenliang123/netty-4.1.89.

Closes #4448

8854bb3ea [liangbowen] bump netty to 4.1.89.Final

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-06 13:05:46 +08:00
liangbowen
489d8a3c66 [KYUUBI #4447] Bump log4j from 2.19.0 to 2.20.0
### _Why are the changes needed?_

- log4j 2.20.0 release notes: https://logging.apache.org/log4j/2.x/release-notes/2.20.0.html

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4447 from bowenliang123/log4j-2.20.0.

Closes #4447

c9bc9aa95 [liangbowen] bump log4j from 2.19.0 to 2.20.0

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-06 11:20:16 +08:00
liangbowen
9d33b849fb
[KYUUBI #4357] Bump Jersey from 2.38 to 2.39
### _Why are the changes needed?_

to close #4357.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4366 from bowenliang123/jersey-2.39.

Closes #4357

bd214e8d [liangbowen] bump jersey from 2.38 to 2.39

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-19 16:33:26 +08:00
odone
41f08059f0 [KYUUBI #3935] Support use Trino client to submit SQL
### _Why are the changes needed?_

Close #3935

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4232 from iodone/kyuubi-3935.

Closes #3935

936ea1f8 [odone] address
e7bd01a1 [odone] support trino client connect kyuubi trino server
9ea8b6af [odone] [WIP] trion request/response implementation

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-02-13 19:28:14 +08:00