### _Why are the changes needed?_
This PR adds a basic Grafana Dashboard template, also updates the metrics docs to guide users to use Prometheus and Grafana to monitor the Kyuubi server.
The Grafana Dashboard template is exported from the Grafana OSS v11.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1484" alt="image" src="https://github.com/user-attachments/assets/417b35fa-cd12-4e51-b73f-2955282aa187" />
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5147 from zhaohehuhu/Improvement-0809.
Closes#5834
f6fc2d71e [Cheng Pan] fix style
465f0546a [Cheng Pan] update dashboard
3fa2d237e [hezhao2] add status chart
4b2bd3dbc [hezhao2] add status chart
185f2cccf [hezhao2] make it compatible with kyuubi 1.8
457085be5 [hezhao2] add REAMDE.md to guide users
45e3ba3e5 [hezhao2] add docker file build a grafana image and load dashboards available
dbc22108b [hezhao2] Add Grafana dashboard template
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 3.5.4 is released https://spark.apache.org/releases/spark-release-3-5-4.html
### How was this patch tested?
Pas GHA
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6842 from pan3793/spark-3.5.4.
Closes#6842
0fb7ad8a0 [Cheng Pan] ga
8eacc9c97 [Cheng Pan] Spark 3.5.4 RC2
0721fa401 [Cheng Pan] fix
49e98a201 [Cheng Pan] maven repo
951db0c82 [Cheng Pan] Spark 3.5.4
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
the prometheus can not discover the podMonitor created by kyuubi helm chart.
### How was this patch tested?
1. enable pod monitor:
```
metrics:
# Enable metrics system, used for 'kyuubi.metrics.enabled' property
enabled: true
# A comma-separated list of metrics reporters, used for 'kyuubi.metrics.reporters' property
reporters: PROMETHEUS
# Prometheus port, used for 'kyuubi.metrics.prometheus.port' property
prometheusPort: 10019
# PodMonitor by Prometheus Operator
podMonitor:
# Enable PodMonitor creation
enabled: true
# List of pod endpoints serving metrics to be scraped by Prometheus, see Prometheus Operator docs for more details
#podMetricsEndpoints: []
podMetricsEndpoints:
- path: /metrics
port: prometheus
```
2. Helm install kyuubi .
3. check prometheus web ui.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6845 from eric9204/add-lab-to-pm.
Closes#6845
4d713a06c [Cheng Pan] Update charts/kyuubi/values.yaml
f8fa51f28 [dongsj] resolve conflicts with master
Lead-authored-by: dongsj <90449228+eric9204@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
I am using the podminitor of kyuubi metrics, But I found using the podmonitor's selector.matchLabels can not select correct pods . So I fix it .
I also changed the values.yaml to add an example of podmonitor to improve the usability.
### How was this patch tested?
```
helm install kyuubi .
# kubectl get podmonitor kyuubi -oyaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
annotations:
meta.helm.sh/release-name: kyuubi
meta.helm.sh/release-namespace: default
creationTimestamp: "2024-12-06T07:46:12Z"
generation: 1
labels:
app.kubernetes.io/instance: kyuubi
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kyuubi
app.kubernetes.io/version: 1.8.0
helm.sh/chart: kyuubi-0.1.0
name: kyuubi
namespace: default
resourceVersion: "7583800"
uid: 0dd11e43-d126-434e-988a-7f1914586e2b
spec:
podMetricsEndpoints:
- path: /metrics
port: prometheus
selector:
matchLabels:
app.kubernetes.io/instance: kyuubi
app.kubernetes.io/name: kyuubi
# kubectl get po -l app.kubernetes.io/instance=kyuubi
NAME READY STATUS RESTARTS AGE
kyuubi-0 1/1 Running 0 24m
kyuubi-1 1/1 Running 0 22m
# kubectl get po -l app.kubernetes.io/name=kyuubi
NAME READY STATUS RESTARTS AGE
kyuubi-0 1/1 Running 0 24m
kyuubi-1 1/1 Running 0 23m
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6840 from zhifanggao/change_podmonitor.
Closes#6840
092f13f87 [zhifanggao] change the podmonitor to select pods correctly
Authored-by: zhifanggao <28681649@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
Currently, Kyuubi supports Trino progress in scala-2.12, but got some error in scala-2.13.
* Make Trino progress available in scala-2.13
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6841 from naive-zhang/trino-scala13.
Closes#6726
0693d281a [Cheng Pan] Update externals/kyuubi-trino-engine/pom.xml
c20c49a3e [native-zhang] modify scala code style
6cfe4453e [native-zhang] unify scala grammar
c0cd92e47 [native-zhang] change total stage num back into 3 in TrinoOperationProgressSuite
852398ddc [naive-zhang] Merge branch 'apache:master' into trino-scala13
6a038ec2b [native-zhang] move trino progress monitor available in profile scala-2.13
f434a2142 [native-zhang] move trino progress monitor into scala-2.12 impl and change state num in some test case
8d291f513 [native-zhang] add scala version diff in pom for trino module
Lead-authored-by: native-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Enable gluten CI for kyuubi apache repo
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6846 from wForget/gluten_ci.
Closes#6846
22ab3ada8 [wforget] Enable gluten CI for kyuubi apache repo
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
Adding an example to `servicemonitor` in values.yaml to improve the usability .
### How was this patch tested?
execute command `helm install kyuubi-test .`
check the servicemonitor after deployment
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6839 from zhifanggao/add_default_endpoints.
Closes#6839
59aac4142 [Cheng Pan] Update charts/kyuubi/values.yaml
869f45e79 [zhifanggao] include the key itself
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: zhifanggao <28681649@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long
2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`.
68a6f48da5/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql (L32)
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6829 from turboFei/batch_pending_time.
Closes#6829
ee4f93125 [Wang, Fei] docs
bf8169ad4 [Wang, Fei] comments
f493a2af8 [Wang, Fei] new config
ab7b6db65 [Wang, Fei] ut
168017587 [Wang, Fei] in memory session
510a30b6a [Wang, Fei] batchSearchWindow opt
1e93dd276 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
I'd like to include `kafka-clients` in the Kyuubi binary distribution tarball to enable the out-of-box support for sinking Kyuubi events to Kafka.
- Kafka is an important component in modern data platforms, and is a defacto message queue implementation, especially in the big data domain
- `kafka-clients` is released under Apache License V2, has no legal issue
- `kafka-clients` is quite a light lib, has no third-party deps except for `slf4j-api` and a few optional compression libs
- `kafka-clients` uses "none" compression as default, in practice, "gzip"(delegate to JDK gzip algorithm, no additional libs are required) already performs well for non-extreme cases
Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking logging to Kafka, which also requires `kafka-clients` in the classpath, I have some initial ideas to forward both Kyuubi server's and engine bootstrap processes log to Kafka in a structured format, will send another PR to add docs to guide users in configuring that.
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6836 from pan3793/kafka-lib.
Closes#6836
b069eb199 [Cheng Pan] Ship kafka-clients in binary distribution tarball without compression libs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Replace the obsolete address with a new one.
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6838 from pan3793/mail.
Closes#6838
858e6cc72 [Cheng Pan] [INFRA] Update archive mailing list address
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The GHA cache `engine-archives` seems frequently to be evicted because of exceeding the project cache capacity(10GiB), and I found the K8s IT job produces fresh large cache objects frequently, we should disable that to let `engine-archives` survive, to avoid downloading from the fragile `archive.apache.org` each time.
```
Connect to archive.apache.org:443 [archive.apache.org/65.108.204.189, archive.apache.org/2a01:4f9:1a:a084:0:0:0:2] failed: Network is unreachable (connect failed)
```
<img width="1306" alt="Xnip2024-12-04_17-36-03" src="https://github.com/user-attachments/assets/7e00ecad-8492-4a3e-b669-03b56a05747c">
PS: I deleted bunches of `buildkit-blob-*` manually to save `engine-archives`
### How was this patch tested?
Monitor GHA after merging.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6837 from pan3793/k8s-it.
Closes#6837
8bab514dc [Cheng Pan] [INFRA] Disable K8s CI image GHA cache
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
Currently, Kyuubi supports JDBC engines with limited dialects, and I extend the dialects to support Oracle.
* Introduce Oracle support in JDBC Engine
* Adding dialects and tests for Oracle
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Add tests of `OperationWithOracleEngineSuite`, `OracleOperationSuite`, `OracleSessionSuite` and `OracleStatementSuite`.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6815 from naive-zhang/jdbc-oracle.
Closes#6815
0ffad5b6b [native-zhang] add some brief comments on the caller side for the implementation of Oracle JDBC engine
6f469a135 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ae70710e6 [Cheng Pan] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
171d06b9e [native-zhang] use another implementation of transform decimal into int, in engine instead of KyuubiBaseResultSet
7cb74d28e [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ccd7cae8b [naive-zhang] remove redundant override methods in OracleSQLDialect.scala
a7da4a646 [naive-zhang] remove redundant impl of getTableTypesOperation in OracleSQLDialect.scala
70b49fcba [naive-zhang] Use the single line string if SQL fits in one line, otherwise write it in a pretty style
e58348460 [naive-zhang] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
b33e97a08 [naive-zhang] remove redundant testcontainers-scala-oracle-xe dependency in pom.xml
4c967b98e [naive-zhang] use gvenzl/oracle-free:23.5-slim with docker-compose for test case
0215e6d49 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
d688b4706 [naive-zhang] change oracle image into gvenzl/oracle-free:23.5-slim
abf983727 [naive-zhang] fix code style checking error in KyuubiConf.scala
d1e82edb1 [naive-zhang] fix code style checking error in settings.md
aa2e2e9ba [naive-zhang] adjust wired space in OracleSQLDialect
b43cea421 [naive-zhang] add oracle configuration for kyuubi.engine.jdbc.connection.provider
397c1cfec [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
2f1b5ed0b [naive-zhang] add jdbc support for Oracle
Lead-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: native-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport 2b0e424daa
The commit does not have a Hive ticket.
### How was this patch tested?
Pass GHA to ensure it breaks nothing, and verify locally that the reported error message does contain the data when failing to parse the illegal date.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6828 from pan3793/hive-jdbc-2b0e424.
Closes#6828
b6e03a661 [Cheng Pan] KyuubiSQLException
63861af99 [Cheng Pan] Backport Hive 2b0e424 - Clean up and improve error message for KyuubiBaseResultSet
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
As pan3793 pointed out that version 2.24.1 had critical bug in https://github.com/apache/kyuubi/pull/6774#issuecomment-2499523805
Bump bump log4j from 2.24.1 to 2.24.2
- Release：https://logging.apache.org/log4j/2.x/release-notes.html#release-notes-2-24-2
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6826 from pionCham/bump-log4j-version.
Closes#6826
150713f02 [chengpeiming] Bump log4j to 2.24.2
Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: chengpeiming <chengpeiming@chengpeimingdeMacBook-Pro.local>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This issue was noticed a few times when the batch `state` was `set` to `ERROR`, but the `appState` kept the non-terminal state forever (e.g. `RUNNING`), even if the application was finished (in this case Yarn Application).
```json
{
"id": "********",
"user": "****",
"batchType": "SPARK",
"name": "*********",
"appStartTime": 0,
"appId": "********",
"appUrl": "********",
"appState": "RUNNING",
"appDiagnostic": "",
"kyuubiInstance": "*********",
"state": "ERROR",
"createTime": 1725343207318,
"endTime": 1725343300986,
"batchInfo": {}
}
```
It seems that this happens when there is some intermittent failure during the monitoring step and the batch ends with ERROR, leaving the application metadata without an update. This can lead to some misinterpretation that the application is still running. We need to set this to `UNKNOWN` state to avoid errors.
## Describe Your Solution 🔧
This is a simple fix that only checks if the batch state is `ERROR` and the appState is not in a terminal state and changes the `appState` to `UNKNOWN`, in these cases (during the batch metadata update).
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
If there is some error between the Kyuubi and the Application request (e.g. YARN client), the batch is finished with `ERROR` state and the application keeps the last know state (e.g. RUNNING).
#### Behavior With This Pull Request 🎉
If there is some error between the Kyuubi and the Application request (e.g. YARN client), the batch is finished with `ERROR `state and the application has a non-terminal state, it is forced to `UNKNOWN` state.
#### Related Unit Tests
I've tried to implement a unit test to replicate this behavior but I didn't make it. We need to force an exception in the Engine Request (e.g. `YarnClient.getApplication`) but we need to wait for the application to be in the RUNNING state before raising this exception, or maybe block the connection between kyuubi and the engine.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6722 from joaopamaral/fix/app-state-on-batch-error.
Closes#6722
8409eacac [Wang, Fei] fix
da8c356a7 [Joao Amaral] format fix
73b77b3f7 [Joao Amaral] use isTerminated
64f96a256 [Joao Amaral] Remove test
1eb80ef73 [Joao Amaral] Remove test
13498fa6b [Joao Amaral] Remove test
60ce55ef3 [Joao Amaral] add todo
3a3ba162b [Joao Amaral] Fix
215ac665f [Joao Amaral] Fix AppState when Engine connection is terminated
Lead-authored-by: Joao Amaral <7281460+joaopamaral@users.noreply.github.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6726
## Describe Your Solution 🔧
Add trino statement progress
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6759 from taylor12805/trino-progress.
Closes#6726
6646c9511 [taylor.fan] [KYUUBI #6726] update test case result
d84904e82 [taylor.fan] [KYUUBI #6726] reformat code
2b1c776e1 [taylor.fan] [KYUUBI #6726] reformat code
f635b38de [taylor.fan] [KYUUBI #6726] add test case
7c29ba6f3 [taylor.fan] [KYUUBI #6726] Support trino stage progress
Authored-by: taylor.fan <taylor.fan@vipshop.com>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
Fix CVE: https://github.com/advisories/GHSA-m6vm-37g8-gqvh
## Describe Your Solution 🔧
Upgrade to 8.4.0, the latest 8.x version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6818 from turboFei/mysql_version.
Closes#6818
954d3c91e [Wang, Fei] Upgrade mysql jdbc version to 8.4.0
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
From my review experience in the past few months, the contributors have less interest in filling the current rich PR template without explicit requests from reviewers.
Instead of keeping a rich template to ask the contributors to answer bunches of questions, I prefer to leave only the most important questions to keep the template clean and short.
### How was this patch tested?
It's not a code change, it requires manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6821 from pan3793/pr-template.
Closes#6821
b1bbdd6c1 [Cheng Pan] nit
ac9a9a59b [Cheng Pan] nit
6fece594c [Cheng Pan] Restore to the plain PULL_REQUEST_TEMPLATE
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
```
export JAVA_HOME=/path/of/openjdk-17
build/mvn clean install -DskipTests -Dmaven.scaladoc.skip=false
```
```
[INFO] --- scala-maven-plugin:4.9.2:doc-jar (attach-scaladocs) kyuubi-server-plugin ---
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.20,1.7.19,null)
error: fatal error: object scala in compiler mirror not found.
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Successfully run the build command
```
export JAVA_HOME=/path/of/openjdk-17
build/mvn clean install -DskipTests -Dmaven.scaladoc.skip=false
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6820 from pan3793/scaladoc.
Closes#6820
f5cee3429 [Cheng Pan] Explicitly disable attach-scaladocs for pure Java modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- fix the compilation error on Zulu JDK8 (Zulu 8.82.0.21-cA-macos-aarch64 with M4 Max chip on MacOS 15.1), when running `build/mvn clean install -pl :kyuubi-util -DskipTests -am` command to build a pure Java module, by replacing version code sytle from 1.8 to 8.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6813 from bowenliang123/java8.
Closes#6813
1a1e3f3a7 [Bowen Liang] set java.version to 8
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6807 from pionCham/bump-maven-plugin-version.
Closes#6807
6582a194c [Bowen Liang] bump to 1.11.3
1505449a6 [chengpeiming] Bump maven plugin download version to 1.11.2
Lead-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump commons-codec from 1.15 (Seq 2020) to 1.17.1 (Jul 2024): https://commons.apache.org/proper/commons-codec/changes-report.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6773 from bowenliang123/commons-codec-1.17.
Closes#6773
347ef0fe9 [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- batching removals to avoid repeatedly reading and updating the apt-get databases
- 50% speed up the execution time for freeing disk space from 2 minutes or more , to about 1 minutues.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6765 from bowenliang123/speedup-freedisk.
Closes#6765
dd475c243 [Bowen Liang] update (+1 squashed commit) Squashed commits: [688793f47] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.7.0 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.7.0
## Describe Your Solution 🔧
- Bump Apache Iceberg to 1.7.0
- As Apache Iceberg 1.7.0 drops support for Java 8 and building with Java 11, keep it in 1.6.x for Java 8
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6804 from pionCham/bump-iceberg-version.
Closes#6804
0896ac768 [Bowen Liang] keep iceberg 1.6.1 in playground
eba16ae6c [chengpeiming] Specify the iceberg version in java-8 profile
3b160ddd6 [chengpeiming] Bump iceberg version
Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- bump log4j from 2.20.0(Feb 2023) to 2.24.1 (Sep 2024) : https://logging.apache.org/log4j/2.x/release-notes.html#release-notes-2-24-1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6774 from bowenliang123/log4j-2.24.
Closes#6774
98a89ef8a [Bowen Liang] bump log4j from 2.20.0 to 2.24.1
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
Fix the warning of `maven-surefire-plugin` that show `Parameter 'systemProperties' is deprecated. Use systemPropertyVariables instead.` in the build.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6808 from pionCham/use-systemPropertyVariables.
Closes#6808
668bfe252 [chengpeiming] Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
0e74b4725 [chengpeiming] Revert "Replace systemProperties with systemPropertyVariables"
797384a5b [chengpeiming] Replace systemProperties with systemPropertyVariables
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Default Kyuubi version in the chart is not up to date with the latest release version.
## Describe Your Solution 🔧
Upgrade default version to the latest 1.10.0 to be up to date with the release version.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Check the chart deploys Kyuubi 1.10.0
Install the chart:
```shell
helm install kyuubi charts/kyuubi
```
Check Kyuubi version:
```shell
kubectl exec kyuubi-0 -- cat /opt/kyuubi/RELEASE
Kyuubi 1.10.0 (git revision bfcd1f1) built for
Java 1.8.0_432
Scala 2.12
Flink 1.20.0
Spark 3.5.2
Kyuubi Hadoop 3.3.6
Hive 3.1.3
Build flags:
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
Closes#6805 from dnskr/helm-default-kyuubi-version-1.10.0.
Closes#6805
ee36f5788 [dnskr] [K8S][HELM] Update default Kyuubi version to 1.10.0
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- previously, `pnpm install` and `npm install` are forced to inherits HTTP proxy configs from Maven by the frontend maven plugin (https://github.com/eirslett/frontend-maven-plugin?tab=readme-ov-file#proxy-settings)
- adding a parameter `maven.plugin.frontend.inheritsProxyConfigFromMave` to control whether it inherits the proxy configs with false as default value
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6778 from bowenliang123/frontend-build-proxy.
Closes#6778
c8eb2e4f1 [Bowen Liang] inheritsProxyConfigFromMaven
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- `TColumnGenerator` is used for generating column-based data. This PR skips repeated checks on checking nullability of the convert function in generating single column.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6786 from bowenliang123/skip-converfunc-check.
Closes#6786
f76f6864a [Bowen Liang] nit
930dfef3a [Bowen Liang] fix
9712274e3 [Bowen Liang] comment
9db16ef6b [Bowen Liang] nit
977d21533 [Bowen Liang] skip repeated checking on convert function
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Fix engine cannot exit when gracefully stopped
This pull request fixes#6790
## Describe Your Solution 🔧
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6792 from wForget/KYUUBI-6790.
Closes#6790
efe7a2ffa [wforget] [KYUUBI-6790] Fix engine cannot exit when gracefully stopped
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#2112
## Describe Your Solution 🔧
Similar to #2113, the query-timeout-thread should verify the Thrift protocol version. For protocol versions <= HIVE_CLI_SERVICE_PROTOCOL_V8, it should convert TIMEDOUT_STATE to CANCELED.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6787 from lsm1/branch-timer-checker-set-cancel.
Closes#6787
9fbe1ac97 [senmiaoliu] add isHive21OrLower method
0c77c6f6f [senmiaoliu] time checker set cancel state
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Preparing v1.11.0-SNAPSHOT after branch-1.10 cut
```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6769 from bowenliang123/bump-1.11.
Closes#6769
6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6034
## Describe Your Solution 🔧
Currently, use beeline to connect kyuubiServer with HA mode, the strategy only support random, this will lead to a high load on the machine. So i make this pr to support choose strategy.
[description]
First, we need know, beeline connect kyuubiServer dependency on kyuubi-hive-jdbc, it is isolated from the kyuubi cluster, so the code only support random choose serverHost from zk node /${namespace}. Because kyuubi-hive-jdbc is a stateless module, only run once, cannot store var about get serverHost from zk node.
[Solution]
This pr, we could implement a interface named ChooseServerStrategy to choose serverHost. I implement two strategy
1. poll: it will create a zk node named ${namespace}-counter, when a beeline client want connect kyuubiServer, the node will increment 1, use this value to take the remainder from serverHosts, like counter % serverHost.size, so we could get a order serverHost
2. random: random get serverHost from serverHosts
3. User Definied Class: implemented the ChooseServerStrategy, then put the jar to beeline-jars, it can use your strategy to choose serverHost
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Test the Strategy in my test Cluster
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
[Use Case]
1. poll: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true`
2. random: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=random?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true` or `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true`
3. YourStrategy: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=xxx.xxx.xxx.XxxChooseServerStrategy?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true`
[Result: The Cluster have two Server (221,233)]
1. poll:
1.1. zkNode: counterValue
1.2. result:
2. random:
3. YourStrategy(the test case only get the first serverHost):
#### Related Unit Tests
There is no Unit Tests.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6213 from davidyuan1223/ha_zk_support_more_strategy.
Closes#6034
961d3e989 [Bowen Liang] rename ServerStrategyFactory to ServerSelectStrategyFactory
353f94059 [Bowen Liang] repeat
8822ad471 [Bowen Liang] repeat
619339402 [Bowen Liang] nit
e94f9e909 [Bowen Liang] nit
40f427ae5 [Bowen Liang] rename StrategyFactory to StrategyFactoryServerStrategyFactory
7668f99cc [Bowen Liang] test name
e194ea62f [Bowen Liang] remove ZooKeeperHiveClientException from method signature of chooseServer
265965e5d [Bowen Liang] polling
b39c56700 [Bowen Liang] style
1ab79b494 [Bowen Liang] strategyName
8f8ca28f2 [Bowen Liang] nit
228bf1091 [Bowen Liang] rename parameter zooKeeperStrategy to serverSelectStrategy
125c82358 [Bowen Liang] rename ChooseServerStrategy to ServerSelectStrategy
b4aeb3dbd [Bowen Liang] repeat testing on pollingChooseStrategy
465548005 [davidyuan] update
09a84f1f9 [david yuan] remove the distirbuted lock
93f4a2699 [davidyuan] remove reset
7b0c1b811 [davidyuan] fix var not valid and counter getAndIncrement
c95382a23 [davidyuan] fix var not valid and counter getAndIncrement
9ed2cac85 [david yuan] remove test comment
8eddd7682 [davidyuan] Add Strategy Unit Test Case and fix the polling strategy counter begin with 0
73952f878 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy
97b959776 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy
ee5a9ad68 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy
6a0445357 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy
1892f148d [davidyuan] add common method to get session level config
7c0c6058d [yuanfuyuan] fix_4186
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: davidyuan <davidyuan1223@gmail.com>
Co-authored-by: david yuan <davidyuan1223@gmail.com>
Co-authored-by: yuanfuyuan <1406957364@qq.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- as titled
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6775 from bowenliang123/gha-cache-v4.
Closes#6775
de7b9fdf0 [Bowen Liang] actions/dependency-review-action@v4
c9e41923a [Bowen Liang] actions/cache@v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
This PR addresses an issue in the ProcessBuilder class where Java options passed as a single string (e.g., "-Dxxx -Dxxx") do not take effect. The command list must separate these options into individual elements to ensure they are recognized correctly by the Java runtime.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6772 from lsm1/branch-fix-processBuilder.
Closes#6772
fb6d53234 [senmiaoliu] fix process builder java opts
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- skip explicitly set the parameter `check-latest` of Github action `actions/setup-java` to false, which is its default value.
- docs: https://github.com/actions/setup-java?tab=readme-ov-file#check-latest
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6768 from bowenliang123/action-java-latest-false.
Closes#6768
a0ecd4688 [Bowen Liang] update
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- fix the dockerfile warning reported by `GitHub Actions
/ Kyuubi Server On Kubernetes Integration Test` :
- Legacy key/value format with whitespace separator should not be used
LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format
More info: https://docs.docker.com/go/dockerfile/rule/legacy-key-value-format/
- The 'as' keyword should match the case of the 'from' keyword
FromAsCasing: 'as' and 'FROM' keywords' casing do not match
More info: https://docs.docker.com/go/dockerfile/rule/from-as-casing/
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6762 from bowenliang123/docker-warning.
Closes#6762
d9d7b7465 [Bowen Liang] cleanup dockerfile warning
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
`Spark Engine On Kubernetes Integration Test` always fails after #6739
The default java version of Ubuntu 24.04 is OpenJDK 21: https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat-release-notes/39890#openjdk , this is a version higher than that supported by spark 3.5.X.
## Describe Your Solution 🔧
Specify JDK 11 for spark engine on kubernetes it.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
test ga: https://github.com/wForget/kyuubi/actions/runs/11441231714/job/31828818049
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6763 from wForget/minor.
Closes#6763
aee39339f [wforget] Specify JDK 11 for spark engine on kubernetes it
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6754
## Describe Your Solution 🔧
Right now in RuleAuthorization we use an ArrayBuffer to collect access requests, which is very slow because each new PrivilegeObject needs to be compared with all access requests.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Add benchmark
Before
```sh
Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark 181863 189434 NaN -0.0 -181863368958.0 1.0X
````
#### Behavior With This Pull Request 🎉
After
```sh
Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark 1281 1310 33 -0.0 -1280563000.0 1.0X
```
#### Related Unit Tests
Exists UT
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6758 from wankunde/ranger2.
Closes#6754
9d7d1964b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
88b9c049b [wankun] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/spark/sql/RuleAuthorizationBenchmark.scala
20c55fbeb [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
f5a3b6ca5 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
9793249de [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
d86b01f9c [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
b904b491b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
aad08a6bb [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
1374604bc [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
01e15c149 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
805e8a9c0 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
e19817943 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
Lead-authored-by: wankunde <wankunde@163.com>
Co-authored-by: wankun <wankun@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Check the uploaded resource files when creating batch via REST API
- add config `kyuubi.batch.resource.file.max.size` for resource file's max size in bytes
- add config `kyuubi.batch.extra.resource.file.max.size` for each extra resource file's max size in bytes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6756 from bowenliang123/resource-maxsize.
Closes#6756
5c409c425 [Bowen Liang] nit
4b16bcfc4 [Bowen Liang] nit
743920d25 [Bowen Liang] check resource file size max size
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- as titled
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6741 from bowenliang123/gha-upload-artifact-v4.
Closes#6741
83a0ee90e [Bowen Liang] update GHA actions/upload-artifact from v3 to v4
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- The Ubuntu 24.04 image for Actions is now generally available since Sep 2024 : https://github.blog/changelog/2024-09-25-actions-new-images-and-ubuntu-latest-changes/
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6739 from bowenliang123/ubuntu-2404.
Closes#6739
2d120d1f6 [Bowen Liang] bump ci image from ubuntu 22.04 to 24.04
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- inspired by comments from #6664 : https://github.com/apache/kyuubi/pull/6664#issuecomment-2344351119
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6687 from bowenliang123/mvn-args.
Closes#6687
6a9170b60 [Cheng Pan] Update .github/workflows/master.yml
3573131d8 [liangbowen] export MAVEN_ARGS
36ced0532 [liangbowen] update
a93fa28ee [liangbowen] MAVEN_ARGS
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Pass build args to extra spark engine build command
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6757 from wForget/minor2.
Closes#6757
5b373d2fa [wforget] Pass build args to extra spark engine build command
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- update GHA docker/build-push-action from v5 to v6 to support build mmary
https://docs.docker.com/build/ci/github-actions/build-summary/
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6740 from bowenliang123/docker-push-v6.
Closes#6740
674cc15fc [Bowen Liang] update GHA docker/build-push-action from v5 to v6 to support build summary
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
