### Why are the changes needed?
Currently range check missing check UnsetTableProperties command, we need add it to the range check.
#6940
### How was this patch tested?
Use paimon removing table properties to test this command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6944 from davidyuan1223/test_remove_table_properties.
Closes#6940
4f24d7d6a [davidyuan] Merge branch 'master' into test_remove_table_properties
11d3773ed [davidyuan] test unset table properties command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column position command, add the test case
#6950
### How was this patch tested?
Test ranger check with paimon changing column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6955 from davidyuan1223/test_changing_column_position.
Closes#6950
520b5377f [davidyuan] Merge branch 'master' into test_changing_column_position
1eed87346 [davidyuan] test changing column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently, ranger check for paimon missing rename column name command, add the test case
#6942
### How was this patch tested?
Test Paimon Rename column name with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6946 from davidyuan1223/test_rename_column_name.
Closes#6942
8e49eb0ab [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
**Why are the changes needed:**
Currently looking to connect to a HiveServer2 behind an NGINX proxy that is requiring mTLS communication. pyHive seems to lack the capability to establish an mTLS connection in applications such as Airflow directly communicating to the HiveServer2 instance.
The change needed is to be able to pass in the parameters for a proper mTLS ssl context to be established. I believe that creating your own ssl_context object is the quickest and cleanest way to do so, leaving the responsibility of configuring it to further implementations and users. Also cuts down on code length.
**How was this patch tested:**
Corresponding pytest fixtures have been added, using the mock module to see if ssl_context object was properly accessed, or if the default one created in the Connection initialization was properly configured.
Was not able to run pytest fixtures specifically, was lacking JDBC driver, first time contributing to open source, happy to run tests if provided guidance. Passed a clean build and test of the entire kyuubi project in local dev environment.
**Was this patch authored or co-authored using generative AI tooling**
Yes, Generated-by Cursor-AI with Claude Sonnet 3.5 agent
Closes#6935 from alexio215/connection-class-ssl-context-param.
Closes#6908
539b29962 [Cheng Pan] Update python/pyhive/tests/test_hive.py
14c607489 [Alex Wojtowicz] Simplified testing, following pattern of other tests, need proper SSL setup with nginx to test ssl_context fully
b947f2454 [Alex Wojtowicz] Added exception handling since JDBC driver will not run in python tests
11f9002bf [Alex Wojtowicz] Passing in fully configured mock object before creating connection
009c5cf24 [Alex Wojtowicz] Added back doc string documentation
e3280bcd8 [Alex Wojtowicz] Python testing
529de8a12 [Alex Wojtowicz] Added ssl_context object. If no obj is provided, then it continues to use default provided parameters
Lead-authored-by: Alex Wojtowicz <awojtowi@akamai.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Create Partitioned Table for PAIMON, check that has support the command
#6923
### How was this patch tested?
est Authz for paimon with create partitioned table command. Check the permission
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6931 from davidyuan1223/support_create_with_parition_for_paimon.
Closes#6923
61f7560d3 [Cheng Pan] Merge branch 'master' into support_create_with_parition_for_paimon
ffb79376f [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
b0829795a [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
4b160d720 [davidyuan] support create partition table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Add/Change Table properties for PAIMON, check that has support the command
https://github.com/apache/kyuubi/issues/6932
### How was this patch tested?
Test Add/Change properties SQL
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6933 from davidyuan1223/test_alter_tableproperties_for_paimin.
Closes#6932
4d64fbf23 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
c861a778b [davidyuan] support add/change table properties for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Without this PR:
- use and conform Scalafmt 3.7.x
- forcing all the imports from the same parent package, which is even violation the max length of the line
With this PR:
- use and conform Scalafmt 3.9.x
- Scalafmt 3.8.2 changes the binpack style: https://github.com/scalameta/scalafmt/releases/tag/v3.8.2
- change to `binPack.importSelectors=singleLine`, minimizing the impacts to existed code
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6674 from bowenliang123/scalafmt38.
Closes#6674
d9f4f9ddc [Bowen Liang] 3.9.0
438a07d61 [Bowen Liang] reformat
2df50669c [liangbowen] bump scalafmt to 3.8
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### Why are the changes needed?
AUTHZ Test CTAS for Paimon to check it support this command, the related issue is https://github.com/apache/kyuubi/issues/6921
### How was this patch tested?
Test Authz for paimon with create table as command. Check the permission.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6922 from davidyuan1223/support_create_table_as_for_paimon_check.
Closes#6921
7bfd6ad49 [david yuan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
a9ce20cc4 [davidyuan] support create table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: david yuan <davidyuan1223@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Paimon does not seem to support Scala 2.13
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6925 from pan3793/authz-paimon-scala212.
Closes#6925
865a7dd72 [Cheng Pan] fix
971d23273 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
499f10ab0 [Cheng Pan] Only run Paimon authz tests with Scala 2.12
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 4.0 continues to receive breaking changes since 4.0.0-preview2, and the 4.0.0 RC1 is scheduled at 20250215, this PR fixes all compatibility for the latest Spark 4.0.0-SNAPSHOT for Spark SQL engine.
### How was this patch tested?
Pass GHA with `spark-master`
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6920 from pan3793/spark4.
Closes#6920
170430e5e [Cheng Pan] Revert "ci"
c6d889350 [Cheng Pan] fix
86ff7ea2e [Cheng Pan] fix
75d0bf563 [Cheng Pan] ci
9d88c8630 [Cheng Pan] fix spark 4.0 compatibility
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
add Utils.CLIENT_IP_ADDRESS to cache local ipAddress
closes#6918
### How was this patch tested?
minor fix
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6919 from wForget/KYUUBI-6918.
Closes#6918
5b0efe722 [Cheng Pan] Update kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java
b8660c83b [wforget] address comment
835dfb8ef [wforget] address comment
29dd13a2e [wforget] fix style
843b29fe8 [wforget] [KYUUBI #6918] Cache client ipAddress in kyuubi jdbc connection
Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
https://iceberg.apache.org/releases/#180-release
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6914 from pan3793/iceberg-1.8.0.
Closes#6914
f7e9b7e51 [Cheng Pan] fix
f44dee4e6 [Cheng Pan] Use Iceberg 1.7 for Spark 3.3
cd0bdda6a [Cheng Pan] rm staging repo
881a01581 [Cheng Pan] Bump Iceberg 1.8.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
HTTP dialect ignores the database specified in the URL and uses the "default" instead.
### How was this patch tested?
Tested manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6906 from developster/pyhive-update1.
Closes#6905
6e21d7259 [Cheng Pan] Update python/pyhive/sqlalchemy_hive.py
ec7d4629e [Octavian Ciubotaru] [KYUUBI #6905] PyHive HTTP/HTTPS dialect to use the database name from url
Lead-authored-by: Octavian Ciubotaru <ociubotaru@developmentgateway.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Why are the changes needed?
## Issue reference:
https://github.com/apache/kyuubi/issues/6912
## How to reproduce the issue?
The changes in this PR will avoid a wrong result when generating the instance of org.apache.kyuubi.plugin.lineage.Lineage, in the certain case as follows:
step 1: create a temporary view from a file
step 2: insert into a table by selecting from the temporary view in step 1
step 3: generate the lineage when executing the insert statement in step 2
In detail, please see the UT code submission in this patch.
## The issue analysis
Let's see the current code when getting the Lineage object by resolving a LogicalPlan object:
<img width="694" alt="image" src="https://github.com/user-attachments/assets/65256a0d-320d-4271-968f-59eafb74de9f" />
According to the above logic, a None org.apache.kyuubi.plugin.lineage.Lineage object will be generated due to "try-catch" self-protection, in this certain case. This None object will lead to problems in the following 2 scenes:
### Unit Test Environment
In Unit Test, when the code runs here a "None.get" exception will be raised:
<img width="682" alt="image" src="https://github.com/user-attachments/assets/102dc9bd-294f-4b1e-b1c6-01b6fee50fed" />
Here's the runtime exception stack:
```
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineageWithoutExecuting(SparkSQLLineageParserHelperSuite.scala:1485)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$83(SparkSQLLineageParserHelperSuite.scala:1465)
```
### Production Environment
This Lineage object cannot be used in the production environment because it has a None value which lacks some necessary lineage information. The right content of the Lineage instance in the above case should be:
```
inputTables(List())
outputTables(List(spark_catalog.test_db.test_table_from_dir))
columnLineage(List(ColumnLineage(spark_catalog.test_db.test_table_from_dir.a0,Set()), ColumnLineage(spark_catalog.test_db.test_table_from_dir.b0,Set())))
```
a newly added test case(test directory to table) passed after this issue is fixed.
# How to fix the issue?
Add a "Empty judgment" logic. In detail, please see the code submission in this patch.
# How was this patch tested?
1. by adding a new test case in UT code and make sure it passes
2. by submitting a Spark application including the SQL of this case in the production environment, and make sure a right Lineage instance is generated, instead of a None object
# Was this patch authored or co-authored using generative AI tooling?
No
Closes#6911 from xglv1985/fix_spark_lineage_runtime_exception.
Closes#6912
13a71075d [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
4e89b95cd [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
59b350bfb [xglv1985] fix a runtime exception when generate column lineage tuple--more readable code
52bc0288d [xglv1985] fix a runtime exception when generate column lineage tuple--spotless sytle
fea6bbc0d [xglv1985] fix a runtime exception when generate column lineage tuple--remove tab from UT code
901879095 [xglv1985] fix a runtime exception when generate column lineage tuple--unit test
fbb4df879 [xglv1985] fix a runtime exception when generate column lineage tuple
Lead-authored-by: xglv1985 <xglv1985@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
https://github.com/delta-io/delta/releases/tag/v3.3.0
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6916 from pan3793/delta-3.3.0.
Closes#6916
51f6c1cdf [Cheng Pan] Bump Delta 3.3.0 for Spark 3.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
https://hudi.apache.org/releases/release-1.0.1
### How was this patch tested?
Pass GHA
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6917 from pan3793/hudi-1.0.1.
Closes#6917
b25414bd3 [Cheng Pan] Bump Hudi 1.0.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
I observed ClickHouse integration test failure in GHA, after some investigation, the root cause is https://github.com/testcontainers/testcontainers-java/pull/9942
```
/entrypoint.sh: neither CLICKHOUSE_USER nor CLICKHOUSE_PASSWORD is set, disabling network access for user 'default'
```
In short, the recent ClickHouse docker image does not allow the `default` user to connect without a password, unfortunately, `testcontainers-scala-clickhosue` does not expose API to set CLICKHOSUE_USER and CLICKHOUSE_PASSWORD, as a workaround, I pin `clickhouse-server:24.3.15`(the latest version has no such restriction) until a fixed version of Testcontainers available.
This PR also switches the `clickhouse-jdbc`'s classifier from `http` to `shaded`, the reason is, `http` does not ship ApacheHttpClient5, previously, it happened to work because `iceberg-runtime-spark3.5_2.12` packaged un-relocated ApacheHttpClient5 classes, but it gets fixed in Iceberg 1.8.0, then `clickhouse-jdbc:http` stop working.
```
java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpRequest
```
Additionally, this PR bumps `clickhouse-jdbc` from 0.6.0 to 0.6.5.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6915 from pan3793/fix-ch-test.
Closes#6915
996f095e0 [Cheng Pan] Pin clickhouse-server:24.3.15
d633df07c [Cheng Pan] Bump clickhouse-jdbc 0.6.5
214c8a227 [Cheng Pan] Fix ClickHouse integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Added a service definition for spark which in turn enables the creation of a default policy for the spark service.
Default policy will block access until another policy is downloaded from Apache Ranger.
### How was this patch tested?
Tested manually.
Configure Kyuubi Authz plugin. Do not start Apache Ranger, it must not be reachable.
Make sure that policy cache is empty.
Start Kyuubi engine and try to query any tables. The default policy should not allow any access.
Previously the access was not restricted because there wasn't a default policy defined.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6902 from developster/master.
Closes#6901
feb6ebf61 [Octavian Ciubotaru] Default policy for spark
Authored-by: Octavian Ciubotaru <ociubotaru@developmentgateway.org>
Signed-off-by: Kent Yao <yao@apache.org>
Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md">cross-spawn's changelog</a>.</em></p>
<blockquote>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.5...v7.0.6">7.0.6</a> (2024-11-18)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>update cross-spawn version to 7.0.5 in package-lock.json (<a href="f700743918">f700743</a>)</li>
</ul>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.4...v7.0.5">7.0.5</a> (2024-11-07)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>fix escaping bug introduced by backtracking (<a href="640d391fde">640d391</a>)</li>
</ul>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.3...v7.0.4">7.0.4</a> (2024-11-07)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>disable regexp backtracking (<a href="https://redirect.github.com/moxystudio/node-cross-spawn/issues/160">#160</a>) (<a href="5ff3a07d9a">5ff3a07</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="77cd97f3ca"><code>77cd97f</code></a> chore(release): 7.0.6</li>
<li><a href="6717de49ff"><code>6717de4</code></a> chore: upgrade standard-version</li>
<li><a href="f700743918"><code>f700743</code></a> fix: update cross-spawn version to 7.0.5 in package-lock.json</li>
<li><a href="9a7e3b2165"><code>9a7e3b2</code></a> chore: fix build status badge</li>
<li><a href="085268352d"><code>0852683</code></a> chore(release): 7.0.5</li>
<li><a href="640d391fde"><code>640d391</code></a> fix: fix escaping bug introduced by backtracking</li>
<li><a href="bff0c87c8b"><code>bff0c87</code></a> chore: remove codecov</li>
<li><a href="a7c6abc6fe"><code>a7c6abc</code></a> chore: replace travis with github workflows</li>
<li><a href="9b9246e096"><code>9b9246e</code></a> chore(release): 7.0.4</li>
<li><a href="5ff3a07d9a"><code>5ff3a07</code></a> fix: disable regexp backtracking (<a href="https://redirect.github.com/moxystudio/node-cross-spawn/issues/160">#160</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.3...v7.0.6">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
Dependabot will merge this PR once CI passes on it, as requested by yaooqinn.
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6814 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/cross-spawn-7.0.6.
Closes#6814
10dafbc6e [dependabot[bot]] ⬆️ Bump cross-spawn from 7.0.3 to 7.0.6 in /kyuubi-server/web-ui
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Correct spelling make the documentation better.
### How was this patch tested?
No need.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6900 from shuyouZZ/new-branch.
Closes#6900
27220abaf [dongshuyou] [DOCS] Correct spelling errors in 'large_query_results' part
Authored-by: dongshuyou <dongshuyou@idea.edu.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For the `com.codahale.metrics.MetricRegistry::gauge`.
It `getOrAdd` the gauge with name.
```
public <T extends Gauge> T gauge(String name) {
return (Gauge)this.getOrAdd(name, MetricRegistry.MetricBuilder.GAUGES);
}
```
So we have to get all the gauges to check whether the gauge exists.
### How was this patch tested?
UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6891 from turboFei/gauge_exists.
Closes#6891
18be2a521 [Wang, Fei] o(1)
039e7b5eb [Wang, Fei] check existing gauge
32dce6fb1 [Wang, Fei] check gauge exists
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this directory from growing indefinitely.
This PR proposes to let `spark.kubernetes.file.upload.path` support placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}` and introduce a switch `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled` to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
For example, the user can configure the below configurations in `kyuubi-defaults.conf` to enable monthly rolling support for `spark.kubernetes.file.upload.path`
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
Note that: spark would create sub dir `s"spark-upload-${UUID.randomUUID()}"` under the `spark.kubernetes.file.upload.path` for each uploading, the administer still needs to clean up the staging directory periodically.
For example:
```
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-f2b71340-dc1d-4940-89e2-c5fc31614eb4
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-173a8653-4d3e-48c0-b8ab-b7f92ae582d6
hdfs://hadoop-cluster/spark-upload-202501/spark-upload-3b22710f-a4a0-40bb-a3a8-16e481038a63
```
Administer can safely delete the `hdfs://hadoop-cluster/spark-upload-202412` after 20250101
### How was this patch tested?
New UTs are added.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6876 from pan3793/rolling-upload.
Closes#6876
6614bf29c [Cheng Pan] comment
5d5cb3eb3 [Cheng Pan] docs
343adaefb [Cheng Pan] review
3eade8bc4 [Cheng Pan] fix
706989778 [Cheng Pan] docs
38953dc3f [Cheng Pan] Support rolling spark.kubernetes.file.upload.path
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Address comments: https://github.com/apache/kyuubi/discussions/6877#discussioncomment-11743818
> I guess this is a Kyuubi implementation issue, we just read the content from the kyuubi.kubernetes.authenticate.oauthTokenFile and call ConfigBuilder.withOauthToken, I guess this approach does not support token refresh...
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6883 from turboFei/k8s_token_provider.
Closes#6883
69dd28d27 [Wang, Fei] comments
a01040f94 [Wang, Fei] withOauthTokenProvider
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
If the session manager's ThreadPoolExecutor refuses to execute the asyncOperation, then we need to shut down the query-timeout-thread in the catch block. This should also be done in JDBC and the CHAT engine.
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#6873 from lsm1/branch-followup-6843.
Closes#6843
aed9088c8 [senmiaoliu] fix query timeout checker leak in chat engine and jdbc engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
### Why are the changes needed?
see https://github.com/apache/kyuubi/issues/6843
If the session manager's ThreadPoolExecutor refuses to execute asyncOperation, then we need to shut down the query-timeout-thread in the catch
### How was this patch tested?
1 Use jstack to view threads on the long-lived engine side
2 Wait for all SQL statements in the engine to finish executing, and then use stack to check the number of query-timeout-thread threads, which should be empty.
### Was this patch authored or co-authored using generative AI tooling?
NO
Closes#6844 from ASiegeLion/master.
Closes#6843
9107a300e [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
4b3417f21 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
ef1f66bb5 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
9e1a015f6 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
78a9fde09 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
Authored-by: liupeiyue <liupeiyue@yy.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Followup for https://github.com/apache/kyuubi/pull/6866
It would throw exception if both thrift binary SSL and thrift http SSL enabled
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6872 from turboFei/duplicate_gauge.
Closes#6866
ea356766e [Wang, Fei] prevent conflicts
982f175fd [Wang, Fei] conflicts
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
This reverts commit e8cbff32d4.
### Why are the changes needed?
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#6871 from turboFei/revert_zk.
Closes#6871Closes#6865
1c61f9d85 [Wang, Fei] Revert "[KYUUBI #6865] [TEST] Set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing"
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Similar with `kyuubi.frontend.bind.host`, it is better to set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing as well.
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6865 from turboFei/zk_bind_host.
Closes#6865
9044e284b [Wang, Fei] [TEST] Set kyuubi.zookeeper.embedded.client.port.address for testing
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Add metrics for SSL keystore expiration, then we can add alert if the keystore will expire in 1 month.
### How was this patch tested?
Integration testing.
<img width="1721" alt="image" src="https://github.com/user-attachments/assets/f4ef6af6-923b-403c-a80d-06dbb80dbe1c" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6866 from turboFei/keystore_expire.
Closes#6866
77c6db0a7 [Wang, Fei] Add metrics for SSL keystore expiration time #6866
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The original dashboard dashboard template does not work for my use case:
In this PR:
1. I add the `baseFilter` variable, default value is `job=~"kyuubi"`.
2. apply the `baseFilter` for all expressions
3. add `baseLegend` for the legendFormat
4. Fix hardcode typos
5. remove the `sum` from the expressions for JVM
### How was this patch tested?
<img width="1723" alt="image" src="https://github.com/user-attachments/assets/2da3366d-4cd7-41fe-9a6b-1159d0b61734" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6867 from turboFei/dash_board.
Closes#6867
d0d1a3612 [Wang, Fei] comments
9fc3b5c20 [Wang, Fei] use kyuubi_jvm_uptime to get instance
280f97c59 [Wang, Fei] green
f8fc35193 [Wang, Fei] all filter
8d1ac2fdf [Wang, Fei] base legend
9c18bfcb0 [Wang, Fei] save
1a4799013 [Wang, Fei] job=kyuubi
ea7634580 [Wang, Fei] instant datasource and remove cluster
19db2b3e1 [Wang, Fei] add filter
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For histogram and timer metrics, it already has label, so need to support apply the instance label with existing ones.
For example:
```
# HELP kyuubi_backend_service_close_operation Generated from Dropwizard metric import (metric=kyuubi.backend_service.close_operation, type=com.codahale.metrics.Timer)
# TYPE kyuubi_backend_service_close_operation summary
kyuubi_backend_service_close_operation{quantile="0.5",}{instance="hadoopkyuubi-1.hadoopkyuubihl.hadoopmaster-dev.svc.140.tess.io:10019"} 0.032923216000000005
```
### How was this patch tested?
UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6868 from turboFei/instance_label_follow.
Closes#6864
4894784e0 [Wang, Fei] ut
b8f227f32 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For my use case, the instances are not human readable, so I prefer to return the FQDN.
<img width="1483" alt="image" src="https://github.com/user-attachments/assets/92045517-456f-4087-8a36-9e3e4bea2f1d" />
### How was this patch tested?
Integration testing.
```
(base) ➜ dist git:(prometheus_label_2) cat conf/kyuubi-defaults.conf
kyuubi.metrics.prometheus.metrics.instance.enabled=true
kyuubi.zookeeper.embedded.client.port.address=localhost
kyuubi.frontend.bind.host=localhost
```
<img width="1692" alt="image" src="https://github.com/user-attachments/assets/0b60d504-62ec-418d-880b-f8a2f00d5550" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6864 from turboFei/prometheus_label_2.
Closes#6864
d24571ccb [Wang, Fei] match
6a6a5110b [Wang, Fei] comments
c3046d4a1 [Wang, Fei] save
fb2021a31 [Wang, Fei] revert
42395945e [Wang, Fei] compatible
17b7007f5 [Wang, Fei] add instance label
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.
To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.
PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.
### How was this patch tested?
Manually test.
Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
<Appenders>
<Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
<JsonTemplateLayout>
<EventTemplateAdditionalField key="app" value="kyuubi"/>
<EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
<EventTemplateAdditionalField key="host" value="${hostName}"/>
</JsonTemplateLayout>
<Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
<Property name="compression.type" value="gzip"/>
</Kafka>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="kafka"/>
</Root>
</Loggers>
</Configuration>
```
Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6861 from pan3793/structured-logging.
Closes#6861
9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/kyuubi/pull/5852 to Spark 3.3, to enhance MaxScanStrategy to include support for the datasourcev2 in Spark 3.3
### How was this patch tested?
Add some UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6862 from zhaohehuhu/dev-1225.
Closes#6862
c745eda14 [zhaohehuhu] MaxScanStrategy supports DSv2 in Spark 3.3
Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Kyuubi uses log4j2 as the logging framework, while I found that the Hive SQL engine module still polls log4j 1.2 to the classpath unexpectedly, we should exclude it to avoid potential issues.
```
build/mvn dependency:tree -pl :kyuubi-hive-sql-engine_2.12
```
```
...
[INFO] +- org.apache.hive:hive-service:jar:3.1.3:provided
[INFO] | +- org.apache.hive:hive-exec:jar:3.1.3:provided
[INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
[INFO] | | | +- log4j:log4j:jar:1.2.16:provided
...
```
### How was this patch tested?
Checks `build/mvn dependency:tree | grep 'log4j:log4j:jar:1.2'` returns nothing and pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6859 from pan3793/exclude-log4j1.
Closes#6859
287cf78af [Cheng Pan] Exclude log4j12 from hive engine module classpath
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6615
## Describe Your Solution 🔧
Add a config item that controls whether Jetty should send its version in response.
This is an additional patch which enables/disables sending Jetty version for prometheus reporter.
Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false)
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Compiled and tested manually.
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6685 from paul8263/KYUUBI-6615-patch.
Closes#6615
0638a5116 [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable
Authored-by: zhang_yao <xzhangyao@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/kyuubi/pull/5852 to Spark 3.4, to enhance MaxScanStrategy to include support for the datasourcev2 in Spark 3.4
### How was this patch tested?
Add some UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6857 from zhaohehuhu/dev-1224.
Closes#6857
c72c62984 [zhaohehuhu] remove the import
dfbf2bc2d [zhaohehuhu] MaxScanStrategy supports DSv2 in Spark 3.4
Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR adds a basic Grafana Dashboard template, also updates the metrics docs to guide users to use Prometheus and Grafana to monitor the Kyuubi server.
The Grafana Dashboard template is exported from the Grafana OSS v11.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1484" alt="image" src="https://github.com/user-attachments/assets/417b35fa-cd12-4e51-b73f-2955282aa187" />
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5147 from zhaohehuhu/Improvement-0809.
Closes#5834
f6fc2d71e [Cheng Pan] fix style
465f0546a [Cheng Pan] update dashboard
3fa2d237e [hezhao2] add status chart
4b2bd3dbc [hezhao2] add status chart
185f2cccf [hezhao2] make it compatible with kyuubi 1.8
457085be5 [hezhao2] add REAMDE.md to guide users
45e3ba3e5 [hezhao2] add docker file build a grafana image and load dashboards available
dbc22108b [hezhao2] Add Grafana dashboard template
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 3.5.4 is released https://spark.apache.org/releases/spark-release-3-5-4.html
### How was this patch tested?
Pas GHA
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6842 from pan3793/spark-3.5.4.
Closes#6842
0fb7ad8a0 [Cheng Pan] ga
8eacc9c97 [Cheng Pan] Spark 3.5.4 RC2
0721fa401 [Cheng Pan] fix
49e98a201 [Cheng Pan] maven repo
951db0c82 [Cheng Pan] Spark 3.5.4
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
the prometheus can not discover the podMonitor created by kyuubi helm chart.
### How was this patch tested?
1. enable pod monitor:
```
metrics:
# Enable metrics system, used for 'kyuubi.metrics.enabled' property
enabled: true
# A comma-separated list of metrics reporters, used for 'kyuubi.metrics.reporters' property
reporters: PROMETHEUS
# Prometheus port, used for 'kyuubi.metrics.prometheus.port' property
prometheusPort: 10019
# PodMonitor by Prometheus Operator
podMonitor:
# Enable PodMonitor creation
enabled: true
# List of pod endpoints serving metrics to be scraped by Prometheus, see Prometheus Operator docs for more details
#podMetricsEndpoints: []
podMetricsEndpoints:
- path: /metrics
port: prometheus
```
2. Helm install kyuubi .
3. check prometheus web ui.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6845 from eric9204/add-lab-to-pm.
Closes#6845
4d713a06c [Cheng Pan] Update charts/kyuubi/values.yaml
f8fa51f28 [dongsj] resolve conflicts with master
Lead-authored-by: dongsj <90449228+eric9204@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
I am using the podminitor of kyuubi metrics, But I found using the podmonitor's selector.matchLabels can not select correct pods . So I fix it .
I also changed the values.yaml to add an example of podmonitor to improve the usability.
### How was this patch tested?
```
helm install kyuubi .
# kubectl get podmonitor kyuubi -oyaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
annotations:
meta.helm.sh/release-name: kyuubi
meta.helm.sh/release-namespace: default
creationTimestamp: "2024-12-06T07:46:12Z"
generation: 1
labels:
app.kubernetes.io/instance: kyuubi
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kyuubi
app.kubernetes.io/version: 1.8.0
helm.sh/chart: kyuubi-0.1.0
name: kyuubi
namespace: default
resourceVersion: "7583800"
uid: 0dd11e43-d126-434e-988a-7f1914586e2b
spec:
podMetricsEndpoints:
- path: /metrics
port: prometheus
selector:
matchLabels:
app.kubernetes.io/instance: kyuubi
app.kubernetes.io/name: kyuubi
# kubectl get po -l app.kubernetes.io/instance=kyuubi
NAME READY STATUS RESTARTS AGE
kyuubi-0 1/1 Running 0 24m
kyuubi-1 1/1 Running 0 22m
# kubectl get po -l app.kubernetes.io/name=kyuubi
NAME READY STATUS RESTARTS AGE
kyuubi-0 1/1 Running 0 24m
kyuubi-1 1/1 Running 0 23m
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6840 from zhifanggao/change_podmonitor.
Closes#6840
092f13f87 [zhifanggao] change the podmonitor to select pods correctly
Authored-by: zhifanggao <28681649@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
Currently, Kyuubi supports Trino progress in scala-2.12, but got some error in scala-2.13.
* Make Trino progress available in scala-2.13
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6841 from naive-zhang/trino-scala13.
Closes#6726
0693d281a [Cheng Pan] Update externals/kyuubi-trino-engine/pom.xml
c20c49a3e [native-zhang] modify scala code style
6cfe4453e [native-zhang] unify scala grammar
c0cd92e47 [native-zhang] change total stage num back into 3 in TrinoOperationProgressSuite
852398ddc [naive-zhang] Merge branch 'apache:master' into trino-scala13
6a038ec2b [native-zhang] move trino progress monitor available in profile scala-2.13
f434a2142 [native-zhang] move trino progress monitor into scala-2.12 impl and change state num in some test case
8d291f513 [native-zhang] add scala version diff in pom for trino module
Lead-authored-by: native-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Enable gluten CI for kyuubi apache repo
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6846 from wForget/gluten_ci.
Closes#6846
22ab3ada8 [wforget] Enable gluten CI for kyuubi apache repo
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
Adding an example to `servicemonitor` in values.yaml to improve the usability .
### How was this patch tested?
execute command `helm install kyuubi-test .`
check the servicemonitor after deployment
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6839 from zhifanggao/add_default_endpoints.
Closes#6839
59aac4142 [Cheng Pan] Update charts/kyuubi/values.yaml
869f45e79 [zhifanggao] include the key itself
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: zhifanggao <28681649@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long
2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`.
68a6f48da5/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql (L32)
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6829 from turboFei/batch_pending_time.
Closes#6829
ee4f93125 [Wang, Fei] docs
bf8169ad4 [Wang, Fei] comments
f493a2af8 [Wang, Fei] new config
ab7b6db65 [Wang, Fei] ut
168017587 [Wang, Fei] in memory session
510a30b6a [Wang, Fei] batchSearchWindow opt
1e93dd276 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
