### Why are the changes needed?
The current mechanism for handling operation timeouts in Kyuubi creates a new `ScheduledExecutorService` with a dedicated thread for each operation. In scenarios with a large number of concurrent operations, this results in excessive thread creation, which consumes substantial system resources and may adversely affect server performance and stability.
This PR introduces a shared `ScheduledThreadPool` within the Operation Manager to centrally schedule operation timeouts. This approach avoids the overhead of creating an excessive number of threads, thereby reducing the system load. Additionally, both the pool size and thread keep-alive time are configurable via the `OPERATION_TIMEOUT_POOL_SIZE` and `OPERATION_TIMEOUT_POOL_KEEPALIVE_TIME` parameters.
### How was this patch tested?
A new unit test for `newDaemonScheduledThreadPool` was added to `ThreadUtilsSuite.scala`. Furthermore, a dedicated `TimeoutSchedulerSuite` was introduced to verify operation timeout behavior.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7121 from wangzhigang1999/master.
Closes#7121
df7688dbf [wangzhigang] Refactor timeout management configuration and improve documentation
2b03b1e68 [wangzhigang] Remove deprecated `ThreadPoolTimeoutExecutor` class following refactor of operation timeout management.
52a8a516a [wangzhigang] Refactor operation timeout management to use per-OperationManager scheduler
7e46d47f8 [wangzhigang] Refactor timeout management by introducing ThreadPoolTimeoutExecutor
f7f10881a [wangzhigang] Add operation timeout management with ThreadPoolTimeoutExecutor
d8cd6c7d4 [wangzhigang] Update .gitignore to exclude .bloop and .metals directories
Lead-authored-by: wangzhigang <wangzhigang1999@live.cn>
Co-authored-by: wangzhigang <wzg443064@alibaba-inc.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This change addresses an issue where the Flink engine in Kyuubi would perform a Hadoop classpath check even when a `flink-shaded-hadoop` jar is already present in the Flink `lib` directory. In such cases, the check is unnecessary and may cause confusion or warnings in environments where the shaded jar is used instead of a full Hadoop classpath. By skipping the check when a `flink-shaded-hadoop` jar exists, we improve compatibility and reduce unnecessary log output.
### How was this patch tested?
The patch was tested by deploying Kyuubi with a Flink environment that includes a `flink-shaded-hadoop` jar in the `lib` directory and verifying that the classpath check is correctly skipped. Additional tests ensured that the check still occurs when neither the Hadoop classpath nor the shaded jar is present. Unit tests and manual verification steps were performed to confirm the fix.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7113 from cutiechi/fix/flink-classpath-missing-hadoop-check.
Closes#7113
99a4bf834 [cutiechi] fix(flink): fix process builder suite
7b9998760 [cutiechi] fix(flink): remove hadoop cp add
ea33258a3 [cutiechi] fix(flink): update flink hadoop classpath doc
6bb3b1dfa [cutiechi] fix(flink): optimize hadoop class path messages
c548ed6a1 [cutiechi] fix(flink): simplify classpath detection by merging hasHadoopJar conditions
9c16d5436 [cutiechi] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala
0f729dcf9 [cutiechi] fix(flink): skip hadoop classpath check if flink-shaded-hadoop jar exists
Authored-by: cutiechi <superchijinpeng@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This pull request fixes#6924
## Describe Your Solution 🔧
Bump ranger version to 2.6.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.6.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#7124 from namanagraw/ranger_upgrade.
Closes#6924
bade24db8 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/README.md
650f27319 [namaagra] [KYUUBI apache#6924] Upgrade Spark Ranger plugin to 2.6.0
Lead-authored-by: namaagra <namaagra@visa.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The metrics `kyuubi_operation_state_LaunchEngine_*` cannot reflect the state of Semaphore after configuring the maximum engine startup limit through `kyuubi.server.limit.engine.startup`, add some metrics to show the relevant permit state.
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#7072 from LennonChin/engine_startup_metrics.
Closes#7072
d6bf3696a [Lennon Chin] Expose metrics of engine startup permit status
Authored-by: Lennon Chin <i@coderap.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
As clarified in https://github.com/apache/kyuubi/issues/6926, there are some scenarios user want to launch engine on each kyuubi server. SERVER_LOCAL engine share level implement this function by extracting local host address as subdomain, in which case each kyuubi server's engine is unique.
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7013 from taylor12805/share_level_server_local.
Closes#6926
ba201bb72 [taylor.fan] [KYUUBI #6926] update format
42f0a4f7d [taylor.fan] [KYUUBI #6926] move host address to subdomain
e06de79ad [taylor.fan] [KYUUBI #6926] Add SERVER_LOCAL engine share level
Authored-by: taylor.fan <taylor.fan@vipshop.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Add an option to allow construct the batch info from metadata directly instead of redirecting the requests to reduce the RPC latency.
### How was this patch tested?
Minor change and Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7043 from turboFei/support_no_redirect.
Closes#7043
7f7a2fb80 [Wang, Fei] comments
bb0e324a1 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
This PR removes the page https://kyuubi.readthedocs.io/en/v1.10.1/client/python/pyspark.html and merges the most content into https://kyuubi.readthedocs.io/en/v1.10.1/extensions/engines/spark/jdbc-dialect.html, some original content of the latter is also modified.
The current docs are misleading, I got asked several times by users why they follow the [Kyuubi PySpark docs](https://kyuubi.readthedocs.io/en/v1.10.1/client/python/pyspark.html) to access data stored in Hive warehouse is too slow.
Actually, accessing HiveServer2/STS from Spark JDBC data source is discouraged by the Spark community, see [SPARK-47482](https://github.com/apache/spark/pull/45609), even though it's technical feasible.
### How was this patch tested?
It's a docs-only change, review is required.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7036 from pan3793/jdbc-ds-docs.
Closes#7036
c00ce0706 [Cheng Pan] style
f2676bd23 [Cheng Pan] [DOCS] Improve docs for kyuubi-extension-spark-jdbc-dialect
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This ensure the Kyuubi server is promptly informed for any Kubernetes resource changes after startup. It is highly recommend to set it for multiple Kyuubi instances mode.
### How was this patch tested?
Existing GA and Integration testing.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7027 from turboFei/k8s_client_init.
Closes#7027
393b9960a [Wang, Fei] server only
a640278c4 [Wang, Fei] refresh
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
The PR resolves the following warning message:
```
../kyuubi/docs/quick_start/quick_start_with_jupyter.md: WARNING: document isn't included in any toctree
```
It removes the empty page `Getting Started with Jupyter Lap` which is also not presented in the documentation menu.
### How was this patch tested?
Built documentation locally and checked there are no warning message anymore.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7005 from dnskr/remove-empty-getting-started-with-jupyter-lap.
Closes#7005
030fb3598 [dnskr] [DOC] Remove empty page "Getting Started with Jupyter Lap"
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
The PR fixes multiple `Pygments lexer name` issues and resolves the following warnings during the documentation build process:
```
../kyuubi/docs/client/advanced/kerberos.md:37: WARNING: Pygments lexer name 'cmd' is not known
../kyuubi/docs/client/bi_tools/hue.md:26: WARNING: Lexing literal_block "Welcome to\n __ __ __\n /\\ \\/\\ \\ /\\ \\ __\n \\ \\ \\/'/' __ __ __ __ __ __\\ \\ \\____/\\_\\\n \\ \\ , < /\\ \\/\\ \\/\\ \\/\\ \\/\\ \\/\\ \\\\ \\ '__`\\/\\ \\\n \\ \\ \\\\`\\\\ \\ \\_\\ \\ \\ \\_\\ \\ \\ \\_\\ \\\\ \\ \\L\\ \\ \\ \\\n \\ \\_\\ \\_\\/`____ \\ \\____/\\ \\____/ \\ \\_,__/\\ \\_\\\n \\/_/\\/_/`/___/> \\/___/ \\/___/ \\/___/ \\/_/\n /\\___/\n \\/__/" as "bash" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/client/jdbc/hive_jdbc.md:27: WARNING: Pygments lexer name 'gradle' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:111: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:134: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:143: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:163: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/connector/spark/delta_lake_with_azure_blob.rst:191: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/deployment/hive_metastore.md:38: WARNING: Pygments lexer name 'shell script' is not known
../kyuubi/docs/deployment/hive_metastore.md:207: WARNING: Lexing literal_block "Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)\n\tat org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:127)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1274)\n\t... 93 more" as "java" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/extensions/server/authentication.rst:75: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/extensions/server/events.rst:76: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/monitor/logging.md:38: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:86: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:222: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/security/kerberos.rst:104: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/security/ldap.md:24: WARNING: Pygments lexer name 'properties example' is not known
../kyuubi/docs/security/ldap.md:40: WARNING: Pygments lexer name 'properties example' is not known
```
Supported languages: [Pygments lexers](https://pygments.org/docs/lexers) and [highlightjs](https://github.com/highlightjs/highlight.js/blob/main/SUPPORTED_LANGUAGES.md).
### How was this patch tested?
Built documentation locally and checked there are related warnings.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6986 from dnskr/fix-unknown-Pygments-lexer-name.
Closes#6986
f5b62f52d [dnskr] [DOC] Fix multiple Pygments lexer name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes few `Unknown target name: "XYZ". [docutils]` issues and resolves the following errors messages:
```
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "github repository". [docutils]
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "read the docs". [docutils]
../kyuubi/docs/contributing/doc/style.rst:66: ERROR: Unknown target name: "directive rubric". [docutils]
```
### How was this patch tested?
Built documentation locally, checked there are no related error messages and doc pages are correct.
##### Page `contributing/doc/get_started.html`
Before changes
<img width="1114" alt="image" src="https://github.com/user-attachments/assets/f1a19c51-3c4c-4268-bf83-7ca0c60315b1" />
After changes
<img width="1113" alt="image" src="https://github.com/user-attachments/assets/437edef1-0fd9-43bf-bd3f-bda43035a2c9" />
##### Page `contributing/doc/style.html`
Before changes
<img width="1128" alt="image" src="https://github.com/user-attachments/assets/39666841-1155-439f-9045-06a9d78624c3" />
After changes
<img width="1117" alt="image" src="https://github.com/user-attachments/assets/2e1f8663-5c1e-4a3c-887e-5f65d01b4cf3" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6987 from dnskr/fix-doc-unknown-target-name.
Closes#6987
391958b4d [dnskr] [DOC] Fix Unknown target name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The feature `spark.sql.watchdog.forcedMaxOutputRows` is a little bit hacky, it's actually a manually implemented "limit pushdown", we already have a simple and more reliable way to achieve that by using `kyuubi.operation.result.max.rows`.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6983 from pan3793/rm-forcedMaxOutputRows.
Closes#6983
5e0707955 [Cheng Pan] Remove support for spark.sql.watchdog.forcedMaxOutputRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes [nested lists formatting](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks) and resolves the following warnings:
```shell
../kyuubi/docs/contributing/doc/style.rst:65: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:67: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:68: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:73: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:106: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:107: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
```
### How was this patch tested?
Built documentation locally and checked nested lists are fixed.
Before changes:
<img width="914" alt="image" src="https://github.com/user-attachments/assets/3ec7079a-e494-4614-9af0-d6e217bcad60" />
After changes:
<img width="1020" alt="image" src="https://github.com/user-attachments/assets/2d3b3231-094d-49bd-b3d7-c6149e13c939" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6981 from dnskr/doc-fix-nested-lists.
Closes#6981
4b425f279 [dnskr] [DOC] Fix nested lists
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
fix typos in docs
### How was this patch tested?
NO
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6964 from wenxin-cn/fix-typos-in-docs.
Closes#6964
5a50a927a [Kent Yao] style
be899c21f [10172] fix typos in docs
Lead-authored-by: wenxin-cn <wen.xin@datasw.com>
Co-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: 10172 <wen.xin@datasw.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The change fixes minor issue that resolves the following error:
```shell
../kyuubi/docs/contributing/doc/get_started.rst:78: ERROR: Content block expected for the "note" directive; none found. [docutils]
```
### How was this patch tested?
Built documentation locally and checked there are no difference and error message.
Before changes:
<img width="1214" alt="image" src="https://github.com/user-attachments/assets/f53398d8-b04a-4367-8040-3e6573cc54f2" />
After changes:
<img width="1197" alt="image" src="https://github.com/user-attachments/assets/3b08e1ee-11c2-4386-b178-35e33d6a56dc" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6977 from dnskr/remove-empty-note-block.
Closes#6977
942a2687e [dnskr] [DOC] Remove empty note block
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR fixes `nonexisting document` issues:
```shell
./kyuubi/docs/client/advanced/features/index.rst:19: WARNING: toctree contains reference to nonexisting document 'client/advanced/features/engine_resources' [toc.not_readable]
./kyuubi/docs/client/odbc/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/odbc/todo' [toc.not_readable]
./kyuubi/docs/client/thrift/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/thrift/hive_beeline' [toc.not_readable]
./kyuubi/docs/index.rst:189: WARNING: toctree contains reference to nonexisting document 'sql/index' [toc.not_readable]
./kyuubi/docs/quick_start/index.rst:23: WARNING: toctree contains reference to nonexisting document 'quick_start/quick_start_with_beeline' [toc.not_readable]
```
### How was this patch tested?
Checked that there are no `nonexisting document` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6970 from dnskr/doc-fix-nonexisting-document.
Closes#6970
a7c2b3617 [dnskr] [DOC] Fix "nonexisting document" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR resolves multiple `"Title underline too short"` warnings to reduce noise during documentation building, for instance:
```shell
./kyuubi/docs/client/jdbc/mysql_jdbc.rst:18: WARNING: Title underline too short.
`MySQL Connectors`_
================ [docutils]
./kyuubi/docs/connector/hive/paimon.rst:17: WARNING: Title underline too short.
`Apache Paimon (Incubating)`_
========== [docutils]
./kyuubi/docs/connector/hive/paimon.rst:31: WARNING: Title underline too short.
Apache Paimon (Incubating) Integration
------------------- [docutils]
```
### How was this patch tested?
Checked that there are no `"Title underline too short"` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6969 from dnskr/doc-fix-title-underline-too-short.
Closes#6969
2007a2440 [dnskr] [DOC] Fix "Title underline too short" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
https://hudi.apache.org/releases/release-1.0.1
### How was this patch tested?
Pass GHA
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6917 from pan3793/hudi-1.0.1.
Closes#6917
b25414bd3 [Cheng Pan] Bump Hudi 1.0.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Correct spelling make the documentation better.
### How was this patch tested?
No need.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6900 from shuyouZZ/new-branch.
Closes#6900
27220abaf [dongshuyou] [DOCS] Correct spelling errors in 'large_query_results' part
Authored-by: dongshuyou <dongshuyou@idea.edu.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this directory from growing indefinitely.
This PR proposes to let `spark.kubernetes.file.upload.path` support placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}` and introduce a switch `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled` to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
For example, the user can configure the below configurations in `kyuubi-defaults.conf` to enable monthly rolling support for `spark.kubernetes.file.upload.path`
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
Note that: spark would create sub dir `s"spark-upload-${UUID.randomUUID()}"` under the `spark.kubernetes.file.upload.path` for each uploading, the administer still needs to clean up the staging directory periodically.
For example:
```
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-f2b71340-dc1d-4940-89e2-c5fc31614eb4
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-173a8653-4d3e-48c0-b8ab-b7f92ae582d6
hdfs://hadoop-cluster/spark-upload-202501/spark-upload-3b22710f-a4a0-40bb-a3a8-16e481038a63
```
Administer can safely delete the `hdfs://hadoop-cluster/spark-upload-202412` after 20250101
### How was this patch tested?
New UTs are added.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6876 from pan3793/rolling-upload.
Closes#6876
6614bf29c [Cheng Pan] comment
5d5cb3eb3 [Cheng Pan] docs
343adaefb [Cheng Pan] review
3eade8bc4 [Cheng Pan] fix
706989778 [Cheng Pan] docs
38953dc3f [Cheng Pan] Support rolling spark.kubernetes.file.upload.path
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For my use case, the instances are not human readable, so I prefer to return the FQDN.
<img width="1483" alt="image" src="https://github.com/user-attachments/assets/92045517-456f-4087-8a36-9e3e4bea2f1d" />
### How was this patch tested?
Integration testing.
```
(base) ➜ dist git:(prometheus_label_2) cat conf/kyuubi-defaults.conf
kyuubi.metrics.prometheus.metrics.instance.enabled=true
kyuubi.zookeeper.embedded.client.port.address=localhost
kyuubi.frontend.bind.host=localhost
```
<img width="1692" alt="image" src="https://github.com/user-attachments/assets/0b60d504-62ec-418d-880b-f8a2f00d5550" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6864 from turboFei/prometheus_label_2.
Closes#6864
d24571ccb [Wang, Fei] match
6a6a5110b [Wang, Fei] comments
c3046d4a1 [Wang, Fei] save
fb2021a31 [Wang, Fei] revert
42395945e [Wang, Fei] compatible
17b7007f5 [Wang, Fei] add instance label
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.
To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.
PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.
### How was this patch tested?
Manually test.
Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
<Appenders>
<Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
<JsonTemplateLayout>
<EventTemplateAdditionalField key="app" value="kyuubi"/>
<EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
<EventTemplateAdditionalField key="host" value="${hostName}"/>
</JsonTemplateLayout>
<Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
<Property name="compression.type" value="gzip"/>
</Kafka>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="kafka"/>
</Root>
</Loggers>
</Configuration>
```
Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6861 from pan3793/structured-logging.
Closes#6861
9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR adds a basic Grafana Dashboard template, also updates the metrics docs to guide users to use Prometheus and Grafana to monitor the Kyuubi server.
The Grafana Dashboard template is exported from the Grafana OSS v11.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1484" alt="image" src="https://github.com/user-attachments/assets/417b35fa-cd12-4e51-b73f-2955282aa187" />
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5147 from zhaohehuhu/Improvement-0809.
Closes#5834
f6fc2d71e [Cheng Pan] fix style
465f0546a [Cheng Pan] update dashboard
3fa2d237e [hezhao2] add status chart
4b2bd3dbc [hezhao2] add status chart
185f2cccf [hezhao2] make it compatible with kyuubi 1.8
457085be5 [hezhao2] add REAMDE.md to guide users
45e3ba3e5 [hezhao2] add docker file build a grafana image and load dashboards available
dbc22108b [hezhao2] Add Grafana dashboard template
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 3.5.4 is released https://spark.apache.org/releases/spark-release-3-5-4.html
### How was this patch tested?
Pas GHA
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6842 from pan3793/spark-3.5.4.
Closes#6842
0fb7ad8a0 [Cheng Pan] ga
8eacc9c97 [Cheng Pan] Spark 3.5.4 RC2
0721fa401 [Cheng Pan] fix
49e98a201 [Cheng Pan] maven repo
951db0c82 [Cheng Pan] Spark 3.5.4
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long
2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`.
68a6f48da5/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql (L32)
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6829 from turboFei/batch_pending_time.
Closes#6829
ee4f93125 [Wang, Fei] docs
bf8169ad4 [Wang, Fei] comments
f493a2af8 [Wang, Fei] new config
ab7b6db65 [Wang, Fei] ut
168017587 [Wang, Fei] in memory session
510a30b6a [Wang, Fei] batchSearchWindow opt
1e93dd276 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
Currently, Kyuubi supports JDBC engines with limited dialects, and I extend the dialects to support Oracle.
* Introduce Oracle support in JDBC Engine
* Adding dialects and tests for Oracle
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Add tests of `OperationWithOracleEngineSuite`, `OracleOperationSuite`, `OracleSessionSuite` and `OracleStatementSuite`.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6815 from naive-zhang/jdbc-oracle.
Closes#6815
0ffad5b6b [native-zhang] add some brief comments on the caller side for the implementation of Oracle JDBC engine
6f469a135 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ae70710e6 [Cheng Pan] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
171d06b9e [native-zhang] use another implementation of transform decimal into int, in engine instead of KyuubiBaseResultSet
7cb74d28e [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ccd7cae8b [naive-zhang] remove redundant override methods in OracleSQLDialect.scala
a7da4a646 [naive-zhang] remove redundant impl of getTableTypesOperation in OracleSQLDialect.scala
70b49fcba [naive-zhang] Use the single line string if SQL fits in one line, otherwise write it in a pretty style
e58348460 [naive-zhang] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
b33e97a08 [naive-zhang] remove redundant testcontainers-scala-oracle-xe dependency in pom.xml
4c967b98e [naive-zhang] use gvenzl/oracle-free:23.5-slim with docker-compose for test case
0215e6d49 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
d688b4706 [naive-zhang] change oracle image into gvenzl/oracle-free:23.5-slim
abf983727 [naive-zhang] fix code style checking error in KyuubiConf.scala
d1e82edb1 [naive-zhang] fix code style checking error in settings.md
aa2e2e9ba [naive-zhang] adjust wired space in OracleSQLDialect
b43cea421 [naive-zhang] add oracle configuration for kyuubi.engine.jdbc.connection.provider
397c1cfec [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
2f1b5ed0b [naive-zhang] add jdbc support for Oracle
Lead-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: native-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.7.0 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.7.0
## Describe Your Solution 🔧
- Bump Apache Iceberg to 1.7.0
- As Apache Iceberg 1.7.0 drops support for Java 8 and building with Java 11, keep it in 1.6.x for Java 8
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6804 from pionCham/bump-iceberg-version.
Closes#6804
0896ac768 [Bowen Liang] keep iceberg 1.6.1 in playground
eba16ae6c [chengpeiming] Specify the iceberg version in java-8 profile
3b160ddd6 [chengpeiming] Bump iceberg version
Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Check the uploaded resource files when creating batch via REST API
- add config `kyuubi.batch.resource.file.max.size` for resource file's max size in bytes
- add config `kyuubi.batch.extra.resource.file.max.size` for each extra resource file's max size in bytes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6756 from bowenliang123/resource-maxsize.
Closes#6756
5c409c425 [Bowen Liang] nit
4b16bcfc4 [Bowen Liang] nit
743920d25 [Bowen Liang] check resource file size max size
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- build python dependencies for docs building to latest versions
- no display or behaviour changes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6752 from bowenliang123/doc-req.
Closes#6752
ffd8782bd [Bowen Liang] update
c328c7584 [Bowen Liang] bump doc build requirements
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- as titled
- add Spark 3.4 and 3.5 to the supported Spark list
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6728 from bowenliang123/doc-authz-build-am.
Closes#6728
f8254bc5c [Bowen Liang] doc
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- add authentication example in REST API docs
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6734 from bowenliang123/rest-doc-auth.
Closes#6734
f9ac9446d [Cheng Pan] Update docs/client/rest/rest_api.md
528e55e79 [Bowen Liang] update doc
371af8806 [Bowen Liang] update doc
e64a08245 [Bowen Liang] update doc
341e7e010 [Bowen Liang] update doc
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6704
## Describe Your Solution 🔧
if periodic gc is set to 0, there is no need to perform an explicit gc.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6725 from taylor12805/master.
Closes#6704
a52ddda62 [Bowen Liang] update doc
b84a32f35 [Bowen Liang] make periodic gc thead pool lazy
2d4bd7c05 [Bowen Liang] update doc in spark style
3e04604b0 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
bf20b134b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
c2b7c3078 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
6182075fc [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
52b1c078b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
ccf19cf24 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
affd67c88 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0
d4ee164d1 [taylor.fan] disable periodic gc if set interval to 0
Lead-authored-by: taylor.fan <taylor.fan@vipshop.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix a couple of typos
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6719 from pionCham/fix-typos.
Closes#6719
71409a875 [chengpeiming] fix violations in jvm-quake.md
de8f0d7b8 [chengpeiming] fix some typos
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6666
## Describe Your Solution 🔧
Bump ranger version to 2.5.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.5.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6692 from Madhukar525722/ranger_upgrade.
Closes#6666
88e1e12c5 [madlnu] [KYUUBI #6666] Upgrade spark ranger plugin to 2.5.0
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
To close#6686
The pod already in failed state, and the driver container is in waiting state.
We shall mark the application terminated and ignore the container state.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6690 from turboFei/pod_state.
Closes#6686
0d4c8a255 [Wang, Fei] comments
d60b901c1 [Wang, Fei] check pod terminated
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes typos in the logging.md documentation file.
## Describe Your Solution 🔧
Fixed typos while reading through the page.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Not needed.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6673 from MLNW/patch-1.
Closes#6673
02ea73113 [Bowen Liang] Update docs/monitor/logging.md
659580ed4 [Lucas Resch] Fix typos in logging.md
Lead-authored-by: Lucas Resch <lucas.resch@gmx.de>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
fix the typo in ENGINE SHARE LEVEL docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6671 from pionCham/fix-typos.
Closes#6671
7dfbd4036 [chengpeiming] Fixed typo in ENGINE SHARE LEVEL docs
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- adding `maximumSize` to support size based eviction for server-side temp files cleanup in `TempFileService`
- size-based eviction is disabled by default , with `maximumSize` set to optional by default
- time-based eviction time is now extended from 14 days to 30 days by default
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6645 from bowenliang123/temp-file-size-evict.
Closes#6645
e1f166b6a [liangbowen] docs
0b2d5aa6e [liangbowen] increase default SERVER_TEMP_FILE_EXPIRE_TIME to 30 days
ee87da56a [liangbowen] make config optional
0607efcd7 [Bowen Liang] import
9cc777660 [liangbowen] update
f9e4de00e [Bowen Liang] docs
55bf238d3 [liangbowen] size
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.6.1 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1
## Describe Your Solution 🔧
In the project POM file, I have updated the Apache Iceberg version from 1.6.0 to 1.6.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6659 from pionCham/upgrate-iceberg-version.
Closes#6659
923019440 [chengpeiming] Update the docs for Spark connector
433981e66 [chengpeiming] Supplement other configurations
1617e36fe [chengpeiming] Upgrate iceberg.version in pom.xml
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
- fix the typo in REST API docs
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6658 from pionCham/fix-typos.
Closes#6658
e8937f1e0 [chengpeiming] Fixed typos in rest_api.md
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5402
## Describe Your Solution 🔧
When facing out-of-control memory management in Spark engine, we typically use JVMkill as a remedy by killing the process and generating a heap dump for post-analysis. However, even with jvmkill protection, we may still encounter issues caused by JVM running out of memory, such as repeated execution of Full GC without performing any useful work during the pause time. Since the JVM does not exhaust 100% of resources, JVMkill will not be triggered.
So introducing JVMQuake provides more granular monitoring of GC behavior, enabling early detection of memory management issues and facilitating fast failure.
You can use the following configuration to enable jvmQuake plugins：
```
spark.plugins=org.apache.spark.kyuubi.jvm.quake.KyuubiJVMQuakePlugin
```
| configuration | default | comment |
| ---- | ---- | ---- |
| spark.driver.jvmQuake.enabled | false | when true, enable driver jvmQuake |
| spark.executor.jvmQuake.enabled | false | when true, enable executor jvmQuake |
| spark.driver.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.executor.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.jvmQuake.dumpThreshold | 100 | The number of seconds to dump memory |
| spark.jvmQuake.killThreshold | 200 | The number of seconds to kill process |
| spark.jvmQuake.exitCode | 502 | The exit code of kill process |
| spark.jvmQuake.heapDumpPath | /tmp/kyuubi_jvm_quake/apps | The path of heap dump |
| spark.jvmQuake.checkInterval | 3 | The number of seconds to check jvmQuake |
| spark.jvmQuake.runTimeWeight | 1.0 | The weight of rum time |
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6572 from yoock/features/kyuubi-jvm-quake.
Closes#5402
84361ce8f [王龙] add jvm quake
Authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Before we only support to list the batches in `ASC` ORDER. It is not user friendly.
## Describe Your Solution 🔧
Support the list the batches in `DESC` order.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6652 from turboFei/latest_batch.
Closes#6652
b3d80f5bd [Wang, Fei] ut
dce0b226d [Wang, Fei] doc
d815ec39c [Wang, Fei] ut
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
-
## Describe Your Solution 🔧
Fix the memory leak on server caused by `Files.deleteOnExit`.
For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook` by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue.
This PR fix this issue by:
1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list
2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup.
3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6587 from bowenliang123/file-expiration.
Closes#6587
e23b72e08 [liangbowen] change to P14D
acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime
6c7ddd527 [liangbowen] import
ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet
fbf73ccb4 [liangbowen] update
34d3fc71c [liangbowen] add guava to common module's dep
49c10e5ef [Bowen Liang] file expiration
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
Fix#6594.
This PR ports HIVE-26633(https://github.com/apache/hive/pull/3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6631 from pan3793/thrift-max-size.
Closes#6594
e4841c88e [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#3897
## Describe Your Solution 🔧
enrich the description for GROUP Share Level.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6628 from futureltl/master.
Closes#6628
ba18bfce4 [futureltl] Improve docs for GROUP Share Level
3b19521e1 [futureltl] Improve docs for GROUP Share Level
da7d9b61e [Cheng Pan] Update docs/deployment/engine_share_level.md
674066a08 [Cheng Pan] Update docs/deployment/engine_share_level.md
c3a373370 [Cheng Pan] Update docs/deployment/engine_share_level.md
7389cedd2 [futureltl] Improve docs for GROUP Share Level
Lead-authored-by: futureltl <futureltl@163.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6618
## Describe Your Solution 🔧
It is a subtask of #6590
This PR is to support http bearer token authentication for REST protocol. In addition to BasicAuthenticationHandler, BearerAuthenticationHandler will be added to handle http bear token authentication. They will both support CUSTOM AuthType. In order to distinguish them, two new configurations are added: kyuubi.authentication.custom.basic.class and kyuubi.authentication.custom.bearer.class. For http bear token custom authentication, users could implement the new 'org.apache.kyuubi.service.authentication.TokenAuthenticationProvider', and specify it in the configuration.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6608 from George314159/authentication.
Closes#6618
d07a30f83 [Wang, Fei] fix UT
6499c9986 [George314159] Update Test Case
da519a9c6 [George314159] Update based on comments
f47160148 [Wang, Fei] Refine UT
544422399 [George314159] Add test suite for custom authentication
f2bbfbf7e [Wang, Fei] comments & refine
a733c0e8f [George314159] Remove unused val
6f669d46c [George314159] Fix
650b88d4e [George314159] Update based on comments
5bc2bac58 [George314159] Update based on comments
1893889db [George314159] Update based on Comments
ddee882e9 [George314159] Fix Style
379a563fa [George314159] Support http bearer token authentication
Lead-authored-by: George314159 <hua16732@gmail.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6615
## Describe Your Solution 🔧
Add a config item that controls whether Jetty should send its version in response.
Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Compiled and tested manually.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6616 from paul8263/KYUUBI-6615.
Closes#6615
c1567fdfa [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable
Authored-by: zhang_yao <xzhangyao@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
