### Why are the changes needed?
Support ranger check with paimon Update & Delete & MergeInti Table Command
### How was this patch tested?
Test Paimon Update & Delete Table Command with ranger check
#6973
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6974 from davidyuan1223/update_table.
Closes#6973
3bd607300 [davidyuan] update
1d68494ce [davidyuan] Test MergeInto
a27ea633f [davidyuan] Test MergeInto
56638f47c [davidyuan] Merge branch 'master' into update_table
1c3464df5 [davidyuan] Test Table Update & Delete
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Simple refactoring to clean up the code for the Spark 3.5 extension, e.g., remove unnecessary `*Base` `*Helper` abstraction layers, remove code for legacy Spark versions.
Note: I don't touch `ForcedMaxOutputRows*` because I'm going to remove it in the next PR.
Preparation for Spark 4.0 support.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6975 from pan3793/spark-ext-35-cleanup.
Closes#6975
b5a94a680 [Cheng Pan] nit
c729e268c [Cheng Pan] fix
1087ac709 [Cheng Pan] Clean up code for Spark 3.5 extension
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The change fixes minor issue that resolves the following error:
```shell
../kyuubi/docs/contributing/doc/get_started.rst:78: ERROR: Content block expected for the "note" directive; none found. [docutils]
```
### How was this patch tested?
Built documentation locally and checked there are no difference and error message.
Before changes:
<img width="1214" alt="image" src="https://github.com/user-attachments/assets/f53398d8-b04a-4367-8040-3e6573cc54f2" />
After changes:
<img width="1197" alt="image" src="https://github.com/user-attachments/assets/3b08e1ee-11c2-4386-b178-35e33d6a56dc" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6977 from dnskr/remove-empty-note-block.
Closes#6977
942a2687e [dnskr] [DOC] Remove empty note block
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check with paimon missing check drop columns command
#6947
### How was this patch tested?
Test ranger check with drop columns commad
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6952 from davidyuan1223/test_dropping_columns.
Closes#6947
27e7b3836 [davidyuan] Merge remote-tracking branch 'origin/master' into test_dropping_columns
83815ef3b [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently, ranger check missing paimon Table Query command, add the test cases
#6971
### How was this patch tested?
Test Paimon Table Query & Batch Time Travel
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6972 from davidyuan1223/test_query.
Closes#6972
7aa821215 [davidyuan] Test Table Query
ae1b75e85 [davidyuan] Test Table Query
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR fixes `nonexisting document` issues:
```shell
./kyuubi/docs/client/advanced/features/index.rst:19: WARNING: toctree contains reference to nonexisting document 'client/advanced/features/engine_resources' [toc.not_readable]
./kyuubi/docs/client/odbc/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/odbc/todo' [toc.not_readable]
./kyuubi/docs/client/thrift/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/thrift/hive_beeline' [toc.not_readable]
./kyuubi/docs/index.rst:189: WARNING: toctree contains reference to nonexisting document 'sql/index' [toc.not_readable]
./kyuubi/docs/quick_start/index.rst:23: WARNING: toctree contains reference to nonexisting document 'quick_start/quick_start_with_beeline' [toc.not_readable]
```
### How was this patch tested?
Checked that there are no `nonexisting document` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6970 from dnskr/doc-fix-nonexisting-document.
Closes#6970
a7c2b3617 [dnskr] [DOC] Fix "nonexisting document" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR resolves multiple `"Title underline too short"` warnings to reduce noise during documentation building, for instance:
```shell
./kyuubi/docs/client/jdbc/mysql_jdbc.rst:18: WARNING: Title underline too short.
`MySQL Connectors`_
================ [docutils]
./kyuubi/docs/connector/hive/paimon.rst:17: WARNING: Title underline too short.
`Apache Paimon (Incubating)`_
========== [docutils]
./kyuubi/docs/connector/hive/paimon.rst:31: WARNING: Title underline too short.
Apache Paimon (Incubating) Integration
------------------- [docutils]
```
### How was this patch tested?
Checked that there are no `"Title underline too short"` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6969 from dnskr/doc-fix-title-underline-too-short.
Closes#6969
2007a2440 [dnskr] [DOC] Fix "Title underline too short" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Scala 2.13.16 release notes: https://github.com/scala/scala/releases/tag/v2.13.16
Also, bump the silencer plugin to 1.7.19 to support Scala 2.13.16
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6965 from pan3793/scala-2.13.6.
Closes#6965
141e75864 [Cheng Pan] Bump silencer 1.7.19
9f09127e7 [Cheng Pan] Bump Scala 2.13.16
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Range check Test case missing paimon changing column comment, add the test case
#6948
### How was this patch tested?
Test Ranger check with paimon changing column comment command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6953 from davidyuan1223/test_changing_column_comment.
Closes#6948
983720e78 [davidyuan] test changing column comment
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently , ranger check missing paimon insert table command, add test cases
#6958
### How was this patch tested?
1. Test INSERT INTO:
1.1 table1OnlyUserForNs could select table1, try to insert table1
1.2 someone has no any permission, try to insert table1
2. Test INSERT OVERWRITE:
2.1 table1OnlyUserForNs could select table1, try to insert table2
2.2 someone has no any permiession, try select table1 then insert table2
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6959 from davidyuan1223/test_insert.
Closes#6958
d1f41ba81 [davidyuan] Merge branch 'master' into test_insert
b56e701d4 [davidyuan] Test Insert Table
8306210ee [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Bump the log4j version to fix below issue:
```
2025-03-04 22:27:58.291 WARN [main-SendThread(xxxx:2181)] org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
: java.lang.ExceptionInInitializerError
at org.apache.log4j.Logger.getLogger(Logger.java:35)
at org.apache.kyuubi.shaded.zookeeper.Login.<init>(Login.java:44)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:228)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:990)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1042)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1657)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1216)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:379)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:344)
at org.apache.log4j.legacy.core.CategoryUtil.setLevel(CategoryUtil.java:131)
at org.apache.log4j.Category.setLevel(Category.java:643)
at org.apache.log4j.Category.setLevel(Category.java:638)
at org.apache.log4j.spi.RootLogger.setLevel(RootLogger.java:60)
at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:70)
... 6 more
```
It is fixed in https://github.com/apache/logging-log4j2/releases/tag/rel%2F2.24.3https://github.com/apache/logging-log4j2/issues/3234
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6960 from turboFei/log4j_version.
Closes#6960
48b076c3e [Wang, Fei] Upgrade log4j version to 2.24.3
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Test Authz Support paimon rename table name command privilege check
#6936
### How was this patch tested?
Test Authz Support paimon rename table name command privilege check
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6937 from davidyuan1223/check_authz_paimon_rename_table.
Closes#6936
797d1c489 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
bc3c823a3 [davidyuan] Merge remote-tracking branch 'origin/master' into check_authz_paimon_rename_table
6205670d2 [davidyuan] add renameTable to command_spec.json
e4b241ef5 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
5fec3bcb7 [davidyuan] test paimon rename table name command
30d09418c [davidyuan] test paimon rename table name command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon adding column position command, add the test case
#6949
### How was this patch tested?
Test ranger check with paimon adding column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6954 from davidyuan1223/test_adding_column_position.
Closes#6949
262ecaaca [davidyuan] Merge remote-tracking branch 'origin/master' into test_adding_column_position
154765fc3 [davidyuan] Merge branch 'master' into test_adding_column_position
4ebf985a9 [davidyuan] test adding column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently range check missing check UnsetTableProperties command, we need add it to the range check.
#6940
### How was this patch tested?
Use paimon removing table properties to test this command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6944 from davidyuan1223/test_remove_table_properties.
Closes#6940
4f24d7d6a [davidyuan] Merge branch 'master' into test_remove_table_properties
11d3773ed [davidyuan] test unset table properties command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column position command, add the test case
#6950
### How was this patch tested?
Test ranger check with paimon changing column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6955 from davidyuan1223/test_changing_column_position.
Closes#6950
520b5377f [davidyuan] Merge branch 'master' into test_changing_column_position
1eed87346 [davidyuan] test changing column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently, ranger check for paimon missing rename column name command, add the test case
#6942
### How was this patch tested?
Test Paimon Rename column name with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6946 from davidyuan1223/test_rename_column_name.
Closes#6942
8e49eb0ab [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
**Why are the changes needed:**
Currently looking to connect to a HiveServer2 behind an NGINX proxy that is requiring mTLS communication. pyHive seems to lack the capability to establish an mTLS connection in applications such as Airflow directly communicating to the HiveServer2 instance.
The change needed is to be able to pass in the parameters for a proper mTLS ssl context to be established. I believe that creating your own ssl_context object is the quickest and cleanest way to do so, leaving the responsibility of configuring it to further implementations and users. Also cuts down on code length.
**How was this patch tested:**
Corresponding pytest fixtures have been added, using the mock module to see if ssl_context object was properly accessed, or if the default one created in the Connection initialization was properly configured.
Was not able to run pytest fixtures specifically, was lacking JDBC driver, first time contributing to open source, happy to run tests if provided guidance. Passed a clean build and test of the entire kyuubi project in local dev environment.
**Was this patch authored or co-authored using generative AI tooling**
Yes, Generated-by Cursor-AI with Claude Sonnet 3.5 agent
Closes#6935 from alexio215/connection-class-ssl-context-param.
Closes#6908
539b29962 [Cheng Pan] Update python/pyhive/tests/test_hive.py
14c607489 [Alex Wojtowicz] Simplified testing, following pattern of other tests, need proper SSL setup with nginx to test ssl_context fully
b947f2454 [Alex Wojtowicz] Added exception handling since JDBC driver will not run in python tests
11f9002bf [Alex Wojtowicz] Passing in fully configured mock object before creating connection
009c5cf24 [Alex Wojtowicz] Added back doc string documentation
e3280bcd8 [Alex Wojtowicz] Python testing
529de8a12 [Alex Wojtowicz] Added ssl_context object. If no obj is provided, then it continues to use default provided parameters
Lead-authored-by: Alex Wojtowicz <awojtowi@akamai.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Create Partitioned Table for PAIMON, check that has support the command
#6923
### How was this patch tested?
est Authz for paimon with create partitioned table command. Check the permission
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6931 from davidyuan1223/support_create_with_parition_for_paimon.
Closes#6923
61f7560d3 [Cheng Pan] Merge branch 'master' into support_create_with_parition_for_paimon
ffb79376f [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
b0829795a [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
4b160d720 [davidyuan] support create partition table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Add/Change Table properties for PAIMON, check that has support the command
https://github.com/apache/kyuubi/issues/6932
### How was this patch tested?
Test Add/Change properties SQL
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6933 from davidyuan1223/test_alter_tableproperties_for_paimin.
Closes#6932
4d64fbf23 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
c861a778b [davidyuan] support add/change table properties for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Without this PR:
- use and conform Scalafmt 3.7.x
- forcing all the imports from the same parent package, which is even violation the max length of the line
With this PR:
- use and conform Scalafmt 3.9.x
- Scalafmt 3.8.2 changes the binpack style: https://github.com/scalameta/scalafmt/releases/tag/v3.8.2
- change to `binPack.importSelectors=singleLine`, minimizing the impacts to existed code
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6674 from bowenliang123/scalafmt38.
Closes#6674
d9f4f9ddc [Bowen Liang] 3.9.0
438a07d61 [Bowen Liang] reformat
2df50669c [liangbowen] bump scalafmt to 3.8
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### Why are the changes needed?
AUTHZ Test CTAS for Paimon to check it support this command, the related issue is https://github.com/apache/kyuubi/issues/6921
### How was this patch tested?
Test Authz for paimon with create table as command. Check the permission.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6922 from davidyuan1223/support_create_table_as_for_paimon_check.
Closes#6921
7bfd6ad49 [david yuan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
a9ce20cc4 [davidyuan] support create table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: david yuan <davidyuan1223@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Paimon does not seem to support Scala 2.13
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6925 from pan3793/authz-paimon-scala212.
Closes#6925
865a7dd72 [Cheng Pan] fix
971d23273 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
499f10ab0 [Cheng Pan] Only run Paimon authz tests with Scala 2.12
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 4.0 continues to receive breaking changes since 4.0.0-preview2, and the 4.0.0 RC1 is scheduled at 20250215, this PR fixes all compatibility for the latest Spark 4.0.0-SNAPSHOT for Spark SQL engine.
### How was this patch tested?
Pass GHA with `spark-master`
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6920 from pan3793/spark4.
Closes#6920
170430e5e [Cheng Pan] Revert "ci"
c6d889350 [Cheng Pan] fix
86ff7ea2e [Cheng Pan] fix
75d0bf563 [Cheng Pan] ci
9d88c8630 [Cheng Pan] fix spark 4.0 compatibility
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
add Utils.CLIENT_IP_ADDRESS to cache local ipAddress
closes#6918
### How was this patch tested?
minor fix
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6919 from wForget/KYUUBI-6918.
Closes#6918
5b0efe722 [Cheng Pan] Update kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java
b8660c83b [wforget] address comment
835dfb8ef [wforget] address comment
29dd13a2e [wforget] fix style
843b29fe8 [wforget] [KYUUBI #6918] Cache client ipAddress in kyuubi jdbc connection
Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
https://iceberg.apache.org/releases/#180-release
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6914 from pan3793/iceberg-1.8.0.
Closes#6914
f7e9b7e51 [Cheng Pan] fix
f44dee4e6 [Cheng Pan] Use Iceberg 1.7 for Spark 3.3
cd0bdda6a [Cheng Pan] rm staging repo
881a01581 [Cheng Pan] Bump Iceberg 1.8.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
HTTP dialect ignores the database specified in the URL and uses the "default" instead.
### How was this patch tested?
Tested manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6906 from developster/pyhive-update1.
Closes#6905
6e21d7259 [Cheng Pan] Update python/pyhive/sqlalchemy_hive.py
ec7d4629e [Octavian Ciubotaru] [KYUUBI #6905] PyHive HTTP/HTTPS dialect to use the database name from url
Lead-authored-by: Octavian Ciubotaru <ociubotaru@developmentgateway.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Why are the changes needed?
## Issue reference:
https://github.com/apache/kyuubi/issues/6912
## How to reproduce the issue?
The changes in this PR will avoid a wrong result when generating the instance of org.apache.kyuubi.plugin.lineage.Lineage, in the certain case as follows:
step 1: create a temporary view from a file
step 2: insert into a table by selecting from the temporary view in step 1
step 3: generate the lineage when executing the insert statement in step 2
In detail, please see the UT code submission in this patch.
## The issue analysis
Let's see the current code when getting the Lineage object by resolving a LogicalPlan object:
<img width="694" alt="image" src="https://github.com/user-attachments/assets/65256a0d-320d-4271-968f-59eafb74de9f" />
According to the above logic, a None org.apache.kyuubi.plugin.lineage.Lineage object will be generated due to "try-catch" self-protection, in this certain case. This None object will lead to problems in the following 2 scenes:
### Unit Test Environment
In Unit Test, when the code runs here a "None.get" exception will be raised:
<img width="682" alt="image" src="https://github.com/user-attachments/assets/102dc9bd-294f-4b1e-b1c6-01b6fee50fed" />
Here's the runtime exception stack:
```
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineageWithoutExecuting(SparkSQLLineageParserHelperSuite.scala:1485)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$83(SparkSQLLineageParserHelperSuite.scala:1465)
```
### Production Environment
This Lineage object cannot be used in the production environment because it has a None value which lacks some necessary lineage information. The right content of the Lineage instance in the above case should be:
```
inputTables(List())
outputTables(List(spark_catalog.test_db.test_table_from_dir))
columnLineage(List(ColumnLineage(spark_catalog.test_db.test_table_from_dir.a0,Set()), ColumnLineage(spark_catalog.test_db.test_table_from_dir.b0,Set())))
```
a newly added test case(test directory to table) passed after this issue is fixed.
# How to fix the issue?
Add a "Empty judgment" logic. In detail, please see the code submission in this patch.
# How was this patch tested?
1. by adding a new test case in UT code and make sure it passes
2. by submitting a Spark application including the SQL of this case in the production environment, and make sure a right Lineage instance is generated, instead of a None object
# Was this patch authored or co-authored using generative AI tooling?
No
Closes#6911 from xglv1985/fix_spark_lineage_runtime_exception.
Closes#6912
13a71075d [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
4e89b95cd [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
59b350bfb [xglv1985] fix a runtime exception when generate column lineage tuple--more readable code
52bc0288d [xglv1985] fix a runtime exception when generate column lineage tuple--spotless sytle
fea6bbc0d [xglv1985] fix a runtime exception when generate column lineage tuple--remove tab from UT code
901879095 [xglv1985] fix a runtime exception when generate column lineage tuple--unit test
fbb4df879 [xglv1985] fix a runtime exception when generate column lineage tuple
Lead-authored-by: xglv1985 <xglv1985@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
https://github.com/delta-io/delta/releases/tag/v3.3.0
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6916 from pan3793/delta-3.3.0.
Closes#6916
51f6c1cdf [Cheng Pan] Bump Delta 3.3.0 for Spark 3.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
https://hudi.apache.org/releases/release-1.0.1
### How was this patch tested?
Pass GHA
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6917 from pan3793/hudi-1.0.1.
Closes#6917
b25414bd3 [Cheng Pan] Bump Hudi 1.0.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
I observed ClickHouse integration test failure in GHA, after some investigation, the root cause is https://github.com/testcontainers/testcontainers-java/pull/9942
```
/entrypoint.sh: neither CLICKHOUSE_USER nor CLICKHOUSE_PASSWORD is set, disabling network access for user 'default'
```
In short, the recent ClickHouse docker image does not allow the `default` user to connect without a password, unfortunately, `testcontainers-scala-clickhosue` does not expose API to set CLICKHOSUE_USER and CLICKHOUSE_PASSWORD, as a workaround, I pin `clickhouse-server:24.3.15`(the latest version has no such restriction) until a fixed version of Testcontainers available.
This PR also switches the `clickhouse-jdbc`'s classifier from `http` to `shaded`, the reason is, `http` does not ship ApacheHttpClient5, previously, it happened to work because `iceberg-runtime-spark3.5_2.12` packaged un-relocated ApacheHttpClient5 classes, but it gets fixed in Iceberg 1.8.0, then `clickhouse-jdbc:http` stop working.
```
java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpRequest
```
Additionally, this PR bumps `clickhouse-jdbc` from 0.6.0 to 0.6.5.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6915 from pan3793/fix-ch-test.
Closes#6915
996f095e0 [Cheng Pan] Pin clickhouse-server:24.3.15
d633df07c [Cheng Pan] Bump clickhouse-jdbc 0.6.5
214c8a227 [Cheng Pan] Fix ClickHouse integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Added a service definition for spark which in turn enables the creation of a default policy for the spark service.
Default policy will block access until another policy is downloaded from Apache Ranger.
### How was this patch tested?
Tested manually.
Configure Kyuubi Authz plugin. Do not start Apache Ranger, it must not be reachable.
Make sure that policy cache is empty.
Start Kyuubi engine and try to query any tables. The default policy should not allow any access.
Previously the access was not restricted because there wasn't a default policy defined.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6902 from developster/master.
Closes#6901
feb6ebf61 [Octavian Ciubotaru] Default policy for spark
Authored-by: Octavian Ciubotaru <ociubotaru@developmentgateway.org>
Signed-off-by: Kent Yao <yao@apache.org>
Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md">cross-spawn's changelog</a>.</em></p>
<blockquote>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.5...v7.0.6">7.0.6</a> (2024-11-18)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>update cross-spawn version to 7.0.5 in package-lock.json (<a href="f700743918">f700743</a>)</li>
</ul>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.4...v7.0.5">7.0.5</a> (2024-11-07)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>fix escaping bug introduced by backtracking (<a href="640d391fde">640d391</a>)</li>
</ul>
<h3><a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.3...v7.0.4">7.0.4</a> (2024-11-07)</h3>
<h3>Bug Fixes</h3>
<ul>
<li>disable regexp backtracking (<a href="https://redirect.github.com/moxystudio/node-cross-spawn/issues/160">#160</a>) (<a href="5ff3a07d9a">5ff3a07</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="77cd97f3ca"><code>77cd97f</code></a> chore(release): 7.0.6</li>
<li><a href="6717de49ff"><code>6717de4</code></a> chore: upgrade standard-version</li>
<li><a href="f700743918"><code>f700743</code></a> fix: update cross-spawn version to 7.0.5 in package-lock.json</li>
<li><a href="9a7e3b2165"><code>9a7e3b2</code></a> chore: fix build status badge</li>
<li><a href="085268352d"><code>0852683</code></a> chore(release): 7.0.5</li>
<li><a href="640d391fde"><code>640d391</code></a> fix: fix escaping bug introduced by backtracking</li>
<li><a href="bff0c87c8b"><code>bff0c87</code></a> chore: remove codecov</li>
<li><a href="a7c6abc6fe"><code>a7c6abc</code></a> chore: replace travis with github workflows</li>
<li><a href="9b9246e096"><code>9b9246e</code></a> chore(release): 7.0.4</li>
<li><a href="5ff3a07d9a"><code>5ff3a07</code></a> fix: disable regexp backtracking (<a href="https://redirect.github.com/moxystudio/node-cross-spawn/issues/160">#160</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/moxystudio/node-cross-spawn/compare/v7.0.3...v7.0.6">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
Dependabot will merge this PR once CI passes on it, as requested by yaooqinn.
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6814 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/cross-spawn-7.0.6.
Closes#6814
10dafbc6e [dependabot[bot]] ⬆️ Bump cross-spawn from 7.0.3 to 7.0.6 in /kyuubi-server/web-ui
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Correct spelling make the documentation better.
### How was this patch tested?
No need.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6900 from shuyouZZ/new-branch.
Closes#6900
27220abaf [dongshuyou] [DOCS] Correct spelling errors in 'large_query_results' part
Authored-by: dongshuyou <dongshuyou@idea.edu.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For the `com.codahale.metrics.MetricRegistry::gauge`.
It `getOrAdd` the gauge with name.
```
public <T extends Gauge> T gauge(String name) {
return (Gauge)this.getOrAdd(name, MetricRegistry.MetricBuilder.GAUGES);
}
```
So we have to get all the gauges to check whether the gauge exists.
### How was this patch tested?
UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6891 from turboFei/gauge_exists.
Closes#6891
18be2a521 [Wang, Fei] o(1)
039e7b5eb [Wang, Fei] check existing gauge
32dce6fb1 [Wang, Fei] check gauge exists
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this directory from growing indefinitely.
This PR proposes to let `spark.kubernetes.file.upload.path` support placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}` and introduce a switch `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled` to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
For example, the user can configure the below configurations in `kyuubi-defaults.conf` to enable monthly rolling support for `spark.kubernetes.file.upload.path`
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
Note that: spark would create sub dir `s"spark-upload-${UUID.randomUUID()}"` under the `spark.kubernetes.file.upload.path` for each uploading, the administer still needs to clean up the staging directory periodically.
For example:
```
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-f2b71340-dc1d-4940-89e2-c5fc31614eb4
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-173a8653-4d3e-48c0-b8ab-b7f92ae582d6
hdfs://hadoop-cluster/spark-upload-202501/spark-upload-3b22710f-a4a0-40bb-a3a8-16e481038a63
```
Administer can safely delete the `hdfs://hadoop-cluster/spark-upload-202412` after 20250101
### How was this patch tested?
New UTs are added.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6876 from pan3793/rolling-upload.
Closes#6876
6614bf29c [Cheng Pan] comment
5d5cb3eb3 [Cheng Pan] docs
343adaefb [Cheng Pan] review
3eade8bc4 [Cheng Pan] fix
706989778 [Cheng Pan] docs
38953dc3f [Cheng Pan] Support rolling spark.kubernetes.file.upload.path
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Address comments: https://github.com/apache/kyuubi/discussions/6877#discussioncomment-11743818
> I guess this is a Kyuubi implementation issue, we just read the content from the kyuubi.kubernetes.authenticate.oauthTokenFile and call ConfigBuilder.withOauthToken, I guess this approach does not support token refresh...
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6883 from turboFei/k8s_token_provider.
Closes#6883
69dd28d27 [Wang, Fei] comments
a01040f94 [Wang, Fei] withOauthTokenProvider
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
If the session manager's ThreadPoolExecutor refuses to execute the asyncOperation, then we need to shut down the query-timeout-thread in the catch block. This should also be done in JDBC and the CHAT engine.
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#6873 from lsm1/branch-followup-6843.
Closes#6843
aed9088c8 [senmiaoliu] fix query timeout checker leak in chat engine and jdbc engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: senmiaoliu <senmiaoliu@trip.com>
### Why are the changes needed?
see https://github.com/apache/kyuubi/issues/6843
If the session manager's ThreadPoolExecutor refuses to execute asyncOperation, then we need to shut down the query-timeout-thread in the catch
### How was this patch tested?
1 Use jstack to view threads on the long-lived engine side
2 Wait for all SQL statements in the engine to finish executing, and then use stack to check the number of query-timeout-thread threads, which should be empty.
### Was this patch authored or co-authored using generative AI tooling?
NO
Closes#6844 from ASiegeLion/master.
Closes#6843
9107a300e [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
4b3417f21 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
ef1f66bb5 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
9e1a015f6 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
78a9fde09 [liupeiyue] [KYUUBI #6843] FIX 'query-timeout-thread' thread leak
Authored-by: liupeiyue <liupeiyue@yy.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Followup for https://github.com/apache/kyuubi/pull/6866
It would throw exception if both thrift binary SSL and thrift http SSL enabled
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6872 from turboFei/duplicate_gauge.
Closes#6866
ea356766e [Wang, Fei] prevent conflicts
982f175fd [Wang, Fei] conflicts
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
This reverts commit e8cbff32d4.
### Why are the changes needed?
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#6871 from turboFei/revert_zk.
Closes#6871Closes#6865
1c61f9d85 [Wang, Fei] Revert "[KYUUBI #6865] [TEST] Set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing"
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Similar with `kyuubi.frontend.bind.host`, it is better to set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing as well.
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6865 from turboFei/zk_bind_host.
Closes#6865
9044e284b [Wang, Fei] [TEST] Set kyuubi.zookeeper.embedded.client.port.address for testing
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Add metrics for SSL keystore expiration, then we can add alert if the keystore will expire in 1 month.
### How was this patch tested?
Integration testing.
<img width="1721" alt="image" src="https://github.com/user-attachments/assets/f4ef6af6-923b-403c-a80d-06dbb80dbe1c" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6866 from turboFei/keystore_expire.
Closes#6866
77c6db0a7 [Wang, Fei] Add metrics for SSL keystore expiration time #6866
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The original dashboard dashboard template does not work for my use case:
In this PR:
1. I add the `baseFilter` variable, default value is `job=~"kyuubi"`.
2. apply the `baseFilter` for all expressions
3. add `baseLegend` for the legendFormat
4. Fix hardcode typos
5. remove the `sum` from the expressions for JVM
### How was this patch tested?
<img width="1723" alt="image" src="https://github.com/user-attachments/assets/2da3366d-4cd7-41fe-9a6b-1159d0b61734" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6867 from turboFei/dash_board.
Closes#6867
d0d1a3612 [Wang, Fei] comments
9fc3b5c20 [Wang, Fei] use kyuubi_jvm_uptime to get instance
280f97c59 [Wang, Fei] green
f8fc35193 [Wang, Fei] all filter
8d1ac2fdf [Wang, Fei] base legend
9c18bfcb0 [Wang, Fei] save
1a4799013 [Wang, Fei] job=kyuubi
ea7634580 [Wang, Fei] instant datasource and remove cluster
19db2b3e1 [Wang, Fei] add filter
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For histogram and timer metrics, it already has label, so need to support apply the instance label with existing ones.
For example:
```
# HELP kyuubi_backend_service_close_operation Generated from Dropwizard metric import (metric=kyuubi.backend_service.close_operation, type=com.codahale.metrics.Timer)
# TYPE kyuubi_backend_service_close_operation summary
kyuubi_backend_service_close_operation{quantile="0.5",}{instance="hadoopkyuubi-1.hadoopkyuubihl.hadoopmaster-dev.svc.140.tess.io:10019"} 0.032923216000000005
```
### How was this patch tested?
UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6868 from turboFei/instance_label_follow.
Closes#6864
4894784e0 [Wang, Fei] ut
b8f227f32 [Wang, Fei] save
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
For my use case, the instances are not human readable, so I prefer to return the FQDN.
<img width="1483" alt="image" src="https://github.com/user-attachments/assets/92045517-456f-4087-8a36-9e3e4bea2f1d" />
### How was this patch tested?
Integration testing.
```
(base) ➜ dist git:(prometheus_label_2) cat conf/kyuubi-defaults.conf
kyuubi.metrics.prometheus.metrics.instance.enabled=true
kyuubi.zookeeper.embedded.client.port.address=localhost
kyuubi.frontend.bind.host=localhost
```
<img width="1692" alt="image" src="https://github.com/user-attachments/assets/0b60d504-62ec-418d-880b-f8a2f00d5550" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6864 from turboFei/prometheus_label_2.
Closes#6864
d24571ccb [Wang, Fei] match
6a6a5110b [Wang, Fei] comments
c3046d4a1 [Wang, Fei] save
fb2021a31 [Wang, Fei] revert
42395945e [Wang, Fei] compatible
17b7007f5 [Wang, Fei] add instance label
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
