### Why are the changes needed?
To fix NPE.
Before, we use below method to get `metadataManager`.
```
private def metadataManager = KyuubiServer.kyuubiServer.backendService
.sessionManager.asInstanceOf[KyuubiSessionManager].metadataManager
```
But before the kyuubi server fully restarted, the `KyuubiServer.kyuubiServer` is null and might throw NPE during batch recovery phase.
For example:
```
:2025-04-23 14:06:24.040 ERROR [KyuubiSessionManager-exec-pool: Thread-231] org.apache.kyuubi.engine.KubernetesApplicationOperation: Failed to get application by label: kyuubi-unique-tag=95116703-4240-4cc1-9886-ccae3a2ac879, due to Cannot invoke "org.apache.kyuubi.server.KyuubiServer.backendService()" because the return value of "org.apache.kyuubi.server.KyuubiServer$.kyuubiServer()" is null
```
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7041 from turboFei/fix_NPE.
Closes#7041
064d88707 [Wang, Fei] Fix NPE
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
It is missed in #6828733d4f0901/jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java (L151-L159)
### How was this patch tested?
Minor change.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7038 from turboFei/fixNPE.
Closes#6828
2785e97be [Wang, Fei] Fix NPE
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
This PR removes the page https://kyuubi.readthedocs.io/en/v1.10.1/client/python/pyspark.html and merges the most content into https://kyuubi.readthedocs.io/en/v1.10.1/extensions/engines/spark/jdbc-dialect.html, some original content of the latter is also modified.
The current docs are misleading, I got asked several times by users why they follow the [Kyuubi PySpark docs](https://kyuubi.readthedocs.io/en/v1.10.1/client/python/pyspark.html) to access data stored in Hive warehouse is too slow.
Actually, accessing HiveServer2/STS from Spark JDBC data source is discouraged by the Spark community, see [SPARK-47482](https://github.com/apache/spark/pull/45609), even though it's technical feasible.
### How was this patch tested?
It's a docs-only change, review is required.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7036 from pan3793/jdbc-ds-docs.
Closes#7036
c00ce0706 [Cheng Pan] style
f2676bd23 [Cheng Pan] [DOCS] Improve docs for kyuubi-extension-spark-jdbc-dialect
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
After https://github.com/apache/spark/pull/34460 (Since Spark 3.3.0), the `spark-app-name` is available.
We shall use it as the application name if it exists.
### How was this patch tested?
Minor change.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7034 from turboFei/k8s_app_name.
Closes#7034
bfa88a436 [Wang, Fei] Get pod app name
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
I found that, for a kyuubi batch on kubernetes.
1. It has been `FINISHED`.
2. then I delete the pod manually, then I check the k8s-audit.log, then the appState became `FAILED`.
```
2025-04-15 11:16:30.453 INFO [-675216314-pool-44-thread-839] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Running containers=[microvault->ContainerState(running=ContainerStateRunning(startedAt=2025-04-15T18:13:48Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FINISHED appError=''
:2025-04-15 11:16:30.854 INFO [-675216314-pool-44-thread-840] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://91654e3ee74e2c31218e14be201b50a4a604c2ad15d3afd84dc6f620e59894b7, exitCode=2, finishedAt=2025-04-15T18:16:30Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T18:13:48Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FAILED appError='{
```
This PR is a followup for #6690 , which ignore the container state if POD is terminated.
It is more reasonable to respect the terminated container state than terminated pod state.
### How was this patch tested?
Integration testing.
```
:2025-04-15 13:53:24.551 INFO [-1077768163-pool-36-thread-3] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: eventType=DELETE label=e0eb4580-3cfa-43bf-bdcc-efeabcabc93c context=97 namespace=dls-prod pod=kyuubi-spark-e0eb4580-3cfa-43bf-bdcc-efeabcabc93c-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://66c42206730950bd422774e3c1b0f426d7879731788cea609bbfe0daab24a763, exitCode=2, finishedAt=2025-04-15T20:53:22Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T20:52:00Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://9179a73d9d9e148dcd9c13ee6cc29dc3e257f95a33609065e061866bb611cb3b, exitCode=0, finishedAt=2025-04-15T20:52:28Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T20:52:01Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-578df0facbfd4958a07f8d1ae79107dc appState=FINISHED appError=''
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7025 from turboFei/container_terminated.
Closes#7025Closes#6686
a3b2a5a56 [Wang, Fei] comments
4356d1bc9 [Wang, Fei] fix the app state logical
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Github has provide subtask creation & auto-linking features, and they are more advanced
### How was this patch tested?
https://github.com/apache/kyuubi/issues/7030
### Was this patch authored or co-authored using generative AI tooling?
no
Closes#7032 from yaooqinn/sb.
Closes#7032
3fd6ccd90 [Kent Yao] remove more
cbf691033 [Kent Yao] Remove Umbrella/Subtask issue template
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
1. Audit the kubernetes resource event type.
2. Fix the process logical for DELETE event.
Before this pr:
I tried to delete the POD manually, then I saw that, kyuubi thought the `appState=PENDING`.
```
:2025-04-15 13:58:20.320 INFO [-1077768163-pool-36-thread-7] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: eventType=DELETE label=3c58e9fd-cf8c-4cc3-a9aa-82ae40e200d8 context=97 namespace=dls-prod pod=kyuubi-spark-3c58e9fd-cf8c-4cc3-a9aa-82ae40e200d8-driver podState=Pending containers=[] appId=spark-cd125bbd9fc84ffcae6d6b5d41d4d8ad appState=PENDING appError=''
```
It seems that, the pod status in the event is the snapshot before pod deleted.
Then we would not receive any event for this POD, and finally the batch FINISHED with application `NOT_FOUND` .
<img width="1389" alt="image" src="https://github.com/user-attachments/assets/5df03db6-0924-4a58-9538-b196fbf87f32" />
Seems we need to process the DELETE event specially.
1. get the app state from the pod/container states
2. if the applicationState got is terminated, return the applicationState directly
3. otherwise, the applicationState should be FAILED, as the pod has been deleted.
### How was this patch tested?
<img width="1614" alt="image" src="https://github.com/user-attachments/assets/11e64c6f-ad53-4485-b8d2-a351bb23e8ca" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7026 from turboFei/k8s_audit.
Closes#7026
4e5695d34 [Wang, Fei] for delete
c16757218 [Wang, Fei] audit the pod event type
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
This ensure the Kyuubi server is promptly informed for any Kubernetes resource changes after startup. It is highly recommend to set it for multiple Kyuubi instances mode.
### How was this patch tested?
Existing GA and Integration testing.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7027 from turboFei/k8s_client_init.
Closes#7027
393b9960a [Wang, Fei] server only
a640278c4 [Wang, Fei] refresh
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Upgrade the kubernetes client, https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.5
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7023 from turboFei/k8s_client.
Closes#7023
3e3ac634f [Wang, Fei] 6.16.5
df5aa011f [Wang, Fei] upgrade
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
To satisfy ASF policy
https://lists.apache.org/thread/89jb1kp77wcv16tph8qlbf5k0fscyz9l
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7022 from pan3793/ann-tmpl.
Closes#7022
7fa64b163 [Cheng Pan] Update announcement mail template to contain download links
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently, if the kyuubi session between client and kyuubi session disconnected without closing properly, it is difficult to debug, and we have to check the kyuubi server log.
It is better that, we can record such kind of information into kyuubi session event.
### How was this patch tested?
IT.
<img width="1264" alt="image" src="https://github.com/user-attachments/assets/d2c5b6d0-6298-46ec-9b73-ce648551120c" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7015 from turboFei/disconnect.
Closes#7015
c95709284 [Wang, Fei] do not post
e46521410 [Wang, Fei] nit
bca7f9b7e [Wang, Fei] post
1cf6f8f49 [Wang, Fei] disconnect
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
To fix the batch kyuubi instance port is negative issue.
<img width="697" alt="image" src="https://github.com/user-attachments/assets/ef992390-8d20-44b3-8640-35496caff85d" />
It happen after I stop the kyuubi service.
We should use variable instead of function for jetty server serverUri.
After the server connector stopped, the localPort would be `-2`.
### How was this patch tested?
Existing UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7017 from turboFei/server_port_negative.
Closes#7017
3d34c4031 [Wang, Fei] warn
e58298646 [Wang, Fei] mutable server uri
2cbaf772a [Wang, Fei] Revert "hard code the server uri"
b64d91b32 [Wang, Fei] hard code the server uri
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Since https://github.com/apache/kyuubi/pull/3618
Kyuubi server could retry opening the engine when encountering a special error.
1937dd93f9/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala (L177-L212)
The `_client` might be reset and closed.
So, we shall set `_client` after open engine session successfully, as the `client` method is a public method.
### How was this patch tested?
Existing UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7011 from turboFei/client_ready.
Closes#7011
3ad57ee91 [Wang, Fei] fix npe
b956394fa [Wang, Fei] close internal engine client
523b48a4d [Wang, Fei] internal client
5baeedec1 [Wang, Fei] Revert "method"
84c808cfb [Wang, Fei] method
8efaa52f6 [Wang, Fei] check engine launched
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Backport https://github.com/apache/hive/pull/3749
It is not possible to create SSL connection with Kerberos authentication when the server certificate is not issued to the canonical host name but to an alternative domain name.
See details about the exception and steps for reproducing in the [HIVE-26723](https://issues.apache.org/jira/browse/HIVE-26723)
Hive JDBC client validates the host name by its canonical name by default. This behaviour leads to SSLHandshakeExcpetion when trying to connect using alias name with Kerberos authentication. To solve this issue a new connection property is introduced to be able disabling canonical host name check: enableCanonicalHostnameCheck having default value true.
When the property is not given in connection string (or its value is true) then the original behaviour is applied i.e. checking canonical host name.
### How was this patch tested?
There are no new unit tests because the fix is in the HiveConnection constructor which contains lot of logic inside and also builds new SSL connections.
IMO it would have been far too much effort to mock the whole environment for creating unit tests against this tiny change. :(
There wasn't any already existing test against HiveConnection that could be extended with this new feature/bugfix. It is misleading that there is a class having name TestHiveConnection but there is no any tests that would test the class HiveConnection itself.
BTW It was tested manually: after this fix when the steps in JIRA are executed again using the new JARs then the SSL connection is created successfully, and I was able to execute queries.
### Does this PR introduce any user-facing change?
A new JDBC connection URL property has been introduced: enableCanonicalHostnameCheck to be able to turn off the canonical host name checking. Its default value is true so if it is not set the canonical host name is checked when building up the SSL connection.
To turn off the canonical host name checking just add this property to the connection string, i.e:
```
./beeline -u "jdbc:hive2://hs2.subdomain.example.com:443/default;transportMode=http;httpPath=cliservice;socketTimeout=60;ssl=true;retries=1;principal=myhiveprincipal/mydomain.example.com;enableCanonicalHostnameCheck=false;"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7009 from turboFei/kerberos_can.
Closes#7009
40cd48814 [Wang, Fei] Backport HIVE-26723: Configurable canonical name checking.
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/hive/pull/4823
We need to setup production tunnel because we can't connect to production environment directly:
```
sh -fN -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -L 127.0.0.1:10001:hiveserver2.prod.company.com:10001 bastion.company.com
JDBC url: jdbc:hive2://127.0.0.1:10001/default;ssl=true
```
But it will throw exception after [HIVE-15025](https://issues.apache.org/jira/browse/HIVE-15025):
```
Exception in thread "main" java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10001/default;ssl=true: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.spark.sql.TestJDBC$.main(TestJDBC.scala:47)
at org.apache.spark.sql.TestJDBC.main(TestJDBC.scala)
Caused by: org.apache.hive.org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
at org.apache.hive.org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at org.apache.hive.org.apache.thrift.transport.TSaslTransport.sendSaslMessage(TSaslTransport.java:166)
at org.apache.hive.org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:100)
at org.apache.hive.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.hive.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:311)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:196)
... 5 more
```
This PR disables ssl hostname verification for 127.0.0.1 to workaround this issue.
### How was this patch tested?
Manual test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7008 from turboFei/ssl.
Closes#7008
6ae1b7b82 [Wang, Fei] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR resolves the following warning message:
```
../kyuubi/docs/quick_start/quick_start_with_jupyter.md: WARNING: document isn't included in any toctree
```
It removes the empty page `Getting Started with Jupyter Lap` which is also not presented in the documentation menu.
### How was this patch tested?
Built documentation locally and checked there are no warning message anymore.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7005 from dnskr/remove-empty-getting-started-with-jupyter-lap.
Closes#7005
030fb3598 [dnskr] [DOC] Remove empty page "Getting Started with Jupyter Lap"
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x, and it causes `ClassNotFoundException` when user use `kyuubi-spark-authz-shaded_2.12-1.10.1.jar`(built with Ranger 2.5.0)
```
java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/base/ProviderBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:208)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:191)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:465)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1321)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1183)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:148)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:172)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:112)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:563)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:138)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:254)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
at org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension.<init>(RangerSparkExtension.scala:44)
```
### How was this patch tested?
```
$ jar tf kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep org/apache/kyuubi/shade/com/fasterxml
org/apache/kyuubi/shade/com/fasterxml/
org/apache/kyuubi/shade/com/fasterxml/jackson/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AbstractTypeResolver.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty$Type.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$XmlExtensions.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/BeanDescription.class
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7004 from pan3793/authz-jackson.
Closes#7004
cbf870516 [Cheng Pan] fix
4312d9fe5 [Cheng Pan] Include FastXML Jackson into authZ shaded jar
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check missing paimom system producers command, need to support these command
1. create_tag
2. delete_tag
3. rollback
#6979
PS: There has a question about paimon, paimon'sparkCatalog need the currentCatalog Env is the paimon's catalog, use default spark_catalog will throw exception, maybe we should add this hint to the documentation.
such as
If you wanna support producers check with paimon, you need use sql `use $paimon_catalog` to ensure the session currentCatalog is paimon_catalog
PS: paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version
### How was this patch tested?
producers test cases
1. create_tag
2. delete_tag
3. rollback
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6980 from davidyuan1223/paimon_producers.
Closes#6979
90f367c6a [davidyuan] update
c0503cb5f [davidyuan] Merge remote-tracking branch 'origin/paimon_producers' into paimon_producers
993d1dcb8 [davidyuan] Merge branch 'master' into paimon_producers
f68edef41 [davidyuan] producers
58224191b [davidyuan] Merge branch 'master' into paimon_producers
57aac600b [davidyuan] update
cbcdd8dbf [davidyuan] producers
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This PR provides an alternative for RANGER-4125 to cut out JNA dependencies for authZ plugin.
### How was this patch tested?
Pass GHA, and I checked the content of authz-shaded jar
```
$ jar tf extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep Hostname
org/apache/kyuubi/shade/com/kstruct/gethostname4j/Hostname.class
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7003 from pan3793/authz-hostname.
Closes#7003
42e246856 [Cheng Pan] Cut out JNA dependencies for authz plugin
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
In Scala [v2.13.16](https://github.com/scala/scala/releases/tag/v2.13.16)
> JNA is no longer a dependency of `scala-compiler.jar`
Since Spark 4.0 upgrades to Scala 2.13.16, JNA deps have gone too.
```
$ spark-3.5.5-bin-hadoop3-scala2.13 cat RELEASE
Spark 3.5.5 (git revision 7c29c664cdc) built for Hadoop 3.3.4
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.13 -Phadoop-3 -Phive -Phive-thriftserver
$ spark-3.5.5-bin-hadoop3-scala2.13 ls jars | grep jna
jna-5.9.0.jar
```
```
$ spark-4.0.0-bin-hadoop3 cat RELEASE
Spark 4.0.0 (git revision ca56e9ce591) built for Hadoop 3.4.1
Build flags: -B -Pyarn -Pkubernetes -Psparkr -Phadoop-3 -Phive -Phive-thriftserver
$ spark-4.0.0-bin-hadoop3 ls jars | grep jna
<no-output>
```
It's rare to use the non-default Scala version with Spark in practice, we shall respect Spark 4 deps for Scala 2.13 cases.
### How was this patch tested?
Review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6999 from pan3793/authz-scala213.
Closes#6999
18230a2d7 [Cheng Pan] Keep JNA in authz-shaded with Scala 2.13
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Fix the missing `assert` in `SparkProcessBuilderSuite - spark process builder`.
Fix the flaky test `SparkProcessBuilderSuite - capture error from spark process builder` by increasing `kyuubi.session.engine.startup.maxLogLines` from 10 to 4096, this is easy to fail, especially in Spark 4.0 due to increased error stack trace. for example, https://github.com/apache/kyuubi/actions/runs/13974413470/job/39290129824
```
SparkProcessBuilderSuite:
- spark process builder
- capture error from spark process builder *** FAILED ***
The code passed to eventually never returned normally. Attempted 167 times over 1.5007926256666668 minutes. Last failure message: "org.apache.kyuubi.KyuubiSQLException: Suppressed: org.apache.spark.util.Utils$OriginalTryStackTraceException: Full stacktrace of original doTryWithCallerStacktrace caller
See more: /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/kyuubi-spark-sql-engine.log.2
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:239)
at java.base/java.lang.Thread.run(Thread.java:1583)
.
FYI: The last 10 line(s) of log are:
25/03/24 12:53:39 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/03/24 12:53:39 INFO MemoryStore: MemoryStore cleared
25/03/24 12:53:39 INFO BlockManager: BlockManager stopped
25/03/24 12:53:39 INFO BlockManagerMaster: BlockManagerMaster stopped
25/03/24 12:53:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/03/24 12:53:39 INFO SparkContext: Successfully stopped SparkContext
25/03/24 12:53:39 INFO ShutdownHookManager: Shutdown hook called
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-18455622-344e-48ac-92eb-4b368c35e697
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/artifacts/spark-7479249b-44a2-4fe5-aa0f-544074f9c356
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-5ba8250f-1ff2-4e0d-a365-27d7518308e1" did not contain "org.apache.hadoop.hive.ql.metadata.HiveException:". (SparkProcessBuilderSuite.scala:77)
```
### How was this patch tested?
Pass GHA, and verified locally with Spark 4.0.0 RC3 by running tests 10 times with constant success.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6998 from pan3793/spark-pb-ut.
Closes#6998
a4290b413 [Cheng Pan] harness SparkProcessBuilderSuite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
We meet below issue:
For spark on yarn:
```
spark.yarn.submit.waitAppCompletion=false
kyuubi.engine.yarn.submit.timeout=PT10M
```
Due to network issue, the application submission was very slow.
It was submitted after 15 minutes.
<img width="1430" alt="image" src="https://github.com/user-attachments/assets/a326c3d1-4d39-42da-b6aa-cad5f8e7fc4b" />
<img width="1350" alt="image" src="https://github.com/user-attachments/assets/8e20056a-bd71-4515-a5e3-f881509a34b2" />
Then the batch failed from PENDING state to ERRO state directly, due to application state NOT_FOUND(exceeds the kyuubi.engine.yarn.submit.timeout).
a54ee39ab3/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala (L99-L106)
<img width="1727" alt="image" src="https://github.com/user-attachments/assets/20a2987c-675c-4136-a107-001f30b1b217" />
Here is the operation event:
<img width="1727" alt="image" src="https://github.com/user-attachments/assets/e2bab9c3-a959-4e2b-a207-813ae6489b30" />
But from the batch log, the current application status should be `PENDING`.
```
:2025-03-21 17:36:19.350 INFO [KyuubiSessionManager-exec-pool: Thread-176922] org.apache.kyuubi.operation.BatchJobSubmission: Batch report for bbba09c8-3704-4a87-8394-9bcbbd39cc34, Some(ApplicationInfo(application_1741747369441_2258235,6042072c-e8fa-425d-a6a3-3d5bbb4ec1e3-275732_6042072c-e8fa-425d-a6a3-3d5bbb4ec1e3-275732.e3a34b86-7fc7-43ea-b4a5-1b6f27df54b5.0_20250322002147.stm,PENDING,Some(https://apollo-rno-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1741747369441_2258235/),Some()))
```
So, we should retrieve the batch application info after the submission process terminated before checking the application failed, to get the current application information to prevent the corner case:
1. the application submission time exceeds the `kyuubi.engine.yarn.submit.timeout` and the app state is NOT FOUND
2. can not get the application report before the submission process terminated
3. then the batch state to ERROR from PENDING directly.
Conclusion:
The application state transition was:
UNKNOWN(before submit timeout) -> NOT_FOUND(reach submit timeout) -> processExit -> batchOpError -> PENDING(updateApplicationInfoMetadataIfNeeded) -> UNKNOWN(batchError but app not terminated)
After this PR, it should be:
UNKNOWN(before submit timeout) -> NOT_FOUND(reach submit timeout) -> processExit-> PENDING(after process terminated) -> ....
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6997 from turboFei/app_not_found_v2.
Closes#6997
370cf49e9 [Wang, Fei] v2
912ec28ca [Wang, Fei] nit
3c376f922 [Wang, Fei] log the op ex
d9cbdb87d [Wang, Fei] fix app not found
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
When users switch from Hive to Spark, for sql like INSERT OVERWRITE DIRECTORY AS SELECT, it would be great if small files could be automatically merged through simple configuration, just like in Hive.
### How was this patch tested?
UnitTest
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6991 from Z1Wu/feat/add_insert_dir_rebalance_support.
Closes#6990
2820bb2d2 [wuziyi] [fix] nit
a69c04191 [wuziyi] [fix] nit
951a7738f [wuziyi] [fix] nit
f75dfcb3a [wuziyi] [Feat] add rebalance before InsertIntoHiveDirCommand and InsertIntoDataSourceDirCommand to align with behaviors of hive
Authored-by: wuziyi <wuziyi02@corp.netease.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
As title.
Fix NPE, because the cleanupTerminatedAppInfoTrigger will be set to `null`.
d3520ddbce/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala (L269)
Also shutdown the ExecutorService when KubernetesApplicationOperation stoped.
## Describe Your Solution 🔧
Shutdown the thread executor service and check the null.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6785 from turboFei/npe_k8s.
Closes#6785
6afd052e6 [Wang, Fei] comments
f0c3e3134 [Wang, Fei] prevent npe
9dffe0125 [Wang, Fei] shutdown
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
[
[KYUUBI #6984] Fix ValueError when rendering MapType data
](https://github.com/apache/kyuubi/issues/6984)
### Why are the changes needed?
The issue was caused by an incorrect iteration of MapType data in the `%table` magic command. When iterating over a `MapType` column, the code used `for k, v in m` directly, which leads to a `ValueError` because raw `Map` entries may not be properly unpacked
### How was this patch tested?
- [x] Manual testing:
Executed a query with a `MapType` column and confirmed that the `%table` command now renders it without errors.
```python
from pyspark.sql import SparkSession
from pyspark.sql.types import MapType, StringType, IntegerType
spark = SparkSession.builder \
.appName("MapFieldExample") \
.getOrCreate()
data = [
(1, {"a": "1", "b": "2"}),
(2, {"x": "10"}),
(3, {"key": "value"})
]
schema = "id INT, map_col MAP<STRING, STRING>"
df = spark.createDataFrame(data, schema=schema)
df.printSchema()
df2=df.collect()
```
using `%table` render table
```python
%table df2
```
result
```python
{'application/vnd.livy.table.v1+json': {'headers': [{'name': 'id', 'type': 'INT_TYPE'}, {'name': 'map_col', 'type': 'MAP_TYPE'}], 'data': [[1, {'a': '1', 'b': '2'}], [2, {'x': '10'}], [3, {'key': 'value'}]]}}
```
### Was this patch authored or co-authored using generative AI tooling?
No
**notice** This PR was co-authored by DeepSeek-R1.
Closes#6985 from JustFeng/patch-1.
Closes#6984
e0911ba94 [Reese Feng] Update PySparkTests for magic cmd
bc3ce1a49 [Reese Feng] Update PySparkTests for magic cmd
200d7ad9b [Reese Feng] Fix syntax error in dict iteration in magic_table_convert_map
Authored-by: Reese Feng <10377945+JustFeng@users.noreply.github.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Avoid unstable test case caused by table size changes, this is likely to happen when upgrading Parquet/ORC/Spark.
### How was this patch tested?
unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6989 from wForget/minor_fix.
Closes#6989
9cdd36973 [wforget] address comments
f79fcca0d [wforget] Calculate expected join partitions based on scanned table size
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
To satisfy the ASF requirements.
> An error occurred while processing the github feature in .asf.yaml:
>
> GitHub discussions can only be enabled if a mailing list target exists for it.
### How was this patch tested?
Review and monitor the master branch after merging.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6988 from pan3793/discussion.
Closes#6988
dd788c054 [Cheng Pan] [INFRA] Foward GitHub discussions to ASF mailing list
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check test case missing check paimon add new column command, add it.
#6941
### How was this patch tested?
Test Add New Column for paimin with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6945 from davidyuan1223/test_add_new_column_for_paimon.
Closes#6941
f865e132a [davidyuan] test add new column
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column type command, add the test case
#6951
### How was this patch tested?
Test ranger check paimon changing column type command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6956 from davidyuan1223/test_changing_column_type.
Closes#6951
9d5140e81 [davidyuan] Merge branch 'master' into test_changing_column_type
e4f8974d8 [davidyuan] test changing column type
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes multiple `Pygments lexer name` issues and resolves the following warnings during the documentation build process:
```
../kyuubi/docs/client/advanced/kerberos.md:37: WARNING: Pygments lexer name 'cmd' is not known
../kyuubi/docs/client/bi_tools/hue.md:26: WARNING: Lexing literal_block "Welcome to\n __ __ __\n /\\ \\/\\ \\ /\\ \\ __\n \\ \\ \\/'/' __ __ __ __ __ __\\ \\ \\____/\\_\\\n \\ \\ , < /\\ \\/\\ \\/\\ \\/\\ \\/\\ \\/\\ \\\\ \\ '__`\\/\\ \\\n \\ \\ \\\\`\\\\ \\ \\_\\ \\ \\ \\_\\ \\ \\ \\_\\ \\\\ \\ \\L\\ \\ \\ \\\n \\ \\_\\ \\_\\/`____ \\ \\____/\\ \\____/ \\ \\_,__/\\ \\_\\\n \\/_/\\/_/`/___/> \\/___/ \\/___/ \\/___/ \\/_/\n /\\___/\n \\/__/" as "bash" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/client/jdbc/hive_jdbc.md:27: WARNING: Pygments lexer name 'gradle' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:111: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:134: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:143: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:163: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/connector/spark/delta_lake_with_azure_blob.rst:191: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/deployment/hive_metastore.md:38: WARNING: Pygments lexer name 'shell script' is not known
../kyuubi/docs/deployment/hive_metastore.md:207: WARNING: Lexing literal_block "Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)\n\tat org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:127)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1274)\n\t... 93 more" as "java" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/extensions/server/authentication.rst:75: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/extensions/server/events.rst:76: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/monitor/logging.md:38: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:86: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:222: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/security/kerberos.rst:104: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/security/ldap.md:24: WARNING: Pygments lexer name 'properties example' is not known
../kyuubi/docs/security/ldap.md:40: WARNING: Pygments lexer name 'properties example' is not known
```
Supported languages: [Pygments lexers](https://pygments.org/docs/lexers) and [highlightjs](https://github.com/highlightjs/highlight.js/blob/main/SUPPORTED_LANGUAGES.md).
### How was this patch tested?
Built documentation locally and checked there are related warnings.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6986 from dnskr/fix-unknown-Pygments-lexer-name.
Closes#6986
f5b62f52d [dnskr] [DOC] Fix multiple Pygments lexer name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes few `Unknown target name: "XYZ". [docutils]` issues and resolves the following errors messages:
```
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "github repository". [docutils]
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "read the docs". [docutils]
../kyuubi/docs/contributing/doc/style.rst:66: ERROR: Unknown target name: "directive rubric". [docutils]
```
### How was this patch tested?
Built documentation locally, checked there are no related error messages and doc pages are correct.
##### Page `contributing/doc/get_started.html`
Before changes
<img width="1114" alt="image" src="https://github.com/user-attachments/assets/f1a19c51-3c4c-4268-bf83-7ca0c60315b1" />
After changes
<img width="1113" alt="image" src="https://github.com/user-attachments/assets/437edef1-0fd9-43bf-bd3f-bda43035a2c9" />
##### Page `contributing/doc/style.html`
Before changes
<img width="1128" alt="image" src="https://github.com/user-attachments/assets/39666841-1155-439f-9045-06a9d78624c3" />
After changes
<img width="1117" alt="image" src="https://github.com/user-attachments/assets/2e1f8663-5c1e-4a3c-887e-5f65d01b4cf3" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6987 from dnskr/fix-doc-unknown-target-name.
Closes#6987
391958b4d [dnskr] [DOC] Fix Unknown target name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The feature `spark.sql.watchdog.forcedMaxOutputRows` is a little bit hacky, it's actually a manually implemented "limit pushdown", we already have a simple and more reliable way to achieve that by using `kyuubi.operation.result.max.rows`.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6983 from pan3793/rm-forcedMaxOutputRows.
Closes#6983
5e0707955 [Cheng Pan] Remove support for spark.sql.watchdog.forcedMaxOutputRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes [nested lists formatting](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks) and resolves the following warnings:
```shell
../kyuubi/docs/contributing/doc/style.rst:65: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:67: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:68: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:73: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:106: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:107: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
```
### How was this patch tested?
Built documentation locally and checked nested lists are fixed.
Before changes:
<img width="914" alt="image" src="https://github.com/user-attachments/assets/3ec7079a-e494-4614-9af0-d6e217bcad60" />
After changes:
<img width="1020" alt="image" src="https://github.com/user-attachments/assets/2d3b3231-094d-49bd-b3d7-c6149e13c939" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6981 from dnskr/doc-fix-nested-lists.
Closes#6981
4b425f279 [dnskr] [DOC] Fix nested lists
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
fix typos in docs
### How was this patch tested?
NO
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6964 from wenxin-cn/fix-typos-in-docs.
Closes#6964
5a50a927a [Kent Yao] style
be899c21f [10172] fix typos in docs
Lead-authored-by: wenxin-cn <wen.xin@datasw.com>
Co-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: 10172 <wen.xin@datasw.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Support ranger check with paimon Update & Delete & MergeInti Table Command
### How was this patch tested?
Test Paimon Update & Delete Table Command with ranger check
#6973
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6974 from davidyuan1223/update_table.
Closes#6973
3bd607300 [davidyuan] update
1d68494ce [davidyuan] Test MergeInto
a27ea633f [davidyuan] Test MergeInto
56638f47c [davidyuan] Merge branch 'master' into update_table
1c3464df5 [davidyuan] Test Table Update & Delete
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Simple refactoring to clean up the code for the Spark 3.5 extension, e.g., remove unnecessary `*Base` `*Helper` abstraction layers, remove code for legacy Spark versions.
Note: I don't touch `ForcedMaxOutputRows*` because I'm going to remove it in the next PR.
Preparation for Spark 4.0 support.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6975 from pan3793/spark-ext-35-cleanup.
Closes#6975
b5a94a680 [Cheng Pan] nit
c729e268c [Cheng Pan] fix
1087ac709 [Cheng Pan] Clean up code for Spark 3.5 extension
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The change fixes minor issue that resolves the following error:
```shell
../kyuubi/docs/contributing/doc/get_started.rst:78: ERROR: Content block expected for the "note" directive; none found. [docutils]
```
### How was this patch tested?
Built documentation locally and checked there are no difference and error message.
Before changes:
<img width="1214" alt="image" src="https://github.com/user-attachments/assets/f53398d8-b04a-4367-8040-3e6573cc54f2" />
After changes:
<img width="1197" alt="image" src="https://github.com/user-attachments/assets/3b08e1ee-11c2-4386-b178-35e33d6a56dc" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6977 from dnskr/remove-empty-note-block.
Closes#6977
942a2687e [dnskr] [DOC] Remove empty note block
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check with paimon missing check drop columns command
#6947
### How was this patch tested?
Test ranger check with drop columns commad
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6952 from davidyuan1223/test_dropping_columns.
Closes#6947
27e7b3836 [davidyuan] Merge remote-tracking branch 'origin/master' into test_dropping_columns
83815ef3b [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently, ranger check missing paimon Table Query command, add the test cases
#6971
### How was this patch tested?
Test Paimon Table Query & Batch Time Travel
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6972 from davidyuan1223/test_query.
Closes#6972
7aa821215 [davidyuan] Test Table Query
ae1b75e85 [davidyuan] Test Table Query
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR fixes `nonexisting document` issues:
```shell
./kyuubi/docs/client/advanced/features/index.rst:19: WARNING: toctree contains reference to nonexisting document 'client/advanced/features/engine_resources' [toc.not_readable]
./kyuubi/docs/client/odbc/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/odbc/todo' [toc.not_readable]
./kyuubi/docs/client/thrift/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/thrift/hive_beeline' [toc.not_readable]
./kyuubi/docs/index.rst:189: WARNING: toctree contains reference to nonexisting document 'sql/index' [toc.not_readable]
./kyuubi/docs/quick_start/index.rst:23: WARNING: toctree contains reference to nonexisting document 'quick_start/quick_start_with_beeline' [toc.not_readable]
```
### How was this patch tested?
Checked that there are no `nonexisting document` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6970 from dnskr/doc-fix-nonexisting-document.
Closes#6970
a7c2b3617 [dnskr] [DOC] Fix "nonexisting document" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR resolves multiple `"Title underline too short"` warnings to reduce noise during documentation building, for instance:
```shell
./kyuubi/docs/client/jdbc/mysql_jdbc.rst:18: WARNING: Title underline too short.
`MySQL Connectors`_
================ [docutils]
./kyuubi/docs/connector/hive/paimon.rst:17: WARNING: Title underline too short.
`Apache Paimon (Incubating)`_
========== [docutils]
./kyuubi/docs/connector/hive/paimon.rst:31: WARNING: Title underline too short.
Apache Paimon (Incubating) Integration
------------------- [docutils]
```
### How was this patch tested?
Checked that there are no `"Title underline too short"` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6969 from dnskr/doc-fix-title-underline-too-short.
Closes#6969
2007a2440 [dnskr] [DOC] Fix "Title underline too short" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Scala 2.13.16 release notes: https://github.com/scala/scala/releases/tag/v2.13.16
Also, bump the silencer plugin to 1.7.19 to support Scala 2.13.16
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6965 from pan3793/scala-2.13.6.
Closes#6965
141e75864 [Cheng Pan] Bump silencer 1.7.19
9f09127e7 [Cheng Pan] Bump Scala 2.13.16
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Range check Test case missing paimon changing column comment, add the test case
#6948
### How was this patch tested?
Test Ranger check with paimon changing column comment command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6953 from davidyuan1223/test_changing_column_comment.
Closes#6948
983720e78 [davidyuan] test changing column comment
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently , ranger check missing paimon insert table command, add test cases
#6958
### How was this patch tested?
1. Test INSERT INTO:
1.1 table1OnlyUserForNs could select table1, try to insert table1
1.2 someone has no any permission, try to insert table1
2. Test INSERT OVERWRITE:
2.1 table1OnlyUserForNs could select table1, try to insert table2
2.2 someone has no any permiession, try select table1 then insert table2
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6959 from davidyuan1223/test_insert.
Closes#6958
d1f41ba81 [davidyuan] Merge branch 'master' into test_insert
b56e701d4 [davidyuan] Test Insert Table
8306210ee [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Bump the log4j version to fix below issue:
```
2025-03-04 22:27:58.291 WARN [main-SendThread(xxxx:2181)] org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
: java.lang.ExceptionInInitializerError
at org.apache.log4j.Logger.getLogger(Logger.java:35)
at org.apache.kyuubi.shaded.zookeeper.Login.<init>(Login.java:44)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:228)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:990)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1042)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1657)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1216)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:379)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:344)
at org.apache.log4j.legacy.core.CategoryUtil.setLevel(CategoryUtil.java:131)
at org.apache.log4j.Category.setLevel(Category.java:643)
at org.apache.log4j.Category.setLevel(Category.java:638)
at org.apache.log4j.spi.RootLogger.setLevel(RootLogger.java:60)
at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:70)
... 6 more
```
It is fixed in https://github.com/apache/logging-log4j2/releases/tag/rel%2F2.24.3https://github.com/apache/logging-log4j2/issues/3234
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6960 from turboFei/log4j_version.
Closes#6960
48b076c3e [Wang, Fei] Upgrade log4j version to 2.24.3
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Test Authz Support paimon rename table name command privilege check
#6936
### How was this patch tested?
Test Authz Support paimon rename table name command privilege check
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6937 from davidyuan1223/check_authz_paimon_rename_table.
Closes#6936
797d1c489 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
bc3c823a3 [davidyuan] Merge remote-tracking branch 'origin/master' into check_authz_paimon_rename_table
6205670d2 [davidyuan] add renameTable to command_spec.json
e4b241ef5 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
5fec3bcb7 [davidyuan] test paimon rename table name command
30d09418c [davidyuan] test paimon rename table name command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon adding column position command, add the test case
#6949
### How was this patch tested?
Test ranger check with paimon adding column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6954 from davidyuan1223/test_adding_column_position.
Closes#6949
262ecaaca [davidyuan] Merge remote-tracking branch 'origin/master' into test_adding_column_position
154765fc3 [davidyuan] Merge branch 'master' into test_adding_column_position
4ebf985a9 [davidyuan] test adding column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
