### Why are the changes needed?
Backport https://github.com/apache/hive/pull/3749
It is not possible to create SSL connection with Kerberos authentication when the server certificate is not issued to the canonical host name but to an alternative domain name.
See details about the exception and steps for reproducing in the [HIVE-26723](https://issues.apache.org/jira/browse/HIVE-26723)
Hive JDBC client validates the host name by its canonical name by default. This behaviour leads to SSLHandshakeExcpetion when trying to connect using alias name with Kerberos authentication. To solve this issue a new connection property is introduced to be able disabling canonical host name check: enableCanonicalHostnameCheck having default value true.
When the property is not given in connection string (or its value is true) then the original behaviour is applied i.e. checking canonical host name.
### How was this patch tested?
There are no new unit tests because the fix is in the HiveConnection constructor which contains lot of logic inside and also builds new SSL connections.
IMO it would have been far too much effort to mock the whole environment for creating unit tests against this tiny change. :(
There wasn't any already existing test against HiveConnection that could be extended with this new feature/bugfix. It is misleading that there is a class having name TestHiveConnection but there is no any tests that would test the class HiveConnection itself.
BTW It was tested manually: after this fix when the steps in JIRA are executed again using the new JARs then the SSL connection is created successfully, and I was able to execute queries.
### Does this PR introduce any user-facing change?
A new JDBC connection URL property has been introduced: enableCanonicalHostnameCheck to be able to turn off the canonical host name checking. Its default value is true so if it is not set the canonical host name is checked when building up the SSL connection.
To turn off the canonical host name checking just add this property to the connection string, i.e:
```
./beeline -u "jdbc:hive2://hs2.subdomain.example.com:443/default;transportMode=http;httpPath=cliservice;socketTimeout=60;ssl=true;retries=1;principal=myhiveprincipal/mydomain.example.com;enableCanonicalHostnameCheck=false;"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7009 from turboFei/kerberos_can.
Closes#7009
40cd48814 [Wang, Fei] Backport HIVE-26723: Configurable canonical name checking.
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/hive/pull/4823
We need to setup production tunnel because we can't connect to production environment directly:
```
sh -fN -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -L 127.0.0.1:10001:hiveserver2.prod.company.com:10001 bastion.company.com
JDBC url: jdbc:hive2://127.0.0.1:10001/default;ssl=true
```
But it will throw exception after [HIVE-15025](https://issues.apache.org/jira/browse/HIVE-15025):
```
Exception in thread "main" java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10001/default;ssl=true: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.spark.sql.TestJDBC$.main(TestJDBC.scala:47)
at org.apache.spark.sql.TestJDBC.main(TestJDBC.scala)
Caused by: org.apache.hive.org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching localhost found.
at org.apache.hive.org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at org.apache.hive.org.apache.thrift.transport.TSaslTransport.sendSaslMessage(TSaslTransport.java:166)
at org.apache.hive.org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:100)
at org.apache.hive.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.hive.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:311)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:196)
... 5 more
```
This PR disables ssl hostname verification for 127.0.0.1 to workaround this issue.
### How was this patch tested?
Manual test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7008 from turboFei/ssl.
Closes#7008
6ae1b7b82 [Wang, Fei] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR resolves the following warning message:
```
../kyuubi/docs/quick_start/quick_start_with_jupyter.md: WARNING: document isn't included in any toctree
```
It removes the empty page `Getting Started with Jupyter Lap` which is also not presented in the documentation menu.
### How was this patch tested?
Built documentation locally and checked there are no warning message anymore.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7005 from dnskr/remove-empty-getting-started-with-jupyter-lap.
Closes#7005
030fb3598 [dnskr] [DOC] Remove empty page "Getting Started with Jupyter Lap"
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x, and it causes `ClassNotFoundException` when user use `kyuubi-spark-authz-shaded_2.12-1.10.1.jar`(built with Ranger 2.5.0)
```
java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/base/ProviderBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:208)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:191)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:465)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1321)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1183)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:148)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:172)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:112)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:563)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:138)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:254)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
at org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension.<init>(RangerSparkExtension.scala:44)
```
### How was this patch tested?
```
$ jar tf kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep org/apache/kyuubi/shade/com/fasterxml
org/apache/kyuubi/shade/com/fasterxml/
org/apache/kyuubi/shade/com/fasterxml/jackson/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AbstractTypeResolver.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty$Type.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$XmlExtensions.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/BeanDescription.class
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7004 from pan3793/authz-jackson.
Closes#7004
cbf870516 [Cheng Pan] fix
4312d9fe5 [Cheng Pan] Include FastXML Jackson into authZ shaded jar
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check missing paimom system producers command, need to support these command
1. create_tag
2. delete_tag
3. rollback
#6979
PS: There has a question about paimon, paimon'sparkCatalog need the currentCatalog Env is the paimon's catalog, use default spark_catalog will throw exception, maybe we should add this hint to the documentation.
such as
If you wanna support producers check with paimon, you need use sql `use $paimon_catalog` to ensure the session currentCatalog is paimon_catalog
PS: paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version
### How was this patch tested?
producers test cases
1. create_tag
2. delete_tag
3. rollback
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6980 from davidyuan1223/paimon_producers.
Closes#6979
90f367c6a [davidyuan] update
c0503cb5f [davidyuan] Merge remote-tracking branch 'origin/paimon_producers' into paimon_producers
993d1dcb8 [davidyuan] Merge branch 'master' into paimon_producers
f68edef41 [davidyuan] producers
58224191b [davidyuan] Merge branch 'master' into paimon_producers
57aac600b [davidyuan] update
cbcdd8dbf [davidyuan] producers
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This PR provides an alternative for RANGER-4125 to cut out JNA dependencies for authZ plugin.
### How was this patch tested?
Pass GHA, and I checked the content of authz-shaded jar
```
$ jar tf extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep Hostname
org/apache/kyuubi/shade/com/kstruct/gethostname4j/Hostname.class
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7003 from pan3793/authz-hostname.
Closes#7003
42e246856 [Cheng Pan] Cut out JNA dependencies for authz plugin
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
In Scala [v2.13.16](https://github.com/scala/scala/releases/tag/v2.13.16)
> JNA is no longer a dependency of `scala-compiler.jar`
Since Spark 4.0 upgrades to Scala 2.13.16, JNA deps have gone too.
```
$ spark-3.5.5-bin-hadoop3-scala2.13 cat RELEASE
Spark 3.5.5 (git revision 7c29c664cdc) built for Hadoop 3.3.4
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.13 -Phadoop-3 -Phive -Phive-thriftserver
$ spark-3.5.5-bin-hadoop3-scala2.13 ls jars | grep jna
jna-5.9.0.jar
```
```
$ spark-4.0.0-bin-hadoop3 cat RELEASE
Spark 4.0.0 (git revision ca56e9ce591) built for Hadoop 3.4.1
Build flags: -B -Pyarn -Pkubernetes -Psparkr -Phadoop-3 -Phive -Phive-thriftserver
$ spark-4.0.0-bin-hadoop3 ls jars | grep jna
<no-output>
```
It's rare to use the non-default Scala version with Spark in practice, we shall respect Spark 4 deps for Scala 2.13 cases.
### How was this patch tested?
Review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6999 from pan3793/authz-scala213.
Closes#6999
18230a2d7 [Cheng Pan] Keep JNA in authz-shaded with Scala 2.13
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Fix the missing `assert` in `SparkProcessBuilderSuite - spark process builder`.
Fix the flaky test `SparkProcessBuilderSuite - capture error from spark process builder` by increasing `kyuubi.session.engine.startup.maxLogLines` from 10 to 4096, this is easy to fail, especially in Spark 4.0 due to increased error stack trace. for example, https://github.com/apache/kyuubi/actions/runs/13974413470/job/39290129824
```
SparkProcessBuilderSuite:
- spark process builder
- capture error from spark process builder *** FAILED ***
The code passed to eventually never returned normally. Attempted 167 times over 1.5007926256666668 minutes. Last failure message: "org.apache.kyuubi.KyuubiSQLException: Suppressed: org.apache.spark.util.Utils$OriginalTryStackTraceException: Full stacktrace of original doTryWithCallerStacktrace caller
See more: /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/kyuubi-spark-sql-engine.log.2
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:239)
at java.base/java.lang.Thread.run(Thread.java:1583)
.
FYI: The last 10 line(s) of log are:
25/03/24 12:53:39 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/03/24 12:53:39 INFO MemoryStore: MemoryStore cleared
25/03/24 12:53:39 INFO BlockManager: BlockManager stopped
25/03/24 12:53:39 INFO BlockManagerMaster: BlockManagerMaster stopped
25/03/24 12:53:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/03/24 12:53:39 INFO SparkContext: Successfully stopped SparkContext
25/03/24 12:53:39 INFO ShutdownHookManager: Shutdown hook called
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-18455622-344e-48ac-92eb-4b368c35e697
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/artifacts/spark-7479249b-44a2-4fe5-aa0f-544074f9c356
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-5ba8250f-1ff2-4e0d-a365-27d7518308e1" did not contain "org.apache.hadoop.hive.ql.metadata.HiveException:". (SparkProcessBuilderSuite.scala:77)
```
### How was this patch tested?
Pass GHA, and verified locally with Spark 4.0.0 RC3 by running tests 10 times with constant success.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6998 from pan3793/spark-pb-ut.
Closes#6998
a4290b413 [Cheng Pan] harness SparkProcessBuilderSuite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
We meet below issue:
For spark on yarn:
```
spark.yarn.submit.waitAppCompletion=false
kyuubi.engine.yarn.submit.timeout=PT10M
```
Due to network issue, the application submission was very slow.
It was submitted after 15 minutes.
<img width="1430" alt="image" src="https://github.com/user-attachments/assets/a326c3d1-4d39-42da-b6aa-cad5f8e7fc4b" />
<img width="1350" alt="image" src="https://github.com/user-attachments/assets/8e20056a-bd71-4515-a5e3-f881509a34b2" />
Then the batch failed from PENDING state to ERRO state directly, due to application state NOT_FOUND(exceeds the kyuubi.engine.yarn.submit.timeout).
a54ee39ab3/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala (L99-L106)
<img width="1727" alt="image" src="https://github.com/user-attachments/assets/20a2987c-675c-4136-a107-001f30b1b217" />
Here is the operation event:
<img width="1727" alt="image" src="https://github.com/user-attachments/assets/e2bab9c3-a959-4e2b-a207-813ae6489b30" />
But from the batch log, the current application status should be `PENDING`.
```
:2025-03-21 17:36:19.350 INFO [KyuubiSessionManager-exec-pool: Thread-176922] org.apache.kyuubi.operation.BatchJobSubmission: Batch report for bbba09c8-3704-4a87-8394-9bcbbd39cc34, Some(ApplicationInfo(application_1741747369441_2258235,6042072c-e8fa-425d-a6a3-3d5bbb4ec1e3-275732_6042072c-e8fa-425d-a6a3-3d5bbb4ec1e3-275732.e3a34b86-7fc7-43ea-b4a5-1b6f27df54b5.0_20250322002147.stm,PENDING,Some(https://apollo-rno-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1741747369441_2258235/),Some()))
```
So, we should retrieve the batch application info after the submission process terminated before checking the application failed, to get the current application information to prevent the corner case:
1. the application submission time exceeds the `kyuubi.engine.yarn.submit.timeout` and the app state is NOT FOUND
2. can not get the application report before the submission process terminated
3. then the batch state to ERROR from PENDING directly.
Conclusion:
The application state transition was:
UNKNOWN(before submit timeout) -> NOT_FOUND(reach submit timeout) -> processExit -> batchOpError -> PENDING(updateApplicationInfoMetadataIfNeeded) -> UNKNOWN(batchError but app not terminated)
After this PR, it should be:
UNKNOWN(before submit timeout) -> NOT_FOUND(reach submit timeout) -> processExit-> PENDING(after process terminated) -> ....
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6997 from turboFei/app_not_found_v2.
Closes#6997
370cf49e9 [Wang, Fei] v2
912ec28ca [Wang, Fei] nit
3c376f922 [Wang, Fei] log the op ex
d9cbdb87d [Wang, Fei] fix app not found
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
When users switch from Hive to Spark, for sql like INSERT OVERWRITE DIRECTORY AS SELECT, it would be great if small files could be automatically merged through simple configuration, just like in Hive.
### How was this patch tested?
UnitTest
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6991 from Z1Wu/feat/add_insert_dir_rebalance_support.
Closes#6990
2820bb2d2 [wuziyi] [fix] nit
a69c04191 [wuziyi] [fix] nit
951a7738f [wuziyi] [fix] nit
f75dfcb3a [wuziyi] [Feat] add rebalance before InsertIntoHiveDirCommand and InsertIntoDataSourceDirCommand to align with behaviors of hive
Authored-by: wuziyi <wuziyi02@corp.netease.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
As title.
Fix NPE, because the cleanupTerminatedAppInfoTrigger will be set to `null`.
d3520ddbce/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala (L269)
Also shutdown the ExecutorService when KubernetesApplicationOperation stoped.
## Describe Your Solution 🔧
Shutdown the thread executor service and check the null.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6785 from turboFei/npe_k8s.
Closes#6785
6afd052e6 [Wang, Fei] comments
f0c3e3134 [Wang, Fei] prevent npe
9dffe0125 [Wang, Fei] shutdown
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
[
[KYUUBI #6984] Fix ValueError when rendering MapType data
](https://github.com/apache/kyuubi/issues/6984)
### Why are the changes needed?
The issue was caused by an incorrect iteration of MapType data in the `%table` magic command. When iterating over a `MapType` column, the code used `for k, v in m` directly, which leads to a `ValueError` because raw `Map` entries may not be properly unpacked
### How was this patch tested?
- [x] Manual testing:
Executed a query with a `MapType` column and confirmed that the `%table` command now renders it without errors.
```python
from pyspark.sql import SparkSession
from pyspark.sql.types import MapType, StringType, IntegerType
spark = SparkSession.builder \
.appName("MapFieldExample") \
.getOrCreate()
data = [
(1, {"a": "1", "b": "2"}),
(2, {"x": "10"}),
(3, {"key": "value"})
]
schema = "id INT, map_col MAP<STRING, STRING>"
df = spark.createDataFrame(data, schema=schema)
df.printSchema()
df2=df.collect()
```
using `%table` render table
```python
%table df2
```
result
```python
{'application/vnd.livy.table.v1+json': {'headers': [{'name': 'id', 'type': 'INT_TYPE'}, {'name': 'map_col', 'type': 'MAP_TYPE'}], 'data': [[1, {'a': '1', 'b': '2'}], [2, {'x': '10'}], [3, {'key': 'value'}]]}}
```
### Was this patch authored or co-authored using generative AI tooling?
No
**notice** This PR was co-authored by DeepSeek-R1.
Closes#6985 from JustFeng/patch-1.
Closes#6984
e0911ba94 [Reese Feng] Update PySparkTests for magic cmd
bc3ce1a49 [Reese Feng] Update PySparkTests for magic cmd
200d7ad9b [Reese Feng] Fix syntax error in dict iteration in magic_table_convert_map
Authored-by: Reese Feng <10377945+JustFeng@users.noreply.github.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
### Why are the changes needed?
Avoid unstable test case caused by table size changes, this is likely to happen when upgrading Parquet/ORC/Spark.
### How was this patch tested?
unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6989 from wForget/minor_fix.
Closes#6989
9cdd36973 [wforget] address comments
f79fcca0d [wforget] Calculate expected join partitions based on scanned table size
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
To satisfy the ASF requirements.
> An error occurred while processing the github feature in .asf.yaml:
>
> GitHub discussions can only be enabled if a mailing list target exists for it.
### How was this patch tested?
Review and monitor the master branch after merging.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6988 from pan3793/discussion.
Closes#6988
dd788c054 [Cheng Pan] [INFRA] Foward GitHub discussions to ASF mailing list
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check test case missing check paimon add new column command, add it.
#6941
### How was this patch tested?
Test Add New Column for paimin with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6945 from davidyuan1223/test_add_new_column_for_paimon.
Closes#6941
f865e132a [davidyuan] test add new column
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column type command, add the test case
#6951
### How was this patch tested?
Test ranger check paimon changing column type command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6956 from davidyuan1223/test_changing_column_type.
Closes#6951
9d5140e81 [davidyuan] Merge branch 'master' into test_changing_column_type
e4f8974d8 [davidyuan] test changing column type
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes multiple `Pygments lexer name` issues and resolves the following warnings during the documentation build process:
```
../kyuubi/docs/client/advanced/kerberos.md:37: WARNING: Pygments lexer name 'cmd' is not known
../kyuubi/docs/client/bi_tools/hue.md:26: WARNING: Lexing literal_block "Welcome to\n __ __ __\n /\\ \\/\\ \\ /\\ \\ __\n \\ \\ \\/'/' __ __ __ __ __ __\\ \\ \\____/\\_\\\n \\ \\ , < /\\ \\/\\ \\/\\ \\/\\ \\/\\ \\/\\ \\\\ \\ '__`\\/\\ \\\n \\ \\ \\\\`\\\\ \\ \\_\\ \\ \\ \\_\\ \\ \\ \\_\\ \\\\ \\ \\L\\ \\ \\ \\\n \\ \\_\\ \\_\\/`____ \\ \\____/\\ \\____/ \\ \\_,__/\\ \\_\\\n \\/_/\\/_/`/___/> \\/___/ \\/___/ \\/___/ \\/_/\n /\\___/\n \\/__/" as "bash" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/client/jdbc/hive_jdbc.md:27: WARNING: Pygments lexer name 'gradle' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:111: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:134: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:143: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/client/jdbc/kyuubi_jdbc.rst:163: WARNING: Pygments lexer name 'jdbc' is not known
../kyuubi/docs/connector/spark/delta_lake_with_azure_blob.rst:191: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/deployment/hive_metastore.md:38: WARNING: Pygments lexer name 'shell script' is not known
../kyuubi/docs/deployment/hive_metastore.md:207: WARNING: Lexing literal_block "Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)\n\tat org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:127)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)\n\tat com.sun.proxy.$Proxy37.getTable(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1274)\n\t... 93 more" as "java" resulted in an error at token: "'". Retrying in relaxed mode. [misc.highlighting_failure]
../kyuubi/docs/extensions/server/authentication.rst:75: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/extensions/server/events.rst:76: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/monitor/logging.md:38: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:86: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/monitor/logging.md:222: WARNING: Pygments lexer name 'log' is not known
../kyuubi/docs/security/kerberos.rst:104: WARNING: Pygments lexer name 'property' is not known
../kyuubi/docs/security/ldap.md:24: WARNING: Pygments lexer name 'properties example' is not known
../kyuubi/docs/security/ldap.md:40: WARNING: Pygments lexer name 'properties example' is not known
```
Supported languages: [Pygments lexers](https://pygments.org/docs/lexers) and [highlightjs](https://github.com/highlightjs/highlight.js/blob/main/SUPPORTED_LANGUAGES.md).
### How was this patch tested?
Built documentation locally and checked there are related warnings.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6986 from dnskr/fix-unknown-Pygments-lexer-name.
Closes#6986
f5b62f52d [dnskr] [DOC] Fix multiple Pygments lexer name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes few `Unknown target name: "XYZ". [docutils]` issues and resolves the following errors messages:
```
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "github repository". [docutils]
../kyuubi/docs/contributing/doc/get_started.rst:27: ERROR: Unknown target name: "read the docs". [docutils]
../kyuubi/docs/contributing/doc/style.rst:66: ERROR: Unknown target name: "directive rubric". [docutils]
```
### How was this patch tested?
Built documentation locally, checked there are no related error messages and doc pages are correct.
##### Page `contributing/doc/get_started.html`
Before changes
<img width="1114" alt="image" src="https://github.com/user-attachments/assets/f1a19c51-3c4c-4268-bf83-7ca0c60315b1" />
After changes
<img width="1113" alt="image" src="https://github.com/user-attachments/assets/437edef1-0fd9-43bf-bd3f-bda43035a2c9" />
##### Page `contributing/doc/style.html`
Before changes
<img width="1128" alt="image" src="https://github.com/user-attachments/assets/39666841-1155-439f-9045-06a9d78624c3" />
After changes
<img width="1117" alt="image" src="https://github.com/user-attachments/assets/2e1f8663-5c1e-4a3c-887e-5f65d01b4cf3" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6987 from dnskr/fix-doc-unknown-target-name.
Closes#6987
391958b4d [dnskr] [DOC] Fix Unknown target name issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The feature `spark.sql.watchdog.forcedMaxOutputRows` is a little bit hacky, it's actually a manually implemented "limit pushdown", we already have a simple and more reliable way to achieve that by using `kyuubi.operation.result.max.rows`.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6983 from pan3793/rm-forcedMaxOutputRows.
Closes#6983
5e0707955 [Cheng Pan] Remove support for spark.sql.watchdog.forcedMaxOutputRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The PR fixes [nested lists formatting](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks) and resolves the following warnings:
```shell
../kyuubi/docs/contributing/doc/style.rst:65: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:67: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:68: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:73: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:106: WARNING: Bullet list ends without a blank line; unexpected unindent. [docutils]
../kyuubi/docs/contributing/doc/style.rst:107: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils]
```
### How was this patch tested?
Built documentation locally and checked nested lists are fixed.
Before changes:
<img width="914" alt="image" src="https://github.com/user-attachments/assets/3ec7079a-e494-4614-9af0-d6e217bcad60" />
After changes:
<img width="1020" alt="image" src="https://github.com/user-attachments/assets/2d3b3231-094d-49bd-b3d7-c6149e13c939" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6981 from dnskr/doc-fix-nested-lists.
Closes#6981
4b425f279 [dnskr] [DOC] Fix nested lists
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
### Why are the changes needed?
fix typos in docs
### How was this patch tested?
NO
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6964 from wenxin-cn/fix-typos-in-docs.
Closes#6964
5a50a927a [Kent Yao] style
be899c21f [10172] fix typos in docs
Lead-authored-by: wenxin-cn <wen.xin@datasw.com>
Co-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: 10172 <wen.xin@datasw.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Support ranger check with paimon Update & Delete & MergeInti Table Command
### How was this patch tested?
Test Paimon Update & Delete Table Command with ranger check
#6973
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6974 from davidyuan1223/update_table.
Closes#6973
3bd607300 [davidyuan] update
1d68494ce [davidyuan] Test MergeInto
a27ea633f [davidyuan] Test MergeInto
56638f47c [davidyuan] Merge branch 'master' into update_table
1c3464df5 [davidyuan] Test Table Update & Delete
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Simple refactoring to clean up the code for the Spark 3.5 extension, e.g., remove unnecessary `*Base` `*Helper` abstraction layers, remove code for legacy Spark versions.
Note: I don't touch `ForcedMaxOutputRows*` because I'm going to remove it in the next PR.
Preparation for Spark 4.0 support.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6975 from pan3793/spark-ext-35-cleanup.
Closes#6975
b5a94a680 [Cheng Pan] nit
c729e268c [Cheng Pan] fix
1087ac709 [Cheng Pan] Clean up code for Spark 3.5 extension
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The change fixes minor issue that resolves the following error:
```shell
../kyuubi/docs/contributing/doc/get_started.rst:78: ERROR: Content block expected for the "note" directive; none found. [docutils]
```
### How was this patch tested?
Built documentation locally and checked there are no difference and error message.
Before changes:
<img width="1214" alt="image" src="https://github.com/user-attachments/assets/f53398d8-b04a-4367-8040-3e6573cc54f2" />
After changes:
<img width="1197" alt="image" src="https://github.com/user-attachments/assets/3b08e1ee-11c2-4386-b178-35e33d6a56dc" />
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6977 from dnskr/remove-empty-note-block.
Closes#6977
942a2687e [dnskr] [DOC] Remove empty note block
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check with paimon missing check drop columns command
#6947
### How was this patch tested?
Test ranger check with drop columns commad
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6952 from davidyuan1223/test_dropping_columns.
Closes#6947
27e7b3836 [davidyuan] Merge remote-tracking branch 'origin/master' into test_dropping_columns
83815ef3b [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently, ranger check missing paimon Table Query command, add the test cases
#6971
### How was this patch tested?
Test Paimon Table Query & Batch Time Travel
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6972 from davidyuan1223/test_query.
Closes#6972
7aa821215 [davidyuan] Test Table Query
ae1b75e85 [davidyuan] Test Table Query
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR fixes `nonexisting document` issues:
```shell
./kyuubi/docs/client/advanced/features/index.rst:19: WARNING: toctree contains reference to nonexisting document 'client/advanced/features/engine_resources' [toc.not_readable]
./kyuubi/docs/client/odbc/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/odbc/todo' [toc.not_readable]
./kyuubi/docs/client/thrift/index.rst:20: WARNING: toctree contains reference to nonexisting document 'client/thrift/hive_beeline' [toc.not_readable]
./kyuubi/docs/index.rst:189: WARNING: toctree contains reference to nonexisting document 'sql/index' [toc.not_readable]
./kyuubi/docs/quick_start/index.rst:23: WARNING: toctree contains reference to nonexisting document 'quick_start/quick_start_with_beeline' [toc.not_readable]
```
### How was this patch tested?
Checked that there are no `nonexisting document` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6970 from dnskr/doc-fix-nonexisting-document.
Closes#6970
a7c2b3617 [dnskr] [DOC] Fix "nonexisting document" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
The PR resolves multiple `"Title underline too short"` warnings to reduce noise during documentation building, for instance:
```shell
./kyuubi/docs/client/jdbc/mysql_jdbc.rst:18: WARNING: Title underline too short.
`MySQL Connectors`_
================ [docutils]
./kyuubi/docs/connector/hive/paimon.rst:17: WARNING: Title underline too short.
`Apache Paimon (Incubating)`_
========== [docutils]
./kyuubi/docs/connector/hive/paimon.rst:31: WARNING: Title underline too short.
Apache Paimon (Incubating) Integration
------------------- [docutils]
```
### How was this patch tested?
Checked that there are no `"Title underline too short"` warnings during the documentation build process.
```shell
make html
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6969 from dnskr/doc-fix-title-underline-too-short.
Closes#6969
2007a2440 [dnskr] [DOC] Fix "Title underline too short" issues
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Scala 2.13.16 release notes: https://github.com/scala/scala/releases/tag/v2.13.16
Also, bump the silencer plugin to 1.7.19 to support Scala 2.13.16
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6965 from pan3793/scala-2.13.6.
Closes#6965
141e75864 [Cheng Pan] Bump silencer 1.7.19
9f09127e7 [Cheng Pan] Bump Scala 2.13.16
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Range check Test case missing paimon changing column comment, add the test case
#6948
### How was this patch tested?
Test Ranger check with paimon changing column comment command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6953 from davidyuan1223/test_changing_column_comment.
Closes#6948
983720e78 [davidyuan] test changing column comment
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently , ranger check missing paimon insert table command, add test cases
#6958
### How was this patch tested?
1. Test INSERT INTO:
1.1 table1OnlyUserForNs could select table1, try to insert table1
1.2 someone has no any permission, try to insert table1
2. Test INSERT OVERWRITE:
2.1 table1OnlyUserForNs could select table1, try to insert table2
2.2 someone has no any permiession, try select table1 then insert table2
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6959 from davidyuan1223/test_insert.
Closes#6958
d1f41ba81 [davidyuan] Merge branch 'master' into test_insert
b56e701d4 [davidyuan] Test Insert Table
8306210ee [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Bump the log4j version to fix below issue:
```
2025-03-04 22:27:58.291 WARN [main-SendThread(xxxx:2181)] org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
: java.lang.ExceptionInInitializerError
at org.apache.log4j.Logger.getLogger(Logger.java:35)
at org.apache.kyuubi.shaded.zookeeper.Login.<init>(Login.java:44)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:228)
at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:990)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1042)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1657)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1216)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:379)
at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:344)
at org.apache.log4j.legacy.core.CategoryUtil.setLevel(CategoryUtil.java:131)
at org.apache.log4j.Category.setLevel(Category.java:643)
at org.apache.log4j.Category.setLevel(Category.java:638)
at org.apache.log4j.spi.RootLogger.setLevel(RootLogger.java:60)
at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:70)
... 6 more
```
It is fixed in https://github.com/apache/logging-log4j2/releases/tag/rel%2F2.24.3https://github.com/apache/logging-log4j2/issues/3234
### How was this patch tested?
Existing GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6960 from turboFei/log4j_version.
Closes#6960
48b076c3e [Wang, Fei] Upgrade log4j version to 2.24.3
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Test Authz Support paimon rename table name command privilege check
#6936
### How was this patch tested?
Test Authz Support paimon rename table name command privilege check
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6937 from davidyuan1223/check_authz_paimon_rename_table.
Closes#6936
797d1c489 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
bc3c823a3 [davidyuan] Merge remote-tracking branch 'origin/master' into check_authz_paimon_rename_table
6205670d2 [davidyuan] add renameTable to command_spec.json
e4b241ef5 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
5fec3bcb7 [davidyuan] test paimon rename table name command
30d09418c [davidyuan] test paimon rename table name command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon adding column position command, add the test case
#6949
### How was this patch tested?
Test ranger check with paimon adding column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6954 from davidyuan1223/test_adding_column_position.
Closes#6949
262ecaaca [davidyuan] Merge remote-tracking branch 'origin/master' into test_adding_column_position
154765fc3 [davidyuan] Merge branch 'master' into test_adding_column_position
4ebf985a9 [davidyuan] test adding column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently range check missing check UnsetTableProperties command, we need add it to the range check.
#6940
### How was this patch tested?
Use paimon removing table properties to test this command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6944 from davidyuan1223/test_remove_table_properties.
Closes#6940
4f24d7d6a [davidyuan] Merge branch 'master' into test_remove_table_properties
11d3773ed [davidyuan] test unset table properties command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column position command, add the test case
#6950
### How was this patch tested?
Test ranger check with paimon changing column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6955 from davidyuan1223/test_changing_column_position.
Closes#6950
520b5377f [davidyuan] Merge branch 'master' into test_changing_column_position
1eed87346 [davidyuan] test changing column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently, ranger check for paimon missing rename column name command, add the test case
#6942
### How was this patch tested?
Test Paimon Rename column name with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6946 from davidyuan1223/test_rename_column_name.
Closes#6942
8e49eb0ab [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
**Why are the changes needed:**
Currently looking to connect to a HiveServer2 behind an NGINX proxy that is requiring mTLS communication. pyHive seems to lack the capability to establish an mTLS connection in applications such as Airflow directly communicating to the HiveServer2 instance.
The change needed is to be able to pass in the parameters for a proper mTLS ssl context to be established. I believe that creating your own ssl_context object is the quickest and cleanest way to do so, leaving the responsibility of configuring it to further implementations and users. Also cuts down on code length.
**How was this patch tested:**
Corresponding pytest fixtures have been added, using the mock module to see if ssl_context object was properly accessed, or if the default one created in the Connection initialization was properly configured.
Was not able to run pytest fixtures specifically, was lacking JDBC driver, first time contributing to open source, happy to run tests if provided guidance. Passed a clean build and test of the entire kyuubi project in local dev environment.
**Was this patch authored or co-authored using generative AI tooling**
Yes, Generated-by Cursor-AI with Claude Sonnet 3.5 agent
Closes#6935 from alexio215/connection-class-ssl-context-param.
Closes#6908
539b29962 [Cheng Pan] Update python/pyhive/tests/test_hive.py
14c607489 [Alex Wojtowicz] Simplified testing, following pattern of other tests, need proper SSL setup with nginx to test ssl_context fully
b947f2454 [Alex Wojtowicz] Added exception handling since JDBC driver will not run in python tests
11f9002bf [Alex Wojtowicz] Passing in fully configured mock object before creating connection
009c5cf24 [Alex Wojtowicz] Added back doc string documentation
e3280bcd8 [Alex Wojtowicz] Python testing
529de8a12 [Alex Wojtowicz] Added ssl_context object. If no obj is provided, then it continues to use default provided parameters
Lead-authored-by: Alex Wojtowicz <awojtowi@akamai.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Create Partitioned Table for PAIMON, check that has support the command
#6923
### How was this patch tested?
est Authz for paimon with create partitioned table command. Check the permission
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6931 from davidyuan1223/support_create_with_parition_for_paimon.
Closes#6923
61f7560d3 [Cheng Pan] Merge branch 'master' into support_create_with_parition_for_paimon
ffb79376f [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
b0829795a [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
4b160d720 [davidyuan] support create partition table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Add/Change Table properties for PAIMON, check that has support the command
https://github.com/apache/kyuubi/issues/6932
### How was this patch tested?
Test Add/Change properties SQL
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6933 from davidyuan1223/test_alter_tableproperties_for_paimin.
Closes#6932
4d64fbf23 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
c861a778b [davidyuan] support add/change table properties for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Without this PR:
- use and conform Scalafmt 3.7.x
- forcing all the imports from the same parent package, which is even violation the max length of the line
With this PR:
- use and conform Scalafmt 3.9.x
- Scalafmt 3.8.2 changes the binpack style: https://github.com/scalameta/scalafmt/releases/tag/v3.8.2
- change to `binPack.importSelectors=singleLine`, minimizing the impacts to existed code
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6674 from bowenliang123/scalafmt38.
Closes#6674
d9f4f9ddc [Bowen Liang] 3.9.0
438a07d61 [Bowen Liang] reformat
2df50669c [liangbowen] bump scalafmt to 3.8
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### Why are the changes needed?
AUTHZ Test CTAS for Paimon to check it support this command, the related issue is https://github.com/apache/kyuubi/issues/6921
### How was this patch tested?
Test Authz for paimon with create table as command. Check the permission.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6922 from davidyuan1223/support_create_table_as_for_paimon_check.
Closes#6921
7bfd6ad49 [david yuan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
a9ce20cc4 [davidyuan] support create table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: david yuan <davidyuan1223@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Paimon does not seem to support Scala 2.13
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6925 from pan3793/authz-paimon-scala212.
Closes#6925
865a7dd72 [Cheng Pan] fix
971d23273 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
499f10ab0 [Cheng Pan] Only run Paimon authz tests with Scala 2.12
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Spark 4.0 continues to receive breaking changes since 4.0.0-preview2, and the 4.0.0 RC1 is scheduled at 20250215, this PR fixes all compatibility for the latest Spark 4.0.0-SNAPSHOT for Spark SQL engine.
### How was this patch tested?
Pass GHA with `spark-master`
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6920 from pan3793/spark4.
Closes#6920
170430e5e [Cheng Pan] Revert "ci"
c6d889350 [Cheng Pan] fix
86ff7ea2e [Cheng Pan] fix
75d0bf563 [Cheng Pan] ci
9d88c8630 [Cheng Pan] fix spark 4.0 compatibility
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
