### Why are the changes needed?
In Catalog which supports row level interface (iceberg etc.), merge into will be rewritten as WriteDelta or ReplaceData by rule . We should support the extraction of lineage relationship under this type.
### How was this patch tested?
add new tests for row-level catalog
### Was this patch authored or co-authored using generative AI tooling?
no
Closes#7127 from yabola/master-listener.
Closes#7126
e39d0d93b [chenliang.lu] add for notMatchedInstructions
1530d7c6c [chenliang.lu] add notMatchedInstructions
00660f83b [chenliang.lu] update spark version
db7d9ca85 [chenliang.lu] update spark version && optimize some code
85de5b069 [chenliang.lu] remove useless logger && fix ut
79d2f9bfa [chenliang.lu] fix check style
2f5e62cdd [chenliang.lu] [KYUUBI #7126][LINEAGE] Support merge into syntax in row level catalog
Authored-by: chenliang.lu <chenlianglu@tencent.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
Fix the lineage plugin cannot capture lineage of view after integrating authz plugin.
closes#7168
### How was this patch tested?
added unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7169 from wForget/KYUUBI-7168.
Closes#7168
42ac01639 [wforget] fix test
208550a3e [wforget] [KYUUBI-7168] Adapt PermanentViewMarker introduced by authz plugin in lineage plugin
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
### Why are the changes needed?
As title.
### How was this patch tested?
UT are modified.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7139 from pan3793/rebalance.
Closes#7139
edb070afd [Cheng Pan] fix
4d3984a92 [Cheng Pan] Fix Spark extension rules to support RebalancePartitions
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Previously, the `HiveScan` class was used to read data. If it is determined to be PARQUET type, the `ParquetScan` from Spark datasourcev2 can be used. `ParquetScan` supports pushfilter down, but `HiveScan` does not yet support it.
The conversation can be controlled by setting `spark.sql.kyuubi.hive.connector.read.convertMetastoreParquet`. When enabled, the data source PARQUET reader is used to process PARQUET tables created by using the HiveQL syntax, instead of Hive SerDe.
close https://github.com/apache/kyuubi/issues/7129
### How was this patch tested?
added unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7130 from flaming-archer/master_parquet_filterdown.
Closes#7129
d7059dca4 [tian bao] Support PARQUET hive table pushdown filter
Authored-by: tian bao <2011xuesong@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Previously, the `HiveScan` class was used to read data. If it is determined to be ORC type, the `ORCScan` from Spark datasourcev2 can be used. `ORCScan` supports pushfilter down, but `HiveScan` does not yet support it.
In our testing, we are able to achieve approximately 2x performance improvement.
The conversation can be controlled by setting `spark.sql.kyuubi.hive.connector.read.convertMetastoreOrc`. When enabled, the data source ORC reader is used to process ORC tables created by using the HiveQL syntax, instead of Hive SerDe.
close https://github.com/apache/kyuubi/issues/7122
### How was this patch tested?
added unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7123 from flaming-archer/master_scanbuilder_new.
Closes#7122
c3f412f90 [tian bao] add case _
2be48909f [tian bao] Merge branch 'master_scanbuilder_new' of github.com:flaming-archer/kyuubi into master_scanbuilder_new
c825d0f8c [tian bao] review change
8a26d6a8a [tian bao] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/KyuubiHiveConnectorConf.scala
68d41969f [tian bao] review change
bed007fea [tian bao] review change
b89e6e67a [tian bao] Optimize UT
5a8941b2d [tian bao] fix failed ut
dc1ba47e3 [tian bao] orc pushdown version 0
Authored-by: tian bao <2011xuesong@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This pull request fixes#6924
## Describe Your Solution 🔧
Bump ranger version to 2.6.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.6.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#7124 from namanagraw/ranger_upgrade.
Closes#6924
bade24db8 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/README.md
650f27319 [namaagra] [KYUUBI apache#6924] Upgrade Spark Ranger plugin to 2.6.0
Lead-authored-by: namaagra <namaagra@visa.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Parent Issue #7040
Support Check Iceberg Alter Table Command
### Why are the changes needed?
- [x] Alter Table Rename To
- [x] Alter Table Set Properties
- [x] Alter Table Unset Properties
- [x] Alter Table Add Column
- [x] Alter Table Rename Column
- [x] Alter Table Alter Column
- [x] Alter Table Drop Column
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#7100 from davidyuan1223/iceberg_alter_table_check.
Closes#7100
4be2210f1 [davidyuan] update
53eda10eb [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
To enhance the MaxScanStrategy in Spark's DSv2 to ensure it only works for relations that support statistics reporting. This prevents Spark from returning a default value of Long.MaxValue, which, leads to some queries failing or behaving unexpectedly.
### How was this patch tested?
It tested out locally.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#7077 from zhaohehuhu/dev-0527.
Closes#7077
64001c94e [zhaohehuhu] fix MaxScanStrategy for datasource v2
Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Iceberg ranger check support branch and tag ddl
### How was this patch tested?
- [x] create branch
- [x] replace branch
- [x] drop branch
- [x] create tag
- [x] replace tag
- [x] drop tag
issue #7068
### Was this patch authored or co-authored using generative AI tooling?
Closes#7069 from davidyuan1223/iceberg_branch_check.
Closes#7068
d060a24e1 [davidyuan] update
1e05018d1 [davidyuan] Merge branch 'master' into iceberg_branch_check
be2684671 [davidyuan] update
231ed3356 [davidyuan] sort spi file
6d2a5bf20 [davidyuan] sort spi file
bc21310cc [davidyuan] update
52ca367f1 [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
#7066
### Why are the changes needed?
Iceberg missing some check, this pr try to fix add partition field check
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
Closes#7065 from davidyuan1223/icerberg_authz.
Closes#7065
be2684671 [davidyuan] update
231ed3356 [davidyuan] sort spi file
6d2a5bf20 [davidyuan] sort spi file
bc21310cc [davidyuan] update
52ca367f1 [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
There were some breaking changes after we fixed compatibility for Spark 4.0.0 RC1 in #6920, but now Spark has reached 4.0.0 RC6, which has less chance to receive more breaking changes.
### How was this patch tested?
Changes are extracted from https://github.com/apache/kyuubi/pull/6928, which passed CI with Spark 4.0.0 RC6
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7061 from pan3793/6920-followup.
Closes#6920
17a1bd9e5 [Cheng Pan] [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x, and it causes `ClassNotFoundException` when user use `kyuubi-spark-authz-shaded_2.12-1.10.1.jar`(built with Ranger 2.5.0)
```
java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/base/ProviderBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:208)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:191)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:465)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1321)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1183)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:148)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:172)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:112)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:563)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:138)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:254)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
at org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension.<init>(RangerSparkExtension.scala:44)
```
### How was this patch tested?
```
$ jar tf kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep org/apache/kyuubi/shade/com/fasterxml
org/apache/kyuubi/shade/com/fasterxml/
org/apache/kyuubi/shade/com/fasterxml/jackson/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AbstractTypeResolver.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty$Type.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$XmlExtensions.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/BeanDescription.class
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7004 from pan3793/authz-jackson.
Closes#7004
cbf870516 [Cheng Pan] fix
4312d9fe5 [Cheng Pan] Include FastXML Jackson into authZ shaded jar
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check missing paimom system producers command, need to support these command
1. create_tag
2. delete_tag
3. rollback
#6979
PS: There has a question about paimon, paimon'sparkCatalog need the currentCatalog Env is the paimon's catalog, use default spark_catalog will throw exception, maybe we should add this hint to the documentation.
such as
If you wanna support producers check with paimon, you need use sql `use $paimon_catalog` to ensure the session currentCatalog is paimon_catalog
PS: paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version
### How was this patch tested?
producers test cases
1. create_tag
2. delete_tag
3. rollback
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6980 from davidyuan1223/paimon_producers.
Closes#6979
90f367c6a [davidyuan] update
c0503cb5f [davidyuan] Merge remote-tracking branch 'origin/paimon_producers' into paimon_producers
993d1dcb8 [davidyuan] Merge branch 'master' into paimon_producers
f68edef41 [davidyuan] producers
58224191b [davidyuan] Merge branch 'master' into paimon_producers
57aac600b [davidyuan] update
cbcdd8dbf [davidyuan] producers
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This PR provides an alternative for RANGER-4125 to cut out JNA dependencies for authZ plugin.
### How was this patch tested?
Pass GHA, and I checked the content of authz-shaded jar
```
$ jar tf extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep Hostname
org/apache/kyuubi/shade/com/kstruct/gethostname4j/Hostname.class
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7003 from pan3793/authz-hostname.
Closes#7003
42e246856 [Cheng Pan] Cut out JNA dependencies for authz plugin
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
In Scala [v2.13.16](https://github.com/scala/scala/releases/tag/v2.13.16)
> JNA is no longer a dependency of `scala-compiler.jar`
Since Spark 4.0 upgrades to Scala 2.13.16, JNA deps have gone too.
```
$ spark-3.5.5-bin-hadoop3-scala2.13 cat RELEASE
Spark 3.5.5 (git revision 7c29c664cdc) built for Hadoop 3.3.4
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.13 -Phadoop-3 -Phive -Phive-thriftserver
$ spark-3.5.5-bin-hadoop3-scala2.13 ls jars | grep jna
jna-5.9.0.jar
```
```
$ spark-4.0.0-bin-hadoop3 cat RELEASE
Spark 4.0.0 (git revision ca56e9ce591) built for Hadoop 3.4.1
Build flags: -B -Pyarn -Pkubernetes -Psparkr -Phadoop-3 -Phive -Phive-thriftserver
$ spark-4.0.0-bin-hadoop3 ls jars | grep jna
<no-output>
```
It's rare to use the non-default Scala version with Spark in practice, we shall respect Spark 4 deps for Scala 2.13 cases.
### How was this patch tested?
Review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6999 from pan3793/authz-scala213.
Closes#6999
18230a2d7 [Cheng Pan] Keep JNA in authz-shaded with Scala 2.13
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
When users switch from Hive to Spark, for sql like INSERT OVERWRITE DIRECTORY AS SELECT, it would be great if small files could be automatically merged through simple configuration, just like in Hive.
### How was this patch tested?
UnitTest
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6991 from Z1Wu/feat/add_insert_dir_rebalance_support.
Closes#6990
2820bb2d2 [wuziyi] [fix] nit
a69c04191 [wuziyi] [fix] nit
951a7738f [wuziyi] [fix] nit
f75dfcb3a [wuziyi] [Feat] add rebalance before InsertIntoHiveDirCommand and InsertIntoDataSourceDirCommand to align with behaviors of hive
Authored-by: wuziyi <wuziyi02@corp.netease.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Avoid unstable test case caused by table size changes, this is likely to happen when upgrading Parquet/ORC/Spark.
### How was this patch tested?
unit test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6989 from wForget/minor_fix.
Closes#6989
9cdd36973 [wforget] address comments
f79fcca0d [wforget] Calculate expected join partitions based on scanned table size
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Cuurently, ranger check test case missing check paimon add new column command, add it.
#6941
### How was this patch tested?
Test Add New Column for paimin with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6945 from davidyuan1223/test_add_new_column_for_paimon.
Closes#6941
f865e132a [davidyuan] test add new column
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column type command, add the test case
#6951
### How was this patch tested?
Test ranger check paimon changing column type command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6956 from davidyuan1223/test_changing_column_type.
Closes#6951
9d5140e81 [davidyuan] Merge branch 'master' into test_changing_column_type
e4f8974d8 [davidyuan] test changing column type
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
The feature `spark.sql.watchdog.forcedMaxOutputRows` is a little bit hacky, it's actually a manually implemented "limit pushdown", we already have a simple and more reliable way to achieve that by using `kyuubi.operation.result.max.rows`.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6983 from pan3793/rm-forcedMaxOutputRows.
Closes#6983
5e0707955 [Cheng Pan] Remove support for spark.sql.watchdog.forcedMaxOutputRows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Support ranger check with paimon Update & Delete & MergeInti Table Command
### How was this patch tested?
Test Paimon Update & Delete Table Command with ranger check
#6973
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6974 from davidyuan1223/update_table.
Closes#6973
3bd607300 [davidyuan] update
1d68494ce [davidyuan] Test MergeInto
a27ea633f [davidyuan] Test MergeInto
56638f47c [davidyuan] Merge branch 'master' into update_table
1c3464df5 [davidyuan] Test Table Update & Delete
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Simple refactoring to clean up the code for the Spark 3.5 extension, e.g., remove unnecessary `*Base` `*Helper` abstraction layers, remove code for legacy Spark versions.
Note: I don't touch `ForcedMaxOutputRows*` because I'm going to remove it in the next PR.
Preparation for Spark 4.0 support.
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6975 from pan3793/spark-ext-35-cleanup.
Closes#6975
b5a94a680 [Cheng Pan] nit
c729e268c [Cheng Pan] fix
1087ac709 [Cheng Pan] Clean up code for Spark 3.5 extension
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Ranger check with paimon missing check drop columns command
#6947
### How was this patch tested?
Test ranger check with drop columns commad
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6952 from davidyuan1223/test_dropping_columns.
Closes#6947
27e7b3836 [davidyuan] Merge remote-tracking branch 'origin/master' into test_dropping_columns
83815ef3b [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently, ranger check missing paimon Table Query command, add the test cases
#6971
### How was this patch tested?
Test Paimon Table Query & Batch Time Travel
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6972 from davidyuan1223/test_query.
Closes#6972
7aa821215 [davidyuan] Test Table Query
ae1b75e85 [davidyuan] Test Table Query
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Range check Test case missing paimon changing column comment, add the test case
#6948
### How was this patch tested?
Test Ranger check with paimon changing column comment command
### Was this patch authored or co-authored using generative AI tooling?
No
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#6953 from davidyuan1223/test_changing_column_comment.
Closes#6948
983720e78 [davidyuan] test changing column comment
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently , ranger check missing paimon insert table command, add test cases
#6958
### How was this patch tested?
1. Test INSERT INTO:
1.1 table1OnlyUserForNs could select table1, try to insert table1
1.2 someone has no any permission, try to insert table1
2. Test INSERT OVERWRITE:
2.1 table1OnlyUserForNs could select table1, try to insert table2
2.2 someone has no any permiession, try select table1 then insert table2
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6959 from davidyuan1223/test_insert.
Closes#6958
d1f41ba81 [davidyuan] Merge branch 'master' into test_insert
b56e701d4 [davidyuan] Test Insert Table
8306210ee [davidyuan] update
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Test Authz Support paimon rename table name command privilege check
#6936
### How was this patch tested?
Test Authz Support paimon rename table name command privilege check
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6937 from davidyuan1223/check_authz_paimon_rename_table.
Closes#6936
797d1c489 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
bc3c823a3 [davidyuan] Merge remote-tracking branch 'origin/master' into check_authz_paimon_rename_table
6205670d2 [davidyuan] add renameTable to command_spec.json
e4b241ef5 [davidyuan] Merge branch 'master' into check_authz_paimon_rename_table
5fec3bcb7 [davidyuan] test paimon rename table name command
30d09418c [davidyuan] test paimon rename table name command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon adding column position command, add the test case
#6949
### How was this patch tested?
Test ranger check with paimon adding column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6954 from davidyuan1223/test_adding_column_position.
Closes#6949
262ecaaca [davidyuan] Merge remote-tracking branch 'origin/master' into test_adding_column_position
154765fc3 [davidyuan] Merge branch 'master' into test_adding_column_position
4ebf985a9 [davidyuan] test adding column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Currently range check missing check UnsetTableProperties command, we need add it to the range check.
#6940
### How was this patch tested?
Use paimon removing table properties to test this command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6944 from davidyuan1223/test_remove_table_properties.
Closes#6940
4f24d7d6a [davidyuan] Merge branch 'master' into test_remove_table_properties
11d3773ed [davidyuan] test unset table properties command
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Ranger check test case missing paimon changing column position command, add the test case
#6950
### How was this patch tested?
Test ranger check with paimon changing column position command
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6955 from davidyuan1223/test_changing_column_position.
Closes#6950
520b5377f [davidyuan] Merge branch 'master' into test_changing_column_position
1eed87346 [davidyuan] test changing column position
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Test Spark 3.5.5 Release Notes
https://spark.apache.org/releases/spark-release-3-5-5.html
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6939 from pan3793/spark-3.5.5.
Closes#6939
8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Currently, ranger check for paimon missing rename column name command, add the test case
#6942
### How was this patch tested?
Test Paimon Rename column name with ranger
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6946 from davidyuan1223/test_rename_column_name.
Closes#6942
8e49eb0ab [davidyuan] test rename column name
Authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
AUTHZ Test Create Partitioned Table for PAIMON, check that has support the command
#6923
### How was this patch tested?
est Authz for paimon with create partitioned table command. Check the permission
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6931 from davidyuan1223/support_create_with_parition_for_paimon.
Closes#6923
61f7560d3 [Cheng Pan] Merge branch 'master' into support_create_with_parition_for_paimon
ffb79376f [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
b0829795a [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
4b160d720 [davidyuan] support create partition table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test Add/Change Table properties for PAIMON, check that has support the command
https://github.com/apache/kyuubi/issues/6932
### How was this patch tested?
Test Add/Change properties SQL
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6933 from davidyuan1223/test_alter_tableproperties_for_paimin.
Closes#6932
4d64fbf23 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
c861a778b [davidyuan] support add/change table properties for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
AUTHZ Test CTAS for Paimon to check it support this command, the related issue is https://github.com/apache/kyuubi/issues/6921
### How was this patch tested?
Test Authz for paimon with create table as command. Check the permission.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6922 from davidyuan1223/support_create_table_as_for_paimon_check.
Closes#6921
7bfd6ad49 [david yuan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
a9ce20cc4 [davidyuan] support create table as for paimon
Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com>
Co-authored-by: david yuan <davidyuan1223@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Paimon does not seem to support Scala 2.13
### How was this patch tested?
Pass GHA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6925 from pan3793/authz-paimon-scala212.
Closes#6925
865a7dd72 [Cheng Pan] fix
971d23273 [Cheng Pan] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
499f10ab0 [Cheng Pan] Only run Paimon authz tests with Scala 2.12
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Why are the changes needed?
## Issue reference:
https://github.com/apache/kyuubi/issues/6912
## How to reproduce the issue?
The changes in this PR will avoid a wrong result when generating the instance of org.apache.kyuubi.plugin.lineage.Lineage, in the certain case as follows:
step 1: create a temporary view from a file
step 2: insert into a table by selecting from the temporary view in step 1
step 3: generate the lineage when executing the insert statement in step 2
In detail, please see the UT code submission in this patch.
## The issue analysis
Let's see the current code when getting the Lineage object by resolving a LogicalPlan object:
<img width="694" alt="image" src="https://github.com/user-attachments/assets/65256a0d-320d-4271-968f-59eafb74de9f" />
According to the above logic, a None org.apache.kyuubi.plugin.lineage.Lineage object will be generated due to "try-catch" self-protection, in this certain case. This None object will lead to problems in the following 2 scenes:
### Unit Test Environment
In Unit Test, when the code runs here a "None.get" exception will be raised:
<img width="682" alt="image" src="https://github.com/user-attachments/assets/102dc9bd-294f-4b1e-b1c6-01b6fee50fed" />
Here's the runtime exception stack:
```
None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineageWithoutExecuting(SparkSQLLineageParserHelperSuite.scala:1485)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$83(SparkSQLLineageParserHelperSuite.scala:1465)
```
### Production Environment
This Lineage object cannot be used in the production environment because it has a None value which lacks some necessary lineage information. The right content of the Lineage instance in the above case should be:
```
inputTables(List())
outputTables(List(spark_catalog.test_db.test_table_from_dir))
columnLineage(List(ColumnLineage(spark_catalog.test_db.test_table_from_dir.a0,Set()), ColumnLineage(spark_catalog.test_db.test_table_from_dir.b0,Set())))
```
a newly added test case(test directory to table) passed after this issue is fixed.
# How to fix the issue?
Add a "Empty judgment" logic. In detail, please see the code submission in this patch.
# How was this patch tested?
1. by adding a new test case in UT code and make sure it passes
2. by submitting a Spark application including the SQL of this case in the production environment, and make sure a right Lineage instance is generated, instead of a None object
# Was this patch authored or co-authored using generative AI tooling?
No
Closes#6911 from xglv1985/fix_spark_lineage_runtime_exception.
Closes#6912
13a71075d [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
4e89b95cd [Cheng Pan] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
59b350bfb [xglv1985] fix a runtime exception when generate column lineage tuple--more readable code
52bc0288d [xglv1985] fix a runtime exception when generate column lineage tuple--spotless sytle
fea6bbc0d [xglv1985] fix a runtime exception when generate column lineage tuple--remove tab from UT code
901879095 [xglv1985] fix a runtime exception when generate column lineage tuple--unit test
fbb4df879 [xglv1985] fix a runtime exception when generate column lineage tuple
Lead-authored-by: xglv1985 <xglv1985@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Added a service definition for spark which in turn enables the creation of a default policy for the spark service.
Default policy will block access until another policy is downloaded from Apache Ranger.
### How was this patch tested?
Tested manually.
Configure Kyuubi Authz plugin. Do not start Apache Ranger, it must not be reachable.
Make sure that policy cache is empty.
Start Kyuubi engine and try to query any tables. The default policy should not allow any access.
Previously the access was not restricted because there wasn't a default policy defined.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6902 from developster/master.
Closes#6901
feb6ebf61 [Octavian Ciubotaru] Default policy for spark
Authored-by: Octavian Ciubotaru <ociubotaru@developmentgateway.org>
Signed-off-by: Kent Yao <yao@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/kyuubi/pull/5852 to Spark 3.3, to enhance MaxScanStrategy to include support for the datasourcev2 in Spark 3.3
### How was this patch tested?
Add some UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6862 from zhaohehuhu/dev-1225.
Closes#6862
c745eda14 [zhaohehuhu] MaxScanStrategy supports DSv2 in Spark 3.3
Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
Backport https://github.com/apache/kyuubi/pull/5852 to Spark 3.4, to enhance MaxScanStrategy to include support for the datasourcev2 in Spark 3.4
### How was this patch tested?
Add some UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6857 from zhaohehuhu/dev-1224.
Closes#6857
c72c62984 [zhaohehuhu] remove the import
dfbf2bc2d [zhaohehuhu] MaxScanStrategy supports DSv2 in Spark 3.4
Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
```
export JAVA_HOME=/path/of/openjdk-17
build/mvn clean install -DskipTests -Dmaven.scaladoc.skip=false
```
```
[INFO] --- scala-maven-plugin:4.9.2:doc-jar (attach-scaladocs) kyuubi-server-plugin ---
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.20,1.7.19,null)
error: fatal error: object scala in compiler mirror not found.
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Successfully run the build command
```
export JAVA_HOME=/path/of/openjdk-17
build/mvn clean install -DskipTests -Dmaven.scaladoc.skip=false
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6820 from pan3793/scaladoc.
Closes#6820
f5cee3429 [Cheng Pan] Explicitly disable attach-scaladocs for pure Java modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Preparing v1.11.0-SNAPSHOT after branch-1.10 cut
```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6769 from bowenliang123/bump-1.11.
Closes#6769
6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6754
## Describe Your Solution 🔧
Right now in RuleAuthorization we use an ArrayBuffer to collect access requests, which is very slow because each new PrivilegeObject needs to be compared with all access requests.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Add benchmark
Before
```sh
Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark 181863 189434 NaN -0.0 -181863368958.0 1.0X
````
#### Behavior With This Pull Request 🎉
After
```sh
Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6
Apple M3
Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
50000 files benchmark 1281 1310 33 -0.0 -1280563000.0 1.0X
```
#### Related Unit Tests
Exists UT
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6758 from wankunde/ranger2.
Closes#6754
9d7d1964b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
88b9c049b [wankun] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/spark/sql/RuleAuthorizationBenchmark.scala
20c55fbeb [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
f5a3b6ca5 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
9793249de [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
d86b01f9c [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
b904b491b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
aad08a6bb [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
1374604bc [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
01e15c149 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
805e8a9c0 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml
e19817943 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests
Lead-authored-by: wankunde <wankunde@163.com>
Co-authored-by: wankun <wankun@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Fix a ClassNotFound issue.
```
java.lang.NoClassDefFoundError: org/apache/kyuubi/shade/javax/ws/rs/core/Cookie
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Verified manually.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6723 from pan3793/6638-followup.
Closes#6638
56e9842e0 [Cheng Pan] [KYUUBI #6638] authz shaded should include jsr311-api
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6666
## Describe Your Solution 🔧
Bump ranger version to 2.5.0
Release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.5.0+-+Release+Notes
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6692 from Madhukar525722/ranger_upgrade.
Closes#6666
88e1e12c5 [madlnu] [KYUUBI #6666] Upgrade spark ranger plugin to 2.5.0
Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Spark 4.0.0-preview2 RC1 passed the vote
https://lists.apache.org/thread/4ctj2mlgs4q2yb4hdw2jy4z34p5yw2b1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6699 from pan3793/spark-4.0.0-preview2.
Closes#6699
2db1f645d [Cheng Pan] 4.0.0-preview2
42055bb1e [Cheng Pan] fix
d29c0ef83 [Cheng Pan] disable delta test
98d323b95 [Cheng Pan] fix
2e782c00b [Cheng Pan] log4j-slf4j2-impl
fde4bb6ba [Cheng Pan] spark-4.0.0-preview2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
- Apache Commons Lang2 is no longer actively maintained and not used by Kyuubi modules
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6672 from bowenliang123/remove-commonlang2.
Closes#6672
34cda170a [liangbowen] remove common lang2
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5402
## Describe Your Solution 🔧
When facing out-of-control memory management in Spark engine, we typically use JVMkill as a remedy by killing the process and generating a heap dump for post-analysis. However, even with jvmkill protection, we may still encounter issues caused by JVM running out of memory, such as repeated execution of Full GC without performing any useful work during the pause time. Since the JVM does not exhaust 100% of resources, JVMkill will not be triggered.
So introducing JVMQuake provides more granular monitoring of GC behavior, enabling early detection of memory management issues and facilitating fast failure.
You can use the following configuration to enable jvmQuake plugins：
```
spark.plugins=org.apache.spark.kyuubi.jvm.quake.KyuubiJVMQuakePlugin
```
| configuration | default | comment |
| ---- | ---- | ---- |
| spark.driver.jvmQuake.enabled | false | when true, enable driver jvmQuake |
| spark.executor.jvmQuake.enabled | false | when true, enable executor jvmQuake |
| spark.driver.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.executor.jvmQuake.heapDump.enabled | false | when true, enable jvm heap dump when jvmQuake rearch the threshold |
| spark.jvmQuake.dumpThreshold | 100 | The number of seconds to dump memory |
| spark.jvmQuake.killThreshold | 200 | The number of seconds to kill process |
| spark.jvmQuake.exitCode | 502 | The exit code of kill process |
| spark.jvmQuake.heapDumpPath | /tmp/kyuubi_jvm_quake/apps | The path of heap dump |
| spark.jvmQuake.checkInterval | 3 | The number of seconds to check jvmQuake |
| spark.jvmQuake.runTimeWeight | 1.0 | The weight of rum time |
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6572 from yoock/features/kyuubi-jvm-quake.
Closes#5402
84361ce8f [王龙] add jvm quake
Authored-by: 王龙 <wanglong16@xiaomi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
