### _Why are the changes needed?_
This PR removes the non-existent query specs in DeleteFromTable and UpdateTable, and all its derives, such as iceberg and hudi.
Timely stop abuse.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
no
Closes#5512 from yaooqinn/du.
Closes#5512
627768666 [Kent Yao] [AuthZ] Remove the non-existent query spec in Deletes and Updates
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#5472
For cases such as `SELECT COUNT(*) FORM XXX`, when `XXX` is table, will extract table's column, if `XXX` is view, only pass the view name
```
test("[KYUUBI #xxx] ") {
val db1 = defaultDb
val table1 = "table1"
val view1 = "view1"
withSingleCallEnabled {
withCleanTmpResources(Seq((s"$db1.$table1", "table"), (s"$db1.$view1", "view"))) {
doAs(admin, sql(s"CREATE TABLE IF NOT EXISTS $db1.$table1 (id int, scope int)"))
doAs(admin, sql(s"CREATE VIEW $db1.$view1 AS SELECT * FROM $db1.$table1"))
val e1 = intercept[AccessControlException](
doAs(someone, sql(s"SELECT count(*) FROM $db1.$table1").show()))
assert(e1.getMessage.contains(
s"does not have [select] privilege on [$db1/$table1/id,$db1/$table1/scope]"))
val e2 = intercept[AccessControlException](
doAs(someone, sql(s"SELECT count(*) FROM $db1.$view1").show()))
assert(e2.getMessage.contains(
s"does not have [select] privilege on [$db1/$view1]"))
}
}
}
```
After this pr, view will also extract columns.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5473 from AngersZhuuuu/KYUUBI-5742.
Closes#5472
6575ad2ba [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
8a264a810 [Angerszhuuuu] Update
afa43d356 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5742
9f0bfb2a7 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5742
275784478 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
88f3d3282 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5742
cd62c8d20 [Angerszhuuuu] update
55be7da95 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
4200ed3ed [Angerszhuuuu] [KYUUBI #5742] Permanent View should pass column when child plan no output
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
It's easy for developers to check a member or method from a Java class using a code viewer or online Java doc.
The current debug msg is kinda noisy for them to locate the key information they want, which is
Java class and field name. It makes our debug log unreadable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5497 from yaooqinn/debug.
Closes#5497
3ad42d523 [Kent Yao] addr comments
23e8e7a09 [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
b7a7fbb7a [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
9f94c62e4 [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
78a66a33c [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To fix#5475
In issue #5417 we fixed the problem that AUTHZ will still check scalar-subquery/in-subquery in permanent will.
But we just ignore the check, the subquery still will run, in this PR, we record the permanent view's visited column to check the permanent view's privilege to avoid extra execution effort.
For the test `[KYUUBI #5417] should not check scalar-subquery in permanent view` I print all the plan that pass to privilege builder as below
<img width="1398" alt="截屏2023-10-19 下午4 05 46" src="https://github.com/apache/kyuubi/assets/46485123/b136bb47-816c-4066-aba7-a74cbe323f7d">
before this pr
<img width="1310" alt="截屏2023-10-19 下午4 15 29" src="https://github.com/apache/kyuubi/assets/46485123/aa2e3cfe-bca7-493d-a364-b2c196c76c3a">
This two graph shows this pr deny the execution of subquery when we don't have the veiw's privilege
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5476 from AngersZhuuuu/KYUUBI-5475.
Closes#5475
e1f7920f3 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5475
3bfd9e677 [Angerszhuuuu] update
6b8c0e6e5 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5475
f7585a451 [Angerszhuuuu] Update PrivilegesBuilder.scala
faea9c699 [Angerszhuuuu] [KYUUBI #5475] Authz check permanent view's subquery should check view's correct privilege
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To fix#5492
When we use saveAsTable and write as a DataSource table, since CreateTableAsDataSource command's catalogTable was directly constructed by identifier and only will check when executing, so here authz will miss db information as below case
This fixes this issue by following the Spark's code
<img width="1256" alt="截屏2023-10-21 下午3 37 37" src="https://github.com/apache/kyuubi/assets/46485123/d7c66db4-400b-4637-b75a-973a8a8a9968">
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5493 from AngersZhuuuu/KYUUBI-5492.
Closes#5492
46867aff2 [Angerszhuuuu] update
7a4c86fe2 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5492
f56f29924 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
7bcfc236a [Angerszhuuuu] Merge branch 'master' into KYUUBI-5492
6f2472811 [Angerszhuuuu] [KYUUBI #5492] saveAsTable create DataSource table miss db info
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#5447
Remove unrelated debug change in tableExtractor
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5494 from AngersZhuuuu/KYUUBI-5447-FOLLOWUP.
Closes#5447
90a0bcd91 [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableExtractors.scala
85941e26c [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Remove unrelated debug change in tableExtractor
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5427 from Yikf/shade-authz.
Closes#5427
d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin
Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
Apache Paimon is an incubating Apache project of data lake platform for high-speed data ingestion, changelog tracking and efficient real-time analytics.
- Initial support for Paimon tables in Authz plugin
- Create Table Command: https://paimon.apache.org/docs/master/engines/spark3/#create-table
- Paimon `0.5.0-incubating` supports Spark 3.1/3.2/3.3/3.4.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5462 from bowenliang123/authz-paimon-createtable.
Closes#5462
8393f6ecc [liangbowen] remove spark-authz-paimon-test profile, use 3.4 as spark binary version for Paimon
a4e01a2a8 [liangbowen] update workflow inclusions
f372882aa [liangbowen] update workflow exclusions
7785dec52 [liangbowen] update workflow include
64ee20106 [liangbowen] update
73edea38f [liangbowen] add spark-authz-paimon-test profile and excluded for Spark 3.5
de58605bb [liangbowen] nit
81a4a4370 [liangbowen] nit
eb2de8359 [Bowen Liang] exclude paimon tests for spark-3.5
4f58d7727 [Bowen Liang] fix typo
cf360e6da [Bowen Liang] drop database
af8ffda14 [Bowen Liang] drop database
dc3ee9462 [Bowen Liang] support Paimon create table
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5457. Kyuubi authz support hudi repair table commands
- RepairHoodieTableCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/RepairHoodieTableCommand.scala
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5458 from AngersZhuuuu/KYUUBI-5457.
Closes#5457
7dcb932ff [Angerszhuuuu] Merge branch 'master' into KYUUBI-5457
85c209fa9 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
4319763df [Angerszhuuuu] [KYUUBI #5457] [AUTHZ] Support RepairTable Commands for Hudi
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#5417
If there is is a view with in-subquery, authz will still request this in-subquery's interval privilege, it's not we want.
For view
```
CREATE VIEW db.view1
AS
WITH temp AS (
SELECT max(scope) max_scope
FROM db1.table1)
SELECT id as new_id FROM db1.table2
WHERE scope in (SELECT max_scope FROM temp)
```
When we query the view
```
SEELCT * FROM db.view1
```
Before this pr, since spark will first execute subquery, it will first request `[default/table1/scope]` then request `[default/view1/new_id]`
after this pr, it only request `[default/view1/new_id]`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
no
Closes#5442 from AngersZhuuuu/KYUUBI-5417-FOLLOWUP.
Closes#5417
6919903cb [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala
5097d8059 [Angerszhuuuu] [KYUUBI #5417] should not check in-subquery in permanent view
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This pr aims to fix https://github.com/apache/kyuubi/issues/5414.
`HiveReader` initialization incorrectly uses the global hadoopConf as hiveconf, which causes reader to pollut the global hadoopConf and cause job read failure.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5424 from Yikf/orc-read.
Closes#5414
d6bdf7be4 [yikaifei] [KYUUBI #5414] Reader should not polluted the global hiveconf instance
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Iceberg 1.3.0 use Spark's `ReplaceData` since 3.4 as Logical plan for "DELETE FROM", instead of Iceberg's `DeleteFromTable` in Spark3.4
- Requiring select privilege for input table, even it's the same with output table
- compatible Spark 3.4 and 3.5, subquies in the plan
- enable iceberg test for authz plugin on Spark 3.4 and 3.5
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4916 from bowenliang123/authz-replacedata.
Closes#4916
658917752 [Bowen Liang] Merge branch 'master' into authz-replacedata
17f5d84dd [liangbowen] update
25009908a [Bowen Liang] ut of user with select only for DELETE FROM
fa77cea15 [Bowen Liang] ut
fd9bb8f3a [Bowen Liang] update
f9cfb98a9 [Bowen Liang] assertion for Spark 3.5
f574e0da3 [Bowen Liang] assertion for Spark 3.5
2449c27de [Bowen Liang] bring back single call ut for someone
78786988d [Bowen Liang] bring back single call ut for someone
b8e4a6319 [wforget] fix style
0e26c08b4 [wforget] fix IcebergCatalogRangerSparkExtensionSuite with spark-3.5
02781781f [wforget] enable iceberg tests in spark-3.5
215e1b861 [wforget] fix TableCommands
d019b1632 [wforget] followup
ae17e076b [wforget] Merge remote-tracking branch 'origin/master' into authz-replacedata
b88f77355 [Bowen Liang] update
febcb3ee5 [Bowen Liang] isSparkV34OrGreater
91d41b438 [Bowen Liang] replace data
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: wforget <643348094@qq.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Fix#5417
If there is is a view with subquery, authz will still request this subquery's interval privilege, it's not we want.
For view
```
CREATE VIEW db.view1
AS
WITH temp AS (
SELECT max(scope) max_scope
FROM db1.table1)
SELECT id as new_id FROM db1.table2
WHERE scope = (SELECT max_scope FROM temp)
```
When we query the view
```
SEELCT * FROM db.view1
```
Before this pr, since spark will first execute subquery, it will first request `[default/table1/scope]` then request `[default/view1/new_id]`
after this pr, it only request `[default/view1/new_id]`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5418 from AngersZhuuuu/KYUUBI-5417.
Closes#5417
e2669faea [Angerszhuuuu] Update tableExtractors.scala
bc72cfc57 [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala
1731b9317 [Angerszhuuuu] Update RuleEliminateViewMarker.scala
282999ee2 [Angerszhuuuu] follow comment
6b37aaa7f [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala
d03354d58 [Angerszhuuuu] Revert "update"
7a96627e4 [Angerszhuuuu] update
78a32b3a5 [Angerszhuuuu] follow comment
79e07ab24 [Angerszhuuuu] Update PrivilegesBuilder.scala
518c2b394 [Angerszhuuuu] update
d033624ea [Angerszhuuuu] update
54ff954f0 [Angerszhuuuu] update.
1119f78f6 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
1918381ac [Angerszhuuuu] Add UT
7723f9002 [Angerszhuuuu] [KYUUBI #5417]Authz will still check source table when persist view contains a subquery
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Remove the deprecated usage.
c780db754e/src/java.base/share/classes/java/lang/Class.java (L534-L535)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5426 from cxzl25/newInstance.
Closes#5426
dcb679b95 [sychen] avoid use class.newInstance directly
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#5284.
Hudi also is a common used data format, since kyuubi already support iceberg and delta, we should also support hudi.
In this pr we support hoodie sql about ALTER COMMAND in authz
In this PR we use default Hudi version 0.14.0.
We support from spark 3.1 to spark 3.4, since Hudi don't support spark 3.5 yet
- [x] spark 3.1
- [x] spark 3.2
- [x] spark 3.3
- [x] spark 3.4
- [ ] spark 3.5
Also since Hudi only supports Scala 2.12, I also made Hudi as a separate profile to avoid importing Hudi when enable Scala 2.13
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5287 from AngersZhuuuu/KYUUBI-5284.
Closes#5284
f171e11af [Angerszhuuuu] Update pom.xml
3f57a3dc5 [Angerszhuuuu] follow comment
f6c764028 [Angerszhuuuu] follow comment
51797e25c [Angerszhuuuu] trigger
b3c059af9 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
3510e7601 [liangbowen] remove scope in dependencyManagement
14ea0d498 [liangbowen] change to use `spark.binary.version` for hudi dependency by default
354260eb0 [liangbowen] remove the abbreviation tite
658bddbab [liangbowen] remove clarification and use ALTERTABLE_PROPERTIES for opType
150edcd40 [Angerszhuuuu] update
30c417b19 [Angerszhuuuu] trigger
56e5cb17b [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
fe9b75270 [Angerszhuuuu] update
888943831 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
db749a277 [Angerszhuuuu] update
9b09e78c2 [Angerszhuuuu] Update HudiCommands.scala
87de62e52 [Angerszhuuuu] follow comment
2d551d112 [Angerszhuuuu] Update master.yml
89082e06b [Angerszhuuuu] Update master.yml
7c7846378 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
d32ca9839 [Angerszhuuuu] Update master.yml
ec43e2a7b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
b3611fd3e [Angerszhuuuu] update
2a0dfa74f [Angerszhuuuu] Update AuthZUtils.scala
45ee9e251 [Angerszhuuuu] update
0560a5e14 [Angerszhuuuu] Update pom.xml
97c50f622 [Angerszhuuuu] update
f57ee0093 [Angerszhuuuu] Update table_command_spec.json
fb72197e6 [Angerszhuuuu] update
2154cf928 [Angerszhuuuu] trigger
44469359f [Angerszhuuuu] trigger
b0e768cb8 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
83795ed63 [Angerszhuuuu] Update pom.xml
eed190f92 [Angerszhuuuu] update
361660145 [Angerszhuuuu] update
1ed1f3ab6 [Angerszhuuuu] Update
7ee3c7dd5 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
ee0916f63 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
010260fa4 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
c11d02def [Angerszhuuuu] update
b84f91f65 [Angerszhuuuu] update
42fbb0ffa [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
c1346adb1 [Angerszhuuuu] update
2ec63ae94 [Angerszhuuuu] Update pom.xml
39bce7468 [Angerszhuuuu] update
c70b0ea2f [Angerszhuuuu] Update pom.xml
e1d85ff77 [Angerszhuuuu] Update pom.xml
59012ac25 [Angerszhuuuu] Update pom.xml
a46de65b5 [Angerszhuuuu] Update HoodieTest.java
b8173b893 [Angerszhuuuu] update
055713329 [Angerszhuuuu] Update table_command_spec.json
d7b21e820 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
0a93ff794 [Angerszhuuuu] [KYUUBI #5284] Kyuubi authz support Hoodie Alter Table Command
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- `DROP TABLE` for Iceberg tables only removes the table from catalog by default, which may contaminates other tests with same table
- Enable PURGE option for dropping Iceberg and Hive table
- Iceberg Spark DDL `DROP TABLE ... PURGE`
- To drop the table from the catalog and delete the table’s contents
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5323 from bowenliang123/iceberg-purge.
Closes#5323
ce4188dd2 [Bowen Liang] purge
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5407 .
Follow up for https://github.com/apache/kyuubi/pull/5248 .
Add some UT for snapshot management procedures. These procedures require alter permissions.
1. rollback_to_snapshot (https://iceberg.apache.org/docs/latest/spark-procedures/#rollback_to_snapshot):
Usage: `CALL catalog_name.system.rollback_to_snapshot('db.sample', 1)
`
Meaning: rollback a table to a specific snapshot ID.
2. rollback_to_timestamp (https://iceberg.apache.org/docs/latest/spark-procedures/#rollback_to_timestamp)
Usage: `CALL catalog_name.system.rollback_to_timestamp('db.sample', TIMESTAMP '2021-06-30 00:00:00')`
Meaning: rollback the table to the latest snapshot less than time.
3. set_current_snapshot (https://iceberg.apache.org/docs/latest/spark-procedures/#set_current_snapshot)
Usage: `CALL catalog_name.system.set_current_snapshot('db.sample', 1)`
Meaning: Set a table to a specific snapshot ID.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5394 from yabola/add_call.
Closes#5407
98b7492e0 [Bowen Liang] split into 3 ut for snapshot management
23fe49ae4 [Bowen Liang] refactor ut
8ba97c6ef [chenliang.lu] Add UT for checking previleges in iceberg call snapshot management
Lead-authored-by: chenliang.lu <marssss2929@gmail.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The cleanup follow-up for #5362, which removed the Spark 3.0 tests for Authz plugin.
Remove the `isSpark31OrGreater` and Spark 3.0 specific implementations in Authz plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5399 from bowenliang123/authz-31greater.
Closes#5399
db4369b13 [Bowen Liang] import
cc38b1c1e [Bowen Liang] fix MASK_SHOW_FIRST_4
cf3ef4e1c [Bowen Liang] remove isSparkV31OrGreater
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.
This PR to fix the performance issue on our side.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5400 from ITzhangqiang/KYUUBI_5365.
Closes#5365
dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#5362 .
Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5363 from AngersZhuuuu/KYUUBI-5362.
Closes#5362
d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar in the following order:
1. `SPARK_SCALA_VERSION` system env
2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5196 from bowenliang123/scala213-test.
Closes#5196
97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
It is basically copied from `kyuubi-extension-spark-3-4`.
### _How was this patch tested?_
Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5336 from wForget/dev_spark_3_5.
Closes#5336
7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/5317#issue-1904751001
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5319 from zhaomin1423/fixhive-connector.
Closes#5317
02e5321dc [Cheng Pan] nit
cadabf4ab [Cheng Pan] nit
d38832f40 [zhaomin] improve
ee5b62d84 [zhaomin] improve
794473468 [zhaomin] improve
e3eca91fb [zhaomin] add tests
d9302e2ba [zhaomin] [KYUUBI #5317] [Bug] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
0bc8ec16f [zhaomin] [KYUUBI #5317] [Bug] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
Lead-authored-by: zhaomin <zhaomin1423@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Since AnalyzeTableCommand also update table's metadata, since alter command also need table write privilege, AnalyzeTableCommand need too
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5272 from AngersZhuuuu/KYUUBI-5271.
Closes#5271
ad5c70403 [Angerszhuuuu] Merge branch 'KYUUBI-5271' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5271
a5932b7e8 [Angerszhuuuu] Update TableCommands.scala
97ee3299c [Angerszhuuuu] Merge branch 'master' into KYUUBI-5271
cdd8100fd [Angerszhuuuu] Update PrivilegesBuilderSuite.scala
5f110563e [Angerszhuuuu] update
92c30454b [Angerszhuuuu] Revert "Update TableCommands.scala"
504ff2a26 [Angerszhuuuu] Update TableCommands.scala
6c4233680 [Angerszhuuuu] [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- Extract common assertion method for verifying file contents
- Ensure integrity of the file by comparing the line count
- Correct the script name for Spark engine KDF doc generation from `gen_kdf.sh` to `gen_spark_kdf_docs.sh`
- Add `gen_hive_kdf_docs.sh` script for Hive engine KDF doc generation
- Fix incorrect hints for Ranger spec file generation
- shows the line number of the incorrect file content
- Streamingly read file content by line with buffered support
- Regeneration hints:
<img width="656" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/d1a7cb70-8b63-4fe9-ae27-80dadbe84799">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5275 from bowenliang123/doc-regen-hint.
Closes#5275
9af97ab86 [Bowen Liang] implicit source position
07020c74d [liangbowen] assertFileContent
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Before this PR, Kyuubi skips authentication for the call command in iceberg. After this PR, it needs `alter` permission to use this command.
This PR supports checking iceberg call rewrite_data_files command:
- CALL ice.system.rewrite_data_files(table => 'ice.lcl.test_ice', options => map('min-input-files','2'))
- some other call statements in iceberg ( https://iceberg.apache.org/docs/latest/spark-procedures ) may be supported but needs to be further determined. The next phase is planned to support all commands as documented.
When the `rewrite_data_files` command is called, there are two situations:
1. When the triggering conditions are not be met(such as input-files < min-input-files) , there is only one logical plan. We need to check one logical plan permission:
```
* == Physical Plan 1 ==
* Call (1)
```
2. When the triggering conditions are met, two logical plan will be triggered. We should check two logical plans permission.
```
* == Physical Plan 1 ==
* Call (1)
*
* == Physical Plan 2 ==
* AppendData (3)
* +- * ColumnarToRow (2)
* +- BatchScan local.iceberg_ns.call_command_table (1)
```
The second plan (`AppendData`) is a bit special, causing us to have to modify the logic of extracting table names in `DataSourceV2RelationTableExtractor`. The `DataSourceV2Relation#identifier` in `AppendData` will be rewrite task id, so it is a more appropriate way to parse the table name from `DataSourceV2Relation#table`
<img width="1318" alt="image" src="https://github.com/apache/kyuubi/assets/31469905/efc815cb-8dfe-4a9f-8e24-d62883ef6eff">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?
No.
Closes#5248 from yabola/master.
Closes#5248
f3a2d736e [liangbowen] initDataFilesCount
6c0da35f7 [Bowen Liang] skip input checks
4a5ab6da4 [Bowen Liang] comment
850100b6b [Bowen Liang] update
c3158d25a [Bowen Liang] change operation type to ALTERTABLE_PROPERTIES
8d5c8b1f8 [Bowen Liang] rewriteDataFiles2
2d79e1730 [Bowen Liang] nit
36f1d9d60 [chenliang.lu] update description for ut
bd99e0dfa [chenliang.lu] update description for ut
a95e1bb4f [Bowen Liang] nit
7a0089907 [Bowen Liang] nit
06566815b [Bowen Liang] revert opType
1dd211b59 [Bowen Liang] withSingleCallEnabled
657c9dcd3 [Bowen Liang] Merge branch 'master' into yabola/master
cc7dda1f7 [Bowen Liang] update policy
aecfd50cf [Bowen Liang] update table_spec
66ddd89b1 [Bowen Liang] update
b1e12789f [Bowen Liang] remove defaultBob
b8f4fabaf [Bowen Liang] change operation type
465fa48a2 [Bowen Liang] change ut to iceberg namespace
0cfe285ea [Bowen Liang] comment
2b835c943 [Bowen Liang] remove implicit casting in StringTableExtractor
280ad029b [Bowen Liang] removed snapshotCommand tests
83be1d0fd [Bowen Liang] removed snapshotCommand tests
49bd8e14c [Bowen Liang] assertion on all required checks
d65915eb9 [Bowen Liang] separate cases
7513e055f [chenliang.lu] update ut
2ac0e1141 [chenliang.lu] add more ut
31a85a9a2 [Bowen Liang] rename ut
a9773de2f [Bowen Liang] introduce TableTableExtractor
da23d838f [Bowen Liang] input table
45129bc3f [Bowen Liang] update
adbe7535b [Bowen Liang] interceptContains
f24e07d9a [Bowen Liang] update table_command_spec.json
8a71a3c17 [Bowen Liang] rename to CallProcedure
5c0fd5689 [Bowen Liang] rename uts
c0c03176e [Bowen Liang] remove blanks
da55dc597 [Bowen Liang] fix imports
450f0779b [Bowen Liang] rename IcebergCallArgsTableExtractor to ExpressionSeqTableExtractor
0deeffd57 [chenliang.lu] Support iceberg call statement in ranger authentication
414ff2a42 [ChenliangLu] Merge branch 'apache:master' into master
c85059a0b [chenliang.lu] fix comments
85ffd0f88 [chenliang.lu] Support iceberg call statement in ranger authentication
66dbfcc35 [chenliang.lu] optimize codes for table extractor
f1e93fcd7 [chenliang.lu] Support iceberg call statement in ranger authentication
Lead-authored-by: yabola <chenlianglu@tencent.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: chenliang.lu <marssss2929@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: ChenliangLu <31469905+yabola@users.noreply.github.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5269 .
1. AnalyzesTablesCommand in table_command_spec is duplicated and won't be processed
2. The extractor desc of AnalyzesTablesCommand in table_command_spec is wrong
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5270 from AngersZhuuuu/KYUUBI-5269.
Closes#5269
75efc1adb [Angerszhuuuu] Update TableCommands.scala
75a6fa223 [Angerszhuuuu] Update table_command_spec.json
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- improvements for testing by extracting method for enabling authorization in single call mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5268 from bowenliang123/authz-withsingle.
Closes#5268
fe6b9d501 [Bowen Liang] extract withSingleCallEnabled method
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to fix
```
- columns lineage extract - CreateTableAsSelect *** FAILED ***
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineage(SparkSQLLineageParserHelperSuite.scala:1452)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24(SparkSQLLineageParserHelperSuite.scala:354)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24$adapted(SparkSQLLineageParserHelperSuite.scala:353)
at org.apache.spark.sql.SparkListenerExtensionTest.withTable(SparkListenerExtensionTest.scala:48)
at org.apache.spark.sql.SparkListenerExtensionTest.withTable$(SparkListenerExtensionTest.scala:46)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.withTable(SparkSQLLineageParserHelperSuite.scala:34)
at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$23(SparkSQLLineageParserHelperSuite.scala:353)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
...
```
the method `CreateTableAsSelect#left` was removed since Spark 3.5, for more details, see https://github.com/apache/spark/pull/40734
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5257 from cfmcgrady/fix-SparkSQLLineageParserHelperSuite.
Closes#5257
aa5e730fc [Fu Chen] fix
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Adding `-DskipTests` and `-am` into maven commands
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5247 from pan3793/minor.
Closes#5247
f373fbd52 [Cheng Pan] Correct building command for Spark authz/lineage plugins
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Closes#3444
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3558 from iodone/kyuubi-3444.
Closes#3444
acaa72afe [odone] remove plugin dependency from kyuubi spark engine
739f7dd5b [odone] remove plugin dependency from kyuubi spark engine
1146eb6e0 [odone] kyuubi-3444
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR aims to unify the exception handling of v1 and v2 during dropDatabase
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5225 from Yikf/hive-connector.
Closes#5225
3be33af76 [yikaifei] [KSHC] Improve test
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR aims to modify the README file of authz to make spark 3.4 the default version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5224 from Yikf/authz-ci-across-spark.
Closes#5224
7e28f8619 [yikaifei] fix authz spark version and ci
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
- Make Spark TPC-H connector compilable on Scala 2.13
- adding spark extensions to scala 2.13 compilation CI test
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5195 from bowenliang123/scala213-conn.
Closes#5195
ad2558a16 [liangbowen] update
c50d29111 [liangbowen] update
d38eb3b6c [liangbowen] adapt spark connector plugin compilable on Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL lineage plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
`mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-spark-lineage_2.13 -Pspark-3.3`
- fix type mismatch, by manually converting Iterable to ListMap
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala:220: type mismatch;
found : scala.collection.immutable.Iterable[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.AttributeSet)]
required: LineageParser.this.AttributeMap[org.apache.spark.sql.catalyst.expressions.AttributeSet]
(which expands to) scala.collection.immutable.ListMap[org.apache.spark.sql.catalyst.expressions.Attribute,org.apache.spark.sql.catalyst.expressions.AttributeSet]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5192 from bowenliang123/scala213-lineage.
Closes#5192
a68ba8457 [liangbowen] adapt spark lineage plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL hive connector plugin compilable on Scala 2.13 with Spark 3.3/3.4
- rename class name `FilePartitionReader` which is copied from Spark to `SparkFilePartitionReader`to fix the class mismatch error
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HivePartitionReaderFactory.scala:83: type mismatch;
found : Iterator[org.apache.kyuubi.spark.connector.hive.read.HivePartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
required: Iterator[org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5193 from bowenliang123/scala213-hivecon.
Closes#5193
d8c6bf5f0 [liangbowen] defer toMap
b20ad4eb1 [liangbowen] adapt spark hive connector plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
- to make Spark extension plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
```
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-2_2.13 -Pspark-3.2 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-3_2.13 -Pspark-3.3 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-4_2.13 -Pspark-3.4 -am
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5191 from bowenliang123/scala213-sparkext.
Closes#5191
156ae26c8 [liangbowen] adapt spark ext plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Make spark authz module compilable on Scala 2.13
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5189 from bowenliang123/scala213-authz.
Closes#5189
c5ee61a24 [liangbowen] adapt spark authz module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix a bug, The `DESCRIBE FUNCTION` syntax should also be checked for USE permissions. However, prior to this PR, the syntax was not checked for any permissions
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5172 from Yikf/auth-desc-function.
Closes#5172
112f4f20b [yikaifei] The DESCRIBE FUNCTION syntax should also be checked for USE permissions
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
In minor cases, Spark Stage hangs forever when spark.sql.finalWriteStage.eagerlyKillExecutors.enabled is true.
The bug occurs if two conditions are met in the same time:
1. All executors are either removed because of idle time out or killed by FinalStageResourceManager.
Target executor num in YarnAllocator will be set to 0 and no more executor will be launched.
2. Target executor num in ExecutorAllocationManager equals to the executor num needed by final stage.
Then ExecutorAllocationManager will not sync target executor num to YarnAllocator.
### _How was this patch tested?_
- [x] Add a new test suite `FinalStageResourceManagerSuite`
Closes#5141 from zhouyifan279/adjust-executors.
Closes#5136
c4403eefa [zhouyifan279] assert adjustedTargetExecutors == 1
ea8f24733 [zhouyifan279] Add comment
5f3ca1d9c [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
12687eee7 [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
9dcbc780d [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
