# 🔍 Description
This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11)
Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6392 from pan3793/servlet.
Closes#6392
27d412599 [Cheng Pan] fix
9f1e72272 [Cheng Pan] other spark modules
f4545dc76 [Cheng Pan] fix
313826fa7 [Cheng Pan] exclude
7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This pull request removes unused dependency management in POM
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6267 from pan3793/clean-pom.
Closes#6267
d19f719bf [Cheng Pan] Remove usued dependency management in POM
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The POM of `kyuubi-spark-authz-shaded` is redundant, just pull `kyuubi-spark-authz` is necessary.
The current dependency management does not work on Ranger 2.1.0, this patch cleans up the POM definition and fixes the compatibility with Ranger 2.1.0
## Describe Your Solution 🔧
Carefully revise the dependency list and exclusion.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
perform packing kyuubi-spark-authz-shaded module.
```
build/mvn clean install -pl extensions/spark/kyuubi-spark-authz-shaded -am -DskipTests
```
before
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default) kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the shaded jar.
```
after
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default) kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugin-classloader:jar:2.4.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the shaded jar.
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6197 from pan3793/authz-dep.
Closes#6197
d0becabce [Cheng Pan] 2.4
47e38502a [Cheng Pan] ranger 2.4
af01f7ed5 [Cheng Pan] test ranger 2.1
203aff3b3 [Cheng Pan] ranger-plugins-cred
974d76b03 [Cheng Pan] Resive dependency management of authz
e5154f30f [Cheng Pan] improve authz deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#5529.
Support create table command for Delta Lake in Authz.
https://docs.delta.io/latest/delta-batch.html#create-a-table
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5530 from zml1206/KYUUBI-5529.
Closes#5529
b8ed2a464 [zml1206] update
f12ff8ff5 [zml1206] Merge branch 'master' into KYUUBI-5529
c02523062 [Bowen Liang] resolve conflicts
036154b04 [Bowen Liang] Merge branch 'master' into KYUUBI-5529
fb4223b2c [zml1206] fix check spec json files
10138d83a [zml1206] Merge branch 'master' into KYUUBI-5529
facd8f7c6 [zml1206] fix show databases ut
b2a9543ec [zml1206] update ut
23f6b8172 [zml1206] Support create table command for Delta Lake
Lead-authored-by: zml1206 <zhuml1206@gmail.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
### _Why are the changes needed?_
Apache Paimon is an incubating Apache project of data lake platform for high-speed data ingestion, changelog tracking and efficient real-time analytics.
- Initial support for Paimon tables in Authz plugin
- Create Table Command: https://paimon.apache.org/docs/master/engines/spark3/#create-table
- Paimon `0.5.0-incubating` supports Spark 3.1/3.2/3.3/3.4.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5462 from bowenliang123/authz-paimon-createtable.
Closes#5462
8393f6ecc [liangbowen] remove spark-authz-paimon-test profile, use 3.4 as spark binary version for Paimon
a4e01a2a8 [liangbowen] update workflow inclusions
f372882aa [liangbowen] update workflow exclusions
7785dec52 [liangbowen] update workflow include
64ee20106 [liangbowen] update
73edea38f [liangbowen] add spark-authz-paimon-test profile and excluded for Spark 3.5
de58605bb [liangbowen] nit
81a4a4370 [liangbowen] nit
eb2de8359 [Bowen Liang] exclude paimon tests for spark-3.5
4f58d7727 [Bowen Liang] fix typo
cf360e6da [Bowen Liang] drop database
af8ffda14 [Bowen Liang] drop database
dc3ee9462 [Bowen Liang] support Paimon create table
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5284.
Hudi also is a common used data format, since kyuubi already support iceberg and delta, we should also support hudi.
In this pr we support hoodie sql about ALTER COMMAND in authz
In this PR we use default Hudi version 0.14.0.
We support from spark 3.1 to spark 3.4, since Hudi don't support spark 3.5 yet
- [x] spark 3.1
- [x] spark 3.2
- [x] spark 3.3
- [x] spark 3.4
- [ ] spark 3.5
Also since Hudi only supports Scala 2.12, I also made Hudi as a separate profile to avoid importing Hudi when enable Scala 2.13
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5287 from AngersZhuuuu/KYUUBI-5284.
Closes#5284
f171e11af [Angerszhuuuu] Update pom.xml
3f57a3dc5 [Angerszhuuuu] follow comment
f6c764028 [Angerszhuuuu] follow comment
51797e25c [Angerszhuuuu] trigger
b3c059af9 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
3510e7601 [liangbowen] remove scope in dependencyManagement
14ea0d498 [liangbowen] change to use `spark.binary.version` for hudi dependency by default
354260eb0 [liangbowen] remove the abbreviation tite
658bddbab [liangbowen] remove clarification and use ALTERTABLE_PROPERTIES for opType
150edcd40 [Angerszhuuuu] update
30c417b19 [Angerszhuuuu] trigger
56e5cb17b [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
fe9b75270 [Angerszhuuuu] update
888943831 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
db749a277 [Angerszhuuuu] update
9b09e78c2 [Angerszhuuuu] Update HudiCommands.scala
87de62e52 [Angerszhuuuu] follow comment
2d551d112 [Angerszhuuuu] Update master.yml
89082e06b [Angerszhuuuu] Update master.yml
7c7846378 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
d32ca9839 [Angerszhuuuu] Update master.yml
ec43e2a7b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
b3611fd3e [Angerszhuuuu] update
2a0dfa74f [Angerszhuuuu] Update AuthZUtils.scala
45ee9e251 [Angerszhuuuu] update
0560a5e14 [Angerszhuuuu] Update pom.xml
97c50f622 [Angerszhuuuu] update
f57ee0093 [Angerszhuuuu] Update table_command_spec.json
fb72197e6 [Angerszhuuuu] update
2154cf928 [Angerszhuuuu] trigger
44469359f [Angerszhuuuu] trigger
b0e768cb8 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
83795ed63 [Angerszhuuuu] Update pom.xml
eed190f92 [Angerszhuuuu] update
361660145 [Angerszhuuuu] update
1ed1f3ab6 [Angerszhuuuu] Update
7ee3c7dd5 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
ee0916f63 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
010260fa4 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
c11d02def [Angerszhuuuu] update
b84f91f65 [Angerszhuuuu] update
42fbb0ffa [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
c1346adb1 [Angerszhuuuu] update
2ec63ae94 [Angerszhuuuu] Update pom.xml
39bce7468 [Angerszhuuuu] update
c70b0ea2f [Angerszhuuuu] Update pom.xml
e1d85ff77 [Angerszhuuuu] Update pom.xml
59012ac25 [Angerszhuuuu] Update pom.xml
a46de65b5 [Angerszhuuuu] Update HoodieTest.java
b8173b893 [Angerszhuuuu] update
055713329 [Angerszhuuuu] Update table_command_spec.json
d7b21e820 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
0a93ff794 [Angerszhuuuu] [KYUUBI #5284] Kyuubi authz support Hoodie Alter Table Command
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4870
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4872 from pan3793/util.
Closes#4870
0b9fe3cba [Cheng Pan] nit
ecc5ee4f2 [Cheng Pan] fix
63be7a20c [Cheng Pan] test
85363c187 [Cheng Pan] style
2227247dd [Cheng Pan] fix package
11d10a081 [Cheng Pan] Add kyuubi-util and kyuubi-util-scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- add ut to check generated Ranger policy file in #4585
- manually activated `genpolicy` profile in CI builds, as the property based activation not auto-triggered as expectedly with property `ranger.version=2.4.0` set in project parent pom
- Support regenerated policy file within the same test suite, by running
`KYUUBI_UPDATE=1 build/mvn clean test -pl :kyuubi-spark-authz_2.12 -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.gen.PolicyJsonFileGenerator -Pgenpolicy`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4677 from bowenliang123/authz-check-policy-gen.
Closes#4677
a372bdfd4 [liangbowen] remove unnecessary profile used in style workflow
7562c88f2 [liangbowen] include in spotless
37b674223 [liangbowen] update policy id
724ec5e28 [liangbowen] replace counter by using zipWithIndex
d322980e7 [liangbowen] extract KRangerPolicyResource object to simplify resource assembly
42c37606a [liangbowen] nit
18a8f4c51 [liangbowen] add usage comments
4ee254d6d [liangbowen] fix issue name in docs
d3cb08d21 [liangbowen] improve file reading
37e4c9c9f [Bowen Liang] Merge branch 'master' into authz-check-policy-gen
6366c50e4 [liangbowen] rename profile to `gen-policy` and remove activation rule by property setting
892faf5ef [liangbowen] update clue
266baa71a [liangbowen] update
cb94e8014 [liangbowen] update
de1f36531 [liangbowen] cleanup
e88c75d46 [liangbowen] check policy file gen
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#4615
- bump Ranger version to 2.4.0, release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.4.0+-+Release+Notes
- #4585 fixed duplication and conflict in policy file
- update docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4675 from bowenliang123/ranger-2.4.0.
Closes#4615
d403bc324 [liangbowen] bump ranger from 2.3.0 to 2.4.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- generate Authz policy file for testing to focus on manage all the policies in the generator
- auto-increased `id` and related `guid` for each policy
- list and reused users, resources, for evaluation impacts when policy changes
- add `policies_base.json` as base template file of authz template including decoration details and service def
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4585 from bowenliang123/authz-policy-gen.
Closes#4585
c8040553b [liangbowen] authz policy json file gen
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- `jerysey-bundle` 1.x , which is depended by `ranger-plugins-common`, contains `javax.ws.rs.*` (like `javax.ws.rs.core.*`) packages that conflicted to `jarkarta.ws.rs-api-2.1.6.jar` in Spark 3.0+ jars
- exclude `jerysey-bundle` 1.x and include `jersey-client` 1.19.4 to align with `ranger-plugins-common`
Diff in Authz dependency jar list:
```diff
gethostname4j-1.0.0.jar
jackson-jaxrs-1.9.13.jar
- jersey-bundle-1.19.3.jar
+ jersey-client-1.19.4.jar
+ jersey-core-1.19.4.jar
jetty-client-9.4.50.v20221201.jar
jetty-http-9.4.50.v20221201.jar
jetty-io-9.4.50.v20221201.jar
jetty-util-9.4.50.v20221201.jar
jna-5.7.0.jar
jna-platform-5.7.0.jar
ranger-plugins-audit-2.3.0.jar
ranger-plugins-common-2.3.0.jar
ranger-plugins-cred-2.3.0.jar
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4437 from bowenliang123/authz-jerseyclient.
Closes#4437
23e7eed5a [liangbowen] fixing conflicts in `javax.ws.rs.*`packages by excluding `jersey-bundle` and including `jersey-client` with `jsr311-api` excluded
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- use `Jackson` instead of `Gson` in Authz tests, where it is the only place `Gson` is used in project
- remove `Gson` as the dependency
- Notice: Gson project is in maintenance mode, (see https://github.com/google/gson/pull/2085)
- Gson is still on `dependencyList`, as depended in kyuubi-ha
<img width="640" alt="image" src="https://user-images.githubusercontent.com/1935105/213956635-ce079638-15c3-4de2-9f46-a93dfe3db199.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4201 from bowenliang123/remove-gson.
Closes#4201
96df2513 [liangbowen] update dependencyList
81303d6e [liangbowen] imports
a73a1f98 [liangbowen] use jackson instead of gson in authz testing and remove gson as dependency
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to fix#4102
- remove usage of ClassTagExtensions for better compatibility to `jackson-module-scala`
- `spark-core` of Spark 3.x's core already has dependency on `com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}`
- change the scope of `com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}` to `provided`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4115 from bowenliang123/authz-jackson-provided.
Closes#4115
5d20eced [liangbowen] style
d4eb245a [liangbowen] Revert "ensure the earliest version of fasterxml.jackson provided in Spark 3.x"
112bb846 [liangbowen] ensure the earliest version of fasterxml.jackson provided in Spark 3.x
b4810142 [liangbowen] remove usage of ClassTagExtensions for better compatibility to `jackson-module-scala` lib
0487674b [liangbowen] change the scope of `com.fasterxml.jackson.module:jackson-module-scala_${scala.binary.version}` to `provided`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR redesigned the authorization part of the spark authz module with a New Authz Plan Serde Layer.
#### Motivation
- add a general layer to describe a command, so that we can add a new command or users can add a third-party command easily according to the specification.
- get rid of the spark version check. The built-in spark commands frequently vary from version to version, which makes us hard to maintain at compile& runtime phase, and the third-party commands are hard to check by spark versions.
#### Data structure
Overall, we introduce 2 general basic data structures:
- CommandSpec: used to describe a command
- `classname` as key for the read-side to get the spec by a particular command
- pre-defined operation type
- descriptors
- Descriptor: used to describe an object, such as table, db, query,
- `fieldName`: the object to get
- `fieldExtractor`: the method to get the object; use SPI to load
- sub-descriptors: such as columns in a table
- etc.
#### SPI
- Extractor: implementations for `fieldExtractor`
- key: the name of the extractor for the read-side to get itself
- func: converting the field value to specific and general objects
#### Code Path
- Write code path
- automatically generated default json configuration files
- custom json configuration files for thrid-party commands
- Read code path
- Load json as maps
- RuleAuthorization -> PrivilegeBuilder.build -> get command desc from maps -> build privileges with the retrieved desc.
#### TODOs
- Add back the ArcticCommand
- Add delta command
- Add ways for loading custom json configuration files
- Add hudi commands
- etc
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3904 from yaooqinn/na.
Closes#3904
efafcbaa2 [Kent Yao] sort
7d2b3e41c [Kent Yao] ci
379e93343 [Kent Yao] style
b52ab418a [Kent Yao] comments
9a24be64d [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
8926f0446 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
b45453a3e [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
f56148ea2 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
e47749d96 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
c32feef63 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
49dbb6842 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
2bced47b6 [Kent Yao] Merge branch 'master' into na
453541bd1 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://iceberg.apache.org/releases/#100-release
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3669 from pan3793/iceberg.
Closes#3669
eb3068c2 [Cheng Pan] scala-collection-compat
23d93616 [Cheng Pan] Bump iceberg from 0.14.1 to 1.0.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3660 from bowenliang123/3635-deletetmp.
Closes#3635
8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3636 from bowenliang123/3635-tempdir.
Closes#3635
5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang] delete files on exit in test suits with Utils.createTempDir
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3515.
By replacing mapChildren in `RuleApplyRowFilterAndDataMasking`to skip head of children query as insterted by iceberg in `IcebergSparkSessionExtensions` .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3520 from bowenliang123/3515-authz-iceberg.
Closes#3515
a10fe43a [Bowen Liang] improve mapPlanChildren
5eb7f845 [Bowen Liang] fix problem after merging from master
bb1eefbd [Bowen Liang] merge from master
6c6b071c [Bowen Liang] Merge commit 'ae2df990a32c695de6f0345ea7e73e51103e87e5' into 3515-authz-iceberg
dbcfe6d8 [Bowen Liang] restrict skipMapchiled to Iceberg command privilege builders and add skipMappedChildren method to IcebergCommands to handling them
ae2df990 [Bowen Liang] nit
0c691798 [liangbowen] update mapPlanChildren and passSparkVersionCheck
161215d0 [liangbowen] nit
0006dee3 [liangbowen] generalize passSparkVersionCheck method to AuthZUtils
44163638 [liangbowen] refactor getFieldValOpt to getFieldValOption
5b8aa40b [Bowen Liang] improvements on skippedMapChildren of mapPlanChildren
11c2e637 [liangbowen] nit
5a971945 [liangbowen] unifiying general way for ensure skipped table is in plan's children
b161d70f [Bowen Liang] nit
952c1e1c [Bowen Liang] reuse MergeIntoTable of v2Commands to MergeIntoIcebergTable of IcebergCommands
0b25bd1b [Bowen Liang] generalize mapPlanChildren for iceberg commands
04fb651f [Bowen Liang] nit
a1f33bc3 [Bowen Liang] update DELETE FROM TABLE ut
34d65e5a [Bowen Liang] introduce IcebergCommands for access checking iceberg table. skip head child for iceberg commands in RuleApplyRowFilterAndDataMasking to prevent marking output tables.
085bfa31 [liangbowen] repalce mapChildren in RuleApplyRowFilterAndDataMasking with mapPlanChildren method, to skip head child for iceberg UpdateIcebergTable/MergeIntoIcebergTable/DeleteFromIcebergTable
6a245019 [liangbowen] init iceberg ut
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3217#issuecomment-1219271537
This pr aims to bump ranger version from 2.2.0 to 2.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3333 from Yikf/pr/3282.
Closes#3333
07b35b9f [Cheng Pan] Update log4j2-test.xml
45a403b4 [yikf] [KYUUBI #3217][FOLLOWUP] Bump ranger version to 2.3.0
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3037 from pan3793/jul.
Closes#3037
fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Some of the commands were migrated to v2 in spark v3.3, this PR is used to match them
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2675 from yaooqinn/authz.
Closes#2675
a55be80f [Kent Yao] Fix compatiblity for spark authz with spark v3.3
defd582e [Kent Yao] Fix compatiblity for spark authz with spark v3.3
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Update jna version to 5.7.0
Before
```
java.lang.UnsatisfiedLinkError: /Users/cathy/Library/Caches/JNA/temp/jna8371704468821938112.tmp: dlopen(/Users/cathy/Library/Caches/JNA/temp/jna8371704468821938112.tmp, 0x0001): tried: '/Users/cathy/Library/Caches/JNA/temp/jna8371704468821938112.tmp' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e'))
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1012)
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
at com.sun.jna.Native.<clinit>(Native.java:195)
at com.kstruct.gethostname4j.Hostname$UnixCLibrary.<clinit>(Hostname.java:12)
at com.kstruct.gethostname4j.Hostname.getHostname(Hostname.java:30)
```
The 5.7.0 version is from https://github.com/mattsheppard/gethostname4j/pull/8
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2310 from ulysses-you/m1-jna.
Closes#2310
3706c871 [ulysses-you] ordering
b9b4e929 [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
