### Why are the changes needed?
RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x, and it causes `ClassNotFoundException` when user use `kyuubi-spark-authz-shaded_2.12-1.10.1.jar`(built with Ranger 2.5.0)
```
java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/base/ProviderBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:208)
at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:191)
at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:465)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1321)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1183)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:148)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:172)
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:112)
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:563)
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:138)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:254)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
at org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension.<init>(RangerSparkExtension.scala:44)
```
### How was this patch tested?
```
$ jar tf kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep org/apache/kyuubi/shade/com/fasterxml
org/apache/kyuubi/shade/com/fasterxml/
org/apache/kyuubi/shade/com/fasterxml/jackson/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AbstractTypeResolver.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty$Type.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector$XmlExtensions.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/AnnotationIntrospector.class
org/apache/kyuubi/shade/com/fasterxml/jackson/databind/BeanDescription.class
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7004 from pan3793/authz-jackson.
Closes#7004
cbf870516 [Cheng Pan] fix
4312d9fe5 [Cheng Pan] Include FastXML Jackson into authZ shaded jar
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
This PR provides an alternative for RANGER-4125 to cut out JNA dependencies for authZ plugin.
### How was this patch tested?
Pass GHA, and I checked the content of authz-shaded jar
```
$ jar tf extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_2.12-1.11.0-SNAPSHOT.jar | grep Hostname
org/apache/kyuubi/shade/com/kstruct/gethostname4j/Hostname.class
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7003 from pan3793/authz-hostname.
Closes#7003
42e246856 [Cheng Pan] Cut out JNA dependencies for authz plugin
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
In Scala [v2.13.16](https://github.com/scala/scala/releases/tag/v2.13.16)
> JNA is no longer a dependency of `scala-compiler.jar`
Since Spark 4.0 upgrades to Scala 2.13.16, JNA deps have gone too.
```
$ spark-3.5.5-bin-hadoop3-scala2.13 cat RELEASE
Spark 3.5.5 (git revision 7c29c664cdc) built for Hadoop 3.3.4
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.13 -Phadoop-3 -Phive -Phive-thriftserver
$ spark-3.5.5-bin-hadoop3-scala2.13 ls jars | grep jna
jna-5.9.0.jar
```
```
$ spark-4.0.0-bin-hadoop3 cat RELEASE
Spark 4.0.0 (git revision ca56e9ce591) built for Hadoop 3.4.1
Build flags: -B -Pyarn -Pkubernetes -Psparkr -Phadoop-3 -Phive -Phive-thriftserver
$ spark-4.0.0-bin-hadoop3 ls jars | grep jna
<no-output>
```
It's rare to use the non-default Scala version with Spark in practice, we shall respect Spark 4 deps for Scala 2.13 cases.
### How was this patch tested?
Review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#6999 from pan3793/authz-scala213.
Closes#6999
18230a2d7 [Cheng Pan] Keep JNA in authz-shaded with Scala 2.13
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Preparing v1.11.0-SNAPSHOT after branch-1.10 cut
```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6769 from bowenliang123/bump-1.11.
Closes#6769
6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
## Issue References 🔗
Fix a ClassNotFound issue.
```
java.lang.NoClassDefFoundError: org/apache/kyuubi/shade/javax/ws/rs/core/Cookie
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Verified manually.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6723 from pan3793/6638-followup.
Closes#6638
56e9842e0 [Cheng Pan] [KYUUBI #6638] authz shaded should include jsr311-api
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
# 🔍 Description
I faced the following error when trying to run authz with Spark 4.0
```
Cause: java.lang.NoClassDefFoundError: javax/ws/rs/core/Cookie
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.ranger.plugin.policyengine.RangerPluginContext.createAdminClient(RangerPluginContext.java:96)
at org.apache.ranger.plugin.util.PolicyRefresher.<init>(PolicyRefresher.java:90)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:251)
at org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin$.initialize(SparkRangerAdminPlugin.scala:68)
```
The `javax.ws.rs:jsr311-api` is the transitive dep of `jersey-client`, we should shade and relocate it correctly.
Why does it work with Spark 3? Spark 3 provides `jakarta.ws.rs:jakarta.ws.rs-api:2.1.6` which provides `java.ws.rs.*` classes, but Spark 4 upgrades to `jakarta.ws.rs:jakarta.ws.rs-api:3.0.0` which changed package name to`jakarta.ws.rs.*`.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA and manually tested with Spark 4
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6638 from pan3793/jsr311.
Closes#6638
5699200cf [Cheng Pan] Shade jsr311-api in Authz
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The POM of `kyuubi-spark-authz-shaded` is redundant, just pull `kyuubi-spark-authz` is necessary.
The current dependency management does not work on Ranger 2.1.0, this patch cleans up the POM definition and fixes the compatibility with Ranger 2.1.0
## Describe Your Solution 🔧
Carefully revise the dependency list and exclusion.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
perform packing kyuubi-spark-authz-shaded module.
```
build/mvn clean install -pl extensions/spark/kyuubi-spark-authz-shaded -am -DskipTests
```
before
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default) kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the shaded jar.
```
after
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default) kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-spark-authz_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12🫙1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugin-classloader:jar:2.4.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the shaded jar.
```
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6197 from pan3793/authz-dep.
Closes#6197
d0becabce [Cheng Pan] 2.4
47e38502a [Cheng Pan] ranger 2.4
af01f7ed5 [Cheng Pan] test ranger 2.1
203aff3b3 [Cheng Pan] ranger-plugins-cred
974d76b03 [Cheng Pan] Resive dependency management of authz
e5154f30f [Cheng Pan] improve authz deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes a class not found issue.
```
Caused by: java.lang.ClassNotFoundException: org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider
...
```
## Describe Your Solution 🔧
`org.apache.ranger:ranger-plugins-cred` was missing in include list.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manual test.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6194 from pan3793/auth-shaded.
Closes#6194
4eae524bd [Cheng Pan] Authz shaded should include ranger-plugins-cred
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Update the NOTICE files for coming 2024.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#5933 from pan3793/notice-2024.
Closes#5933
25e85e5f5 [Cheng Pan] notice
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As description https://github.com/apache/kyuubi/pull/5427#discussion_r1367907906. This PR aims to:
- Authz shaded include the transitive jackson libs
- Add LICENSE and NOTICE file to exactly match the content of jar
- Add scala-2.13 profile to make authz shade module support scala 2.13
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5496 from Yikf/jackson.
Closes#5496
6f442497c [yikaifei] trigger
8eeec2b21 [yikaifei] trigger
6a3cc0c8d [yikaifei] license
758d834b8 [yikaifei] shade jackson
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5427 from Yikf/shade-authz.
Closes#5427
d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin
Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
