### _Why are the changes needed?_
This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5427 from Yikf/shade-authz.
Closes#5427
d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin
Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
To close#5362 .
Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5363 from AngersZhuuuu/KYUUBI-5362.
Closes#5362
d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Move the configuration docs to the top level of docs, which is most commonly used and referenced
- update relevant doc links
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5154 from bowenliang123/config-doc-first.
Closes#5154
b49ed3f8b [liangbowen] nit
db7f0d14d [liangbowen] update doc links
f8fd697a2 [liangbowen] move config docs to the top level
7448e4487 [liangbowen] change title of settings doc
40214ddd8 [liangbowen] move config doc in the front of deployment
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add ut to check generated Ranger policy file in #4585
- manually activated `genpolicy` profile in CI builds, as the property based activation not auto-triggered as expectedly with property `ranger.version=2.4.0` set in project parent pom
- Support regenerated policy file within the same test suite, by running
`KYUUBI_UPDATE=1 build/mvn clean test -pl :kyuubi-spark-authz_2.12 -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.gen.PolicyJsonFileGenerator -Pgenpolicy`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4677 from bowenliang123/authz-check-policy-gen.
Closes#4677
a372bdfd4 [liangbowen] remove unnecessary profile used in style workflow
7562c88f2 [liangbowen] include in spotless
37b674223 [liangbowen] update policy id
724ec5e28 [liangbowen] replace counter by using zipWithIndex
d322980e7 [liangbowen] extract KRangerPolicyResource object to simplify resource assembly
42c37606a [liangbowen] nit
18a8f4c51 [liangbowen] add usage comments
4ee254d6d [liangbowen] fix issue name in docs
d3cb08d21 [liangbowen] improve file reading
37e4c9c9f [Bowen Liang] Merge branch 'master' into authz-check-policy-gen
6366c50e4 [liangbowen] rename profile to `gen-policy` and remove activation rule by property setting
892faf5ef [liangbowen] update clue
266baa71a [liangbowen] update
cb94e8014 [liangbowen] update
de1f36531 [liangbowen] cleanup
e88c75d46 [liangbowen] check policy file gen
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#4615
- bump Ranger version to 2.4.0, release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.4.0+-+Release+Notes
- #4585 fixed duplication and conflict in policy file
- update docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4675 from bowenliang123/ranger-2.4.0.
Closes#4615
d403bc324 [liangbowen] bump ranger from 2.3.0 to 2.4.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As issue [RANGER-4672](https://github.com/apache/kyuubi/issues/4672) reported unresovled failures on Ranger 0.6.0, update docs for unsupported Ranger version 0.6.0.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4673 from bowenliang123/drop-ranger-0.6.0.
Closes#4672
5049596c0 [liangbowen] update README.md
f9ea53449 [liangbowen] drop support for Ranger 0.6.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Prefer `https://` URLs in docs, and all changed URLs are validated.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4235 from bowenliang123/https-link.
Closes#4235
f114dde2 [liangbowen] update AllKyuubiConfiguration
ad8aaedf [liangbowen] style
e973be5a [liangbowen] update
2370f4bf [liangbowen] prefer https URLs in docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This is part of https://github.com/apache/kyuubi/issues/4020, and should finalize the update for repo https://github.com/apache/kyuubi
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4089 from pan3793/graduate.
Closes#4089
60ece0c1 [Cheng Pan] [INFRA] Update release scripts and templates because of graduation
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
ban end-user from security settings
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3564 from cfmcgrady/kyuubi-3563.
Closes#3545Closes#3563
9d912b11 [Fu Chen] rename
ee44f7df [Fu Chen] fix ci
158f1552 [Fu Chen] address comment
3cbf4794 [Fu Chen] doc
0125a862 [Fu Chen] address comment
e4e554f5 [Kent Yao] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckExtension.scala
a8d35efa [Fu Chen] ban end-user from security settings
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#3371
Adding an ranger plugin config for enabling full access privileges, Authz checks all access privileges and throw exception with message of unsatisfied privileges on different resources in natural order in execution plan.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3380 from bowenliang123/3371-authz-full-check.
Closes#3371
9326196d [Bowen Liang] add RangerConfigProvider for getting plugin config of different Ranger versions. add invokeStatic in AuthZUtils.
65d765dd [Bowen Liang] use ArrayBuffer for value of accessTypeToResource map. import scala LinkedHashMap.
959cd7dc [liangbowen] nit: update ut
dbfd3479 [liangbowen] 1.use LinkedHashMap and LinkedHashSet for keeping in natual order for consistent error msg output, 2. update ut
3e0a5feb [liangbowen] revert to not enforcing output error message in alphabet order
1df4130d [Bowen Liang] add ut with multiple input tables and one output table
5241fd6a [Bowen Liang] refactor verify method in more scala style
d7bc6841 [Bowen Liang] nit
32b2dd6d [Bowen Liang] nit: docs
d9787be4 [Bowen Liang] nit: docs
87b095cf [Bowen Liang] simplify verify method
a0d27e0c [Bowen Liang] simplify verify method and update docs
c9b4e065 [Bowen Liang] rename config to ranger.plugin.plugin.authorize.in.single.call, simplify access checking, and related docs
f6ccc1ac [liangbowen] move verify to SparkRangerAdminPlugin and change config name to ranger.plugin.spark.enable.full.access.violation.msg
c531cadb [Bowen Liang] add config doc in Authz
041c49b1 [Bowen Liang] move config to ranger plugin conf
f4fbeb23 [liangbowen] support for full access checking
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3217#issuecomment-1219271537
This pr aims to bump ranger version from 2.2.0 to 2.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3333 from Yikf/pr/3282.
Closes#3333
07b35b9f [Cheng Pan] Update log4j2-test.xml
45a403b4 [yikf] [KYUUBI #3217][FOLLOWUP] Bump ranger version to 2.3.0
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Currently, Apache Kyuubi support Spark 3.0.0 and above, but Kyuubi-AuthZ support Spark 2.4, Spark 3.0 and above, to avoid subsequent maintenance costs, we plan to move out support for Spark 2.4.
This pr aims to modify the documentation to indicate that version 2.4 is not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3305 from Yikf/authz-spark2.4-notsupport.
Closes#3226
5c756f51 [yikf] authz not support spark2.4
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support macros in Row-filter condition expression, introduced in Ranger 2.3 ([release notes](https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.3.0+-+Release+Notes)), is an major feature to significantly simplify the row-filter condition expression in practice by replacing explicit condition query by using user/group's attributes.
- [RANGER-3605](https://issues.apache.org/jira/browse/RANGER-3605) : Support macros in row-filter/condition expressions
- [RANGER-3550](https://issues.apache.org/jira/browse/RANGER-3550) : support for using user/tag attributes in row-filter expressions and conditions
Consider user liangtiancheng with attribute born_city = guangzhou, we can define the row filter condition with city='${{USER.born_city}}' with the macro feature.
However, This feature implicit relies on an config named `ranger.plugin.spark.enable.implicit.userstore.enricher` and the default value false will prevent RangerUserStoreEnricher fetching user/group and their attributes. Macros in row-filter condition will fallback to null value (as lack of user attributes value in UserStore of auth context) in script transformation unexpectedly and imperceptibly.
Improving doc of ranger-spark-security.xml to aware of this feature and related config.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3267 from bowenliang123/doc-ranger-macros.
Closes#3217
aee4b201 [liangbowen] plain text
a7ec3bc9 [liangbowen] update docs and clearify difference between ranger.plugin.hive.policy.cache.dir and ranger.plugin.spark.policy.cache.dir configs
4887bd1f [liangbowen] simplify with "row filter expressions"
fa62402f [liangbowen] skip list items
9dd1cd4c [liangbowen] h5. Using Marcos in Row Level Filters
849bed5b [liangbowen] add docs for Additional configs for using Marcos in row-level filter
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Build the content for extension points documentation, pre-work for #3100
<img width="1767" alt="image" src="https://user-images.githubusercontent.com/8326978/179930987-1accbbb7-e804-4230-871f-6c4b1152f4a1.png">
1. the extensions are divided into 2: server side and engine side extensions. (Do we have client side extension support?)
2. the server side authentication page is cross-referenced by the security section, see 1 in the picture.
3. the engine side ones are grouped by different compute frameworks.
4. connector is one type of extension, so we cross-reference the connector pages directly, see 2 & 3 in the picture.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3103 from yaooqinn/3101.
Closes#3101
a9ae3e32 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
3b7367e9 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
b5eda13e [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2395 from yaooqinn/doc2.
Closes#2395
109440bf [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
852e7fd5 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
dfeef884 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
