### _Why are the changes needed?_
It was reported in https://github.com/apache/kyuubi/discussions/4063
When run `SHOW DATABASES;` on Databricks Runtime, the error occurs
```
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.UnsupportedOperationException
at org.apache.spark.sql.catalyst.plans.logical.LeafNode.computeStats(LogicalPlan.scala:183)
at org.apache.spark.sql.catalyst.plans.logical.LeafNode.computeStats$(LogicalPlan.scala:183)
at org.apache.kyuubi.plugin.spark.authz.util.ObjectFilterPlaceHolder.computeStats(ObjectFilterPlaceHolder.scala:23)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.$anonfun$stats$1(QueryPlanStats.scala:39)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.stats(QueryPlanStats.scala:38)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.stats$(QueryPlanStats.scala:38)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.stats(LogicalPlan.scala:31)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4076 from pan3793/stats.
Closes#4076
8238f317 [Cheng Pan] ut
a3c5f7d3 [Cheng Pan] [AUTHZ] Modified query plan should correctly report stats
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add serde layer for queries, which extracts table information from a scan relation, such as HiveTableRelation, logical relation, etc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4044 from yaooqinn/scan.
Closes#4044
d2d830c50 [Kent Yao] addr comments
3cac8dd54 [Kent Yao] [Authz] Serde for queries
fd5425679 [Kent Yao] [Authz] Serde for queries
d42778999 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into scan2
e19282e1f [Kent Yao] [Authz] Serde for queries
50ee2335d [Kent Yao] [Authz] Serde for queries
78ce87855 [Kent Yao] [Authz] Serde for queries
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
close#3978
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4033 from iodone/kyuubi-3978.
Closes#4033
2a3ae68b [odone] add datasource v2 support
72feb958 [odone] add datasourcev2 support
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
add unit test of row filter for MergeIntoIcebergTable of Iceberg catalog.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4047 from bowenliang123/iceberg-rowfilter-ut.
Closes#4047
c6b7d5e17 [liangbowen] update ut name
c26c61b0d [liangbowen] add e2e ut for row filter of iceberg
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
subtask of #3586 .
1. passing catalog field from `Table` and `Database` to PrivilegeObject for privilege checks. As for`Function` related commands, no catalog supported other than spark_catalog or v1 catalog in Spark 3.1-3.3.
2. extract catalog from `DataSourceRelationV2` in buildQuery with added `CatalogPluginOptionCatalogExtractor`
3. add ut for v2commands SetNamespaceProperties/CreateNamespace/SetNamespaceLocation/DescribeNamespace/DropNameSpace to verify related PrivilegeObject extraction with catalog in spark 3.3
4. in future PR, SparkRangerAdminPlugin will use catalog-related instances for privilege checks using catalog info from PrivilegeObject
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4032 from bowenliang123/catalog-priviobj.
Closes#4032
f66e2f40f [liangbowen] typo
24abdb1dc [liangbowen] add ut for v2commands SetNamespaceProperties/CreateNamespace/SetNamespaceLocation/DescribeNamespace/DropNameSpace
01af55f5f [liangbowen] change assertion to sparkCatalogVal to sparkSessionCatalogName of string type
72dd64e0e [liangbowen] fix catalog assertion for some spark version variation
25160b120 [liangbowen] fix catalog assertion for some spark version variation
4a89b87f4 [liangbowen] fix catalog assertion for some spark version variation
333d47b9e [liangbowen] add catalog assertion to PrivilegesBuilderSuite and V2CommandsPrivilegesSuite
a02b6803e [liangbowen] pass catalog from Database in databasePrivileges
b68a0dc8a [liangbowen] fix
30c7e7de9 [Bowen Liang] Merge branch 'master' into catalog-priviobj
598c6bfef [liangbowen] implement CatalogPluginOptionCatalogExtractor
003c97a58 [liangbowen] add catalog support in method tablePrivileges/v2TablePrivileges of PrivilegesBuilder and pass catalog to PrivilegeObject
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Detect and inject a tag if plan is for writing, then skip doing final stage isolation at query preparation phase.
To make final stage config more flexible with complex Spark application.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3988 from ulysses-you/final-stage.
Closes#3988
d0f2b622 [ulysses-you] fix
e5351fd5 [ulysses-you] nit
39082b20 [ulysses-you] Final stage config isolation support write only
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
1. make DatabaseDesc return added `Database` type instance
2. add `catalogDesc` to DatabaseDesc and `catalog` filed in Database
3. extract catalog for ResolvedNamespaceDatabaseExtractor and ResolvedDBObjectName in databaseExtractors of DSV2
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4015 from bowenliang123/catalog-v2database.
Closes#4015
2d925f046 [liangbowen] remove ut for SetCatalogAndNamespace
cd56ca6a7 [liangbowen] move ut to V2CommandsPrivilegesSuite
36e44415b [liangbowen] update
8ca4f7f9d [liangbowen] add ut for SetCatalogAndNamespace
a29068d21 [liangbowen] move ut for SetNamespaceCommand to V2CommandsPrivilegesSuite
bce9471f2 [liangbowen] revert workaround for blank fieldName of CatalogDesc
63c341797 [Bowen Liang] Merge branch 'master' into catalog-v2database
7d043d7db [liangbowen] remove CurrentCatalogExtractor
49e52ff14 [liangbowen] fix None catalog of `USE NAMESPACE` in spark 3.1/3.2 by filling current catalog name into it
ae1f84030 [liangbowen] implement CurrentCatalogExtractor to fetch current catalog from session state, and fill catalog for SetNamespaceCommand
d1654b936 [liangbowen] implement StringOptionCatalogExtractor for using with StringSeqOptionDatabaseExtractor in SetCatalogAndNamespace
ff464828f [liangbowen] change var order moving catalogSpec before isInput
f924bbc88 [liangbowen] regenerate db spec
a6c58accf [liangbowen] add CatalogPluginCatalogExtractor to catalogDesc for CreateNamespace following StringSeqDatabaseExtractor
696e38c51 [liangbowen] update
08c1c7edd [liangbowen] add ut for ResolvedNamespaceDatabaseExtractor
4101a7294 [liangbowen] use CatalogPluginCatalogExtractor in ResolvedNamespaceDatabaseExtractor
e97f1dd2b [liangbowen] use CatalogPluginCatalogExtractor in ResolvedDBObjectNameDatabaseExtractor
d1b68c25d [liangbowen] support catalog in Database and databaseExtractors
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
extract catalog for ResolvedTable and DataSourceV2Relation in tableExtractors
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3989 from bowenliang123/catalog-v2table.
Closes#3989
88392051c [liangbowen] update
e32dbd54a [liangbowen] revert catalogExtractor and remove spark session for input
0e36e2fb3 [liangbowen] style
90a671640 [liangbowen] remove CurrentCatalogExtractor
77e0f864a [liangbowen] add package name in comments for extractor
5c2642b38 [liangbowen] update CurrentCatalogExtractor
8bdb7d767 [liangbowen] fix ut error and cancel fetching catalog inside IdentifierTableExtractor
6be3496f3 [liangbowen] check catalog for TableIdentifierTableExtractor with DescribeColumnCommand
5ee473de3 [liangbowen] fill catalog in IdentifierTableExtractor/CatalogTableTableExtractor/TableIdentifierTableExtractor
802ea065d [liangbowen] 1. fill catalog for table with current catalog name if None catalog return after catalogdesc extracted 2. implement CurrentCatalogExtractor
273875ae1 [liangbowen] use CatalogPluginCatalogExtractor in ResolvedDbObjectNameTableExtractor
0368ac513 [liangbowen] remove unnecessary seq in ut
0b0ca7802 [liangbowen] revert relevant changes in ResolvedDbObjectNameTableExtractor
7ebcc92f0 [liangbowen] revert relevant changes in ResolvedDbObjectNameTableExtractor
bd13a1ef8 [liangbowen] lint
551153d0d [liangbowen] add ut for DataSourceV2RelationTableExtractor
e13987f86 [liangbowen] extract catalog for DataSourceV2Relation
cdbc705d8 [liangbowen] add ut for ResolvedTableTableExtractor
85f26947a [liangbowen] extract catalog for ResolvedTable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Update outdated docs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4024 from QianyongY/features/update-outdated-docs.
Closes#4024
0ad7173f [yongqian] [DOCS] Update the rules documentation for Kyuubi Spark SQL extension
Authored-by: yongqian <yongqian@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3605.
1. add `catalog` in AccessResource for Ranger plugin auth by catalog
2. add `catalog` in PrivilegeObject, and pass it to AccessResource
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3981 from bowenliang123/3605-poar-cat.
Closes#3605
ab057c7fc [liangbowen] update ut
4fae686b7 [liangbowen] update ut name
f63f6cf59 [liangbowen] fix " multiple overloaded alternatives of method apply define default arguments." problem in AccessResource
2135aaba2 [liangbowen] add catalog in PrivilegeObject and passing to AccessResource
99d808ba8 [liangbowen] add catalog in AccessResource
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This PR is part of work form #3606 by bowenliang123, with UTs added
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3984 from yaooqinn/bw.
Closes#3984
db1e27291 [Kent Yao] style
8563f3ce3 [liangbowen] Please enter the commit message for your changes. Lines starting
808131d52 [liangbowen] CatalogDesc
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
add two conditions to decide if we should add shuffle.
1. make sure AQE is enabled, otherwise it is no meaning to add a shuffle
2. try to reduce the performance regression if add a shuffle
for condition 2: we do not add shuffle if the original plan does not have shuffle
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3962 from ulysses-you/no-shuffle.
Closes#3962
a084cccc [ulysses-you] address comment
9d0aab1b [ulysses-you] address comment
09fc9b21 [ulysses-you] fix ut
06f249a2 [ulysses-you] Reduce the performance regression
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To simplify the method calling via reflection in Authz's `AccessRequest`, including codes from pull #3308 introduced.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3945 from bowenliang123/3300-followup.
Closes#3300
3e3f2bf8e [liangbowen] move exception throwing in `invoke` method
ff1c9a05e [liangbowen] import java.util.{HashMap => JHashMap}
2848083e4 [liangbowen] import java.util.{Set => JSet}
a9b4240fb [liangbowen] import java.util.{Set => JSet}
408977e89 [liangbowen] correct error handling in AccessRequest
c785a251c [liangbowen] use invokeAs
9f9c6a1d9 [liangbowen] rename invokeWithCast to invokeAs
258be28f8 [liangbowen] introduce `invokeWithCast` in AuthZUtils for simplifying object casting
f9902099a [liangbowen] simplify reflection method calling in AccessRequest
da767db88 [liangbowen] lint
3b82c3112 [liangbowen] unused imports
d4a71dac5 [liangbowen] unused imports
0c57d0de9 [liangbowen] simply getUserGroupsFromUserStore with reflection methods
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#3942 .
Adapt Spark's JDBC data type to Hive data type definitions.
1. adapt `DoubleType` to "DOUBLE" for compatibility with Hive 2.1.x and below
- "DOUBLE PRECISION" mapped in [`JdbcUtils.getCommonJDBCType` ](https://github.com/apache/spark/blob/v3.3.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L145) is a pure alias for DOUBLE, only available starting with Hive 2.2.0.
2. adapt `IntegerType` to "INT" instead of "INTEGER" for compatibility with Hive 2.1.x and below
- "INTEGER" is synonym for INT since Hive 2.2.0
3. fix the unsupported Spark data type mapping to the Hive data type definition with correct mappings:
- FloatType to "FLOAT" instead of "REAL"
- StringType to "STRING" instead of "TEXT"
- BooleanType to "BOOLEAN" instead of "BIT(1)"
- BinaryType to "BINARY" instead of "BLOB"
- ByteType to "TINYINT" instead of "BYTE"
Hive Data Type docs refers to https://cwiki.apache.org/confluence/display/hive/languagemanual+types .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3943 from bowenliang123/3942-hivetype.
Closes#3942
10c3ddea [liangbowen] Add ut for non adapted data types
5f6ce015 [liangbowen] Add links to doc and issues for Hive data type definition
1ae0fd1d [liangbowen] mapping IntegerType to "INT" instead of "INTEGER" for compatibility with Hive 2.1.x and below
13486bb2 [liangbowen] update ut
cb0a0537 [liangbowen] mapping ByteType to "TINYINT"
c8a02828 [liangbowen] ut
25862584 [liangbowen] mapping BinaryType to "BINARY"
e8b65a00 [liangbowen] adapt to Hive data type definitions
3f5cc75a [liangbowen] add ut
3a818da9 [liangbowen] comments
a471466a [liangbowen] mapping FloatType to "FLOAT", BooleanType to "BOOLEAN"
b9506dc9 [liangbowen] mapping DoubleType to "DOUBLE" and StringType to "STRING"
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3735
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3736 from iodone/kyuubi-3735.
Closes#3735
89823b85 [odone] using sparkPlan.localLink to replace cache table's view plan
9e1ac5cc [odone] fix using `option.get` which maybe bring some potential risk
34511843 [odone] parse InMemoryRelation
c29b7d51 [odone] [KYUUBI-3735] fix
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fix#3300
1. support overriding usergroup in AccessRequest by fetching user-group relation mapping from UserStore (which introduced since Ranger 2.1+) in RangerBasePlugin.
2. policy condition authorized by usergroup enabled, if any expression used with in filter or condition
3. add a switch config for it, disabled by default.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3308 from bowenliang123/feature-ranger-groups.
Closes#3300
74400f78f [liangbowen] rename config to `ranger.plugin.spark.use.usergroups.from.userstore.enabled`and move it to SparkRangerAdminPlugin
636137a12 [liangbowen] invert condition
af1444fe7 [liangbowen] nit
398b1130c [liangbowen] removed ut related config and policy rules
f3d7f1e78 [liangbowen] unused imports
90d98e001 [liangbowen] remove ut in AccessRequestSuite.scala and getServicePoliciesIfUpdated in RangerLocalClient, which required by Ranger 2.1 and could break the compatibility in compilation
bba74ce3f [liangbowen] extract `getUserGroupsFromUserStore` and `getUserGroupsFromUgi`
94ef31867 [liangbowen] remove userNamePath of getUserGroups method
2e5f7bfe6 [liangbowen] use getRangerConf instead of getConfig to prevent breaking compatibility
4205426e4 [liangbowen] nit
58c9048b4 [liangbowen] nit
38f57081f [liangbowen] format imports
62b17b429 [liangbowen] nit
5e042c42e [liangbowen] update ut for group allow condition
bf5ac1dd5 [liangbowen] restore ut
73968b6b9 [Bowen Liang] nit
28bd10f88 [Bowen Liang] refactoring with getUserGroups method
4ec147913 [Bowen Liang] add ref to java.util.Set
333b4d576 [Bowen Liang] change to config name to ranger.plugin.spark.enable.override.usergroup.from.userstore
015b9c4a9 [Bowen Liang] fix type
91b55cc2b [Bowen Liang] fix type
f28b13033 [Bowen Liang] add spark config 'kyuubi.authz.use.usergroup.in.userstore' for enable/disable usergroup overriding
e319409ea [Bowen Liang] fix reflection error
d87e2caa5 [liangbowen] overriding userGroups by usergroup mapping from UserStore (Ranger 2.3+)
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This PR redesigned the authorization part of the spark authz module with a New Authz Plan Serde Layer.
#### Motivation
- add a general layer to describe a command, so that we can add a new command or users can add a third-party command easily according to the specification.
- get rid of the spark version check. The built-in spark commands frequently vary from version to version, which makes us hard to maintain at compile& runtime phase, and the third-party commands are hard to check by spark versions.
#### Data structure
Overall, we introduce 2 general basic data structures:
- CommandSpec: used to describe a command
- `classname` as key for the read-side to get the spec by a particular command
- pre-defined operation type
- descriptors
- Descriptor: used to describe an object, such as table, db, query,
- `fieldName`: the object to get
- `fieldExtractor`: the method to get the object; use SPI to load
- sub-descriptors: such as columns in a table
- etc.
#### SPI
- Extractor: implementations for `fieldExtractor`
- key: the name of the extractor for the read-side to get itself
- func: converting the field value to specific and general objects
#### Code Path
- Write code path
- automatically generated default json configuration files
- custom json configuration files for thrid-party commands
- Read code path
- Load json as maps
- RuleAuthorization -> PrivilegeBuilder.build -> get command desc from maps -> build privileges with the retrieved desc.
#### TODOs
- Add back the ArcticCommand
- Add delta command
- Add ways for loading custom json configuration files
- Add hudi commands
- etc
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3904 from yaooqinn/na.
Closes#3904
efafcbaa2 [Kent Yao] sort
7d2b3e41c [Kent Yao] ci
379e93343 [Kent Yao] style
b52ab418a [Kent Yao] comments
9a24be64d [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
8926f0446 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
b45453a3e [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
f56148ea2 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
e47749d96 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
c32feef63 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
49dbb6842 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
2bced47b6 [Kent Yao] Merge branch 'master' into na
453541bd1 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3897 from pan3793/group.
Closes#3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. to fix#3893
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3894 from FireFoxAhri/master.
Closes#3893
da15a000 [firefox] [KYUUBI #3893] [BUG] Fix spark extension: UnspecifiedDistribution does not have default partitioning.
Authored-by: firefox <309637962@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3839 .
The session signing feature is introduced with asymmetric encryption to prevent manipulation of session user identity or other key session elements, which could cause privilege leaking in scripts.
1. Server: Server creates and holds the singleton `keypair` for session signing if feature enabled
2. Server -> Engine: Server passes the `public key` to Engine when launching
3. Server -> Engine: Server generates session signing on `session user` when opening Kyuubi Session to Engine
4. Session -> Statement: Kyuubi session create Statement with context bringing `publickey` and `session user signature`
5. Engine: Engine verify `session user signature` with signature wherever necessary, e.g. in Authz for session name authentication.
ECDSA, with a shorter key length and better performance than RSA, is the supported asymmetric encryption in the initial implementation, which is widely supported on JDK7+ in HotSpot or OpenJDK. The session signature is generated with `SHA256withECDSA`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3838 from bowenliang123/check-sessionuser.
Closes#3839
925eee47 [liangbowen] Revert "generalize setSparkLocalProperty"
d726d661 [liangbowen] generalize setSparkLocalProperty
470176db [liangbowen] allow setSparkLocalProperty to set value null
8f2a6e38 [liangbowen] nit
cb5891c9 [liangbowen] nit
edbe3c9c [liangbowen] only set to opensession conf when true, since kyuubi.session.user.sign.enabled is default to false
60546c82 [liangbowen] remove setting setSessionSigningPublicKey in KyuubiSessionManager
9a2a60dd [liangbowen] fix asserts
2e2a51b4 [liangbowen] nit
f37f4dd6 [Bowen Liang] Merge branch 'master' into check-sessionuser
9cd22003 [liangbowen] generalize illegalAccessWithUnverifiedUser in AuthZUtils
c1f27afa [liangbowen] generalize setSessionUserSign and clearSessionUserSign in SparkOperation
3683150f [liangbowen] update error message with `Invalid user identifier`
e143ea77 [liangbowen] fast fail for getting missing session configs
866fc821 [liangbowen] rename param name of base64 encoded pubkey with `Base64` suffix
280a95ef [liangbowen] refactor config key name to new class `ReservedKeys` in Authz
8592070e [liangbowen] nit
bf227e35 [liangbowen] remove redundant ut case
9ed14feb [liangbowen] remove redundant ut case
33a723d0 [liangbowen] add AuthzSessionSigningSuite with session user sign enabled
b77d53e3 [liangbowen] remove config setting in ut
2d56bd30 [liangbowen] remove config setting in ut
a25c1c0b [liangbowen] fail w/ AccessControlException, when kyuubi.session.user.sign.enabled is true and kyuubi.session.user is absent
c8a88fe9 [liangbowen] check not null of userPubKeyStr, userSign
c8bc590a [liangbowen] rename param to publicKeyBase64
b55beb60 [liangbowen] revert to use EC secp256k1 for compatibility
8acfd41d [liangbowen] nit
842b3698 [liangbowen] nit
983585bc [liangbowen] nit
d1003cd7 [liangbowen] nit
f9d6cfb6 [liangbowen] make generateKeyPair return (PrivateKey, PublicKey)
52eaaddc [liangbowen] add ut for SignUtils
b4a44687 [liangbowen] general keypair algorithm
7d40da49 [liangbowen] change to use secp192r1 curve for EC key pair for better performance
1ceed876 [liangbowen] sync settings.md with `false` default value
07d23602 [liangbowen] update ut
eec9d44a [liangbowen] move to _confIgnoreList
b7969446 [liangbowen] create SessionSigningSuite and add ut for 1. verifying user sign , 2. conf kyuubi.session.user.sign.enabled restriction
a50c71a3 [liangbowen] set kyuubi.session.user.sign.enabled to openEngineSessionConf
a2f1ed67 [liangbowen] set kyuubi.session.user.sign.enabled to _confRestrictList
5a12182e [liangbowen] make kyuubi.session.user.sign.enabled default to false, and removed from serverOnlyConfEntries
991a4569 [liangbowen] put SESSION_USER_SIGN_ENABLED in serverOnlyConfEntries
3e863af4 [liangbowen] nit
b232e5c0 [liangbowen] npe
13a046be [liangbowen] nit
2d2a6659 [liangbowen] nit
a0d4721d [liangbowen] move session signing keypair generation to KyuubiSessionImpl on server side. and rename config to KYUUBI_SESSION_SIGN_PUBLICKEY
ab430c39 [liangbowen] make generateKeyPair return Key pair and accept algorithm param
42ee2fe0 [liangbowen] nit
ce5f4af7 [liangbowen] refactoring session pubkey generation on server side (instead of engine side), and passing it to engine via OpenSession op
c2b9d897 [liangbowen] nit: rename to verifySignWithECDSA
d0b2cddb [liangbowen] nit
33a044d0 [liangbowen] add ut
2dc1f57b [liangbowen] change to use spark conf `kyuubi.session.user.sign.enabled` to decide whether verify kyuubi seesion user
b11ba5a5 [liangbowen] clear local prop `kyuubi.session.user.public.key` and `kyuubi.session.user.sign` after execution
2c8b4bf6 [liangbowen] move throwing AccessControlException inside verifyKyuubiSessionUser method
82f5c265 [liangbowen] update conf doc
7cf0d481 [liangbowen] rename config name to `kyuubi.session.user.sign.enabled`
cc2c3570 [liangbowen] typo: fix KYUUBI_SESSION_USER_PUBIC_KEY
bdec509d [liangbowen] update settings.md for config doc
2d00163a [liangbowen] nit
102561a8 [liangbowen] update
af99ea84 [liangbowen] move algorithmSpec to ecKeyPairGenerator
966a327e [liangbowen] update
41064712 [liangbowen] update
9d276799 [liangbowen] add config `kyuubi.session.user.verify.enabled` to control Whether to verify the integrity of session user name in Spark Authz
8a8840f6 [liangbowen] nit
bafd85e9 [liangbowen] replace RSA with ECDSA
2f0c87a5 [liangbowen] KeyPairGenerator init with new SecureRandom instance
7cb31204 [liangbowen] shorten the key size to 1024
5011cf49 [liangbowen] remove unused imports
045fd822 [liangbowen] add to SparkOperation for scala
ce6d394d [liangbowen] move generateRSAKeyPair and signWithRSA to SignUtils.scala
5f295792 [liangbowen] update
1d7f3191 [liangbowen] initial support for signing and verifying `kyuubi.session.user`
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Avoid too much logs on console in CI.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3864 from pan3793/nit.
Closes#3864
39687176 [Cheng Pan] Add missing log4j2-test.xml for Kyuubi Spark Hive Connector
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3858
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3859 from zhouyifan279/3858.
Closes#3859
af5d7e56 [zhouyifan279] [Bug] Error when authz command ReplaceArcticData
5d54eb91 [zhouyifan279] [Bug] Error when authz command ReplaceArcticData
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3840.
- bump scalafmt from 3.1.1 to 3.6.1
- bump spotless maven plugin from 2.41.1 to 2.72.2
- add `scalaMajorVersion` setting to spotless plugin configs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3841 from bowenliang123/bump-scalafmt.
Closes#3840
1714012f [liangbowen] affect exist code with executing dev/reformat
9bafbb48 [liangbowen] change to scalaMajorVersion config name
56bf677d [liangbowen] bump scala fmt version to 3.6.1 and spotless maven plugin to 2.27.2
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Set Apache Spark3.3 as the default in kyuubi-spark-authz, kyuubi-spark-lineage's readme documentation
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3826 from Yikf/default-spark3.
Closes#3441
b5ba46b7 [Yikf] Make Spark 3.3 as default
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Remove kyuubi dependency of the spark lineage plugin.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3537 from wForget/dev-lineage.
Closes#3537
1cd2f549 [Wang Zhen] comment
71b11506 [Wang Zhen] fix and add test
d7465799 [Wang Zhen] fix test
9e44840f [Wang Zhen] Remove kyuubi dependency of the spark lineage plugin
Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3441;
This pr aims to two points as follow:
- Bump Spark 3.3.0 to Spark 3.3.1 in spark-3.3 profile
- Change default Spark version to 3.3.1
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3507 from Yikf/pr/3471.
Closes#3441
7449280b [yikf] Change spark 3.3.0 as default version
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Subtask of #3607
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3683 from zhouyifan279/3675.
Closes#3675
45b960a4 [zhouyifan279] [KYUUBI #3675][Subtask] Get table owner for DDL/DML v2 commands
d1b4a35f [zhouyifan279] [KYUUBI #3675][Subtask] Get table owner for DDL/DML v2 commands
0ef3b986 [zhouyifan279] [KYUUBI #3675][Subtask] Get table owner for DDL/DML v2 commands
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Subtask of #3607
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3678 from zhouyifan279/3672.
Closes#3672
c621db77 [zhouyifan279] Query in AlterViewAsCommand can not be resolved before SPARK-34698
93cb02cc [zhouyifan279] Table may not exist
59215969 [zhouyifan279] [KYUUBI #3672][Subtask] Get table owner for DDL/DML v1 commands
6359525c [zhouyifan279] [KYUUBI #3672][Subtask] Get table owner for DDL/DML v1 commands
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Subtask of #3607
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3674 from zhouyifan279/3666.
Closes#3666
09f85adb [zhouyifan279] Run test on Spark 3.2 or greater
e33551cd [zhouyifan279] [KYUUBI #3666] Support {OWNER} variable for queries run on CatalogV2
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://iceberg.apache.org/releases/#100-release
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3669 from pan3793/iceberg.
Closes#3669
eb3068c2 [Cheng Pan] scala-collection-compat
23d93616 [Cheng Pan] Bump iceberg from 0.14.1 to 1.0.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3592
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3593 from packyan/branch-spark-sql-authz-only-check-permanent-functions.
Closes#3592
fb59e3d8 [Deng An] isPersistentFunction use current db when db name is missing.
4e708976 [Deng An] use Option other than Some as info.getDb may return null.
5aefc66d [Deng An] fix unit test failed
ac38e0c2 [Deng An] generalizing isPersistentFunction method.
fcef751b [Deng An] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
532fd73d [Deng An] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
7877d916 [Deng An] optimize the code
c7005b36 [packyan] Spark SQL authz only consider permanent functions
Lead-authored-by: Deng An <packyande@gmail.com>
Co-authored-by: Deng An <36296995+packyan@users.noreply.github.com>
Co-authored-by: packyan <packyande@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Subtask of #3607
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3665 from zhouyifan279/3608.
Closes#3608
bcea68d8 [zhouyifan279] DataSourceV2Relation
a50f0a72 [zhouyifan279] Empty-Commit
5c8e2e7b [zhouyifan279] Fix typo
a2c3ab70 [zhouyifan279] Fix typo
31e730ea [zhouyifan279] Move test from RangerSparkExtensionSuite to HiveCatalogRangerSparkExtensionSuite
35c627ec [zhouyifan279] Add `hasTableOwner` field in `SparkSessionProvider`
49a08e0d [zhouyifan279] Improve code style
2d7e6be1 [zhouyifan279] Run test "[KYUUBI #3607] Support {OWNER} variable defined in Ranger Policy" if owner can be resolved
e975f36a [zhouyifan279] [Subtask] Get table owner from CatalogTable #3608
da9f52ce [zhouyifan279] [Subtask] Get table owner from CatalogTable #3608
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3660 from bowenliang123/3635-deletetmp.
Closes#3635
8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Improve the rebalance before writing rule.
The rebalance before writing rule adds a rebalance at the top of query for data writing command, however the default partitioning of rebalance uses RoundRobinPartitioning which would break the original partitioning of data. It may cause the the output data size bigger than before.
This pr supports infer the columns from join and aggregate for rebalance and sort to improve the compression ratio.
Note that, this improvement only works for static partition writing.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3601 from ulysses-you/smart-order.
Closes#3601
c190dc1a [ulysses-you] docs
995969b5 [ulysses-you] view
ea23c417 [ulysses-you] Support infer columns for rebalance and sort
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3636 from bowenliang123/3635-tempdir.
Closes#3635
5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang] delete files on exit in test suits with Utils.createTempDir
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove duplicate and shadowed v2Commands `DropNamespace` and `MergeIntoTable` from `PrivilegesBuilder`, since they are already implemented in #3424 .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3639 from bowenliang123/3424-followup2.
Closes#3424
232dd751 [Bowen Liang] remove `DropNamespace` and `MergeIntoTable` in `PrivilegesBuilder` which shadowed and implemented in v2Commands.scala
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3529
The intent of this PR is the following:
- Add tests related to catalog, including the listTables, loadTable, and listNamespaces methods;
- Initialize the DDL test framework.
- Add CreateNamespaceSuite, DropNamespaceSuite and ShowTablesSuite to check for consistency with V1 in hive connector.
- Rectify the fault that namespaces are deleted in cascades. During cascades, ignore the exception that the table exists in the namespace.
- Fix the tableName problem of HiveTable, which should contain namespace name.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3530 from Yikf/hivev2-test.
Closes#3529
d0af0760 [yikf] Add tests to check for consistency with V1
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3515.
By replacing mapChildren in `RuleApplyRowFilterAndDataMasking`to skip head of children query as insterted by iceberg in `IcebergSparkSessionExtensions` .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3520 from bowenliang123/3515-authz-iceberg.
Closes#3515
a10fe43a [Bowen Liang] improve mapPlanChildren
5eb7f845 [Bowen Liang] fix problem after merging from master
bb1eefbd [Bowen Liang] merge from master
6c6b071c [Bowen Liang] Merge commit 'ae2df990a32c695de6f0345ea7e73e51103e87e5' into 3515-authz-iceberg
dbcfe6d8 [Bowen Liang] restrict skipMapchiled to Iceberg command privilege builders and add skipMappedChildren method to IcebergCommands to handling them
ae2df990 [Bowen Liang] nit
0c691798 [liangbowen] update mapPlanChildren and passSparkVersionCheck
161215d0 [liangbowen] nit
0006dee3 [liangbowen] generalize passSparkVersionCheck method to AuthZUtils
44163638 [liangbowen] refactor getFieldValOpt to getFieldValOption
5b8aa40b [Bowen Liang] improvements on skippedMapChildren of mapPlanChildren
11c2e637 [liangbowen] nit
5a971945 [liangbowen] unifiying general way for ensure skipped table is in plan's children
b161d70f [Bowen Liang] nit
952c1e1c [Bowen Liang] reuse MergeIntoTable of v2Commands to MergeIntoIcebergTable of IcebergCommands
0b25bd1b [Bowen Liang] generalize mapPlanChildren for iceberg commands
04fb651f [Bowen Liang] nit
a1f33bc3 [Bowen Liang] update DELETE FROM TABLE ut
34d65e5a [Bowen Liang] introduce IcebergCommands for access checking iceberg table. skip head child for iceberg commands in RuleApplyRowFilterAndDataMasking to prevent marking output tables.
085bfa31 [liangbowen] repalce mapChildren in RuleApplyRowFilterAndDataMasking with mapPlanChildren method, to skip head child for iceberg UpdateIcebergTable/MergeIntoIcebergTable/DeleteFromIcebergTable
6a245019 [liangbowen] init iceberg ut
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
ban end-user from security settings
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3564 from cfmcgrady/kyuubi-3563.
Closes#3545Closes#3563
9d912b11 [Fu Chen] rename
ee44f7df [Fu Chen] fix ci
158f1552 [Fu Chen] address comment
3cbf4794 [Fu Chen] doc
0125a862 [Fu Chen] address comment
e4e554f5 [Kent Yao] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckExtension.scala
a8d35efa [Fu Chen] ban end-user from security settings
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#3581 .
support row filter and data masking on permanent views,
1. matching permanent views with `PermanentViewMarker` injected by `RuleApplyPermanentViewMarker`
2. reusing attributes's exprId in new Alias for permanent views for data masking
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3576 from bowenliang123/authz-permview-rowfilter.
Closes#3581
84d79a0d [Bowen Liang] restrict ut to spark 3.1+, as permanent view
eb3c9fa9 [liangbowen] update id for ranger policy
634eff0a [liangbowen] nit
48ec536f [liangbowen] add ut for row filter on perm view
892fe207 [liangbowen] only reusing attr.exprId in Alias for permanent view
ae796343 [liangbowen] simplify getting view identifier from PermanentViewMarker
b907f6e4 [Bowen Liang] apply Alias with exprId from original atttribute to solve atttribute confliction problem
5a44cddb [Bowen Liang] update
f08cf694 [liangbowen] add ut and try to adapt conversion
3059c71e [liangbowen] support row filter and data masking on permanent views
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#3488 .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3495 from bowenliang123/3488-authz-createviewusing.
Closes#3488
63812cd3 [liangbowen] skip checking output view of CreateTempViewUsing
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
…and register to JdbcDialects
### _Why are the changes needed?_
close#3487 .
1. add kyuubi-extension-spark-client_2.12 module, and introduce KyuubiSparkClientExtension
2. implement HiveDialect and register to JdbcDialects
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3489 from bowenliang123/3487-hive-jdbc-dialect.
Closes#3487
3ed8be75 [Bowen Liang] nit
47be0ba6 [Bowen Liang] update docs for hive jdbc dialect
84623a35 [Bowen Liang] update pom in minor details
b7edc6cf [Bowen Liang] add ut
968bb722 [Bowen Liang] move to package org.apache.spark.sql.dialect
03eab323 [Bowen Liang] renamed to kyuubi-extension-spark-jdbc-dialect module and moved to extensions/spark
9a4eaf44 [Bowen Liang] add kyuubi-extension-spark-client_2.12 module， implement HiveDialect and register to JdbcDialects
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3464, currently, Kyuubi supports hive connector for read/write hive table, it is implemented based on the [Apache Spark DataSource V2](https://www.databricks.com/session/apache-spark-data-source-v2), but there's a potential issue;
Kyuubi use `kyuubi.engine.single.spark.session`=[false](https://kyuubi.apache.org/docs/latest/deployment/settings.html#:~:text=kyuubi.engine.single.spark.session) to provide concurrency sql execution in context isolation, this cause spark.newSession invoked for each transaction, in spark v1 catalog, externalCatalog is shared in the mutiple session, but in catalog v2 architecture, it's big different with v1, v2 catalogs are managed by `CatalogManager` which is [session level](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala#L97), this means that each session will have a separate catalogManager, so in the case of multiple sessions, hivecatalog will be initialized multiple times. this causes two problems: 1 multiple sessions may be wasted initializing multiple HiveExternalCatalog, which may cause the JVM namespace to swell. 2 multiple HiveClient connections may be initialized;
This issue aims to pool externalCatalog to address the potential issues mentioned above.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3465 from Yikf/catalog-pool.
Closes#3464
5e8a94dd [yikf] Support for pooling external catalog
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#3424.
Covering the following V2 commands,
- v2Commands
- CreateNamespace: CREATE DATABASE
- DropNamespace: DROP DATABASE
- CommentOnNamespace
- CreateTable
- CreateV2Table
- CreateAsSelect
- UpdateTable
- AppendData: Insert Using a SELECT Statement / VALUES / TABLE
- OverwriteByExpression: Insert Overwrite
- OverwritePartitionsDynamic
- CacheTable
- DropTable
- TruncateTable
- AlterTable
- MergeIntoTable
- RepairTable
- v2AlterTableCommands
- CommentOnTable
- AddColumns
- DropColumns
- RenameColumn
- AlterColumn
- ReplaceColumns
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3425 from bowenliang123/3424-authz-access-dsv2.
Closes#3424
c35ee4ef [Bowen Liang] fix: renamed to HasQueryAsLogicPlan. add comment for name rule
8b4324b6 [Bowen Liang] fix import
4c67dd25 [Bowen Liang] adding comments of class and methods
dc23b68c [Bowen Liang] nit: rename field name cmdTypes to commandTypes, param name p to plan
3e05617d [liangbowen] fix enum withName error by reverting command enum to extend super.Val
4c099063 [liangbowen] rename V2Command to CmdPrivilegeBuilder
ef9613c4 [liangbowen] rename all the CommandTypes without "v2" prefix . move V2CommandType into v2Command object and rename to CommandType.
11f06454 [Bowen Liang] nit: reorder fields of V2Command
f27f732b [Bowen Liang] nit: remove extends super.Val
6008e20c [Bowen Liang] nit: rename operType to operationType of V2Commmand
9727102d [Bowen Liang] change leastVer and mostVer from String to Option[String]
ef3cb0d1 [Bowen Liang] moved implmented v2Commands name from OperationType match cases
00eb90c3 [Bowen Liang] support AddPartitions,DropPartitions,RenamePartitions,TruncatePartition in v2Commands
84813a5d [Bowen Liang] nit
29cb69a2 [Bowen Liang] support MergeIntoTable in v2Commands
c9bf97fa [Bowen Liang] support RepairTable in v2Commands
f31abee3 [Bowen Liang] remove operType for TruncateTable
a48f6e23 [Bowen Liang] rename handle method name of v2Commands to buildCommand
afe03dbc [Bowen Liang] nit: optimize imports of v2Commands
f4d3a0ac [Bowen Liang] revert mergeProjectionV1Table method name to mergeProjection for less change to existed code
99ac63c7 [Bowen Liang] support CommentOnNamespace, CommentOnTable in v2Commands
b17bcc69 [Bowen Liang] update ut for TRUNCATE TABLE to isSparkV32OrGreater
35843948 [Bowen Liang] overload applyFilterAndMasking method for V2 Identifier
3e31a339 [Bowen Liang] support TruncatedTable in v2Commands
01f6d6b0 [Bowen Liang] revert DropNamespace to private. add ut for create database / drop database
49453081 [Bowen Liang] revert tablePrivileges to private
b8380086 [Bowen Liang] refactor OverwriteByExpression, OverwritePartitionsDynamic to v2Commands
023210c6 [Bowen Liang] refactor AlterTable to v2Commands
b499d75b [Bowen Liang] revert method name `v1TablePrivileges` back to tablePrivileges
eb678cc7 [Bowen Liang] move v2TablePrivileges method to v2Commands
40c5a1cd [Bowen Liang] nit
a6d2224b [Bowen Liang] refactor AlterColumn, DropColumns, ReplaceColumns, RenameColumn to v2Commands
aaadee32 [Bowen Liang] nit
c307ee76 [Bowen Liang] refactor CacheTable, CacheTableAsSelect to v2Commands
974001db [Bowen Liang] refactor AppendData, UpdateTable, DeleteFromTable to v2Commands
ed82898c [Bowen Liang] refactor DropTable to v2Commands
0eabfb09 [Bowen Liang] refactor support for CreateTableAsSelect, ReplaceTable, ReplaceTableAsSelect command. add support for HasQuery cmdType.
22535b29 [Bowen Liang] introduce cmdTypes in v2Commands and support V2CreateTablePlan. support CreateTable, CreateV2Table command.
39e4aaa1 [Bowen Liang] generalized privileges builder for v2commands with enum and reusable builder for inputObjs and outputObjs
e32ffdf5 [Bowen Liang] nit
e09bacc3 [Bowen Liang] change projectV2Table and mergeProjectionWithIdentifier to mergeProjectionV1Table and mergeProjectionV2Table
f36d5b77 [Bowen Liang] change tablePrivileges and tablePrivilegesWithIdentifier to v1TablePrivileges and v2TablePrivileges
7820b13f [Bowen Liang] optimize getTableIdentifierFromIdentifier
906270c3 [Bowen Liang] use the spark version as condition instead
32d108e2 [Bowen Liang] fix ut for changes in mergeProjectionWithIdentifier
3ab1ed88 [Bowen Liang] reuse plan for columns in mergeProjectionWithIdentifier
8ad9f712 [Bowen Liang] remove redundant actionType param for mergeProjectionWithIdentifier method
094a297a [Bowen Liang] remove test-jar dependency on spark-catalyst_${scala.binary.version}
53d16391 [Bowen Liang] remove test-jar dependency on spark-catalyst_${scala.binary.version}
70595e34 [Bowen Liang] reformat
a3a2429c [Bowen Liang] fix
aa26b566 [Bowen Liang] move ut on JdbcTableCatalog to V2JdbcTableCatalogRangerSparkExtensionSuite.scala file
0f0dc41f [liangbowen] change to use class name in string for org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
82ee98af [liangbowen] nit
128c092f [liangbowen] nit
af2462ce [liangbowen] change to use JDBCTableCatalog as V2TableCatalog. ut pass for spark 3.1+
e612b82b [liangbowen] nit
98eae971 [liangbowen] fix ut
f1b37a3f [Bowen Liang] support AddColumns/DropColumns/RenameColumn/AlterColumn and assume spark3.2+ for ut
7cdbcfe4 [Bowen Liang] Revert "change to import org.apache.spark.sql.catalyst.catalog.InMemoryCatalog for 3.0+"
4afa3178 [Bowen Liang] change to import org.apache.spark.sql.catalyst.catalog.InMemoryCatalog for 3.0+
4b68ea00 [Bowen Liang] separate dsv2 asserts into cases.
06cf1681 [Bowen Liang] update getDatasourceV2Identifier and mergeProjectionWithIdentifier to use Identifier directly
af81a308 [Bowen Liang] update getDatasourceV2Identifier and mergeProjectionWithIdentifier to use Identifier directly
179e5aff [liangbowen] ut for cache table
4b64d7e8 [liangbowen] remove redundant "CreateTableAsSelect" in buildCommand
d4eda229 [liangbowen] nit
3e02eba0 [Bowen Liang] add support and ut for delete v2 catalog table
4718fc29 [Bowen Liang] add support and ut for create/drop/select v2 catalog table
ccaf744a [liangbowen] separate getDatasourceV2ColumnNames method
1d0d5131 [liangbowen] initial support for access checks for dsv2 queries
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3445
temp view should be deleted after test suite, otherwise, some tests which purpose to test show tables will be failure
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3446 from Yikf/flaky-authz-test.
Closes#3445
4f543704 [yikf] clean view temp resource
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Cleanup
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3453 from pan3793/minor.
Closes#3453
e0212223 [Cheng Pan] [MINOR] Remove unused version
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3312
### _Why are the changes needed?_
SQL supported like:
```sql
-- ScalarQuery
select (select a from table0) as aa, b as bb from table0
select (select count(*) from table0) as aa, b as bb from table0
-- Left Semi or Anti Join
select * from table0 where table0.a in (select a from table1)
select * from table0 where table0.a not in (select a from table1)
select * from table0 where exists (select * from table1 where table0.c = table1.c)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3384 from iodone/kyuubi-3312.
Closes#3312
e2af4e1c [odone] change lineage column __aggregate__ to __count__ if exist count(*)
d9c46c34 [odone] add aggregate expression lineage extracting
2fd63482 [odone] add subquery support
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3437
This pr aims to refactory class location of the hive connector
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3438 from Yikf/hive-connector-rename.
Closes#3437
a41dd15b [yikf] Refactory class location of the hive connector
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
