### _Why are the changes needed?_
Fix#4917
- support extracting table owner from `ResolvedIdentifier`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
<img width="1266" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/e3066d0e-7a14-41da-96f6-032a5c53780f">
Closes#4941 from zhouyifan279/drop-table.
Closes#4917
b2207ed17 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
bc4661a13 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Update doc to mark the spark plugin's config `spark.sql.optimizer.insertRepartitionNum` used for Spark 3.1 only
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4933 from bowenliang123/insert-num.
Closes#4933
5ed6e2867 [liangbowen] comment and style
280a6af03 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
7f01cf3b6 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4936 from zhouyifan279/kill-executor.
Closes#4936
2aaa84cb1 [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- apply the usage of `ReflectUtils` and `Dyn*` to the modules of engines and plugins (eg. Spark engine, Authz plugin, lineage plugin, beeline)
- remove similar redundant methods for calling reflected methods or getting field values
- unified reflection helper methods with type casting support, as `getField[T]` for getting field values from `getFields`, `invokeAs[T]` for invoking methods in `getMethods`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4879 from bowenliang123/reflect-use.
Closes#4879
c685fb67d [liangbowen] bug fix for "Cannot bind static field options" when executing "bin/beeline"
fc1fdf1de [liangbowen] import
59c3dd032 [liangbowen] comment
c435c131d [liangbowen] reflect util usage
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Reuse extractor singleton instance for less memory footprint, as Authz's extractors are stateless and ready for sharing
- Reneralized getter crossing supported extractor types
- get extractor by class type
- get extractor by explicit class name
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4914 from bowenliang123/authz-get-extractor.
Closes#4914
11dde777f [liangbowen] update
77ce00276 [liangbowen] make extractorKey of lookupExtractor not null by default
5f5b6e580 [liangbowen] Revert "extractorKey: String = null => extractorKey: Option[String] = None"
400c3b054 [liangbowen] extractorKey: String = null => extractorKey: Option[String] = None
60acd27ec [liangbowen] rename `getExtractor` to `lookupExtractor`
e6fbb450f [liangbowen] generalize getExtractor for getting instance of supported types of extractors
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Eliminate unnecessary output in ut "union an unmasked table"
<img width="433" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/f179b827-6144-4887-b1fb-ba11da5a5f2b">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4913 from bowenliang123/authz-remove-show.
Closes#4913
2f6f43080 [liangbowen] remove unnecessary output in ut "union an unmasked table"
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- replacing callings to Scala's assert method by Scalatest's `Assertions.assert`
- While Scala's assert method just throws a simple Java's Assertion Error ,
```
def assert(assertion: Boolean) {
if (!assertion)
throw new java.lang.AssertionError("assertion failed")
}
```
the Scalatest's `Assertions.assert` prettifies the error message, eg.,
`assert(a == b || c >= d) // Error message: 1 did not equal 2, and 3 was not greater than or equal to 4`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4912 from bowenliang123/scalatest-assert.
Closes#4912
e1d2ce3e0 [liangbowen] use Scalatest's assert for better error message
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Generalize util method for loading class from service loader in `kyuubi-util-scala` module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4905 from bowenliang123/service-load-util.
Closes#4905
545183fbf [liangbowen] nit
8714e0591 [liangbowen] rename loadClassFromServiceLoader to loadFromServiceLoader
11936419e [liangbowen] nit
81584e335 [liangbowen] fix loadExtractorsToMap
1d64b662d [liangbowen] fix
b7d8895d3 [liangbowen] update
e15b7d22c [liangbowen] optimize JpsApplicationOperationSuite
c58ef573c [liangbowen] simplify ConnectionProvider.loadProviders
31de53df8 [liangbowen] nit
fca265998 [liangbowen] simplify
1fada9516 [liangbowen] import
323b2bd0e [liangbowen] generalize util method for loading class from service loader
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adapting changes in logical plan of DropTable in Spark 3.4 by extracting table object from ResolvedIdntifier, to fix test w/ Spark 3.4 ut "DropTable"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4910 from bowenliang123/authz-resolved-idtable.
Closes#4910
53c76f66d [liangbowen] Extract table from ResolvedIdentifier for DropTable in Spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Fix#4902
We changed `ObjectFilterPlaceHolder` to extend `UnaryNode` so that `CatalogImpl#listDatabases()` can get `ShowNamespaces` object in LogicalPlan.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4903 from zhouyifan279/ShowNamespaces.
Closes#4903
8bf3e1391 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
8698b4a48 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
a9ad36051 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
78d3d6336 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- move `AssertionUtils` class to `kyuubi-util-scala` module's test source
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4904 from bowenliang123/assert-utils.
Closes#4904
ee0c0ad88 [liangbowen] Move AssertionUtils to kyuubi-util-scala module
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adapting changes in logical plan of DropTable in Spark 3.4 by extracting table object from `ResolvedIdntifier`, to fix test w/ Spark 3.4
- ut "DropTable"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4900 from bowenliang123/authz-resolved-idtable.
Closes#4900
560bb6288 [liangbowen] apply ResolvedIdentifierTableExtractor to CreateTable and DropTable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adapting changes in logical plan of CreateFunction/DropFunction in Spark 3.4 by extracting table object from `FunctionIdentifier`, to fix tests on Spark 3.4
- ut "CreateFunctionCommand"
- ut "DropFunctionCommand"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4899 from bowenliang123/authz-functionid.
Closes#4899
464d3eb3d [liangbowen] apply FunctionIdentifierFunctionTypeExtractor to CreateFunction/DropFunction
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- the identifier parts are turned into lower case by default as `spark.sql.caseSensitive` config (including catalog, database, table, function name), in [`SessionCatalog.qualifyIdentifier`](https://github.com/apache/spark/pull/37415/files#diff-9dd0899e5406230aeff96654432da54f35255f6dc60eecb87264a5c508a8c826R161) of <https://github.com/apache/spark/pull/37415>
- fix failed ut in Authz pluin tested w/ Spark 3.4
- AlterTableRenameCommand
- AlterTableAddPartitionCommand
- AlterViewAsCommand
- AlterTableDropPartitionCommand
- RefreshFunctionCommand
- AlterTableRenamePartitionCommand
- AlterTableSetLocationCommand
- AlterTable(Un)SetPropertiesCommand
- TruncateTableCommand
- AlterTableAddColumnsCommand
- AlterTableChangeColumnCommand
- ShowCreateTableAsSerdeCommand
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4892 from bowenliang123/authz-assert-incase.
Closes#4892
8500dd8ed [liangbowen] case insenstive assertion to identifer part name
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- introduce ResolvedIdentifierTableExtractor for extracting table from `org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier` in Spark 3.4
- fixing ut failures w/ Spark 3.4
- ut CreateTable / CreateTableAsSelect / ReplaceTable / ReplaceTableAsSelect
- ut "Extracting table info with ResolvedDbObjectNameTableExtractor"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4869 from bowenliang123/resolved.
Closes#4869
0bf65cd60 [liangbowen] introduce ResolvedIdentifierTableExtractor for spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove FilteredShowDatabasesCommand for filtering "show databases" in Spark 2.x, as ShowDatabasesCommand is removed since Spark 3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4888 from bowenliang123/remove-showdatabasecommand.
Closes#4888
d4ae60bc1 [liangbowen] remove FilteredShowDatabasesCommand for filtering show database in spark 2.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- namespace changed from `Seq[String]` to `ResolvedNamespace` in Spark 3.4
- fixing uts in the name of `CreateNamespace` and `Extracting database info with ResolvedDBObjectNameDatabaseExtractor` w/ Spark 3.4 for CreateNamespace and SetCatalogAndNamespace commands
Testing by running `build/mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.IcebergTest`.
Before:
```
- CreateNamespace *** FAILED ***
0 did not equal 1 (V2CommandsPrivilegesSuite.scala:707)
...
- Extracting database info with ResolvedDBObjectNameDatabaseExtractor *** FAILED ***
java.lang.NullPointerException:
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$24(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:151)
at org.scalatest.Assertions.withClue(Assertions.scala:1065)
at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$21(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:150)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$20(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:143)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
```
After:
```
- CreateNamespace
- Extracting database info with ResolvedDBObjectNameDatabaseExtractor
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4871 from bowenliang123/create-ns.
Closes#4871
fa141fb5a [liangbowen] update spec json
2cd18f829 [liangbowen] use meaningful desc names
142224d1e [liangbowen] Adapt changes for CreateNamespace and SetCatalogAndNamespace
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add reflection utils in kyuubi-util-scala, using kyuubi-util's DynMethods
- continue to provided simplified reflection calling in scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4873 from bowenliang123/authz-reflect.
Closes#4873
d0a508400 [liangbowen] import
95d4760ad [Cheng Pan] Update kyuubi-util-scala/src/main/scala/org/apache/kyuubi/util/reflect/ReflectUtils.scala
83e70f09b [liangbowen] authz reflect
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove unused methods, passSparkVersionCheck and isSparkVersionEqualTo
- extract sparkSemanticVersion singleton
- move spark version helper to AuthzUtils
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4874 from bowenliang123/authz-util-improve.
Closes#4874
36a8bb157 [Bowen Liang] Merge branch 'master' into authz-util-improve
28e798e88 [liangbowen] import
1c345b984 [liangbowen] blank line
0797143da [liangbowen] blank line
2f368b838 [liangbowen] remove unused method passSparkVersionCheck and isSparkVersionEqualTo, extract sparkSemanticVersion
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- remove assuming Spark v2 in Authz testing, since Spark v2 is marked not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4875 from bowenliang123/authz-remove-spark2.
Closes#4875
6686a4d01 [liangbowen] remove checking spark v2
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- latest released Iceberg plugin does not support Spark 3.4, and it also blocks other tests with exception thrown :
```
IcebergCatalogRangerSparkExtensionSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/expressions/AnsiCast
at org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.$anonfun$apply$6(IcebergSparkSessionExtensions.scala:54)
```
- introduce annotation AuthzIcebergTest for Iceberg ut in Authz plugin, for skipping Iceberg test when testing w/ Spark 3.4 locally
-` build/mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.IcebergTest`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4866 from bowenliang123/tag-authz-iceberg.
Closes#4866
3c4348f14 [liangbowen] change to @IcebergTest
ddd1b885c [liangbowen] add AuthzIcebergTest
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4870
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4872 from pan3793/util.
Closes#4870
0b9fe3cba [Cheng Pan] nit
ecc5ee4f2 [Cheng Pan] fix
63be7a20c [Cheng Pan] test
85363c187 [Cheng Pan] style
2227247dd [Cheng Pan] fix package
11d10a081 [Cheng Pan] Add kyuubi-util and kyuubi-util-scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Reduce the scope of method references
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4834 from wForget/minor.
Closes#4834
2061dc942 [wforget] [MINOR] Reduce the scope of method references
Authored-by: wforget <643348094@qq.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Add CI for Spark 3.4
- Kyuubi Spark TPC-DS/H connectors support Spark 3.4
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4741 from pan3793/spark-3.4.
Closes#4741
84a2d6ad7 [Cheng Pan] log
b9b2ec1fb [Cheng Pan] Add spark-3.4 profile
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add MaxFileSizeStrategy to limit max scan file size.
close#4641
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4642 from wForget/KYUUBI-4641.
Closes#4641
14a680f8e [wforget] comment
d2a393d97 [wforget] comment
b1ef4c52c [wforget] fix
d9e94bd8e [wforget] fix style
8a9121131 [wforget] use optional value
094eb61e3 [wforget] combine
89e2cb4d0 [wforget] [KYUUBI-4641] Add MaxFileSizeStrategy to limit max scan file size
Authored-by: wforget <643348094@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#4715
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4717 from packyan/imporve_authz_spec_json_should_be_generated_in_each_build.
Closes#4717
88e70daa7 [Deng An] Update JsonSpecFileGenerator.scala
d195a6db7 [Deng An] Merge branch 'master' into imporve_authz_spec_json_should_be_generated_in_each_build
a078c8c53 [packyan] add ut for check or generate spec json files.
Lead-authored-by: packyan <packyande@gmail.com>
Co-authored-by: Deng An <36296995+packyan@users.noreply.github.com>
Co-authored-by: Deng An <packy@Dengs-MBP.lan>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- align the same list of users and namespaces used in tests and in policy file generation, as users and namespaces are the most important elements of Ranger policy's conditions and resources.
- help to improve and simplify the decision in Authz testing and make a clear view of what's exactly tested and authorized, and very handy and easy to see the usage link in IDE
- reduce possible abuse and untracable uses of authorized and unauthorized users, rules, resources. (We have up to 4 unauthorized users in separated tests!)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4676 from bowenliang123/authz-gen-common.
Closes#4676
dc535a4d8 [liangbowen] authz-gen-common
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Update outdated docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4727 from pan3793/lineage-doc.
Closes#4727
b6843b282 [Cheng Pan] [DOC] kyuubi-spark-lineage has no transitive deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: odone <odone.zhang@gmail.com>
### _Why are the changes needed?_
to close#4715
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4716 from packyan/improve_prevent_edit_auto_generated_files.
Closes#4716
b6fff8fe7 [Deng An] fix the inconsistency in the spec json file
Authored-by: Deng An <packyande@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Fixes#4712
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4718 from anurag-rajawat/upgrade-spark.
Closes#4712
79dcf1b79 [Anurag Rajawat] Bump Spark from 3.2.3 to 3.2.4
Authored-by: Anurag Rajawat <anuragsinghrajawat22@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add ut to check generated Ranger policy file in #4585
- manually activated `genpolicy` profile in CI builds, as the property based activation not auto-triggered as expectedly with property `ranger.version=2.4.0` set in project parent pom
- Support regenerated policy file within the same test suite, by running
`KYUUBI_UPDATE=1 build/mvn clean test -pl :kyuubi-spark-authz_2.12 -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.gen.PolicyJsonFileGenerator -Pgenpolicy`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4677 from bowenliang123/authz-check-policy-gen.
Closes#4677
a372bdfd4 [liangbowen] remove unnecessary profile used in style workflow
7562c88f2 [liangbowen] include in spotless
37b674223 [liangbowen] update policy id
724ec5e28 [liangbowen] replace counter by using zipWithIndex
d322980e7 [liangbowen] extract KRangerPolicyResource object to simplify resource assembly
42c37606a [liangbowen] nit
18a8f4c51 [liangbowen] add usage comments
4ee254d6d [liangbowen] fix issue name in docs
d3cb08d21 [liangbowen] improve file reading
37e4c9c9f [Bowen Liang] Merge branch 'master' into authz-check-policy-gen
6366c50e4 [liangbowen] rename profile to `gen-policy` and remove activation rule by property setting
892faf5ef [liangbowen] update clue
266baa71a [liangbowen] update
cb94e8014 [liangbowen] update
de1f36531 [liangbowen] cleanup
e88c75d46 [liangbowen] check policy file gen
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#4615
- bump Ranger version to 2.4.0, release notes: https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.4.0+-+Release+Notes
- #4585 fixed duplication and conflict in policy file
- update docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4675 from bowenliang123/ranger-2.4.0.
Closes#4615
d403bc324 [liangbowen] bump ranger from 2.3.0 to 2.4.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This pr change two things:
1. add a config to kill executors if the plan contains table caches. It's not always safe to kill executors if the cache is referenced by two write-like plan.
2. force adjustTargetNumExecutors when killing executors. YarnAllocator` might re-request original target executors if DRA has not updated target executors yet. Note, DRA would re-adjust executors if there are more tasks to be executed, so we are safe. It's better to adjuest target num executor once we kill executors.
### _How was this patch tested?_
These issues are found during my POC
Closes#4678 from ulysses-you/skip-cache.
Closes#4678
b12620954 [ulysses-you] Improve kill executors
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
To fix https://github.com/apache/kyuubi/issues/4530.
1. The reason for issue https://github.com/apache/kyuubi/issues/4530 is that MASK_SHOW_FIRST_4 and MASK_SHOW_LAST_4 mask types are currently implemented using the regexp_replace method, which only replaces English letters and digits, but ignores other languages, such as Chinese.
2. To fix this issue, I modified the regexp_replace method to replace no-english characters to 'U' letters, so they will also be masked properly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4643 from huangzhir/fixbug-datamask.
Closes#4530
abe45b278 [huangzhir] fix nit
f74e582ed [huangzhir] Move the data preparation to setup，some tests were modified due to changes in the data.
fb3f89e15 [huangzhir] 1. Modified test methods to perform end-to-end testing. 2. Mask data should not ignore spaces.
bb6406c81 [huangzhir] Rollback unnecessary changes, add tests using SQL queries, and modify the Scala style checking code.
7754d74fd [huangzhir] Switching the plan.Replace all characters except English letters and numbers with a single character 'U'.Preserve the " " character.
a905817a0 [huangzhir] fix
ce23bcd1b [huangzhir] Regression testing is to keep the original tests unchanged, and only add the "regexp_replace" test method.
a39f185dd [huangzhir] 1. Use a ‘密’ replacer for it Chinese chars 2. Use a separate ut cases for testing this regexp_replace method.
94b05db89 [huangzhir] [KYUUBI #4530] [AUTHZ] fixbug support MASK_SHOW_FIRST_4 和 MASK_SHOW_FIRST_4 chinese data mask
0fc1065ca [huangzhir] fixbug support MASK_SHOW_FIRST_4 和 MASK_SHOW_FIRST_4 chinese data mask
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To allow up the reviewer's comment in https://github.com/apache/kyuubi/issues/4660.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4661 from Karsonnel/4658-authz-insert-follow-up.
Closes#4658
3ce7efc3d [Karsonnel] add e2e test for InsertIntoDatasourceCommand
2c8e3469a [Karsonnel] rename test
1349c2b02 [Karsonnel] fix test assert text
d2f04ca45 [Karsonnel] fix test
8f86bb14b [Karsonnel] Resolve reviewer's comment in pr #4660
Authored-by: Karsonnel <747100667@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- generate Authz policy file for testing to focus on manage all the policies in the generator
- auto-increased `id` and related `guid` for each policy
- list and reused users, resources, for evaluation impacts when policy changes
- add `policies_base.json` as base template file of authz template including decoration details and service def
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4585 from bowenliang123/authz-policy-gen.
Closes#4585
c8040553b [liangbowen] authz policy json file gen
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As issue [RANGER-4672](https://github.com/apache/kyuubi/issues/4672) reported unresovled failures on Ranger 0.6.0, update docs for unsupported Ranger version 0.6.0.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4673 from bowenliang123/drop-ranger-0.6.0.
Closes#4672
5049596c0 [liangbowen] update README.md
f9ea53449 [liangbowen] drop support for Ranger 0.6.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This pr fixes a corner case when repartition on a local relation. e.g.,
```
Repartition
|
LocalRelation
```
it would throw exception since there is no a actually shuffle happen
```
java.util.NoSuchElementException: key not found: 3
at scala.collection.MapLike.default(MapLike.scala:235)
at scala.collection.MapLike.default$(MapLike.scala:234)
at scala.collection.AbstractMap.default(Map.scala:63)
at scala.collection.MapLike.apply(MapLike.scala:144)
at scala.collection.MapLike.apply$(MapLike.scala:143)
at scala.collection.AbstractMap.apply(Map.scala:63)
at org.apache.spark.sql.FinalStageResourceManager.findExecutorToKill(FinalStageResourceManager.scala:122)
at org.apache.spark.sql.FinalStageResourceManager.killExecutors(FinalStageResourceManager.scala:175)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4664 from ulysses-you/kill-executors-followup.
Closes#4664
3811eaee9 [ulysses-you] Fix empty relation
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
To fix https://github.com/apache/kyuubi/issues/4658.
### _How was this patch tested?_
Add ut that will run a InsertHiveDirCommand which query from a no permission table
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4660 from Karsonnel/4658-authz-insert.
Closes#4658
1dfb60ea4 [Karsonnel] fix style
8063ec067 [Karsonnel] Update PrivilegesBuilderSuite.scala
4c6c8e1e2 [Karsonnel] add a test in privilegeBuilderSuite
5c652d3df [root] fix InsertIntoHiveDirCommand classname
Lead-authored-by: Karsonnel <747100667@qq.com>
Co-authored-by: root <root@example.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix#4617.
- The reason for issue #4617 is that delegated SparkPlan is not serilizable when execution
- Collect results for filtered show objects ahead in FilterDataSourceV2Strategy to prevent holding the delegated plan
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4634 from bowenliang123/4617-filter.
Closes#4617
fe00ef58a [liangbowen] rename results to result
65ce03a51 [liangbowen] fix 4617
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This pr improves the behavoir of kill redundant executors.
1. support kill executors even if AQE can not optimize shuffle read. e.g., people call `.repartition(2)`
2. fix a issue that avoid always kill executors which holds shuffle data
### _How was this patch tested?_
test manually
Closes#4636 from ulysses-you/kill-executors.
Closes#4636
19ac808d3 [ulysses-you] Improve eagerly kill redundant executors
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add a new rule `InjectCustomResourceProfile` to support custom resource profile for final write stage.
It now supports executor configs:
```
executor core
executor memory
executor memory overhead
executor off heap memory
```
### _How was this patch tested?_
add test and manully test
<img width="778" alt="image" src="https://user-images.githubusercontent.com/12025282/226606147-82a29b8c-1a31-4842-97a7-fe702d80e190.png">
Closes#4615 from ulysses-you/resource-profile.
Closes#4615
852b207cd [ulysses-you] Support stage level schedule for final write stag
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
close#4430
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4566 from iodone/kyuubi-4430.
Closes#4430
9986113b7 [odone] add lateral view support
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
Remove all transitive dependencies to make the down stream project easy to consume.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4620 from pan3793/kshc.
Closes#4620
407f669f5 [Cheng Pan] [KSHC] Cut off transitive dependenices
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr adds a new rule `FinalStageResourceManager` to eagerly kill redundant executors
We first get the final stage partition which is the actually required cores, then kill the redundant executors. The priority of kill executors follow:
1. kill executor who is younger than other (The older the JIT works better)
2. kill executor who produces less shuffle data first
The reason why add this feature is that, if the previous stage contains lots executors but final stage has less, then the tasks of final stage would be scheduled randomly in all exists executors which may cause resource waste. e.g., each executor only run 1 or 2 tasks but holds 4 or 5 cores.
### _How was this patch tested?_
test manually
- test for the kill executor
<img width="755" alt="image" src="https://user-images.githubusercontent.com/12025282/227203809-9fe0731c-f97f-40d2-ac7f-b892a2a35289.png">
Closes#4592 from ulysses-you/eagerly-kill-executors.
Closes#4592
f35208bfd [ulysses-you] nit
ec627ee4f [ulysses-you] nit
28d4230f8 [ulysses-you] address comments
f2492cec6 [ulysses-you] nit
f44e48451 [ulysses-you] Support eagerly kill redundant executors
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR aims to make Kyuubi Spark Hive Connector(KSHC) support kerberized HMS in cluster mode w/o keytab(which is the typical use case in Kyuubi) by implementing a `HadoopDelegationTokenProvider`.
To enable access to an kerberized HMS using KSHC, the minimal configurations are
```
spark.sql.catalog.warm=org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
spark.sql.catalog.warm.hive.metastore.uris=<thrift-uris>
```
then it's able to run federation query across metastores
```
SELECT * FROM spark_catalog.db1.tbl1 JOIN warm.db2.tbl2 ON ...
```
In addition, it allows disabling token renewal for each catalog explicitly
```
spark.sql.catalog.warm.delegation.token.renewal.enabled=false
```
The current implementation has some limitations:
the catalog configuration must be present on the Spark application bootstrap, which means the catalog configurations should be set in `spark-defaults.conf` or append as `--conf` like:
```
spark-[sql|shell|submit] \
--conf spark.sql.catalog.xxx=org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
--conf spark.sql.catalog.xxx.hive.abc=xyz
```
but does not work for dynamic registering through SET statement, e.g. `SET spark.sql.catalog.xxx=`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
> (select count(*) from hive_2.mammut.test_7) union ( select count(*) from spark_catalog.test.test01 limit 1);
+-----------+
| count(1) |
+-----------+
| 4 |
| 1 |
+-----------+
2 rows selected (8.378 seconds)
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4560 from pan3793/shc-token.
Closes#4560
fe8cd0c6d [Cheng Pan] Centralized metastore token signature fallback logic
851159559 [Cheng Pan] comments
fc3b4d596 [Cheng Pan] hive.metastore.token.signature fallback to hive.metastore.uris
fb7eb033f [Cheng Pan] unused import
858b39024 [Cheng Pan] New catalog property delegation.token.renewal.enabled
28ec5a543 [Cheng Pan] disable hms client retry
52044d474 [Cheng Pan] update comments
33b241831 [Cheng Pan] [KSHC] Support Kerberos by implementing KyuubiHiveConnectorDelegationTokenProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
compatible with Spark3.5
see https://issues.apache.org/jira/browse/SPARK-42101
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4587 from ulysses-you/FinalStageConfigIsolation.
Closes#4587
d456f9c99 [Cheng Pan] Update extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala
34db59d6e [ulysses-you] take care of table cache query stage
Lead-authored-by: ulysses-you <ulyssesyou18@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
