### _Why are the changes needed?_
Fix#4917
- support extracting table owner from `ResolvedIdentifier`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
<img width="1266" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/e3066d0e-7a14-41da-96f6-032a5c53780f">
Closes#4941 from zhouyifan279/drop-table.
Closes#4917
b2207ed17 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
bc4661a13 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Update doc to mark the spark plugin's config `spark.sql.optimizer.insertRepartitionNum` used for Spark 3.1 only
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4933 from bowenliang123/insert-num.
Closes#4933
5ed6e2867 [liangbowen] comment and style
280a6af03 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
7f01cf3b6 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4936 from zhouyifan279/kill-executor.
Closes#4936
2aaa84cb1 [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Bump vite from 3.0.0 to 4.2.3
- Bump vitest from 0.22.0 to 0.32.0
- change package vitest/coverage-c8 to vitest/coverage-v8 as required by vitest (https://github.com/vitest-dev/vitest/pull/3339)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4931 from bowenliang123/vite4.
Closes#4931
4221e2ab0 [liangbowen] bump vite from 3.0.0 to 4.2.3 and vitest from 0.22.0 to 0.32.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- apply the usage of `ReflectUtils` and `Dyn*` to the modules of engines and plugins (eg. Spark engine, Authz plugin, lineage plugin, beeline)
- remove similar redundant methods for calling reflected methods or getting field values
- unified reflection helper methods with type casting support, as `getField[T]` for getting field values from `getFields`, `invokeAs[T]` for invoking methods in `getMethods`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4879 from bowenliang123/reflect-use.
Closes#4879
c685fb67d [liangbowen] bug fix for "Cannot bind static field options" when executing "bin/beeline"
fc1fdf1de [liangbowen] import
59c3dd032 [liangbowen] comment
c435c131d [liangbowen] reflect util usage
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Kyuubi Server Proxy Engine UI
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4795 from zwangsheng/KYUUBI_3420.
Closes#3420
079dc1c60 [zwangsheng] fix frontend unit test case
6e71b4518 [Cheng Pan] fix
cf7ca5145 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala
9a91d62a0 [Cheng Pan] polish
a5dcfae18 [zwangsheng] fix
5d4a8c239 [zwangsheng] Rebase
71d22fc9a [zwangsheng] fix
3b0152f33 [zwangsheng] [KYUUBI #3420][UI] Proxy Engnie UI
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4732
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4813 from huangzhir/operation_log.
Closes#4813
39dcab61d [huangzhir] remove unrelated code
545efaee6 [huangzhir] remove object lock
f0d090f9a [huangzhir] throw exception if user requests FETCH_PRIOR
162e008a1 [huangzhir] remove FETCH_PRIOR test
3b40f6bac [huangzhir] fix style
78e49698c [huangzhir] add extra log test
68154fecb [huangzhir] Merge remote-tracking branch 'origin/master' into operation_log
21c46c06c [huangzhir] code rewritten ,fetch log only support FETCH_NEXT and FETCH_FIRST
cbd714a2b [huangzhir] Add the operationHandle parameter to the /v1/operations/:operationId/log interface.
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
to adapt Spark 3.5, the new conf `spark.sql.execution.arrow.useLargeVarType` was introduced in https://github.com/apache/spark/pull/39572
the signature of function `ArrowUtils#toArrowSchema` before
```scala
def toArrowSchema(
schema: StructType,
timeZoneId: String,
errorOnDuplicatedFieldNames: Boolean): Schema
```
after
```scala
def toArrowSchema(
schema: StructType,
timeZoneId: String,
errorOnDuplicatedFieldNames: Boolean,
largeVarTypes: Boolean = false): Schema
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4923 from cfmcgrady/arrow-toArrowSchema.
Closes#4923
3806494a5 [Fu Chen] Update Arguments of ArrowUtils#toArrowSchema Function
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Reuse extractor singleton instance for less memory footprint, as Authz's extractors are stateless and ready for sharing
- Reneralized getter crossing supported extractor types
- get extractor by class type
- get extractor by explicit class name
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4914 from bowenliang123/authz-get-extractor.
Closes#4914
11dde777f [liangbowen] update
77ce00276 [liangbowen] make extractorKey of lookupExtractor not null by default
5f5b6e580 [liangbowen] Revert "extractorKey: String = null => extractorKey: Option[String] = None"
400c3b054 [liangbowen] extractorKey: String = null => extractorKey: Option[String] = None
60acd27ec [liangbowen] rename `getExtractor` to `lookupExtractor`
e6fbb450f [liangbowen] generalize getExtractor for getting instance of supported types of extractors
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Fix the flaky tests in the "open batch session" functionality within the "BatchesResourceSuite"
to close https://github.com/apache/kyuubi/issues/4918
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4919 from huangzhir/fix_flaky_tests_open_batch_sessio.
Closes#4919
e23087ba8 [huangzhir] To fix the flaky tests in the "open batch session" functionality within the "BatchesResourceSuite"
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Iceberg 1.3.0 was released a few days ago, it brings support for Spark 3.4
https://iceberg.apache.org/releases/#130-release
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4883 from pan3793/iceberg-1.3.
Closes#4883
06545ce0e [Cheng Pan] ignore iceberg test for spark 3.4
6fd4901a5 [Cheng Pan] Bump Iceberg 1.3.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark is migrating to the error class framework, in SPARK-40360(fixed in 3.4.0) SCHEMA_NOT_FOUND was introduced, which changes the output of the error message on switching a not existing database.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4915 from pan3793/schema-not-found.
Closes#4915
c0e03feaf [Cheng Pan] nit
0ee0deada [Cheng Pan] SCHEMA_NOT_FOUND
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Remove test annotations `org.apache.kyuubi.tags.*` which duplicated with `kyuubi-util-scala`'s in `kyuubi-common` module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4911 from bowenliang123/remove-test-tags.
Closes#4911
8244410d5 [liangbowen] remove duplicated tags `org.apache.kyuubi.tags.*` in the common module
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Eliminate unnecessary output in ut "union an unmasked table"
<img width="433" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/f179b827-6144-4887-b1fb-ba11da5a5f2b">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4913 from bowenliang123/authz-remove-show.
Closes#4913
2f6f43080 [liangbowen] remove unnecessary output in ut "union an unmasked table"
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- replacing callings to Scala's assert method by Scalatest's `Assertions.assert`
- While Scala's assert method just throws a simple Java's Assertion Error ,
```
def assert(assertion: Boolean) {
if (!assertion)
throw new java.lang.AssertionError("assertion failed")
}
```
the Scalatest's `Assertions.assert` prettifies the error message, eg.,
`assert(a == b || c >= d) // Error message: 1 did not equal 2, and 3 was not greater than or equal to 4`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4912 from bowenliang123/scalatest-assert.
Closes#4912
e1d2ce3e0 [liangbowen] use Scalatest's assert for better error message
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Authz plugin's tests now pass with Spark3.4.
` mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -am -DskipTests
&& mvn test -pl :kyuubi-spark-authz_2.12 -Pspark-3.4`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4909 from bowenliang123/authz-enabletest-3.4.
Closes#4909
b0bdc106e [liangbowen] enable authz plugin tests with Spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Generalize util method for loading class from service loader in `kyuubi-util-scala` module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4905 from bowenliang123/service-load-util.
Closes#4905
545183fbf [liangbowen] nit
8714e0591 [liangbowen] rename loadClassFromServiceLoader to loadFromServiceLoader
11936419e [liangbowen] nit
81584e335 [liangbowen] fix loadExtractorsToMap
1d64b662d [liangbowen] fix
b7d8895d3 [liangbowen] update
e15b7d22c [liangbowen] optimize JpsApplicationOperationSuite
c58ef573c [liangbowen] simplify ConnectionProvider.loadProviders
31de53df8 [liangbowen] nit
fca265998 [liangbowen] simplify
1fada9516 [liangbowen] import
323b2bd0e [liangbowen] generalize util method for loading class from service loader
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adapting changes in logical plan of DropTable in Spark 3.4 by extracting table object from ResolvedIdntifier, to fix test w/ Spark 3.4 ut "DropTable"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4910 from bowenliang123/authz-resolved-idtable.
Closes#4910
53c76f66d [liangbowen] Extract table from ResolvedIdentifier for DropTable in Spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Scalafmt 3.7.4 release note: https://github.com/scalameta/scalafmt/releases/tag/v3.7.4
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4908 from bowenliang123/scalafmt-3.7.4.
Closes#4908
466686a97 [liangbowen] Bump scalafmt to 3.7.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Fix#4902
We changed `ObjectFilterPlaceHolder` to extend `UnaryNode` so that `CatalogImpl#listDatabases()` can get `ShowNamespaces` object in LogicalPlan.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4903 from zhouyifan279/ShowNamespaces.
Closes#4903
8bf3e1391 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
8698b4a48 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
a9ad36051 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
78d3d6336 [zhouyifan279] [KYUUBI#4902] Fix NoSuchElementException when listing database in CatalogImpl in Spark 3.4
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
IntelliJ IDEA can not use the right Spark version to run UT when we switches between multiple Spark versions.
This PR adds a maven profile 'remote-debug' to let UT open a debug port. So we can run UI using mvn cli and remote debug in IDEA.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1273" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/439164d7-c1b3-4233-a417-b34aa15b428a">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4907 from zhouyifan279/ut-debug.
Closes#4907
ad2b06dbc [zhouyifan279] [KYUUBI#4906][Improvement] Add a maven profile 'remote-debug' to ease unit test remote debugging
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- move `AssertionUtils` class to `kyuubi-util-scala` module's test source
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4904 from bowenliang123/assert-utils.
Closes#4904
ee0c0ad88 [liangbowen] Move AssertionUtils to kyuubi-util-scala module
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adapting changes in logical plan of DropTable in Spark 3.4 by extracting table object from `ResolvedIdntifier`, to fix test w/ Spark 3.4
- ut "DropTable"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4900 from bowenliang123/authz-resolved-idtable.
Closes#4900
560bb6288 [liangbowen] apply ResolvedIdentifierTableExtractor to CreateTable and DropTable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to adapt Spark 3.5
the signature of function `ArrowConverters#fromBatchIterator` is changed in [SPARK-43528](https://github.com/apache/spark/pull/41190) (since Spark 3.5)
Spark 3.4 or previous
```scala
private[sql] def fromBatchIterator(
arrowBatchIter: Iterator[Array[Byte]],
schema: StructType,
timeZoneId: String,
context: TaskContext): Iterator[InternalRow]
```
Spark 3.5 or later
```scala
private[sql] def fromBatchIterator(
arrowBatchIter: Iterator[Array[Byte]],
schema: StructType,
timeZoneId: String,
errorOnDuplicatedFieldNames: Boolean,
context: TaskContext): Iterator[InternalRow]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4895 from cfmcgrady/arrow-spark35.
Closes#4895
87d5b7240 [Fu Chen] fix ci
b37b321d5 [Fu Chen] adapt Spark 3.5
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4897
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4898 from huangzhir/operation_log_bug.
Closes#4898
5d630dd31 [huangzhir] use do while style
87fc2d23d [huangzhir] fix
6cb0e5561 [huangzhir] fix logOperation multiple read with missing line
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- adapting changes in logical plan of CreateFunction/DropFunction in Spark 3.4 by extracting table object from `FunctionIdentifier`, to fix tests on Spark 3.4
- ut "CreateFunctionCommand"
- ut "DropFunctionCommand"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4899 from bowenliang123/authz-functionid.
Closes#4899
464d3eb3d [liangbowen] apply FunctionIdentifierFunctionTypeExtractor to CreateFunction/DropFunction
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- the identifier parts are turned into lower case by default as `spark.sql.caseSensitive` config (including catalog, database, table, function name), in [`SessionCatalog.qualifyIdentifier`](https://github.com/apache/spark/pull/37415/files#diff-9dd0899e5406230aeff96654432da54f35255f6dc60eecb87264a5c508a8c826R161) of <https://github.com/apache/spark/pull/37415>
- fix failed ut in Authz pluin tested w/ Spark 3.4
- AlterTableRenameCommand
- AlterTableAddPartitionCommand
- AlterViewAsCommand
- AlterTableDropPartitionCommand
- RefreshFunctionCommand
- AlterTableRenamePartitionCommand
- AlterTableSetLocationCommand
- AlterTable(Un)SetPropertiesCommand
- TruncateTableCommand
- AlterTableAddColumnsCommand
- AlterTableChangeColumnCommand
- ShowCreateTableAsSerdeCommand
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4892 from bowenliang123/authz-assert-incase.
Closes#4892
8500dd8ed [liangbowen] case insenstive assertion to identifer part name
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add unit tests for `ClassUtils`
- refactor `ClassUtils.createInstance` method with DynConstructors
- move `classIsLoadable` method to `ReflectUtils.isClassLoadable`, refeactor to use DynClasses
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4887 from bowenliang123/classutil.
Closes#4887
c39f9a0a4 [liangbowen] simplify ut
633e21ddc [liangbowen] Refactor and add ut for ClassUtils
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4893 from turboFei/minor_dto.
Closes#4893
aeb5a25a9 [fwang12] prevent null:
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support to list server with kyuubi-admin/kyuubi rest client.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- UT for AdminRestAPI
- UT for AminCtlArgument
- UT for AdminCtl
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4889 from turboFei/list_server.
Closes#4889
bfd13fbde [fwang12] nit
0a0131552 [fwang12] list server
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- introduce ResolvedIdentifierTableExtractor for extracting table from `org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier` in Spark 3.4
- fixing ut failures w/ Spark 3.4
- ut CreateTable / CreateTableAsSelect / ReplaceTable / ReplaceTableAsSelect
- ut "Extracting table info with ResolvedDbObjectNameTableExtractor"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4869 from bowenliang123/resolved.
Closes#4869
0bf65cd60 [liangbowen] introduce ResolvedIdentifierTableExtractor for spark 3.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
test against delta-2.4.0
https://github.com/delta-io/delta/releases/tag/v2.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4878 from cfmcgrady/delta-2.4.0.
Closes#4878
537e556f5 [Fu Chen] Update pom.xml
e89b97125 [Fu Chen] bump delta 2.4.0
ae32f62f4 [Fu Chen] fix style
9789a5e04 [Fu Chen] delta support for Spark-3.4
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4877 from link3280/bump-flink-1.17.1.
Closes#4877
c94c00559 [Paul Lin] Bump Flink 1.17.1
bfd865ac1 [Paul Lin] Fix invalid download url
f2b09559d [Paul Lin] Fix flink jar not found with no profile specified
f63f3842c [Paul Lin] Fix flink download error with no profile specified
3055b6c88 [Paul Lin] Fix spotless
07b62fd95 [Paul Lin] Bump Flink 1.17.1-rc1
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4891 from link3280/bump-flink-1.16.2.
Closes#4891
953f83014 [Paul Lin] Bump Flink 1.16.2
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Return operation metrics and fix typo.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4890 from turboFei/op_data_metrics.
Closes#4890
ef2ed36eb [fwang12] return metrics and fix bug
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- remove FilteredShowDatabasesCommand for filtering "show databases" in Spark 2.x, as ShowDatabasesCommand is removed since Spark 3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4888 from bowenliang123/remove-showdatabasecommand.
Closes#4888
d4ae60bc1 [liangbowen] remove FilteredShowDatabasesCommand for filtering show database in spark 2.4
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- namespace changed from `Seq[String]` to `ResolvedNamespace` in Spark 3.4
- fixing uts in the name of `CreateNamespace` and `Extracting database info with ResolvedDBObjectNameDatabaseExtractor` w/ Spark 3.4 for CreateNamespace and SetCatalogAndNamespace commands
Testing by running `build/mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.IcebergTest`.
Before:
```
- CreateNamespace *** FAILED ***
0 did not equal 1 (V2CommandsPrivilegesSuite.scala:707)
...
- Extracting database info with ResolvedDBObjectNameDatabaseExtractor *** FAILED ***
java.lang.NullPointerException:
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$24(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:151)
at org.scalatest.Assertions.withClue(Assertions.scala:1065)
at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$21(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:150)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.kyuubi.plugin.spark.authz.V2JdbcTableCatalogPrivilegesBuilderSuite.$anonfun$new$20(V2JdbcTableCatalogPrivilegesBuilderSuite.scala:143)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
```
After:
```
- CreateNamespace
- Extracting database info with ResolvedDBObjectNameDatabaseExtractor
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4871 from bowenliang123/create-ns.
Closes#4871
fa141fb5a [liangbowen] update spec json
2cd18f829 [liangbowen] use meaningful desc names
142224d1e [liangbowen] Adapt changes for CreateNamespace and SetCatalogAndNamespace
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To speed up the making distribution.
Refer https://github.com/apache/spark/pull/41141
This PR add more skip properties when making distribution:
- -Dmaven.javadoc.skip=true to skip generating javadoc
- -Dmaven.scaladoc.skip=true to skip generating scaladoc. Please see: https://davidb.github.io/scala-maven-plugin/doc-jar-mojo.html#skip
- -Dmaven.source.skip to skip generating sources.jar
### _How was this patch tested?_
Manual test:
<img width="267" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/af4e7bb5-10f1-4fa0-b212-b779b77c9ff3">
```
./build/dist --spark-provided --hive-provided --flink-provided
```
Before this pr:
```
[INFO] Total time: 09:20 min
```
After this pr:
```
[INFO] Total time: 06:46 min
```
Closes#4880 from turboFei/speedup_dist.
Closes#4880
47f5ef2ac [fwang12] [BUILD] Skip compiling useless files when making distribution
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- unify config value capitalization for String and Seq[String] configs
- Generalize `transformToUpperCase` and `transformToLowerCase` for config entry
- simplify transformation for configs of `kyuubi.authentication` and `kyuubi.frontend.protocols`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4812 from bowenliang123/conf-upper.
Closes#4812
747c2955c [bowenliang] upper and lower case for config values
Authored-by: bowenliang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add reflection utils in kyuubi-util-scala, using kyuubi-util's DynMethods
- continue to provided simplified reflection calling in scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4873 from bowenliang123/authz-reflect.
Closes#4873
d0a508400 [liangbowen] import
95d4760ad [Cheng Pan] Update kyuubi-util-scala/src/main/scala/org/apache/kyuubi/util/reflect/ReflectUtils.scala
83e70f09b [liangbowen] authz reflect
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove unused methods, passSparkVersionCheck and isSparkVersionEqualTo
- extract sparkSemanticVersion singleton
- move spark version helper to AuthzUtils
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4874 from bowenliang123/authz-util-improve.
Closes#4874
36a8bb157 [Bowen Liang] Merge branch 'master' into authz-util-improve
28e798e88 [liangbowen] import
1c345b984 [liangbowen] blank line
0797143da [liangbowen] blank line
2f368b838 [liangbowen] remove unused method passSparkVersionCheck and isSparkVersionEqualTo, extract sparkSemanticVersion
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- remove assuming Spark v2 in Authz testing, since Spark v2 is marked not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4875 from bowenliang123/authz-remove-spark2.
Closes#4875
6686a4d01 [liangbowen] remove checking spark v2
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- latest released Iceberg plugin does not support Spark 3.4, and it also blocks other tests with exception thrown :
```
IcebergCatalogRangerSparkExtensionSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/expressions/AnsiCast
at org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.$anonfun$apply$6(IcebergSparkSessionExtensions.scala:54)
```
- introduce annotation AuthzIcebergTest for Iceberg ut in Authz plugin, for skipping Iceberg test when testing w/ Spark 3.4 locally
-` build/mvn clean install -pl :kyuubi-spark-authz_2.12 -Pspark-3.4 -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.IcebergTest`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4866 from bowenliang123/tag-authz-iceberg.
Closes#4866
3c4348f14 [liangbowen] change to @IcebergTest
ddd1b885c [liangbowen] add AuthzIcebergTest
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4870
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4872 from pan3793/util.
Closes#4870
0b9fe3cba [Cheng Pan] nit
ecc5ee4f2 [Cheng Pan] fix
63be7a20c [Cheng Pan] test
85363c187 [Cheng Pan] style
2227247dd [Cheng Pan] fix package
11d10a081 [Cheng Pan] Add kyuubi-util and kyuubi-util-scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
