### _Why are the changes needed?_
Close#5381
change default metrics reporter to prometheus since Kyuubi 1.8
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5344 from zhaohehuhu/Improvement-0928.
Closes#5381
84f4c8208 [hezhao2] reset METRICS_REPORTERS for test case
b9ee5f711 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
86165a6fe [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
a3605b626 [hezhao2] set METRICS_PROMETHEUS_PORT to 0 for test cases
f1a4d2861 [hezhao2] restore version number for kyuubi.metrics.reporters in doc
dae40e1a2 [hezhao2] change default metrics reporter to prometheus
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- `DROP TABLE` for Iceberg tables only removes the table from catalog by default, which may contaminates other tests with same table
- Enable PURGE option for dropping Iceberg and Hive table
- Iceberg Spark DDL `DROP TABLE ... PURGE`
- To drop the table from the catalog and delete the table’s contents
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5323 from bowenliang123/iceberg-purge.
Closes#5323
ce4188dd2 [Bowen Liang] purge
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Update the regeneration hits for `setting.md` in `develop.md`, as the config doc`settings.md` is checked and able to regenearted by `dev/gen/gen_all_config_docs.sh` script.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5409 from bowenliang123/regen-settings.
Closes#5409
985f39e51 [Bowen Liang] update docs
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5407 .
Follow up for https://github.com/apache/kyuubi/pull/5248 .
Add some UT for snapshot management procedures. These procedures require alter permissions.
1. rollback_to_snapshot (https://iceberg.apache.org/docs/latest/spark-procedures/#rollback_to_snapshot):
Usage: `CALL catalog_name.system.rollback_to_snapshot('db.sample', 1)
`
Meaning: rollback a table to a specific snapshot ID.
2. rollback_to_timestamp (https://iceberg.apache.org/docs/latest/spark-procedures/#rollback_to_timestamp)
Usage: `CALL catalog_name.system.rollback_to_timestamp('db.sample', TIMESTAMP '2021-06-30 00:00:00')`
Meaning: rollback the table to the latest snapshot less than time.
3. set_current_snapshot (https://iceberg.apache.org/docs/latest/spark-procedures/#set_current_snapshot)
Usage: `CALL catalog_name.system.set_current_snapshot('db.sample', 1)`
Meaning: Set a table to a specific snapshot ID.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5394 from yabola/add_call.
Closes#5407
98b7492e0 [Bowen Liang] split into 3 ut for snapshot management
23fe49ae4 [Bowen Liang] refactor ut
8ba97c6ef [chenliang.lu] Add UT for checking previleges in iceberg call snapshot management
Lead-authored-by: chenliang.lu <marssss2929@gmail.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The cleanup follow-up for #5362, which removed the Spark 3.0 tests for Authz plugin.
Remove the `isSpark31OrGreater` and Spark 3.0 specific implementations in Authz plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5399 from bowenliang123/authz-31greater.
Closes#5399
db4369b13 [Bowen Liang] import
cc38b1c1e [Bowen Liang] fix MASK_SHOW_FIRST_4
cf3ef4e1c [Bowen Liang] remove isSparkV31OrGreater
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.
This PR to fix the performance issue on our side.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5400 from ITzhangqiang/KYUUBI_5365.
Closes#5365
dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#5362 .
Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5363 from AngersZhuuuu/KYUUBI-5362.
Closes#5362
d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To add a limitation for batch session lifecycle. otherwise, the batch session might always alive.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5391 from turboFei/close_session_with_maxtimeout.
Closes#5391
5639c7d6b [fwang12] session no op
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar in the following order:
1. `SPARK_SCALA_VERSION` system env
2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5196 from bowenliang123/scala213-test.
Closes#5196
97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Now, the specified engine type and share level does not work
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5353 from turboFei/list_all_engines.
Closes#4994
49ad1acdb [fwang12] refactor
295665bb1 [fwang12] respect the engine share level and type
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Followup #4498
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5354 from turboFei/list_sessions.
Closes#4498
5a3020ac2 [fwang12] list users sessions
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
It is basically copied from `kyuubi-extension-spark-3-4`.
### _How was this patch tested?_
Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5336 from wForget/dev_spark_3_5.
Closes#5336
7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5358 from wForget/bump_iceberg_1_4.
Closes#5358
e03d8d672 [wforget] comment
db0b6b978 [wforget] Bump iceberg 1.4.0
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Prepare for 2023 Kyuubi Code Contribution Program
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1002" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/b6f6ea69-75fa-41b6-b58a-b76e3d87f316">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5356 from pan3793/task.
Closes#5356
6bc5add3d [Cheng Pan] nit
9a68250be [Cheng Pan] nit
44df6ed37 [Cheng Pan] am1
0fa8c9fff [Cheng Pan] nit
498ef8a29 [Cheng Pan] [INFRA] GitHub Issue template for 2023 Kyuubi Code Contribution Program
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add `priority` field in `metadata` table definition
In the current batch v2 implementation, Kyuubi supports the FIFO schedule strategy, we are planning the enhance it with `priority` support, those with a higher integer value of `priority` have a better opportunity to be scheduled.
It refers priority definition in [Apache Hadoop YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setup_for_application_priority) and [Apache YuniKorn](https://yunikorn.apache.org/docs/user_guide/priorities/#application-priority) to define the `priority` as:
1. use a signed 4 bits integer to represent priority with the default value of 10
2. a higher integer value of `priority` has a better opportunity to be scheduled.
In practice, we face some issues when use MySQL as metastore:
MySQL 5.7 could not benefit from index in query mixes `ASC` and `DESC`, for example: `SELECT * FROM metadata ORDER BY priority DESC, create_time ASC`, which is mentioned in [MySQL ORDER BY optimization](https://dev.mysql.com/doc/refman/5.7/en/order-by-optimization.html)
> In some cases, MySQL cannot use indexes to resolve the `ORDER BY`, although it may still use indexes to find the rows that match the `WHERE` clause. Examples:
> - ...
> - The query mixes `ASC` and `DESC`
> - ...
Fortunately, it was enhanced in MySQL 8.0
> Two columns in an ORDER BY can sort in the same direction (both ASC, or both DESC) or in opposite directions (one ASC, one DESC). A condition for index use is that the index must have the same homogeneity, but need not have the same actual direction.
> If a query mixes ASC and DESC, the optimizer can use an index on the columns if the index also uses corresponding mixed ascending and descending columns:
Thus, we create a mixed `ORDER BY` index `ALTER TABLE metadata ADD INDEX priority_create_time_index(priority DESC, create_time ASC)` to improve the query efficiency and avoid slowness in the case of large data volumes.
According to [MySQL 5.7 Index](https://dev.mysql.com/doc/refman/5.7/en/create-index.html), this won't cause SQL error when creating this index in MySQL 5.7 but not take effect.
> A key_part specification can end with ASC or DESC. These keywords are permitted for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.
Close#5327
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Test MySQL & Derby with DBeaver.
MySQL:
Derby:
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5329 from zwangsheng/KYUUBI#5327.
Closes#5327
8688d6bdc [zwangsheng] fix comments
107221b83 [zwangsheng] fix comments
3e4083deb [zwangsheng] fix comments
313b42ba3 [zwangsheng] EOF
eb6dc9c42 [zwangsheng] [KYUUBI #5327][Umbrella][V2] Kyuubi server pick task from metadata db with priority
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
There were obvious errors in the case code, so these cases were fixed.
### _How was this patch tested?_
Just fix case code, no test need. Of course, I have run the case code and have confirmed that there is no problem.
Closes#5347 from mrtisttt/fix-hive-jdbc-example-code.
Closes#5347
ec6ecf758 [mrtisttt] [DOCS] Fix java code errors in quick_start_with_jdbc
Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
### _Why are the changes needed?_
In the `Getting Started with Hive JDBC`documentation, section`Connect to non-kerberized Kyuubi Server`, the code description is incorrect:
```
The below java code is using a keytab file to login and connect to Kyuubi server by JDBC.
```
Obviously, the sections and code here are describing how to connect to a kyuubi service without kerberos authentication.
### _How was this patch tested?_
Just fix code description, no test need.
Closes#5345 from mrtisttt/fix-non-kerberized-code-description.
Closes#5345
fa0418f00 [mrtisttt] [DOCS] Fix non-kerberized code description in quick_start_with_jdbc
Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5340
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5341 from cxzl25/KYUUBI-5340.
Closes#5340
4000b267c [sychen] trigger test
0122e8d7c [sychen] trigger test
86affd3e6 [sychen] trigger test
83cb967ad [sychen] trigger test
37fda1113 [sychen] AdminResourceSuite make engine idle time 3min
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
Replace string literal with constant variable
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5339 from cxzl25/use_engine_init_timeout_key.
Closes#5339
bef2eaa4a [sychen] fix
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We are planning to participate https://github.com/bayoss/Hacktoberfest2023
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5342 from pan3793/hacktoberfest.
Closes#5342
b4860f869 [Cheng Pan] Add label hacktoberfest to project
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Set markdown file EOL avoids spotless check failure in windows.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5335 from wForget/dev.
Closes#5335
7be3ae514 [wforget] Set markdown file EOL
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
After https://github.com/apache/kyuubi/pull/3354, thrift `out of sequence` exception will be thrown when using a higher version of beeline to access a lower version of Kyuubi Server.
```
2023-09-26 11:10:16.426 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error fetching results:
org.apache.thrift.protocol.TProtocolException: Required field 'operationHandle' is unset! Struct:TFetchResultsReq(operationHandle:null, orientation:FETCH_NEXT, maxRows:1000, fetchType:0)
at org.apache.hive.service.rpc.thrift.TFetchResultsReq.validate(TFetchResultsReq.java:548) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_args.validate(TCLIService.java:15755) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_args$FetchResults_argsStandardScheme.write(TCLIService.java:15812) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_args$FetchResults_argsStandardScheme.write(TCLIService.java:15781) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_args.write(TCLIService.java:15732) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_FetchResults(TCLIService.java:561) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Client.FetchResults(TCLIService.java:553) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$fetchResults$1(KyuubiSyncThriftClient.scala:393) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:136) ~[kyuubi-server_2.12-1.6.1-incubating.jar:1.6.1-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_152]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_152]
```
This PR introduces a negotiation mechanism to make the JDBC client adaptively call `fetchLaunchEngineResult` to address such a compatible issue.
It supposes that the higher version of JDBC client could seamlessly communicate with the lower version of Kyuubi Server after this patch, but the JDBC client fetches the engine information only when the connected Server applies this patch too.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5333 from pan3793/launch-engine-result.
Closes#5333
ca9bd5361 [Cheng Pan] [JDBC] Adaptive call fetchLaunchEngineResult
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Follow [Kyuubi Release Guide](https://kyuubi.readthedocs.io/en/master/community/release.html) to update versions for playground and helm charts
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5324 from wForget/dev.
Closes#5324
7110e56bd [wforget] Bump latest 1.7.3 in playground and helm charts
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to enable [GitHub Projects](https://docs.github.com/en/issues/planning-and-tracking-with-projects/learning-about-projects/quickstart-for-projects) for project management boards, we are planning to use this feature for the upcoming 2023 Kyuubi Code Contribution Program.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5322 from pan3793/infra-projects.
Closes#5322
a229b3c4a [Cheng Pan] [INFRA] Enable GitHub Projects
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4994
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5157 from lsm1/branch-kyuubi_4994.
Closes#4994
02e7eb5fa [senmiaoliu] use abbr a
6d001e519 [senmiaoliu] replace 'allengine' with 'all'
f9c548299 [senmiaoliu] fix style
df64e773e [senmiaoliu] fix style
6a7d40e63 [senmiaoliu] remove allenginecommand
6306dd8a2 [senmiaoliu] support list all engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
create a Hive catalog in Flink SQL Client as follows:
```
CREATE CATALOG myhive WITH (
'type' = 'hive',
'hive-conf-dir' = '/opt/hive-conf'
)
```
we should propagate `hive-conf-dir` and pass `HIVE_CONF_DIR` as an env variable to the AppMaster
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5316 from hadoopkandy/KYUUBI-5315.
Closes#5315
3f12acb59 [kandy01.wang] [KYUUBI #5315] [Improvement] Propagate HIVE_CONF_DIR on launching Flink engine
Authored-by: kandy01.wang <kandy01.wang@vipshop.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/5317#issue-1904751001
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5319 from zhaomin1423/fixhive-connector.
Closes#5317
02e5321dc [Cheng Pan] nit
cadabf4ab [Cheng Pan] nit
d38832f40 [zhaomin] improve
ee5b62d84 [zhaomin] improve
794473468 [zhaomin] improve
e3eca91fb [zhaomin] add tests
d9302e2ba [zhaomin] [KYUUBI #5317] [Bug] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
0bc8ec16f [zhaomin] [KYUUBI #5317] [Bug] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
Lead-authored-by: zhaomin <zhaomin1423@163.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
- [x] Manually review.
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5277 from pan3793/spark-3.4-release.
Closes#5202
ce513b333 [Cheng Pan] [KYUUBI #5202][FOLLOWUP] Update upload_nexus_staging
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The current maven-source-plugin version(3.3.0) inherits from `org.apache:parent:pom:30`, unfortunately, MSOURCES-121 breaks shaded modules to create source artifacts.
```
[ERROR] We have duplicated artifacts attached.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-source-plugin:3.3.0:jar-no-fork (attach-sources) on project kyuubi-hive-jdbc-shaded: Presumably you have configured maven-source-plugn to execute twice times in your build. You have to configure a classifier for at least on of them. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-source-plugin:3.3.0:jar-no-fork (attach-sources) on project kyuubi-hive-jdbc-shaded: Presumably you have configured maven-source-plugn to execute twice times in your build. You have to configure a classifier for at least on of them.
```
We should skip upgrading until MSOURCES-141 gets fixed.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5318 from pan3793/mvn-source.
Closes#5318
7d12d12e3 [Cheng Pan] [KYUUBI #5318] Pin maven-source-plugin 3.2.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
For the secured YARN cluster, the Kyuubi Server's user typically has no permission to kill the application. Proxy user or admin should be used instead.
https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_acl.html#concept_yarn_app_acls__section_killing_an_app
> For YARN, the following three groups of users are allowed to kill a running application:
> - The application owner
> - A cluster administrator defined in yarn.admin.acl
> - A queue administrator defined in aclAdministerApps for the queue in which the application is running
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Verified ADMIN mode in internal deployment. (output message is formatted for readable)
```
Error: Batch e351185f-1ed8-437a-91bf-da2174e611e2 failed:
{
"id":"e351185f-1ed8-437a-91bf-da2174e611e2",
"user":"da_music",
"batchType":"SPARK",
"name":"SparkPi",
"appStartTime":0,
"appId":"application_1694730881181_58306",
"appUrl":"http://xxxx-rm-2.xxxx:8088/cluster/app/application_1694730881181_58306",
"appState":"KILLED",
"appDiagnostic":"Application application_1694730881181_58306 was killed by user yarn at 10.49.59.149",
"kyuubiInstance":"kyuubi-1.kyuubi-headless.spark.svc.cluster.local:10099",
"state":"CANCELED",
"createTime":1695102138188,
"endTime":1695102163341,
"batchInfo":{}
}
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5306 from pan3793/kill-proxy-user.
Closes#5306
2b2e54307 [Cheng Pan] address comments
e7e9a9c57 [Cheng Pan] nit
9cf2afc61 [Cheng Pan] polish
ff82d1230 [Cheng Pan] polish
bf0057b41 [Cheng Pan] ApplicationManager supports proxy user
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
In rare cases, when JettyServer is not fully started, we start the recovery task, will get the wrong port.
As a result, we cannot get the task that is not properly submitted by the current kyuubi server instance correctly.
Like :
```
mysql> select identifier, kyuubi_instance from metadata where kyuubi_instance like '%:-1';
+--------------------------------------+----------------------+
| identifier | kyuubi_instance |
+--------------------------------------+----------------------+
| b6e88262-fddb-3ccd-abdf-95ef206e612b | XXXX:-1 |
| 6a02b526-ed10-3b76-ba36-d61fbaf6b28d | XXXX:-1 |
| 70a0446e-d3d0-3b13-9d34-4cc90369c2d9 | XXXX:-1 |
| d290ee2e-b8cd-3bc4-b6c6-72f5b5dfcb9b | XXXX:-1 |
| 9bb10f4-24b4-3eec-b13b-7932b310cb5cd | XXXX:-1 |
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
In local env, start kyuubi server with this PR, kyuubi server start as excpeted.
And i remove start server and runInternal, kyuubi server will wait here.
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5310 from zwangsheng/rest_client/wait_server_start_start_recovery.
Closes#5310
15282c67e [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
d51476b6e [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
a276e39fa [zwangsheng] remove useless thread
2f91f4c85 [zwangsheng] fix comment
a73fa45e8 [zwangsheng] [REST] RestService should wait jetty server started then start to recovery batch session
Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Avoid releasing new version with incorrect java version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5308 from wForget/release_check_java_version.
Closes#5308
8bf3c13ba [Cheng Pan] Update build/release/release.sh
90ddd145d [wforget] comment
b6ab1a480 [wforget] Check java version for release script
Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix UT fail introduced by #5305
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5307 from zwangsheng/KYUUBI#5305-follow-up.
Closes#5305
82a050493 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
5c192c908 [zwangsheng] [KYUUBI #5305][FOLLOWUP] Fix get local log ut
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
…sion manager
### _Why are the changes needed?_
As title
Current we open session first then hand this session with batch by SessionManager:
4136835070/kyuubi-common/src/main/scala/org/apache/kyuubi/session/SessionManager.scala (L97-L121)
We may face `No local log `exception, when kyuubi server picked this batch job to open session but not handled by SessionManager, due to time gap and code gap.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5305 from zwangsheng/server/improve_localLog_when_job_under_submmit.
Closes#5305
14e27c115 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
a5b3ca1f0 [zwangsheng] [V2] Retunr dump log when batch job was picked but not handler by session manager
Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
See more in close#5300
Remove update metadata kyuubi_instance, when open session under batch v2 implementation.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5301 from zwangsheng/server/v2_should_not_update_connect_url.
Closes#5300
0a67bcda1 [zwangsheng] [V2][REST] Not update kyuubi instace when open session
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi v1.7.2 is available now, follow the [release guide](https://kyuubi.readthedocs.io/en/master/community/release.html#keep-other-artifacts-up-to-date) to update version in playground and helm charts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5298 from pan3793/172.
Closes#5298
8af52636e [Cheng Pan] Bump latest 1.7.2 in playgound and helm charts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Due to #5078, we marked kyuubi_instance in metadata can be null.
We should append kyuubi_instance after we check it's non-null.
Otherwise we may face the following error when we use v2 submit job
```
2023-09-14 19:05:43 [INFO] [main] org.apache.kyuubi.client.RetryableRestClient#74 - Current connect server uri http://null/api/v1
2023-09-14 19:05:43 [ERROR] [main] org.apache.kyuubi.client.RestClient#189 - Error:
java.net.UnknownHostException: null: Name or service not known
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5297 from zwangsheng/client/resetclient_get_null.
Closes#5297
2b4b56020 [zwangsheng] [CLIENT] Get null from metadata.kyuubiInstance
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Spark 3.5.0 released in few days ago, this PR aims to add cross-version verification to ensure that Kyuubi engine built against Spark 3.4.1 works on Spark 3.5.0 runtime.
https://spark.apache.org/releases/spark-release-3-5-0.html
For users who want to try Kyuubi with Spark 3.5.0, please compile master/branch-1.8 via
```
./build/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided
```
And then follow the [Quick Start](https://kyuubi.readthedocs.io/en/master/quick_start/quick_start.html) to setup Kyuubi and Spark properly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```log
Connected to: Spark SQL (version 3.5.0)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0-SNAPSHOT)
Beeline version 1.8.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://0.0.0.0:10009/default> select version();
+-------------------------------------------------+
| version() |
+-------------------------------------------------+
| 3.5.0 ce5ddad990373636e94071e7cef2f31021add07b |
+-------------------------------------------------+
1 row selected (0.39 seconds)
0: jdbc:hive2://0.0.0.0:10009/default> select kyuubi_version();
+-------------------+
| kyuubi_version() |
+-------------------+
| 1.8.0-SNAPSHOT |
+-------------------+
1 row selected (0.16 seconds)
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5291 from pan3793/cross-3.5.
Closes#5291
24107a414 [Cheng Pan] Update .github/workflows/master.yml
1554e7131 [Cheng Pan] Verify Spark engine on Spark 3.5
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
upgrade snakeyaml from 1.33 to 2.2 reducing direct CVE vulnerabilities, see (https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes)
[CVE-2022-1471](https://nvd.nist.gov/vuln/detail/CVE-2022-1471)
SnakeYaml's Constructor() class does not restrict types which can be instantiated during deserialization. Deserializing yaml content provided by an attacker can lead to remote code execution. We recommend using SnakeYaml's SafeConsturctor when parsing untrusted content to restrict deserialization. We recommend upgrading to version 2.0 and beyond.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5293 from dev-lpq/snakeyaml_critical.
Closes#5293
5b2412d8e [pengqli] upgrade snakeyaml from 1.33 to 2.2
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Previously, the RESTful audit log did not contain HTTP requests that threw non-AuthenticationException during the process.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5289 from pan3793/auth-log.
Closes#5289
9e292793e [Cheng Pan] RESTful API should always print audit log
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title
We should ref to FIFO design, pick oldest task to run.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5290 from zwangsheng/KYUUBI/metadata_pick_order.
Closes#5290
2d3b09604 [zwangsheng] [REST-V2] Should use ASC order when kyuubi server submitter pick task from metadata
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5211 from ASiegeLion/master.
Closes#5210
6366e962d [peiyue liu] Merge branch 'apache:master' into master
34dcc57dc [liupeiyue] [KYUUBI #5210]Cancel operation will cause the log file node to leak
Lead-authored-by: peiyue liu <945076608@qq.com>
Co-authored-by: liupeiyue <liupeiyue@yy.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR aims to fix the `SparkSubmit` concurrency limit implemented in #5220.
"submitted" judgment in #5220 only considered `OperationState`, actually, `ApplicationState` should be counted too. For instance, if a batch is pending in `ACCEPTED` state, the `SparkSubmit` process won't exit until changed to `RUNNING` or `FAILED` state, in such case, the `OperationState` is `RUNNING` and `ApplicationState` is `PENDING`, it should not be treated as "submitted".
Additionally, this PR treats metastore as the single of truth for batch instead of `SessionManager`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5279 from pan3793/5220-followup.
Closes#5220
903abc6d4 [Cheng Pan] Fix
0af6738d2 [Cheng Pan] [KYUUBI #5220][FOLLOWUP] Batch submmited should not contain application pending state
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Since AnalyzeTableCommand also update table's metadata, since alter command also need table write privilege, AnalyzeTableCommand need too
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5272 from AngersZhuuuu/KYUUBI-5271.
Closes#5271
ad5c70403 [Angerszhuuuu] Merge branch 'KYUUBI-5271' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5271
a5932b7e8 [Angerszhuuuu] Update TableCommands.scala
97ee3299c [Angerszhuuuu] Merge branch 'master' into KYUUBI-5271
cdd8100fd [Angerszhuuuu] Update PrivilegesBuilderSuite.scala
5f110563e [Angerszhuuuu] update
92c30454b [Angerszhuuuu] Revert "Update TableCommands.scala"
504ff2a26 [Angerszhuuuu] Update TableCommands.scala
6c4233680 [Angerszhuuuu] [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- Extract common assertion method for verifying file contents
- Ensure integrity of the file by comparing the line count
- Correct the script name for Spark engine KDF doc generation from `gen_kdf.sh` to `gen_spark_kdf_docs.sh`
- Add `gen_hive_kdf_docs.sh` script for Hive engine KDF doc generation
- Fix incorrect hints for Ranger spec file generation
- shows the line number of the incorrect file content
- Streamingly read file content by line with buffered support
- Regeneration hints:
<img width="656" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/d1a7cb70-8b63-4fe9-ae27-80dadbe84799">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5275 from bowenliang123/doc-regen-hint.
Closes#5275
9af97ab86 [Bowen Liang] implicit source position
07020c74d [liangbowen] assertFileContent
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Before this PR, Kyuubi skips authentication for the call command in iceberg. After this PR, it needs `alter` permission to use this command.
This PR supports checking iceberg call rewrite_data_files command:
- CALL ice.system.rewrite_data_files(table => 'ice.lcl.test_ice', options => map('min-input-files','2'))
- some other call statements in iceberg ( https://iceberg.apache.org/docs/latest/spark-procedures ) may be supported but needs to be further determined. The next phase is planned to support all commands as documented.
When the `rewrite_data_files` command is called, there are two situations:
1. When the triggering conditions are not be met(such as input-files < min-input-files) , there is only one logical plan. We need to check one logical plan permission:
```
* == Physical Plan 1 ==
* Call (1)
```
2. When the triggering conditions are met, two logical plan will be triggered. We should check two logical plans permission.
```
* == Physical Plan 1 ==
* Call (1)
*
* == Physical Plan 2 ==
* AppendData (3)
* +- * ColumnarToRow (2)
* +- BatchScan local.iceberg_ns.call_command_table (1)
```
The second plan (`AppendData`) is a bit special, causing us to have to modify the logic of extracting table names in `DataSourceV2RelationTableExtractor`. The `DataSourceV2Relation#identifier` in `AppendData` will be rewrite task id, so it is a more appropriate way to parse the table name from `DataSourceV2Relation#table`
<img width="1318" alt="image" src="https://github.com/apache/kyuubi/assets/31469905/efc815cb-8dfe-4a9f-8e24-d62883ef6eff">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?
No.
Closes#5248 from yabola/master.
Closes#5248
f3a2d736e [liangbowen] initDataFilesCount
6c0da35f7 [Bowen Liang] skip input checks
4a5ab6da4 [Bowen Liang] comment
850100b6b [Bowen Liang] update
c3158d25a [Bowen Liang] change operation type to ALTERTABLE_PROPERTIES
8d5c8b1f8 [Bowen Liang] rewriteDataFiles2
2d79e1730 [Bowen Liang] nit
36f1d9d60 [chenliang.lu] update description for ut
bd99e0dfa [chenliang.lu] update description for ut
a95e1bb4f [Bowen Liang] nit
7a0089907 [Bowen Liang] nit
06566815b [Bowen Liang] revert opType
1dd211b59 [Bowen Liang] withSingleCallEnabled
657c9dcd3 [Bowen Liang] Merge branch 'master' into yabola/master
cc7dda1f7 [Bowen Liang] update policy
aecfd50cf [Bowen Liang] update table_spec
66ddd89b1 [Bowen Liang] update
b1e12789f [Bowen Liang] remove defaultBob
b8f4fabaf [Bowen Liang] change operation type
465fa48a2 [Bowen Liang] change ut to iceberg namespace
0cfe285ea [Bowen Liang] comment
2b835c943 [Bowen Liang] remove implicit casting in StringTableExtractor
280ad029b [Bowen Liang] removed snapshotCommand tests
83be1d0fd [Bowen Liang] removed snapshotCommand tests
49bd8e14c [Bowen Liang] assertion on all required checks
d65915eb9 [Bowen Liang] separate cases
7513e055f [chenliang.lu] update ut
2ac0e1141 [chenliang.lu] add more ut
31a85a9a2 [Bowen Liang] rename ut
a9773de2f [Bowen Liang] introduce TableTableExtractor
da23d838f [Bowen Liang] input table
45129bc3f [Bowen Liang] update
adbe7535b [Bowen Liang] interceptContains
f24e07d9a [Bowen Liang] update table_command_spec.json
8a71a3c17 [Bowen Liang] rename to CallProcedure
5c0fd5689 [Bowen Liang] rename uts
c0c03176e [Bowen Liang] remove blanks
da55dc597 [Bowen Liang] fix imports
450f0779b [Bowen Liang] rename IcebergCallArgsTableExtractor to ExpressionSeqTableExtractor
0deeffd57 [chenliang.lu] Support iceberg call statement in ranger authentication
414ff2a42 [ChenliangLu] Merge branch 'apache:master' into master
c85059a0b [chenliang.lu] fix comments
85ffd0f88 [chenliang.lu] Support iceberg call statement in ranger authentication
66dbfcc35 [chenliang.lu] optimize codes for table extractor
f1e93fcd7 [chenliang.lu] Support iceberg call statement in ranger authentication
Lead-authored-by: yabola <chenlianglu@tencent.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: chenliang.lu <marssss2929@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: ChenliangLu <31469905+yabola@users.noreply.github.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5269 .
1. AnalyzesTablesCommand in table_command_spec is duplicated and won't be processed
2. The extractor desc of AnalyzesTablesCommand in table_command_spec is wrong
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5270 from AngersZhuuuu/KYUUBI-5269.
Closes#5269
75efc1adb [Angerszhuuuu] Update TableCommands.scala
75a6fa223 [Angerszhuuuu] Update table_command_spec.json
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Previous [version bumping](6061a05f24) missed web-ui, see details #5273
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5274 from pan3793/web-ui-version.
Closes#5274
2c12dbd6e [Cheng Pan] Bump web-ui version to 1.9.0-SNAPSHOT
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5273 from pan3793/doc-release.
Closes#5273
c4f96c69f [Cheng Pan] 1
2266af358 [Cheng Pan] nit
1d52c1a86 [Cheng Pan] Bump version for web-ui and update release docs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
<img width="1080" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/cc118574-c752-449a-9b20-04d2cc59b118">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5227 from hanna-liashchuk/patch-2.
Closes#5227
baf391ba9 [Hanna Liashchuk] fix DOC style
e46dfddc3 [Cheng Pan] Update docs/deployment/engine_on_kubernetes.md
0746a350a [hanna-liashchuk] upd
233bba00b [hanna-liashchuk] Added deploy mode info
Lead-authored-by: Hanna Liashchuk <g.liashchuk@temabit.com>
Co-authored-by: hanna-liashchuk <47921651+hanna-liashchuk@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
