### _Why are the changes needed?_
To provide more insights for kyuubi admin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5064 from turboFei/k8s_log.
Closes#5064
0c072d08e [fwang12] k8s audit
d5f2b44a5 [fwang12] save
c07306283 [fwang12] audit pod
a98234aba [fwang12] log more info
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
When waitCompletion is false, we need to determine whether it is in cluster mode to avoid killing the engine running locally.
close#5043
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5044 from wForget/KYUUBI-5043.
Closes#5043
9d26aea15 [wforget] fix style
aad322f1a [wforget] fix test
20d082ae1 [wforget] [KYUUBI-5043] Destroy the build process when waitCompletion is false and the engine is running in cluster mode
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR allows the user to set the priorityClass used by kyuubi server.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5058 from camper42/priority.
Closes#5058
aa5a9bfe4 [camper42] remove redundant comment
6785270f5 [camper42] remove description field in kyuubi-priorityclass
b8c225524 [camper42] feat(chart): Allows the user to set the priorityClass used by kyuubi.
Authored-by: camper42 <camper.xlii@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now the kubernetes client version in apache/spark is 6.7.2(https://github.com/apache/spark/pull/41490), I think it is better to align with that.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5059 from turboFei/k8s_client_version.
Closes#5059
79955d17a [fwang12] deps
4a68cc820 [fwang12] bump
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR resolves https://github.com/apache/kyuubi/issues/5010, by introducing a new configuration property that makes it possible to override the hostname or ip of the Kyuubi server's thrift frontend service that will be registered for the service discovery ensemble.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
Before applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0`, the znode becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=0.0.0.0:10009;version=1.8.0-SNAPSHOT;sequence=0000000007]
```
After applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0` and `kyuubi.frontend.advertised.host=some_kyuubi_host`, it becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=some_kyuubi_host:10009;version=1.8.0-SNAPSHOT;sequence=0000000009]
```
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
`./build/mvn clean install -pl kyuubi-common` succeeded.
Closes#5015 from touchida/issue-5010-connection-url.
Closes#5010
4598c2baf [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
bf53a8a13 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
0ddf09f0f [toshihiko.uchida] Use pattern matching instead of if-else
34c61d90d [toshihiko.uchida] Rename to kyuubi.frontend.advertised.host and apply to all frontends
7fd980baf [toshihiko.uchida] [KYUUBI #5010] Make Kyuubi server's thrift frontend connection URL configurable
Lead-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Please refer to #4997
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
1. connect to KyuubiServer with beeline
2. Confirm the Application is ACCEPTed in ResourceManager, Restart KyuubiServer
3. Confirmed that Engine was terminated shortly
```
23/06/28 10:44:59 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Current open session is 0
23/06/28 10:45:00 ERROR spark.SparkSQLEngine: Spark engine has been terminated because no incoming connection for more than 60000 ms, deregistering from engine discovery space.
23/06/28 10:45:00 WARN zookeeper.ZookeeperDiscoveryClient: This Kyuubi instance lniuhpi1616.nhnjp.ism:46588 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5002 from risyomei/feature/failfast.
Closes#5002
402d6c01f [Xieming LI] Changed runInNewThread based on comment
58f11e157 [Xieming LI] Changed runInNewThread to non-blocking
c6bb02d6a [Xieming LI] Fixed Unit Test
168d996d0 [Xieming LI] Start countdown after engine is started
48ee819f2 [Xieming LI] Fixed a typo
a8d305942 [Xieming LI] Using runInNewThread ported from Spark
21f0671df [Xieming LI] Updated document
a7d5d1082 [Xieming LI] Changed the default value to turn off this feature
437be512d [Xieming LI] Trigger CI to test agagin
42a847e84 [Xieming LI] Added Configuration for timeout, changed to ThreadPoolExecutor
639bd5239 [Xieming LI] Fail the engine fast when no incoming connection in CONNECTION mode
Authored-by: Xieming LI <risyomei@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Closes#5050.
Check the realUser is an admin or not before verifying proxy access.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5051 from HaoYang670/5050_check_admin_in_get_proxy_user.
Closes#5050
cdabd58d2 [remzi] add admin check
Authored-by: remzi <13716567376yh@gmail.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
To prevent conflicts in case that other dependency also has a resource named `version.properties`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5054 from turboFei/version_conflicts.
Closes#5054
fe1333dbe [fwang12] prevent version properties conflicts
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Return mode helpful information to http client side.
Before this pr, the exception stack is only visible in server side log.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5056 from turboFei/info_err.
Closes#5056
cdda80557 [fwang12] refine
2da2b2072 [fwang12] pretty print
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
To fix#5036, Operation.close() (maybe caused by query timeout) does not update operation complete time.
User may find it confusing and think the query is still running because each refresh on UI will show a different timestamp
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5042 from CavemanIV/kyuubi-5036.
Closes#5036
2e68c0b54 [zhangliang] [KYUUBI-5036] fix Operation.close not update complete timestamp
Authored-by: zhangliang <zhangliang@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
I came cross an issue that is the the value of EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH is 0 when the param is accessed in generateExecutorPodNamePrefixForK8s method.
I tried to move the EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH ahead of generateExecutorPodNamePrefixForK8s method. Then, I found this issue was gone.
So is it necessary to declare the EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH variable before the method definition?
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5045 from zhaohehuhu/Improvement.
Closes#5045
c74732f0c [hezhao2] recover the blank line
99aa14b0c [hezhao2] fix the code style
29929a2cb [hezhao2] declare EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH param before generateExecutorPodNamePrefixForK8s method
Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to support `getQueryId` in Spark engine. It get `sparl.sql.execution.id` by adding a Listener.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5037 from Yikf/spark-queryid.
Closes#5030
9f2b5a3cb [yikaifei] Support get query id in Spark engine
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5035
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5038 from lsm1/features/kyuubi_5035.
Closes#5035
9ee06f113 [senmiaoliu] fix style
c68fd65ab [senmiaoliu] fix style
5d3d86972 [senmiaoliu] show session end time
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to close#4167 which parent issue is #3632
Introduce HiveFunctionPrivilegeBuilder refactored with Serde layers.
The core logic to this is scanning logical plan by transformAllExpressions and build function privileges for UDFs.
As a SQL gateway, Kyuubi also needs to support control user UDF usage behavior.
Therefore, the Spark SQL Authz module needs to add suport for extracting Privilege Objects for UDF usage in queries from a spark logical plans.
In Spark SQL, the hive permanent UDF is wrapped into the following types based on different types:
Org.apache.spark.sql.live.HiveSimpleUDF
Org.apache.spark.sql.live.HiveGenericUDF
Org. apache. park. SQL. live. HiveUDAFFunction
Org.apache.spark.sql.live.HiveGenericUDTF
Based on the existing serde module, we can define a ScanSpec to extract hive permanent udf expression's info, the most important info is the udf type.
And the udf type decide whether to skip the construction of the privilege object based on whether the function type is a system function or a temporary function, as we only consider the permanent udf uasge.
So we should define QualifiedNameStringFunctionExtractor and FunctionNameFunctionTypeExtractor to achieve the goal. And in `org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder#buildFunctions`, we define a method `buildFunctions`, use `transformAllExpressions` to gather all hive udf expression, and building input privilege objects for thoese permanent hive udf.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4168 from packyan/feature_introduce_hive_function_privileges_builder.
Closes#4167
1276ac7f6 [Deng An] [KYUUBI #4167] [Authz] Introduce function support in PrivilegeBuilder with Serde layers
Authored-by: Deng An <packyande@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5014 from link3280/KYUUBI-4938.
Closes#4938
c43d480f9 [Paul Lin] [KYUUBI #4938][FLINK] Update function description
0dd991f03 [Paul Lin] [KYUUBI #4938][FLINK] Fix compatibility problems with Flink 1.16
7e6a3b184 [Paul Lin] [KYUUBI #4938][FLINK] Fix inconsistent istant in engine names
6ecde4c60 [Paul Lin] [KYUUBI #4938][FLINK] Implement Kyuubi UDF in Flink engine
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If the session between kyuubi server and kyuubi engine has been inactive, we should close the kyuubi session as well.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5031 from turboFei/close_session_inactive.
Closes#4847
2eea080b5 [fwang12] fix
964ead778 [fwang12] check engine connection alive
62642f734 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
followup : #4187
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5032 from cxzl25/PR_4187_followup.
Closes#4187
f0b6c89df [sychen] cmd revision_time
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Adding compilation details info in SparkUI's Engine tab
- shows details of kyuubi version, including revision, revision time and branch
- show compilation version of Spark, Scala, Hadoop and Hive
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5019 from bowenliang123/enginetab-info.
Closes#5019
3ea108bd6 [liangbowen] shows Compilation Info in SparkUI's Engine tab
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR aims to fix a bug, In KSHC, `catalog.createTable` should use the correct provider.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5022 from Yikf/KSHC-createTable.
Closes#5022
cd8cb1cf2 [yikaifei] CreateTable should use the correct provider
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
Now the server side session idle time is controlled by kyuubi admin, and the engine session idle time is controlled by users.
If the engine session idle time is short than that of the server session.
The engine session might be closed after engine session idle time, and now the alive probe session will still check the engine alive until the server session idle than the server session idle timeout.
In this pr, we will close the alive probe if the engine session transport is not open.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5029 from turboFei/close_alive_session.
Closes#5029
6ac3d9c1f [fwang12] close alive probe session after engine session closed
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR amins to support Parquet/Orc provider is splitable.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5017 from Yikf/KSHC-support-split.
Closes#5017
9dc3d3d56 [yikaifei] Support Parquet/Orc provider is splitable
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
As title, In KSHC, HiveTable's identify does not attach the catalog to prevent an incorrect catalogName. default catalog is "spark_catalog"
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5023 from Yikf/tableName2.
Closes#5023
86b6a58d0 [yikaifei] KSHC v1IdentifierNoCatalog in spark3.4
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5028 from zhaomin1423/fix_hive_connector.
Closes#5028
d9c7e9c8a [zhaomin] Update session hadoop conf to catalog hadoop conf
Authored-by: zhaomin <zhaomin1423@163.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
The main change is copy code from `kyuubi-extension-spark-common_2.12` and `kyuubi-extension-spark-3-3`. The reason copy `kyuubi-extension-spark-common_2.12` is that, Spark3.4 v1 writes does not have ctas...
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5018 from ulysses-you/extension.
Closes#5018
1ca4951e1 [ulysses-you] fix
fb20bb1f0 [ulysses-you] fix
338a3120a [ulysses-you] rebase
241a436fd [ulysses-you] fix
70618f368 [ulysses-you] fix
5f5e2b7c8 [ulysses-you] fix
301ae5c04 [ulysses-you] fix
bb9416811 [ulysses-you] fix
a38595d0a [ulysses-you] copy
9dec27d6c [ulysses-you] copy
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#1265
After this PR, the following case will work
```sql
CREATE TABLE p (c1 INT, c2 INT, c3 INT) PARTITIONED BY (event_date DATE);
OPTIMIZE p where event_date = current_date() ZORDER BY c1, c2;
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2893 from cfmcgrady/where-expression-support.
Closes#1265
97ac710f0 [Fu Chen] Merge remote-tracking branch 'apache/master' into where-expression-support
c188f0b3d [Fu Chen] fix style
e5f7409d6 [Fu Chen] move verifyPartitionPredicates to KyuubiSparkSQLAstBuilder
f7234abba [Fu Chen] fix style
95d314122 [Fu Chen] fork PredicateHelper.isLikelySelective
1e596e3dd [Fu Chen] partition predicates constraint
541e373cc [Fu Chen] fix
06d9efdf0 [Fu Chen] adapt to spark-3.1/spark-3.2 suite
867263673 [Fu Chen] fix style
b6801b279 [Fu Chen] add test case
79ab60554 [Fu Chen] fix suite bug
cf1b16ee7 [Fu Chen] fix style
dc0ebd908 [Fu Chen] add ut
286d94cc6 [Fu Chen] fix style
1736d18f6 [Fu Chen] adapt to spark-3.1/spark-3.2
04e88a5aa [Fu Chen] fix nep
59103095b [Fu Chen] simplify logical
59fba01e4 [Fu Chen] adapt to spark-3.1
e6477a9c5 [Fu Chen] remove unused
855283e20 [Fu Chen] where clause expression support
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
### _Why are the changes needed?_
This pr amins to make KSHC support Apache Spark 3.4.
- KSHC support Apache Spark 3.4
- Make Apache kyuubi `codecov` module contain the spark-3.4 profile. so that Apache kyubbi CI can cover some modules.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4999 from Yikf/kudu-spark3.4.
Closes#4999
6a35e54b8 [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
66bb742eb [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
7be517c7f [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
ae23133d1 [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
dda5e6521 [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
e43a25dff [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
54f52f16d [Cheng Pan] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveConnectorUtils.scala
0955b544b [Cheng Pan] Update pom.xml
38a1383d9 [yikaifei] codecov module should contain the spark 3.4 profile
Lead-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changes are needed to improve the chart docs and make it self-documented.
The following command prints Helm chart `values.yaml` file:
```shell
helm show values charts/kyuubi
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5003 from dnskr/helm_add_value_docs.
Closes#5003
5938374ec [dnskr] [K8S][HELM] Add doc strings to the chart values.yaml
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- remove unnecessary blank classSparkSimpleStatsReportListener
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5008 from bowenliang123/5007-followup.
Closes#5007
34e19f664 [liangbowen] remove blank SparkSimpleStatsReportListener
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Scalafmt 3.7.5 release note: https://github.com/scalameta/scalafmt/releases/tag/v3.7.5
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5007 from bowenliang123/scalafmt-3.7.5.
Closes#5007
f3f7163a4 [liangbowen] Bump Scalafmt from 3.7.4 to 3.7.5
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
close#5005
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5006 from QianyongY/features/kyuubi-5005.
Closes#5005
3bc8b7482 [yongqian] [KYUUBI #5005] Remove default settings
Authored-by: yongqian <yongqian@trip.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
I met below exception.
```
2023-06-29 00:10:43.174 ERROR org.apache.kyuubi.engine.KubernetesApplicationOperation: Failed to get application with 3c036904-ffef-480b-89e9-2bc6439a6d04, due to Kubernetes context Some(97) is not in the allowed list[ArrayBuffer(97, 116)]
```
The context and namespace are type of `Option[String]` and the allowed lists are type of `Seq[String]`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5004 from turboFei/context_check.
Closes#5004
3a3e8c878 [fwang12] ut
55766313a [fwang12] add ut
68f582bfc [fwang12] fix context and namespace check
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Bump Scala to 2.12.18
- release note: https://github.com/scala/scala/releases/tag/v2.12.18
- Bump silencer-plugin from 1.7.10 to 1.7.13 for compatibility
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5001 from bowenliang123/scala-2.12.18.
Closes#5001
e4a5b9b0d [liangbowen] update dependencyList
e847bbced [liangbowen] Bump silencer-plugin from 1.7.10 to 1.7.13
daceece66 [liangbowen] Bump scala from 2.12.17 to 2.12.18
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This is a followup of #4843
To support load kubernetes conf during runtime, so that we can support more kuberntes contexts without restarting the kyuubi server.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4996 from turboFei/refresh_kubernetes_conf.
Closes#4996
807fb92e4 [fwang12] comments
d42d25af7 [fwang12] from conf
809a7d3df [fwang12] refresh
22743f9e5 [fwang12] save
dedebbe71 [fwang12] api
d6f58cfc7 [fwang12] refresh kubernetes config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#4843
Support to submit kyuubi engine/batch to multiple kubernetes contexts and namespaces.
In this pr, the user can config the kubernetes conf for specified kubernetes context and namespace likes below.
```
kyuubi.kubernetes.<context>.master.address
kyuubi.kubernetes.<context>.<namespace>.authenticate.oauthTokenFile
```
For example:
```
kyuubi.kubernetes.28.master.address=k8s://master
kyuubi.kubernetes.28.ns1.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
kyuubi.kubernetes.28.ns2.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
for k8s context=28, namespace=ns1, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
```
for k8s context=28, namespace=ns2, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
So that, kyuubi server can build kubernetes client for each context and namespace.
### _How was this patch tested?_
Existing kubernetes integration testing.
Closes#4984 from turboFei/k8s_client_yaml.
Closes#4843
f8ffaeeb9 [fwang12] nit
d25774288 [fwang12] comments
5ae7c8433 [fwang12] save into request conf
fd6c363db [fwang12] save
ff004a529 [fwang12] procebuilder method
6b9520bfd [fwang12] save
58850387e [fwang12] save
98df67e5f [fwang12] ut
da811697c [fwang12] fix
aa568aaa4 [fwang12] save
89656f463 [fwang12] check init
a0ef6894b [fwang12] code style
00abb6568 [fwang12] default namespace
295512987 [fwang12] k8s context namespace
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
There are hdfs-site.xml, hive-site, etc in spark job classpath, but we should use hadoop conf and hive conf from catalog options.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4995 from zhaomin1423/fix_hive_connector.
Closes#4995
64429fdcb [Xiao Zhao] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
d921be750 [zhaomin] fix
375934d65 [zhaomin] Using hadoop conf and hive conf from catalog options
Lead-authored-by: zhaomin <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4920
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4921 from lightning-L/kyuubi-4920.
Closes#4920
d395f8a8a [Tianlin Liao] remove KyuubiEvent
3a6730f9b [Tianlin Liao] [KYUUBI #4920] add OperationRestApi
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final
Currently, we can not upgrade to 4.1.94.Final or above because of https://github.com/netty/netty/pull/13408
```
java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4992 from pan3793/netty.
Closes#4992
9dd8f9ad0 [Cheng Pan] nit
e6953144e [Cheng Pan] Bump Netty 4.1.93.Final
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4982
The `targetURL` should include the query string so that kyuubi server can forward request to right page, eg:
```
/engine-ui/spark-43efae88d766226a-driver-svc.kyuubi-dlssjc-canned.svc:4045/jobs/job/?id=4
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4983 from zhaohehuhu/Improvement-4982.
Closes#4982
9140a8a82 [Cheng Pan] simple
3140bd9be [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/EngineUIProxyServlet.scala
421152268 [hezhao2] refactor the logic of append
892c5511e [hezhao2] add queryString as job suffix
53f1ce4b6 [hezhao2] [Improvement] [UI] Add query string to forward requests to Engine UI
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This method is forked from Apache Spark, and recently, [SPARK-44074](https://github.com/apache/spark/issues/41663) identified and fixed the issue about loglevel restore behavior.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4985 from pan3793/test-log.
Closes#4985
4c05bc522 [Cheng Pan] Fix loglevel restore behavior of `KyuubiFunSuite#withLogAppender`
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The assertions are fragile because the session only has 5s as the idle timeout.
```
assert(lastAccessTime === session.lastAccessTime)
assert(sessionManager.getOpenSessionCount === 1)
```
https://github.com/apache/kyuubi/actions/runs/5312620487/jobs/9617377736?pr=4980
```
- close expired operations *** FAILED ***
0 did not equal 1 (TFrontendServiceSuite.scala:544)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4987 from pan3793/flaytest.
Closes#4978
3feacafe1 [Cheng Pan] nit
317f04576 [Cheng Pan] [KYUUBI #4978][FOLLOWUP] Fix flaky test: close expired operations
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://spark.apache.org/releases/spark-release-3-4-1.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4990 from pan3793/spark-3.4.1.
Closes#4990
68203b2c0 [Cheng Pan] Bump Spark 3.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Always use `Files#deleteIfExists` to replace `Files#delete` to suppress the stacktrace like
```
17:14:47.167 ERROR org.apache.kyuubi.operation.BatchJobSubmission: Failed to remove corresponding log file of operation: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
java.io.IOException: Failed to remove corresponding log file of operation: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
at org.apache.kyuubi.operation.log.OperationLog.trySafely(OperationLog.scala:272) ~[classes/:?]
at org.apache.kyuubi.operation.log.OperationLog.close(OperationLog.scala:257) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$2(BatchJobSubmission.scala:328) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$2$adapted(BatchJobSubmission.scala:328) ~[classes/:?]
at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.17.jar:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$1(BatchJobSubmission.scala:328) ~[classes/:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.17.jar:?]
at org.apache.kyuubi.Utils$.withLockRequired(Utils.scala:415) ~[classes/:?]
at org.apache.kyuubi.operation.AbstractOperation.withLockRequired(AbstractOperation.scala:51) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.close(BatchJobSubmission.scala:326) ~[classes/:?]
at org.apache.kyuubi.session.KyuubiBatchSession.close(KyuubiBatchSession.scala:185) ~[classes/:?]
at org.apache.kyuubi.session.KyuubiSessionManager.openBatchSession(KyuubiSessionManager.scala:181) ~[classes/:?]
at org.apache.kyuubi.server.KyuubiBatchService.$anonfun$start$1(KyuubiBatchService.scala:96) ~[classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.nio.file.NoSuchFileException: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) ~[?:?]
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105) ~[?:?]
at java.nio.file.Files.delete(Files.java:1142) ~[?:?]
at org.apache.kyuubi.operation.log.OperationLog.$anonfun$close$4(OperationLog.scala:257) ~[classes/:?]
at org.apache.kyuubi.operation.log.OperationLog.trySafely(OperationLog.scala:263) ~[classes/:?]
... 18 more
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4986 from pan3793/delete.
Closes#4986
7d49bfec0 [Cheng Pan] Always use Files#deleteIfExists
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Please see: https://github.com/apache/kyuubi/issues/4946
Simply put, the kinit service needs to be started before the initLoggerEventHandler, so that it can obtain the DELEGATION TOKEN, which is necessary for accessing the log path on HDFS.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
After this PR, confirmed the LoggerEventHandler can be initiated even kdestroy was executed beforehand.
```
sudo -u kyuubi kdestroy
sudo -u kyuubi /opt/kyuubi/bin/kyuubi start
...
2023-06-19 22:18:36.581 INFO KyuubiTHttpFrontendService: Thread-101 org.eclipse.jetty.server.AbstractConnector: Started ServerConnector50110971{HTTP/1.1, (http/1.1)}{0.0.0.0:10010}
2023-06-19 22:18:36.581 INFO KyuubiTHttpFrontendService: Thread-101 org.eclipse.jetty.server.Server: Started 6081ms
2023-06-19 22:18:36.669 INFO main org.apache.kyuubi.events.handler.ServerJsonLoggingEventHandler: Logging kyuubi events to viewfs://<namespace>/tmp/kyuubi_hive/events/kyuubi_server_info/day=20230619/server-<servername>.json
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Passed the test with kyuubi-server
```
./build/mvn clean install -pl kyuubi-server
```
Closes#4980 from risyomei/fix/kyuubi-4946.
Closes#4946
d729d2384 [Xieming LI] Alter the order of initLoggerEventHandler
Authored-by: Xieming LI <risyomei@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to refactor the method `closeBatchSession`, by
- extracting sub-methods `checkPermission` and `forceKill`
- explicitly handling redirection on `metadata.kyuubiInstance != fe.connectionUrl` to make the logic clearer
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4981 from pan3793/cancel.
Closes#4981
0d2e85acc [Cheng Pan] Refactor code of closeBatchSession
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Currently, there is no error message on Kyuubi server's log when REST API invocation error occurs, and since we are building REST API, all response media types should be JSON.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4977 from pan3793/error-log.
Closes#4977
00c2b0c0f [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/trino/api/api.scala
5fe00474f [Cheng Pan] trino
f988329f8 [Cheng Pan] nit
74f9ed762 [Cheng Pan] log error message for REST api invocation
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/kyuubi/actions/runs/5307034060/jobs/9605209396
```
- support to interrupt the thrift request if remote engine is broken *** FAILED ***
"org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "java.net.SocketException", and "org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "org.apache.thrift.transport.TTransportException", and "org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "connection does not exist" (KyuubiOperationPerConnectionSuite.scala:330)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4979 from pan3793/test-1.
Closes#4979
86807e751 [Cheng Pan] Fix flaky test: support to interrupt the thrift request if remote engine is broken
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
