### _Why are the changes needed?_
`branch-1.5` and `branch-1.6` don't active for a while, the currently active maintained branches are `master` and `branch-1.7`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5235 from pan3793/snapshot.
Closes#5235
d63e1a570 [Cheng Pan] [INFRA] Daily deploy snapshot version on master and branch-1.7
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Closes#3444
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3558 from iodone/kyuubi-3444.
Closes#3444
acaa72afe [odone] remove plugin dependency from kyuubi spark engine
739f7dd5b [odone] remove plugin dependency from kyuubi spark engine
1146eb6e0 [odone] kyuubi-3444
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR aims to unify the exception handling of v1 and v2 during dropDatabase
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5225 from Yikf/hive-connector.
Closes#5225
3be33af76 [yikaifei] [KSHC] Improve test
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
The generated application name is not effective in Flink app mode. The PR moves the name generating to the `ProcessBuilder`.
The generated app name would be like `kyuubi_USER_FLINK_SQL_myuser_default_382c0371-8cc1-4aec-90bd-a2acf4de6fac`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5200 from link3280/flink_app_name.
Closes#5200
bf06d1c16 [Paul Lin] Fix engine name udf test
6aa09e462 [Paul Lin] Filter out unused conf in app mode
957d18c42 [Paul Lin] Fix test error in local mode
eaa5de9b4 [Paul Lin] Fix engine name missing in tests
109ff46f5 [Paul Lin] Fix test error
efb1cda82 [Paul Lin] Fix compatibility with YARN and local
65e6759b2 [Paul Lin] Remove unused import
49860f65e [Paul Lin] Optimize Flink application name generating
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR changes the block phase of the batch submitter
- before: from PENDING until TERMINATED
- after: from PENDING until RUNNING or TERMINATED
Usually, we submit Spark batch applications in cluster mode with waitAppCompletion disabled, after a Spark application goes into the RUNNING or TERMINATED stage, the `spark-submit` process exits and the batch session does not occupy too many resources. Thus, limiting the concurrency on the submitting phase instead of the whole lifecycle of the Spark app makes more sense.
In practice, we use 16 threads for Kyuubi instance with 8C32G. A larger concurrency may result in CPU resources being exhausted and `spark-submit` process hanging.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5220 from pan3793/batch-submitter-threads.
Closes#5220
543c31cd1 [Cheng Pan] nit
dc5d0c816 [Cheng Pan] nit
9b499b0eb [Cheng Pan] Batch submmitter should only block during submit stage
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We shall create session limiter if user unlimited list or deny list nonempty.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5215 from turboFei/typo_deny.
Closes#4540
c26dec74e [fwang12] comments
01fc61619 [fwang12] fix typo
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to modify the README file of authz to make spark 3.4 the default version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5224 from Yikf/authz-ci-across-spark.
Closes#5224
7e28f8619 [yikaifei] fix authz spark version and ci
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
- Remove the provided dependency `flink-table-planner_${scala.binary.version}` which provides a legacy table planner API on Scala, but is never used in Kyuubi's source code or in runtime directly.
- `** The legacy planner is deprecated and will be dropped in Flink 1.14.**` according to [Flink 1.13's doc of Legacy Planner](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/table/legacy_planner/)
- Kyuubi has dropped support for Flink 1.14 and before in #4588
- Remove the unused provided dependency `flink-sql-parser`
- All tests on Scala 2.12 work fine without them, as `flink-table-runtime` dependency provides enough Java API for usage.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5222 from bowenliang123/flink-remove-planner.
Closes#5222
716ec06e9 [liangbowen] remove flink-sql-parser dependency
0922ba5af [liangbowen] Remove unnecessary dependency flink-table-planner
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- check config values in the range of targeted enum values
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5213 from bowenliang123/config-checkenum.
Closes#5213
857af7c6b [Bowen Liang] is valid enum
2862e5e7c [Bowen Liang] embrace
afe6a5333 [Bowen Liang] assert
00d22f73b [liangbowen] support checking config in the range of enumeration values
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR improves the log message of `KubernetesApplicationOperation`, by adding `kyuubi-unique-tag=${tag_value}` to each message, which is important for analyzing while the thing goes wrong.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5218 from pan3793/log-tag.
Closes#5218
263fab9ff [Cheng Pan] revert irrelevant changes
610738b20 [Cheng Pan] nit
ad3bf97a5 [Cheng Pan] improve logs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Move Scala tests in `kyuubi-util-scala` module from `src/test/java`back to the correct path `src/test/scala`
- Apply Scala style fixes
- Add handy assertion methods for intercepting exception
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5214 from bowenliang123/scalautil-test.
Closes#5214
55bc38cde [liangbowen] correct file path for scala tests in util-scala module
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Improvement: https://github.com/apache/kyuubi/issues/4540
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5182 from zhuyaogai/issue-4540.
Closes#4540
73652ae6d [Fantasy-Jay] Merge remote-tracking branch 'origin/issue-4540' into issue-4540
ae7a3b5bb [Fantasy-Jay] Support to deny some user to make connection.
46ee2f158 [Fantasy-Jay] Support to limit some user to make connection.
3d7e5e534 [Fantasy-Jay] Support to limit some user to make connection.
1f173ffe4 [Fantasy-Jay] Merge branch 'master' into issue-4540
1d3a3d47f [Fantasy-Jay] Support to limit some user to make connection.
e4a1b3eb3 [Fantasy-Jay] Support to limit some user to make connection.
1c120a992 [Fantasy-Jay] Support to limit some user to make connection.
6b657aa99 [Fantasy-Jay] Support to limit some user to make connection.
506c4751b [Fantasy-Jay] Support to limit some user to make connection.
f99072ef5 [Fantasy-Jay] Support to limit some user to make connection.
Authored-by: Fantasy-Jay <13631435453@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now for `KubernetesApplicationOperation`, it rely on the appInfoStore.
For batch rest api, if the closeBatch request can not be send to the kyuubiInstance that created the batch, the current kyuubiInstance will try to kill the batch.
I wonder that, in this case, the applicationInfo might can not be found in the appInfoStore.
It is better to try the best to kill the pod with `kyuubi-unique-tag` label to prevent pod leak.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5206 from turboFei/k8s_pod_kill.
Closes#5206
630bcd04b [fwang12] warning
a9c22e03f [fwang12] delete with label
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- adding source code in `src/main/scala-2.12/**/*.scala` and `src/main/scala-2.13/**/*.scala` to including paths for Scala spotless styling
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5209 from bowenliang123/scala-reformat.
Closes#5209
84b4205e9 [liangbowen] use wildcard for scala versions
0c32c582d [liangbowen] reformat scala-2.12 specific code
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR fixes a NPE when client opens session with null configuration using Thrift HTTP protocol
```
2023-08-28 02:03:56.512 ERROR org.apache.kyuubi.server.KyuubiTHttpFrontendService: Error opening session:
java.lang.NullPointerException: null
at org.apache.kyuubi.service.TFrontendService.getProxyUser(TFrontendService.scala:130) ~[kyuubi-common_2.12-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at org.apache.kyuubi.server.KyuubiTHttpFrontendService.getRealUserAndSessionUser(KyuubiTHttpFrontendService.scala:281) ~[kyuubi-server_2.12-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at org.apache.kyuubi.service.TFrontendService.getSessionHandle(TFrontendService.scala:168) ~[kyuubi-common_2.12-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at org.apache.kyuubi.service.TFrontendService.OpenSession(TFrontendService.scala:190) ~[kyuubi-common_2.12-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482) ~[hive-service-rpc-3.1.3.jar:3.1.3]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.kyuubi.server.http.ThriftHttpServlet.doPost(ThriftHttpServlet.scala:146) ~[kyuubi-server_2.12-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) ~[jakarta.servlet-api-4.0.4.jar:4.0.4]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5204 from pan3793/http-null.
Closes#5204
1116b5efc [Cheng Pan] Thrift HTTP FE checks req conf nullable before evaluating proxy user
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi fully supports Spark 3.4 now, it's time to move forward.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5202 from pan3793/default-3.4.
Closes#5202
a0efccdbf [Cheng Pan] nit
30456dbb9 [Cheng Pan] nit
1cc83c871 [Cheng Pan] enable lineage test
d8ca7c7d8 [Cheng Pan] Switch to Spark 3.4 by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It makes no sense to enable batch impl v2 when `kyuubi.batch.submitter.enabled` is disabled, to avoid misuse, globally disable batch impl v2 in such case.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5201 from pan3793/batch-v2-switch.
Closes#5201
2cc24032f [Cheng Pan] nit
f5c8bbf44 [Cheng Pan] nit
d280a209b [Cheng Pan] Allow disabling batch impl v2 on globally
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Since https://github.com/apache/kyuubi/issues/4843, we need the `requestConf` column to get application manager info.
So now the stateOnly option for metadata select is not needed.
```
private val METADATA_ALL_COLUMNS = Seq(
METADATA_STATE_ONLY_COLUMNS,
"resource",
"class_name",
"request_conf",
"request_args").mkString(",")
```
In this pr, I remove the `stateOnly` flag to select all metadata columns to prevent column missing and make it easy to maintain.
### _How was this patch tested?_
Existing UT.
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5199 from turboFei/metadata_state_only.
Closes#5199
8e02e6a5e [fwang12] all
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Use `Set` collection for order-insensitive configs instead of `Seq`
- kyuubi.frontend.thrift.binary.ssl.disallowed.protocols
- kyuubi.authentication
- kyuubi.authentication.ldap.groupFilter
- kyuubi.authentication.ldap.userFilter
- kyuubi.kubernetes.context.allow.list
- kyuubi.kubernetes.namespace.allow.list
- kyuubi.session.conf.ignore.list
- kyuubi.session.conf.restrict.list
- kyuubi.session.local.dir.allow.list
- kyuubi.batch.conf.ignore.list
- kyuubi.engine.deregister.exception.classes
- kyuubi.engine.deregister.exception.messages
- kyuubi.operation.plan.only.excludes
- kyuubi.server.limit.connections.user.unlimited.list
- kyuubi.server.administrators
- kyuubi.metrics.reporters
- Support skipping blank elements
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5185 from bowenliang123/conf-toset.
Closes#5185
c656af78a [liangbowen] conf to set
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Currently, the name of flink bootstrap SQL is auto-generated 'collect'.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5190 from link3280/bootstrap_job_name.
Closes#5190
ac769295c [Paul Lin] Explicit name Flink bootstrap sql
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5134 from link3280/KYUUBI-4806.
Closes#4806
a1b74783c [Paul Lin] Optimize code style
546cfdf5b [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperation.scala
b6eb7af4f [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/ResultSet.scala
1563fa98b [Paul Lin] Remove explicit StartRowOffset for Flink
4e61a348c [Paul Lin] Add comments
c93294650 [Paul Lin] Improve code style
6bd0c8e69 [Paul Lin] Use dedicated thread pool
15412db3a [Paul Lin] Improve logging
d6a2a9cff [Paul Lin] [KYUUBI #4806][FLINK] Implement incremental result fetching
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
https://spark.apache.org/news/spark-3-3-3-released.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5150 from pan3793/spark-3.3.3.
Closes#5150
61583609b [Cheng Pan] image
3021dd80b [Cheng Pan] remove staging repo
71b8aa843 [Cheng Pan] Revert "tgz"
d9125e63e [Cheng Pan] tgz
ebe3107c9 [Cheng Pan] Bump Spark 3.3.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Make Spark TPC-H connector compilable on Scala 2.13
- adding spark extensions to scala 2.13 compilation CI test
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5195 from bowenliang123/scala213-conn.
Closes#5195
ad2558a16 [liangbowen] update
c50d29111 [liangbowen] update
d38eb3b6c [liangbowen] adapt spark connector plugin compilable on Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL lineage plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
`mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-spark-lineage_2.13 -Pspark-3.3`
- fix type mismatch, by manually converting Iterable to ListMap
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala:220: type mismatch;
found : scala.collection.immutable.Iterable[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.AttributeSet)]
required: LineageParser.this.AttributeMap[org.apache.spark.sql.catalyst.expressions.AttributeSet]
(which expands to) scala.collection.immutable.ListMap[org.apache.spark.sql.catalyst.expressions.Attribute,org.apache.spark.sql.catalyst.expressions.AttributeSet]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5192 from bowenliang123/scala213-lineage.
Closes#5192
a68ba8457 [liangbowen] adapt spark lineage plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adding a basic compilation CI test on Scala 2.13 , skipping test runs
- separate versions of `KyuubiSparkILoop` for Scala 2.12 and 2.13, adapting the changes of Scala interpreter packages
- rename `export` variable
```
[Error] /Users/bw/dev/kyuubi/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala:563: Wrap `export` in backticks to use it as an identifier, it will become a keyword in Scala 3.
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5188 from bowenliang123/sparksql-213.
Closes#5188
04f192064 [liangbowen] update
9e764271b [liangbowen] add ci for compilation with server module
a465375bd [liangbowen] update
5c3f24fdf [liangbowen] update
4b6a6e339 [liangbowen] use Iterable for the row in MySQLTextResultSetRowPacket
f09d61d26 [liangbowen] use ListMap.newBuilder
6b5480872 [liangbowen] Use Iterable for collections
1abfa29d8 [liangbowen] 2.12's KyuubiSparkILoop
b1c9da591 [liangbowen] rename export variable
b6a6e077b [liangbowen] remove original KyuubiSparkILoop
15438b503 [liangbowen] move back scala 2.12's KyuubiSparkILoop
dd3244351 [liangbowen] adapt spark sql module to 2.13
62d3abbf0 [liangbowen] adapt server module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL hive connector plugin compilable on Scala 2.13 with Spark 3.3/3.4
- rename class name `FilePartitionReader` which is copied from Spark to `SparkFilePartitionReader`to fix the class mismatch error
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HivePartitionReaderFactory.scala:83: type mismatch;
found : Iterator[org.apache.kyuubi.spark.connector.hive.read.HivePartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
required: Iterator[org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5193 from bowenliang123/scala213-hivecon.
Closes#5193
d8c6bf5f0 [liangbowen] defer toMap
b20ad4eb1 [liangbowen] adapt spark hive connector plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
- to make Spark extension plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
```
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-2_2.13 -Pspark-3.2 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-3_2.13 -Pspark-3.3 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-4_2.13 -Pspark-3.4 -am
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5191 from bowenliang123/scala213-sparkext.
Closes#5191
156ae26c8 [liangbowen] adapt spark ext plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Make spark authz module compilable on Scala 2.13
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5189 from bowenliang123/scala213-authz.
Closes#5189
c5ee61a24 [liangbowen] adapt spark authz module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- introducing `scala-2.13` for development use only
- adapt the common modules compilable on Scala 2.13, including
- kyuubi-common
- kyuubi-ctl
- kyuubi-ha
- use separate compile options for Scala 2.13 , skipping deprecated scala compiler option `-Yno-adapted-args`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5183 from bowenliang123/scala213-common.
Closes#5183
1e65a5d03 [Bowen Liang] Align scala 2.13 version with Spark to 2.13.8
0003c8bed [liangbowen] duplicate scala-maven-plugin configs in profile
8b448a5f0 [liangbowen] 2.13.11
36d2284c4 [liangbowen] comment
28b515a0c [liangbowen] comment
865c5e55b [liangbowen] update
52e381200 [liangbowen] ensure kyuubi-ctl module compilable on scala 2.13
70dbc7ae2 [liangbowen] ensure kyuubi-ha module compilable on scala 2.13
84c1e4b4d [liangbowen] ensure kyuubi-common module compilable on scala 2.13
e116d77f5 [liangbowen] skip deprecated compiler option "-Yno-adapted-args" in 2.13
26c27a2ee [liangbowen] introduce new profile scala-2.13
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- fix info message with correct string placeholder of user in UserFilter
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5187 from bowenliang123/userfilter-str.
Closes#5187
25d370ee3 [liangbowen] fix info message in UserFilter
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Fix class mismatch when trying to compilation on Scala 2.13, due to implicit class reference to `StageInfo`. The compilation fails by type mismatching, if the compiler classloader loads Spark's `org.apache.spark.schedulerStageInfo` ahead of Kyuubi's `org.apache.spark.kyuubi.StageInfo`.
- Change var integer to AtomicInteger for `numActiveTasks` and `numCompleteTasks`, preventing possible concurrent inconsistency
```
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:56: type mismatch;
found : java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.scheduler.StageInfo]
required: java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.kyuubi.StageInfo]
[INFO] [Info] : java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.scheduler.StageInfo] <: java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.kyuubi.StageInfo]?
[INFO] [Info] : false
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:126: not enough arguments for constructor StageInfo: (stageId: Int, attemptId: Int, name: String, numTasks: Int, rddInfos: Seq[org.apache.spark.storage.RDDInfo], parentIds: Seq[Int], details: String, taskMetrics: org.apache.spark.executor.TaskMetrics, taskLocalityPreferences: Seq[Seq[org.apache.spark.scheduler.TaskLocation]], shuffleDepId: Option[Int], resourceProfileId: Int, isPushBasedShuffleEnabled: Boolean, shuffleMergerCount: Int): org.apache.spark.scheduler.StageInfo.
Unspecified value parameters name, numTasks, rddInfos...
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:148: value numActiveTasks is not a member of org.apache.spark.scheduler.StageInfo
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:156: value numActiveTasks is not a member of org.apache.spark.scheduler.StageInfo
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:158: value numCompleteTasks is not a member of org.apache.spark.scheduler.StageInfo
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5184 from bowenliang123/spark-stage-info.
Closes#5184
fd0b9b564 [liangbowen] update
d410491f3 [liangbowen] rename Kyuubi's StageInfo to SparkStageInfo preventing class mismatch
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Use `Iterable` instead of `Seq` in kyuubi-ctl commands for Scala compatibility, as
1. in Scala 2.13, the `scala.Seq` is now an alias for `scala.collection.immutable.Seq` (instead of `scala.collection.Seq`)
2. in Scala 2.13, `scala.collection.mutable.ListBuffer` (or Buffers) does not extend `scala.collection.immutable.Seq` according to [Scala collection migration guide](https://docs.scala-lang.org/overviews/core/collections-migration-213.html)
3. in both Scala 2.12 and 2.13, `ListBuffer` (or Buffers) extends `scala.collection.mutable.Iterable`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5179 from bowenliang123/ctlcmd-iter.
Closes#5179
3265e3b99 [liangbowen] Use Iterable instead of Seq in kyuubi-ctl commands
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Replace deprecated class alias `scala.tools.nsc.interpreter.IR` and `scala.tools.nsc.interpreter.JPrintWriter` in Scala 2.13 with equivalent classes `scala.tools.nsc.interpreter.Results` and `java.io.PrintWriter`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5180 from bowenliang123/interpreter-213.
Closes#5180
e76f1f05a [liangbowen] prevent to use deprecated classes in the package scala.tools.nsc.interpreter of Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- As `scala.collection.generic.Growable` is marked as deprecated and made `+=` is marked as a final method in Scala 2.13, MarkdownBuilder which extends Growable is a blocker issue for Scala 2.13 in the future
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5178 from bowenliang123/markdown-notgrow.
Closes#5178
c34b95649 [liangbowen] update
565fdf6e5 [liangbowen] make MarkdownBuilder not extending Growable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
1. Print those pod name/prefix in kyuubi server log
2. Make sure driver pod name in length limit
close#5165
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5168 from zwangsheng/KYUUBI#5165.
Closes#5165
290a46860 [Cheng Pan] nit
9c9dd2d38 [Cheng Pan] Revert "Add app Name"
50187cddb [zwangsheng] Add app Name
f704a9839 [Cheng Pan] app key
463904bec [Cheng Pan] revise
33d1ffd9e [Cheng Pan] Apply suggestions from code review
0b7e64da5 [Cheng Pan] Apply suggestions from code review
b3b1e08fd [zwangsheng] Fix Test
742e78461 [zwangsheng] modify long app name length to 266
90b165e95 [zwangsheng] fix test
191447058 [zwangsheng] fix test fail
05698fdb1 [zwangsheng] fix test fail
39637ae68 [zwangsheng] fix comments
23513810f [zwangsheng] fix comments
a56630179 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkBatchProcessBuilder.scala
8599a1430 [zwangsheng] fix style
73aec9a03 [zwangsheng] [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix a bug, The `DESCRIBE FUNCTION` syntax should also be checked for USE permissions. However, prior to this PR, the syntax was not checked for any permissions
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5172 from Yikf/auth-desc-function.
Closes#5172
112f4f20b [yikaifei] The DESCRIBE FUNCTION syntax should also be checked for USE permissions
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
In minor cases, Spark Stage hangs forever when spark.sql.finalWriteStage.eagerlyKillExecutors.enabled is true.
The bug occurs if two conditions are met in the same time:
1. All executors are either removed because of idle time out or killed by FinalStageResourceManager.
Target executor num in YarnAllocator will be set to 0 and no more executor will be launched.
2. Target executor num in ExecutorAllocationManager equals to the executor num needed by final stage.
Then ExecutorAllocationManager will not sync target executor num to YarnAllocator.
### _How was this patch tested?_
- [x] Add a new test suite `FinalStageResourceManagerSuite`
Closes#5141 from zhouyifan279/adjust-executors.
Closes#5136
c4403eefa [zhouyifan279] assert adjustedTargetExecutors == 1
ea8f24733 [zhouyifan279] Add comment
5f3ca1d9c [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
12687eee7 [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
9dcbc780d [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5170
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5171 from iodone/kyuubi-5170.
Closes#5170
7cd0e2cbf [odone] identifier compatible with sprk3.4
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Integrating Prometheus as the metrics collection method in the Kyuubi Helm Chart simplifies the configuration and deployment process, provides a standardized monitoring solution, and makes monitoring in the Kubernetes cluster more convenient and reliable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5132 from dev-lpq/monitor.
Closes#5132
75da2d491 [pengqli] upgrade the kyuubi monitoring
76c2db130 [pengqli] add monitoring api check
5a4c8ee88 [pengqli] add monitoring api check
1ce8c3b7d [pengqli] add monitoring api check
7362aaf0d [pengqli] Merge branch 'master' into monitor
76e5b95b6 [pengqli] update helm chart monitor
50439b603 [pengqli] update helm chart monitor
5fafe6516 [pengqli] add helm chart monitor
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5162
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5163 from lsm1/features/kyuubi-5162.
Closes#5162
c5bba8c20 [senmiaoliu] fix
af5e816fb [Kent Yao] Update docs/contributing/code/developer.md
ceb44358d [senmiaoliu] Add statements about contributions co-authored with AIGC in developer guide page
Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR provides a new configuration `kyuubi.zookeeper.embedded.client.use.hostname` to control whether use hostname or IP address on binding, to improve the out-of-box experience of K8s case, similar to `kyuubi.frontend.connection.url.use.hostname`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5158 from pan3793/zk-ip.
Closes#5158
260dae987 [Cheng Pan] nit
3ac0c430e [Cheng Pan] doc
de50f65e8 [Cheng Pan] false
890b13bd9 [Cheng Pan] Allow embedded Zookeeper binding IP address
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Currently, `getNextRowSetInternal` returns `TRowSet` which is not friendly to explicit EOS in streaming result fetch.
This PR changes the return type to `TFetchResultsResp` to allow the engines to determine the EOS.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5160 from link3280/refactor_result.
Closes#5160
09822f2ee [Paul Lin] Fix hasMoreRows missing
c94907e2b [Paul Lin] Explicitly set `resp.setHasMoreRows(false)` for operations
4d193fb1d [Paul Lin] Revert unrelated changes in FlinkOperation
ffd0367b3 [Paul Lin] Refactor getNextRowSetInternal to support fetch streaming data
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- extract development scripts for regenerating and verifying the golden files
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5121 from bowenliang123/gen-golden.
Closes#5121
a8fb20046 [liangbowen] add golden result file path hint in comment
444e5aff9 [liangbowen] nit
eec4fe84f [liangbowen] use the generation scripts for running test suites.
5a97785e6 [liangbowen] fix class name in gen_ranger_spec_json.sh
3be9aacf5 [liangbowen] comments
32993de43 [liangbowen] comments
ca0090909 [liangbowen] update scripts
6439ad3e9 [liangbowen] update scripts
f26d77935 [liangbowen] add scripts for golden files' generation
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Correct a typo in this doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5153 from zhaohehuhu/Improvement-0810.
Closes#5153
b2b593d94 [hezhao2] Update quick start hive jdbc instruction
Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Show prompt if milestone or assignees not set when merging pull requests.
<img width="611" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/4f4df661-14ab-45e4-bcfe-9050549048e6">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5152 from bowenliang123/merge-check.
Closes#5152
3a1731d40 [liangbowen] nit
20eb0e2ad [liangbowen] print
51a268e71 [liangbowen] check and print milestone and assignees of pull requests
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change the embedded Zookeeper server to method variable
- Throw runtime exception for failures when initializing the embedded Zookeeper
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5145 from bowenliang123/embed-zk.
Closes#5145
2a78957e5 [liangbowen] nit
86087117c [liangbowen] update
0f523f265 [liangbowen] throw runtime exception for failures when initializing embedded zookeeper
b33e7a497 [liangbowen] make the embedded zookeeper from member attributes to method-local variable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Move the configuration docs to the top level of docs, which is most commonly used and referenced
- update relevant doc links
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5154 from bowenliang123/config-doc-first.
Closes#5154
b49ed3f8b [liangbowen] nit
db7f0d14d [liangbowen] update doc links
f8fd697a2 [liangbowen] move config docs to the top level
7448e4487 [liangbowen] change title of settings doc
40214ddd8 [liangbowen] move config doc in the front of deployment
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Currently, overloaded methods are considered the same and deduplicated in ReflectUtils, thus not easy to tell why no method is found.
This PR fixes the problem by adding the argument lists. In addition, it fixes the problem that the arg classes are not printed correctly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5149 from link3280/reflect_exception_msg.
Closes#5149
b27d743fc [Paul Lin] Update test case
0c04f2709 [Paul Lin] Improve error msg of ReflectUtils
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
