### _Why are the changes needed?_
It makes no sense to enable batch impl v2 when `kyuubi.batch.submitter.enabled` is disabled, to avoid misuse, globally disable batch impl v2 in such case.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5201 from pan3793/batch-v2-switch.
Closes#5201
2cc24032f [Cheng Pan] nit
f5c8bbf44 [Cheng Pan] nit
d280a209b [Cheng Pan] Allow disabling batch impl v2 on globally
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Since https://github.com/apache/kyuubi/issues/4843, we need the `requestConf` column to get application manager info.
So now the stateOnly option for metadata select is not needed.
```
private val METADATA_ALL_COLUMNS = Seq(
METADATA_STATE_ONLY_COLUMNS,
"resource",
"class_name",
"request_conf",
"request_args").mkString(",")
```
In this pr, I remove the `stateOnly` flag to select all metadata columns to prevent column missing and make it easy to maintain.
### _How was this patch tested?_
Existing UT.
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5199 from turboFei/metadata_state_only.
Closes#5199
8e02e6a5e [fwang12] all
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Use `Set` collection for order-insensitive configs instead of `Seq`
- kyuubi.frontend.thrift.binary.ssl.disallowed.protocols
- kyuubi.authentication
- kyuubi.authentication.ldap.groupFilter
- kyuubi.authentication.ldap.userFilter
- kyuubi.kubernetes.context.allow.list
- kyuubi.kubernetes.namespace.allow.list
- kyuubi.session.conf.ignore.list
- kyuubi.session.conf.restrict.list
- kyuubi.session.local.dir.allow.list
- kyuubi.batch.conf.ignore.list
- kyuubi.engine.deregister.exception.classes
- kyuubi.engine.deregister.exception.messages
- kyuubi.operation.plan.only.excludes
- kyuubi.server.limit.connections.user.unlimited.list
- kyuubi.server.administrators
- kyuubi.metrics.reporters
- Support skipping blank elements
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5185 from bowenliang123/conf-toset.
Closes#5185
c656af78a [liangbowen] conf to set
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Currently, the name of flink bootstrap SQL is auto-generated 'collect'.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5190 from link3280/bootstrap_job_name.
Closes#5190
ac769295c [Paul Lin] Explicit name Flink bootstrap sql
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
As titled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5134 from link3280/KYUUBI-4806.
Closes#4806
a1b74783c [Paul Lin] Optimize code style
546cfdf5b [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperation.scala
b6eb7af4f [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/ResultSet.scala
1563fa98b [Paul Lin] Remove explicit StartRowOffset for Flink
4e61a348c [Paul Lin] Add comments
c93294650 [Paul Lin] Improve code style
6bd0c8e69 [Paul Lin] Use dedicated thread pool
15412db3a [Paul Lin] Improve logging
d6a2a9cff [Paul Lin] [KYUUBI #4806][FLINK] Implement incremental result fetching
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
https://spark.apache.org/news/spark-3-3-3-released.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5150 from pan3793/spark-3.3.3.
Closes#5150
61583609b [Cheng Pan] image
3021dd80b [Cheng Pan] remove staging repo
71b8aa843 [Cheng Pan] Revert "tgz"
d9125e63e [Cheng Pan] tgz
ebe3107c9 [Cheng Pan] Bump Spark 3.3.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Make Spark TPC-H connector compilable on Scala 2.13
- adding spark extensions to scala 2.13 compilation CI test
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5195 from bowenliang123/scala213-conn.
Closes#5195
ad2558a16 [liangbowen] update
c50d29111 [liangbowen] update
d38eb3b6c [liangbowen] adapt spark connector plugin compilable on Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL lineage plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
`mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-spark-lineage_2.13 -Pspark-3.3`
- fix type mismatch, by manually converting Iterable to ListMap
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala:220: type mismatch;
found : scala.collection.immutable.Iterable[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.AttributeSet)]
required: LineageParser.this.AttributeMap[org.apache.spark.sql.catalyst.expressions.AttributeSet]
(which expands to) scala.collection.immutable.ListMap[org.apache.spark.sql.catalyst.expressions.Attribute,org.apache.spark.sql.catalyst.expressions.AttributeSet]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5192 from bowenliang123/scala213-lineage.
Closes#5192
a68ba8457 [liangbowen] adapt spark lineage plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adding a basic compilation CI test on Scala 2.13 , skipping test runs
- separate versions of `KyuubiSparkILoop` for Scala 2.12 and 2.13, adapting the changes of Scala interpreter packages
- rename `export` variable
```
[Error] /Users/bw/dev/kyuubi/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala:563: Wrap `export` in backticks to use it as an identifier, it will become a keyword in Scala 3.
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5188 from bowenliang123/sparksql-213.
Closes#5188
04f192064 [liangbowen] update
9e764271b [liangbowen] add ci for compilation with server module
a465375bd [liangbowen] update
5c3f24fdf [liangbowen] update
4b6a6e339 [liangbowen] use Iterable for the row in MySQLTextResultSetRowPacket
f09d61d26 [liangbowen] use ListMap.newBuilder
6b5480872 [liangbowen] Use Iterable for collections
1abfa29d8 [liangbowen] 2.12's KyuubiSparkILoop
b1c9da591 [liangbowen] rename export variable
b6a6e077b [liangbowen] remove original KyuubiSparkILoop
15438b503 [liangbowen] move back scala 2.12's KyuubiSparkILoop
dd3244351 [liangbowen] adapt spark sql module to 2.13
62d3abbf0 [liangbowen] adapt server module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to make Spark SQL hive connector plugin compilable on Scala 2.13 with Spark 3.3/3.4
- rename class name `FilePartitionReader` which is copied from Spark to `SparkFilePartitionReader`to fix the class mismatch error
```
[ERROR] [Error] /Users/bw/dev/incubator-kyuubi/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HivePartitionReaderFactory.scala:83: type mismatch;
found : Iterator[org.apache.kyuubi.spark.connector.hive.read.HivePartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
required: Iterator[org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader[org.apache.spark.sql.catalyst.InternalRow]]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5193 from bowenliang123/scala213-hivecon.
Closes#5193
d8c6bf5f0 [liangbowen] defer toMap
b20ad4eb1 [liangbowen] adapt spark hive connector plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
- to make Spark extension plugin compilable on Scala 2.13 with Spark 3.2/3.3/3.4 (Spark 3.1 does not support Scala 2.13)
```
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-2_2.13 -Pspark-3.2 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-3_2.13 -Pspark-3.3 -am
mvn clean install -DskipTests -Pflink-provided,hive-provided,spark-provided -Pscala-2.13 -pl :kyuubi-extension-spark-3-4_2.13 -Pspark-3.4 -am
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5191 from bowenliang123/scala213-sparkext.
Closes#5191
156ae26c8 [liangbowen] adapt spark ext plugin to Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Make spark authz module compilable on Scala 2.13
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5189 from bowenliang123/scala213-authz.
Closes#5189
c5ee61a24 [liangbowen] adapt spark authz module to 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- introducing `scala-2.13` for development use only
- adapt the common modules compilable on Scala 2.13, including
- kyuubi-common
- kyuubi-ctl
- kyuubi-ha
- use separate compile options for Scala 2.13 , skipping deprecated scala compiler option `-Yno-adapted-args`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5183 from bowenliang123/scala213-common.
Closes#5183
1e65a5d03 [Bowen Liang] Align scala 2.13 version with Spark to 2.13.8
0003c8bed [liangbowen] duplicate scala-maven-plugin configs in profile
8b448a5f0 [liangbowen] 2.13.11
36d2284c4 [liangbowen] comment
28b515a0c [liangbowen] comment
865c5e55b [liangbowen] update
52e381200 [liangbowen] ensure kyuubi-ctl module compilable on scala 2.13
70dbc7ae2 [liangbowen] ensure kyuubi-ha module compilable on scala 2.13
84c1e4b4d [liangbowen] ensure kyuubi-common module compilable on scala 2.13
e116d77f5 [liangbowen] skip deprecated compiler option "-Yno-adapted-args" in 2.13
26c27a2ee [liangbowen] introduce new profile scala-2.13
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- fix info message with correct string placeholder of user in UserFilter
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5187 from bowenliang123/userfilter-str.
Closes#5187
25d370ee3 [liangbowen] fix info message in UserFilter
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Fix class mismatch when trying to compilation on Scala 2.13, due to implicit class reference to `StageInfo`. The compilation fails by type mismatching, if the compiler classloader loads Spark's `org.apache.spark.schedulerStageInfo` ahead of Kyuubi's `org.apache.spark.kyuubi.StageInfo`.
- Change var integer to AtomicInteger for `numActiveTasks` and `numCompleteTasks`, preventing possible concurrent inconsistency
```
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:56: type mismatch;
found : java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.scheduler.StageInfo]
required: java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.kyuubi.StageInfo]
[INFO] [Info] : java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.scheduler.StageInfo] <: java.util.concurrent.ConcurrentHashMap[org.apache.spark.kyuubi.StageAttempt,org.apache.spark.kyuubi.StageInfo]?
[INFO] [Info] : false
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:126: not enough arguments for constructor StageInfo: (stageId: Int, attemptId: Int, name: String, numTasks: Int, rddInfos: Seq[org.apache.spark.storage.RDDInfo], parentIds: Seq[Int], details: String, taskMetrics: org.apache.spark.executor.TaskMetrics, taskLocalityPreferences: Seq[Seq[org.apache.spark.scheduler.TaskLocation]], shuffleDepId: Option[Int], resourceProfileId: Int, isPushBasedShuffleEnabled: Boolean, shuffleMergerCount: Int): org.apache.spark.scheduler.StageInfo.
Unspecified value parameters name, numTasks, rddInfos...
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:148: value numActiveTasks is not a member of org.apache.spark.scheduler.StageInfo
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:156: value numActiveTasks is not a member of org.apache.spark.scheduler.StageInfo
[ERROR] [Error] /Users/bw/dev/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala:158: value numCompleteTasks is not a member of org.apache.spark.scheduler.StageInfo
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5184 from bowenliang123/spark-stage-info.
Closes#5184
fd0b9b564 [liangbowen] update
d410491f3 [liangbowen] rename Kyuubi's StageInfo to SparkStageInfo preventing class mismatch
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Use `Iterable` instead of `Seq` in kyuubi-ctl commands for Scala compatibility, as
1. in Scala 2.13, the `scala.Seq` is now an alias for `scala.collection.immutable.Seq` (instead of `scala.collection.Seq`)
2. in Scala 2.13, `scala.collection.mutable.ListBuffer` (or Buffers) does not extend `scala.collection.immutable.Seq` according to [Scala collection migration guide](https://docs.scala-lang.org/overviews/core/collections-migration-213.html)
3. in both Scala 2.12 and 2.13, `ListBuffer` (or Buffers) extends `scala.collection.mutable.Iterable`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5179 from bowenliang123/ctlcmd-iter.
Closes#5179
3265e3b99 [liangbowen] Use Iterable instead of Seq in kyuubi-ctl commands
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Replace deprecated class alias `scala.tools.nsc.interpreter.IR` and `scala.tools.nsc.interpreter.JPrintWriter` in Scala 2.13 with equivalent classes `scala.tools.nsc.interpreter.Results` and `java.io.PrintWriter`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5180 from bowenliang123/interpreter-213.
Closes#5180
e76f1f05a [liangbowen] prevent to use deprecated classes in the package scala.tools.nsc.interpreter of Scala 2.13
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning: Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning: 'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5175 from bowenliang123/artifactId-scala.
Closes#5177
2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- As `scala.collection.generic.Growable` is marked as deprecated and made `+=` is marked as a final method in Scala 2.13, MarkdownBuilder which extends Growable is a blocker issue for Scala 2.13 in the future
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5178 from bowenliang123/markdown-notgrow.
Closes#5178
c34b95649 [liangbowen] update
565fdf6e5 [liangbowen] make MarkdownBuilder not extending Growable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
1. Print those pod name/prefix in kyuubi server log
2. Make sure driver pod name in length limit
close#5165
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5168 from zwangsheng/KYUUBI#5165.
Closes#5165
290a46860 [Cheng Pan] nit
9c9dd2d38 [Cheng Pan] Revert "Add app Name"
50187cddb [zwangsheng] Add app Name
f704a9839 [Cheng Pan] app key
463904bec [Cheng Pan] revise
33d1ffd9e [Cheng Pan] Apply suggestions from code review
0b7e64da5 [Cheng Pan] Apply suggestions from code review
b3b1e08fd [zwangsheng] Fix Test
742e78461 [zwangsheng] modify long app name length to 266
90b165e95 [zwangsheng] fix test
191447058 [zwangsheng] fix test fail
05698fdb1 [zwangsheng] fix test fail
39637ae68 [zwangsheng] fix comments
23513810f [zwangsheng] fix comments
a56630179 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkBatchProcessBuilder.scala
8599a1430 [zwangsheng] fix style
73aec9a03 [zwangsheng] [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix a bug, The `DESCRIBE FUNCTION` syntax should also be checked for USE permissions. However, prior to this PR, the syntax was not checked for any permissions
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5172 from Yikf/auth-desc-function.
Closes#5172
112f4f20b [yikaifei] The DESCRIBE FUNCTION syntax should also be checked for USE permissions
Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
In minor cases, Spark Stage hangs forever when spark.sql.finalWriteStage.eagerlyKillExecutors.enabled is true.
The bug occurs if two conditions are met in the same time:
1. All executors are either removed because of idle time out or killed by FinalStageResourceManager.
Target executor num in YarnAllocator will be set to 0 and no more executor will be launched.
2. Target executor num in ExecutorAllocationManager equals to the executor num needed by final stage.
Then ExecutorAllocationManager will not sync target executor num to YarnAllocator.
### _How was this patch tested?_
- [x] Add a new test suite `FinalStageResourceManagerSuite`
Closes#5141 from zhouyifan279/adjust-executors.
Closes#5136
c4403eefa [zhouyifan279] assert adjustedTargetExecutors == 1
ea8f24733 [zhouyifan279] Add comment
5f3ca1d9c [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
12687eee7 [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
9dcbc780d [zhouyifan279] [KYUUBI #5136][Bug] Spark App may hang forever if FinalStageResourceManager killed all executors
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5170
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5171 from iodone/kyuubi-5170.
Closes#5170
7cd0e2cbf [odone] identifier compatible with sprk3.4
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Integrating Prometheus as the metrics collection method in the Kyuubi Helm Chart simplifies the configuration and deployment process, provides a standardized monitoring solution, and makes monitoring in the Kubernetes cluster more convenient and reliable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5132 from dev-lpq/monitor.
Closes#5132
75da2d491 [pengqli] upgrade the kyuubi monitoring
76c2db130 [pengqli] add monitoring api check
5a4c8ee88 [pengqli] add monitoring api check
1ce8c3b7d [pengqli] add monitoring api check
7362aaf0d [pengqli] Merge branch 'master' into monitor
76e5b95b6 [pengqli] update helm chart monitor
50439b603 [pengqli] update helm chart monitor
5fafe6516 [pengqli] add helm chart monitor
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5162
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5163 from lsm1/features/kyuubi-5162.
Closes#5162
c5bba8c20 [senmiaoliu] fix
af5e816fb [Kent Yao] Update docs/contributing/code/developer.md
ceb44358d [senmiaoliu] Add statements about contributions co-authored with AIGC in developer guide page
Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR provides a new configuration `kyuubi.zookeeper.embedded.client.use.hostname` to control whether use hostname or IP address on binding, to improve the out-of-box experience of K8s case, similar to `kyuubi.frontend.connection.url.use.hostname`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5158 from pan3793/zk-ip.
Closes#5158
260dae987 [Cheng Pan] nit
3ac0c430e [Cheng Pan] doc
de50f65e8 [Cheng Pan] false
890b13bd9 [Cheng Pan] Allow embedded Zookeeper binding IP address
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Currently, `getNextRowSetInternal` returns `TRowSet` which is not friendly to explicit EOS in streaming result fetch.
This PR changes the return type to `TFetchResultsResp` to allow the engines to determine the EOS.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5160 from link3280/refactor_result.
Closes#5160
09822f2ee [Paul Lin] Fix hasMoreRows missing
c94907e2b [Paul Lin] Explicitly set `resp.setHasMoreRows(false)` for operations
4d193fb1d [Paul Lin] Revert unrelated changes in FlinkOperation
ffd0367b3 [Paul Lin] Refactor getNextRowSetInternal to support fetch streaming data
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- extract development scripts for regenerating and verifying the golden files
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5121 from bowenliang123/gen-golden.
Closes#5121
a8fb20046 [liangbowen] add golden result file path hint in comment
444e5aff9 [liangbowen] nit
eec4fe84f [liangbowen] use the generation scripts for running test suites.
5a97785e6 [liangbowen] fix class name in gen_ranger_spec_json.sh
3be9aacf5 [liangbowen] comments
32993de43 [liangbowen] comments
ca0090909 [liangbowen] update scripts
6439ad3e9 [liangbowen] update scripts
f26d77935 [liangbowen] add scripts for golden files' generation
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Correct a typo in this doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5153 from zhaohehuhu/Improvement-0810.
Closes#5153
b2b593d94 [hezhao2] Update quick start hive jdbc instruction
Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Show prompt if milestone or assignees not set when merging pull requests.
<img width="611" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/4f4df661-14ab-45e4-bcfe-9050549048e6">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5152 from bowenliang123/merge-check.
Closes#5152
3a1731d40 [liangbowen] nit
20eb0e2ad [liangbowen] print
51a268e71 [liangbowen] check and print milestone and assignees of pull requests
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Change the embedded Zookeeper server to method variable
- Throw runtime exception for failures when initializing the embedded Zookeeper
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5145 from bowenliang123/embed-zk.
Closes#5145
2a78957e5 [liangbowen] nit
86087117c [liangbowen] update
0f523f265 [liangbowen] throw runtime exception for failures when initializing embedded zookeeper
b33e7a497 [liangbowen] make the embedded zookeeper from member attributes to method-local variable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Move the configuration docs to the top level of docs, which is most commonly used and referenced
- update relevant doc links
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5154 from bowenliang123/config-doc-first.
Closes#5154
b49ed3f8b [liangbowen] nit
db7f0d14d [liangbowen] update doc links
f8fd697a2 [liangbowen] move config docs to the top level
7448e4487 [liangbowen] change title of settings doc
40214ddd8 [liangbowen] move config doc in the front of deployment
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Currently, overloaded methods are considered the same and deduplicated in ReflectUtils, thus not easy to tell why no method is found.
This PR fixes the problem by adding the argument lists. In addition, it fixes the problem that the arg classes are not printed correctly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5149 from link3280/reflect_exception_msg.
Closes#5149
b27d743fc [Paul Lin] Update test case
0c04f2709 [Paul Lin] Improve error msg of ReflectUtils
Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
Before this PR, the ip address obtained in the K8s environment is incorrect, and `spark.driver.host` cannot be manually specified.
This time pr will adjust the way the IP is obtained and support the external parameter `spark.driver.host`
### _How was this patch tested?_
It has been tested in the local environment and verified as expected
Add screenshots for manual tests if appropriate
Closes#5148 from fantasticKe/feat_k8s_driver_ip.
Closes#5148
330f5f93d [marcoluo] feat: 修改k8s模式下获取spark.driver.host的方式
Authored-by: marcoluo <marcoluo@verizontal.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4925
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4932 from iodone/kyuubi-4868.
Closes#4925
ff3195772 [odone] remove the catalog with atlas supporting
fda2fe321 [odone] add default catalog to v1 table when parsing lineage
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
update error link
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5146 from liunaijie/master.
Closes#5146
9d07aa753 [jarvis] [Doc] update link
Authored-by: jarvis <liunaijie1996@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add support for function privileges building in 3.4 for #4167
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5055 from packyan/PR_4167_follow_up_support_spark_3.4.
Closes#5055
46fe89e01 [Deng An] add support for function privileges building in 3.4
Authored-by: Deng An <packyande@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- To avoid the generated web-ui's coverage report files triggering Rat plugin checks on Maven
```
[WARNING] Files with unapproved licenses:
kyuubi-server/web-ui/coverage/index.html
kyuubi-server/web-ui/coverage/locales/en_US/index.html
kyuubi-server/web-ui/coverage/locales/zh_CN/index.html
...
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.15:check (default) on project kyuubi-parent: Too many files with unapproved license: 21 See RAT report in: /kyuubi/target/rat.txt ->
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5135 from bowenliang123/rat-exclude.
Closes#5135
1ec4411cb [liangbowen] add web-ui/coverage/* to rat exclusions
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- allow using the session's user/password to connect database in the JDBC engine
- it is allowed to be applied on any share level of engines, since every kyuubi session maintains a dedicated JDBC connection in the current implementation
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5120 from bowenliang123/jdbc-user.
Closes#5120
7b8ebd137 [liangbowen] Use session's user and password to connect to database in JDBC engine
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- use `ListBuffer` instead of `ArrayBuffer` for inner string buffer, to minimize allocations for resizing
- handy `+=` operator in chaining style without explicit quotes, to make the user focus on content assembly and less distraction of quoting
- make `MarkdownBuilder` extending `Growable`, to utilize semantic operators like `+=` and `++=` which is unified inside for single or batch operation
- use `this +=` rather than `line(...)` , to reflect side effects in semantic way
- change list to stream for output comparison
- remove unused methods
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5127 from bowenliang123/md-buff.
Closes#5127
458e18c3d [liangbowen] Improvements for markdown builder
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
close#5105
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5128 from lsm1/features/kyuubi_5105.
Closes#5105
84be7fd6d [senmiaoliu] distinct default namespace
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support SSL for trino engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3374 from hddong/support-trino-password.
Closes#3374
f39daaf78 [Cheng Pan] improve
6308c4cf7 [hongdongdong] Support SSL for trino engine
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
One of the basic ideas of Kerberos authentication is that the Client MUST provide the Server Principal Name (SPN).
[ZOOKEEPER-1467](https://issues.apache.org/jira/browse/ZOOKEEPER-1467) (fixed in 3.6.0, 3.5.7) allows to configure Zookeeper SPN at the client side.
Previously, there is no way but only `-Dzookeeper.server.principal=zookeeper/_HOSTREALM` to set ZK SPN, this PR allows the user to set it via Kyuubi configuration `kyuubi.ha.zookeeper.auth.serverPrincipal`.
In default, if `zookeeper.server.principal` is not configured, it assumes that the Zookeeper server uses `zookeeper/_HOST<SAME_REALM_WITH_CLIENT_PRINCIPAL>`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5130 from pan3793/zk-spn.
Closes#5130
6f0a1bbdc [Cheng Pan] doc
290e172aa [Cheng Pan] docs
a43b0c67a [Cheng Pan] nit
9a33446b7 [Cheng Pan] Allow set Zookeeper server principal
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Otherwise we can not see JDK logs like Krb5.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5129 from pan3793/beeline-log.
Closes#5129
100094823 [Cheng Pan] KyuubiBeeline should redirect JDK logging
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
In Batch implementation v2, the following query is frequently executed to pick the job.
```
SELECT identifier FROM metadata WHERE state='INITIALIZED' ORDER BY create_time DESC LIMIT 1
```
Create an index for `create_time` could speed up the query and reduce the pressure on MySQL server.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Test the MySQL upgrading SQLs
```
mysql> CREATE TABLE IF NOT EXISTS metadata(
-> key_id bigint PRIMARY KEY AUTO_INCREMENT COMMENT 'the auto increment key id',
-> identifier varchar(36) NOT NULL COMMENT 'the identifier id, which is an UUID',
-> session_type varchar(32) NOT NULL COMMENT 'the session type, SQL or BATCH',
-> real_user varchar(255) NOT NULL COMMENT 'the real user',
-> user_name varchar(255) NOT NULL COMMENT 'the user name, might be a proxy user',
-> ip_address varchar(128) COMMENT 'the client ip address',
-> kyuubi_instance varchar(1024) NOT NULL COMMENT 'the kyuubi instance that creates this',
-> state varchar(128) NOT NULL COMMENT 'the session state',
-> resource varchar(1024) COMMENT 'the main resource',
-> class_name varchar(1024) COMMENT 'the main class name',
-> request_name varchar(1024) COMMENT 'the request name',
-> request_conf mediumtext COMMENT 'the request config map',
-> request_args mediumtext COMMENT 'the request arguments',
-> create_time BIGINT NOT NULL COMMENT 'the metadata create time',
-> engine_type varchar(32) NOT NULL COMMENT 'the engine type',
-> cluster_manager varchar(128) COMMENT 'the engine cluster manager',
-> engine_open_time bigint COMMENT 'the engine open time',
-> engine_id varchar(128) COMMENT 'the engine application id',
-> engine_name mediumtext COMMENT 'the engine application name',
-> engine_url varchar(1024) COMMENT 'the engine tracking url',
-> engine_state varchar(32) COMMENT 'the engine application state',
-> engine_error mediumtext COMMENT 'the engine application diagnose',
-> end_time bigint COMMENT 'the metadata end time',
-> peer_instance_closed boolean default '0' COMMENT 'closed by peer kyuubi instance',
-> UNIQUE INDEX unique_identifier_index(identifier),
-> INDEX user_name_index(user_name),
-> INDEX engine_type_index(engine_type)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
Query OK, 0 rows affected (0.03 sec)
mysql> ALTER TABLE metadata MODIFY kyuubi_instance varchar(1024) COMMENT 'the kyuubi instance that creates this';
Query OK, 0 rows affected (0.06 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE metadata ADD INDEX create_time_index(create_time);
Query OK, 0 rows affected (0.03 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table metadata;
+----------+--------------------------------------------------------------------------------+
| Table | Create Table |
+----------+--------------------------------------------------------------------------------+
| metadata | CREATE TABLE `metadata` (
`key_id` bigint NOT NULL AUTO_INCREMENT COMMENT 'the auto increment key id',
`identifier` varchar(36) NOT NULL COMMENT 'the identifier id, which is an UUID',
`session_type` varchar(32) NOT NULL COMMENT 'the session type, SQL or BATCH',
`real_user` varchar(255) NOT NULL COMMENT 'the real user',
`user_name` varchar(255) NOT NULL COMMENT 'the user name, might be a proxy user',
`ip_address` varchar(128) DEFAULT NULL COMMENT 'the client ip address',
`kyuubi_instance` varchar(1024) DEFAULT NULL COMMENT 'the kyuubi instance that creates this',
`state` varchar(128) NOT NULL COMMENT 'the session state',
`resource` varchar(1024) DEFAULT NULL COMMENT 'the main resource',
`class_name` varchar(1024) DEFAULT NULL COMMENT 'the main class name',
`request_name` varchar(1024) DEFAULT NULL COMMENT 'the request name',
`request_conf` mediumtext COMMENT 'the request config map',
`request_args` mediumtext COMMENT 'the request arguments',
`create_time` bigint NOT NULL COMMENT 'the metadata create time',
`engine_type` varchar(32) NOT NULL COMMENT 'the engine type',
`cluster_manager` varchar(128) DEFAULT NULL COMMENT 'the engine cluster manager',
`engine_open_time` bigint DEFAULT NULL COMMENT 'the engine open time',
`engine_id` varchar(128) DEFAULT NULL COMMENT 'the engine application id',
`engine_name` mediumtext COMMENT 'the engine application name',
`engine_url` varchar(1024) DEFAULT NULL COMMENT 'the engine tracking url',
`engine_state` varchar(32) DEFAULT NULL COMMENT 'the engine application state',
`engine_error` mediumtext COMMENT 'the engine application diagnose',
`end_time` bigint DEFAULT NULL COMMENT 'the metadata end time',
`peer_instance_closed` tinyint(1) DEFAULT '0' COMMENT 'closed by peer kyuubi instance',
PRIMARY KEY (`key_id`),
UNIQUE KEY `unique_identifier_index` (`identifier`),
KEY `user_name_index` (`user_name`),
KEY `engine_type_index` (`engine_type`),
KEY `create_time_index` (`create_time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+----------+--------------------------------------------------------------------------------+
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5131 from pan3793/metastore-create-time-index.
Closes#5131
fc18041f2 [Cheng Pan] ALTER TABLE ADD INDEX
c2261edb2 [Cheng Pan] update upgrade script
4f94be5ca [Cheng Pan] Create index on metastore.create_time
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5122
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5125 from lsm1/features/kyuubi_5122.
Closes#5122
02d0769cc [senmiaoliu] add hive kdf docs
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Use StatefulSet instead of Deployment, add a headless service for statefulset
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5062 from camper42/statefulset.
Closes#4788
a1a7f1b0e [camper42] style: remove redudant Global variable `$`
5286f4ff4 [camper42] fix: set statefulset podManagementPolicy
ed83ae2e8 [camper42] style: move headless service to separate file
97b76ea24 [camper42] use `clusterIP: None` for headless serivce
d2078ffe5 [Cheng Pan] Update charts/kyuubi/templates/kyuubi-statefulset.yaml
35c7e0f90 [Cheng Pan] Update charts/kyuubi/templates/kyuubi-statefulset.yaml
8d970d21d [camper42] style: indent
3cf22748f [camper42] [KYUUBI #4788][K8S][HELM] Use StatefulSet instead of Deployment
Lead-authored-by: camper42 <camper.xlii@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://hadoop.apache.org/release/3.3.6.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5116 from pan3793/hadoop-3.3.6.
Closes#5116
c3717e7fb [Cheng Pan] Bump Hadoop 3.3.6
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4940
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5110 from lsm1/features/kyuubi_4940.
Closes#4940
6c0a9a37f [senmiaoliu] add kdf for hive engine
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It was planned but actually delayed, remove this dummy module to save CI and avoid confusing users and release managers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5113 from pan3793/remove-kudu.
Closes#5113
ff8fd2e6a [Cheng Pan] Remove Spark Kudu connector
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
