### _Why are the changes needed?_
Close#4843
Support to submit kyuubi engine/batch to multiple kubernetes contexts and namespaces.
In this pr, the user can config the kubernetes conf for specified kubernetes context and namespace likes below.
```
kyuubi.kubernetes.<context>.master.address
kyuubi.kubernetes.<context>.<namespace>.authenticate.oauthTokenFile
```
For example:
```
kyuubi.kubernetes.28.master.address=k8s://master
kyuubi.kubernetes.28.ns1.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
kyuubi.kubernetes.28.ns2.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
for k8s context=28, namespace=ns1, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns1
```
for k8s context=28, namespace=ns2, its kubernetes config is:
```
kyuubi.kubernetes.master.address=k8s://master
kyuubi.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/token.ns2
```
So that, kyuubi server can build kubernetes client for each context and namespace.
### _How was this patch tested?_
Existing kubernetes integration testing.
Closes#4984 from turboFei/k8s_client_yaml.
Closes#4843
f8ffaeeb9 [fwang12] nit
d25774288 [fwang12] comments
5ae7c8433 [fwang12] save into request conf
fd6c363db [fwang12] save
ff004a529 [fwang12] procebuilder method
6b9520bfd [fwang12] save
58850387e [fwang12] save
98df67e5f [fwang12] ut
da811697c [fwang12] fix
aa568aaa4 [fwang12] save
89656f463 [fwang12] check init
a0ef6894b [fwang12] code style
00abb6568 [fwang12] default namespace
295512987 [fwang12] k8s context namespace
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
There are hdfs-site.xml, hive-site, etc in spark job classpath, but we should use hadoop conf and hive conf from catalog options.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4995 from zhaomin1423/fix_hive_connector.
Closes#4995
64429fdcb [Xiao Zhao] Update extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
d921be750 [zhaomin] fix
375934d65 [zhaomin] Using hadoop conf and hive conf from catalog options
Lead-authored-by: zhaomin <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4920
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4921 from lightning-L/kyuubi-4920.
Closes#4920
d395f8a8a [Tianlin Liao] remove KyuubiEvent
3a6730f9b [Tianlin Liao] [KYUUBI #4920] add OperationRestApi
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final
Currently, we can not upgrade to 4.1.94.Final or above because of https://github.com/netty/netty/pull/13408
```
java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4992 from pan3793/netty.
Closes#4992
9dd8f9ad0 [Cheng Pan] nit
e6953144e [Cheng Pan] Bump Netty 4.1.93.Final
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4982
The `targetURL` should include the query string so that kyuubi server can forward request to right page, eg:
```
/engine-ui/spark-43efae88d766226a-driver-svc.kyuubi-dlssjc-canned.svc:4045/jobs/job/?id=4
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4983 from zhaohehuhu/Improvement-4982.
Closes#4982
9140a8a82 [Cheng Pan] simple
3140bd9be [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/EngineUIProxyServlet.scala
421152268 [hezhao2] refactor the logic of append
892c5511e [hezhao2] add queryString as job suffix
53f1ce4b6 [hezhao2] [Improvement] [UI] Add query string to forward requests to Engine UI
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This method is forked from Apache Spark, and recently, [SPARK-44074](https://github.com/apache/spark/issues/41663) identified and fixed the issue about loglevel restore behavior.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4985 from pan3793/test-log.
Closes#4985
4c05bc522 [Cheng Pan] Fix loglevel restore behavior of `KyuubiFunSuite#withLogAppender`
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The assertions are fragile because the session only has 5s as the idle timeout.
```
assert(lastAccessTime === session.lastAccessTime)
assert(sessionManager.getOpenSessionCount === 1)
```
https://github.com/apache/kyuubi/actions/runs/5312620487/jobs/9617377736?pr=4980
```
- close expired operations *** FAILED ***
0 did not equal 1 (TFrontendServiceSuite.scala:544)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4987 from pan3793/flaytest.
Closes#4978
3feacafe1 [Cheng Pan] nit
317f04576 [Cheng Pan] [KYUUBI #4978][FOLLOWUP] Fix flaky test: close expired operations
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://spark.apache.org/releases/spark-release-3-4-1.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4990 from pan3793/spark-3.4.1.
Closes#4990
68203b2c0 [Cheng Pan] Bump Spark 3.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Always use `Files#deleteIfExists` to replace `Files#delete` to suppress the stacktrace like
```
17:14:47.167 ERROR org.apache.kyuubi.operation.BatchJobSubmission: Failed to remove corresponding log file of operation: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
java.io.IOException: Failed to remove corresponding log file of operation: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
at org.apache.kyuubi.operation.log.OperationLog.trySafely(OperationLog.scala:272) ~[classes/:?]
at org.apache.kyuubi.operation.log.OperationLog.close(OperationLog.scala:257) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$2(BatchJobSubmission.scala:328) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$2$adapted(BatchJobSubmission.scala:328) ~[classes/:?]
at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.17.jar:?]
at org.apache.kyuubi.operation.BatchJobSubmission.$anonfun$close$1(BatchJobSubmission.scala:328) ~[classes/:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.17.jar:?]
at org.apache.kyuubi.Utils$.withLockRequired(Utils.scala:415) ~[classes/:?]
at org.apache.kyuubi.operation.AbstractOperation.withLockRequired(AbstractOperation.scala:51) ~[classes/:?]
at org.apache.kyuubi.operation.BatchJobSubmission.close(BatchJobSubmission.scala:326) ~[classes/:?]
at org.apache.kyuubi.session.KyuubiBatchSession.close(KyuubiBatchSession.scala:185) ~[classes/:?]
at org.apache.kyuubi.session.KyuubiSessionManager.openBatchSession(KyuubiSessionManager.scala:181) ~[classes/:?]
at org.apache.kyuubi.server.KyuubiBatchService.$anonfun$start$1(KyuubiBatchService.scala:96) ~[classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.nio.file.NoSuchFileException: /Users/chengpan/Projects/apache-kyuubi/server_operation_logs/9a96f19d-93a9-474c-b170-f957ed82c502/3fe99873-134b-4307-a286-94494ae847f2
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) ~[?:?]
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105) ~[?:?]
at java.nio.file.Files.delete(Files.java:1142) ~[?:?]
at org.apache.kyuubi.operation.log.OperationLog.$anonfun$close$4(OperationLog.scala:257) ~[classes/:?]
at org.apache.kyuubi.operation.log.OperationLog.trySafely(OperationLog.scala:263) ~[classes/:?]
... 18 more
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4986 from pan3793/delete.
Closes#4986
7d49bfec0 [Cheng Pan] Always use Files#deleteIfExists
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Please see: https://github.com/apache/kyuubi/issues/4946
Simply put, the kinit service needs to be started before the initLoggerEventHandler, so that it can obtain the DELEGATION TOKEN, which is necessary for accessing the log path on HDFS.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
After this PR, confirmed the LoggerEventHandler can be initiated even kdestroy was executed beforehand.
```
sudo -u kyuubi kdestroy
sudo -u kyuubi /opt/kyuubi/bin/kyuubi start
...
2023-06-19 22:18:36.581 INFO KyuubiTHttpFrontendService: Thread-101 org.eclipse.jetty.server.AbstractConnector: Started ServerConnector50110971{HTTP/1.1, (http/1.1)}{0.0.0.0:10010}
2023-06-19 22:18:36.581 INFO KyuubiTHttpFrontendService: Thread-101 org.eclipse.jetty.server.Server: Started 6081ms
2023-06-19 22:18:36.669 INFO main org.apache.kyuubi.events.handler.ServerJsonLoggingEventHandler: Logging kyuubi events to viewfs://<namespace>/tmp/kyuubi_hive/events/kyuubi_server_info/day=20230619/server-<servername>.json
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Passed the test with kyuubi-server
```
./build/mvn clean install -pl kyuubi-server
```
Closes#4980 from risyomei/fix/kyuubi-4946.
Closes#4946
d729d2384 [Xieming LI] Alter the order of initLoggerEventHandler
Authored-by: Xieming LI <risyomei@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to refactor the method `closeBatchSession`, by
- extracting sub-methods `checkPermission` and `forceKill`
- explicitly handling redirection on `metadata.kyuubiInstance != fe.connectionUrl` to make the logic clearer
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4981 from pan3793/cancel.
Closes#4981
0d2e85acc [Cheng Pan] Refactor code of closeBatchSession
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Currently, there is no error message on Kyuubi server's log when REST API invocation error occurs, and since we are building REST API, all response media types should be JSON.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4977 from pan3793/error-log.
Closes#4977
00c2b0c0f [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/trino/api/api.scala
5fe00474f [Cheng Pan] trino
f988329f8 [Cheng Pan] nit
74f9ed762 [Cheng Pan] log error message for REST api invocation
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/kyuubi/actions/runs/5307034060/jobs/9605209396
```
- support to interrupt the thrift request if remote engine is broken *** FAILED ***
"org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "java.net.SocketException", and "org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "org.apache.thrift.transport.TTransportException", and "org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: Socket for SessionHandle [68f22014-c484-4279-8f79-81bd2e7f0eca] is closed" did not contain "connection does not exist" (KyuubiOperationPerConnectionSuite.scala:330)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4979 from pan3793/test-1.
Closes#4979
86807e751 [Cheng Pan] Fix flaky test: support to interrupt the thrift request if remote engine is broken
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/kyuubi/actions/runs/5302176227/jobs/9596926968
```
- close expired operations *** FAILED ***
The code passed to eventually never returned normally. Attempted 70 times over 1.00037702345 minutes. Last failure message: 1687069174227 was not greater than 1687069174227. (TFrontendServiceSuite.scala:540)
```
Key change is
from
```
assert(session.lastIdleTime > lastAccessTime)
```
to
```
assert(lastAccessTime <= session.lastIdleTime)
```
because there are updated nearly at the same time.
```
private def release(userAccess: Boolean): Unit = {
if (userAccess) {
_lastAccessTime = System.currentTimeMillis
}
if (opHandleSet.isEmpty) {
_lastIdleTime = System.currentTimeMillis
}
}
```
This PR also changes some assertion statements from `assert(actual == expected)` to `assert(actual === expected)`, the former is Scala assert syntax, the latter is scalatest method
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4978 from pan3793/test.
Closes#4978
84dd8334d [Cheng Pan] nit
565b9c0b4 [Cheng Pan] Fix flaky test: close expired operations
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Eliminate rawtypes compilation warning in `KyuubiBeeLine.java` by adding `SuppressWarnings` annotation of rule `rawtypes` to BeelineParser implementation
```
Warning: [Warn] /home/runner/work/kyuubi/kyuubi/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/KyuubiBeeLine.java:148:53: [rawtypes] found raw type: ListIterator
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4974 from bowenliang123/beeline-rawtypes.
Closes#4974
994a2d3e0 [liangbowen] Fix rawtypes compilation warning in KyuubiBeeLine
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Remove redundant quoteIfNeeded method
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4973 from bowenliang123/redundant-quoteifneeded.
Closes#4937
acec0fb09 [liangbowen] Remove redundant quoteIfNeeded method
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The run test link in the template is broken.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4976 from mans2singh/ISSUE-4975.
Closes#4975
593172786 [Cheng Pan] Update .github/PULL_REQUEST_TEMPLATE
83103e26f [Cheng Pan] Update .github/PULL_REQUEST_TEMPLATE
ddec082d4 [mans2singh] ISSUE-4975 - Reverted unrelated change
37342792d [mans2singh] ISSUE-4975 - Fixed run tests link
Lead-authored-by: mans2singh <mans2singh@yahoo.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- To improve Scala code with corrections, simplification, scala style, redundancy cleaning-up. No feature changes introduced.
Corrections:
- Class doesn't correspond to file name (SparkListenerExtensionTest)
- Correct package name in ResultSetUtil and PySparkTests
Improvements:
- 'var' could be a 'val'
- GetOrElse(null) to orNull
Cleanup & Simplification:
- Redundant cast inspection
- Redundant collection conversion
- Simplify boolean expression
- Redundant new on case class
- Redundant return
- Unnecessary parentheses
- Unnecessary partial function
- Simplifiable empty check
- Anonymous function convertible to a method value
Scala Style:
- Constructing range for seq indices
- Get and getOrElse to getOrElse
- Convert expression to Single Abstract Method (SAM)
- Scala unnecessary semicolon inspection
- Map and getOrElse(false) to exists
- Map and flatten to flatMap
- Null initializer can be replaced by _
- scaladoc link to method
Other Improvements:
- Replace map and getOrElse(true) with forall
- Unit return type in the argument of map
- Size to length on arrays and strings
- Type check can be pattern matching
- Java mutator method accessed as parameterless
- Procedure syntax in method definition
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4959 from bowenliang123/scala-Improve.
Closes#4959
2d36ff351 [liangbowen] code improvement for Scala
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add `maven-surefire-plugin` to beeline module for running JUnit tests.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4969 from bowenliang123/scalatest-junit.
Closes#4969
995c83c07 [liangbowen] surefire on beeline
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- comment https://github.com/apache/kyuubi/pull/4963#discussion_r1230490326
- simplify reflection calling with unified `invokeAs` / `getField` method for either declared, inherited, or static methods / fields
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4970 from bowenliang123/unify-invokeas.
Closes#4970
592833459 [liangbowen] Revert "dedicate invokeStaticAs method"
ad45ff3fd [liangbowen] dedicate invokeStaticAs method
f08528c0f [liangbowen] nit
42aeb9fcf [liangbowen] add ut case
b5b384120 [liangbowen] nit
072add599 [liangbowen] add ut
8d019ab35 [liangbowen] unified invokeAs and getField
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Backport https://github.com/apache/hive/pull/1814
related kyuubi issues
#4305#4333#4406
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4972 from turboFei/beeline_head_command.
Closes#4305
04b235fb3 [fwang12] [KYUUBI #4305][Bug] Backport HIVE-15820: comment at the head of beeline -e
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#4803
The beeline has regression because of 70590f71ef (diff-993fdbefe9fe3d1c91fcedba99362a8c8d9b94793ec16cbfbc989e750367ea89)
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4965 from turboFei/revert_beeline_python_change.
Closes#4965
856d92391 [fwang12] trim for non python mdoe
f8464606b [fwang12] Revert "[KYUUBI #4619] Fix beeline with -e When there are other SQL statements before the source statement, the source statement cannot be executed normally"
e5e3c31b3 [fwang12] revert trim
bec09c254 [fwang12] migration guide
585da6fc1 [fwang12] fix'
f3fcfe97e [fwang12] save
8cb8cb9d0 [fwang12] save
e1539775a [fwang12] comments
814c970a2 [fwang12] save
b1baa773b [fwang12] save
3337ca8fa [fwang12] options
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Remove unnecessary conversion between `value = valueOpt.orNull` and `valueOpt = Option(value)`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4968 from pan3793/option.
Closes#4968
882181372 [Cheng Pan] fix
c79a72116 [Cheng Pan] nit
fdc95f221 [Cheng Pan] nit
f9e189fc0 [Cheng Pan] nit
2bdfe9a13 [Cheng Pan] nit
d02e2cfa1 [Cheng Pan] nit
ac8b5fb5d [Cheng Pan] Simplify Option convertion
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#4937.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4963 from bowenliang123/remove-spark-shim.
Closes#4937
e6593b474 [liangbowen] remove unnecessary row
2481e3317 [liangbowen] remove SparkCatalogShim
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Apache parent POM version 30 release note: https://github.com/apache/maven-apache-parent/releases/tag/apache-30
- remove `maven.plugin.surefire.version` property and bump the surefire plugin from 3.0.0-M8 to 3.1.2
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4966 from bowenliang123/parent-pom-30.
Closes#4966
970e48584 [liangbowen] Bump Apache parent pom from 29 to 30
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
FYI:
For hive-3.1.3, if failed to get connection, the return code of `beeline -f` is 0, the correct one should be non-zero.
And it works well for hive-4.0.0-alpha-2.
And the hive-beeline dependency version for kyuubi is 3.1.3.
This pr backports HIVE-19048: Initscript errors are ignored.
https://issues.apache.org/jira/browse/HIVE-19048006bf8a1a4
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4962 from turboFei/beeline_exit.
Closes#4962
85f497ad2 [fwang12] add ut
21e22cf6f [fwang12] save
49432c256 [fwang12] save
69f90ae37 [fwang12] save
b1afa9e26 [fwang12] backport
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It resolves https://github.com/apache/kyuubi/issues/4951.
It would be also possible to resolve it by adding log4j 1 slf4j binding (e.g., adding `slf4j-api-1.7.30.jar` and `slf4j-log4j12-1.7.30`) to the Spark jars directory, but it would be helpful if Kyuubi could support reload4j to avoid log4j 1's security vulnerabilities.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Assume
- we use Spark SQL engine in YARN cluster deploy mode;
- we use Hadoop-free Spark 3.2.4 and provides the classpath of Hadoop in YARN cluster, which uses reload4j, with `SPARK_DIST_CLASSPATH`.
Before applying the patch, Hive beeline shows operation logs only from Kyuubi server, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 22:55:35.584 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 5409e4cb-abe0-457f-b3c8-c3f1c5467c60
2023-06-14 22:55:35.604 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 22:55:35.662 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[013b8294-b395-4e3d-945e-3594b0603f83] in FINISHED_STATE
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.857 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.113 seconds)
```
After applying the patch, Hive beeline shows operation logs from both Kyuubi server and Spark SQL engine, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 23:05:57.424 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 146aa800-6047-471e-a480-5683197a84cd
2023-06-14 23:05:57.453 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 23:05:57.512 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement:
Spark application name: kyuubi_CONNECTION_SPARK_SQL_some_user_146aa800-6047-471e-a480-5683197a84cd
application ID: application_1686295570868_0065
application web UI: some_yarn_application_url
master: yarn
deploy mode: cluster
version: 3.2.4
Start time: 2023-06-14T23:05:02.490
User: some_user
(omitted)
2023-06-14 23:06:00.285 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[1c209bea-6f62-4854-acdc-a1945867a65d] in FINISHED_STATE
2023-06-14 23:06:00.286 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.774 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.156 seconds)
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4964 from touchida/issue-4951-reload4j.
Closes#4951
d7444be17 [toshihiko.uchida] [KYUUBI #4951] Add reload4j support
Authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3648
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4793 from zwangsheng/KYUUBI_3648.
Closes#3648
e5fe04f7a [zwangsheng] debug
a14c6d325 [zwangsheng] debug
eb96cf0dc [zwangsheng] retest
5cfbb64c5 [zwangsheng] Add UT
7a83a01b0 [zwangsheng] [KYUUBI #3648][UI] Add Session Detail Page
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
SQLite is added since 1.8.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4955 from pan3793/sqlite-init.
Closes#4950
24de565cf [Cheng Pan] [KYUUBI #4950][FOLLOWUP] SQLite schema files should start from 1.8.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Improve log on executing init schema DDL.
- Add `IF NOT EXISTS` on `CREATE INDEX` statement.
- Fix `kyuubi.metadata.store.jdbc.url` configuration description.
- Delete SQLite files on performing `build/mvn clean`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4953 from pan3793/sqlite-followup.
Closes#4950
a0b44f37c [Cheng Pan] [KYUUBI #4950][FOLLOWUP] Misc improvement for usage of SQLite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
fix typo, replace kyuubiJdbcUrl with kyuubiJdbcUrlTemplate.
### _Why are the changes needed?_
fix typo, should be String.format(kyuubiJdbcUrlTemplate, ...)
### _How was this patch tested?_
just fix typo, no test need.
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4954 from uicosp/patch-1.
Closes#4954
8abf04d87 [鱼肚] Update quick_start_with_jdbc.md
Authored-by: 鱼肚 <uicosp@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
When the default databases of glue do not exist, initializing the engine session failed.
```
Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:iam::xxxxx:user/vault-token-wap-udp-int-readwrite-1686560253-eVgl3oauuaB7v0V6wX9 is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:us-east-1:xxxxx:database/default because no identity-based policy allows the glue:GetDatabase action (Service: Glue, Status Code: 400, Request ID: 3f816608-0cb2-467b-9181-05c5bfcd29b3, Extended Request ID: null)
```
The default initialize sql "SHOW DATABASES" when Kyuubi spark accesses the glue catalog. Try to change the initialize sql "use glue.wap" has the same error.
```
kyuubi.engine.initialize.sql=use glue.wap
kyuubi.engine.session.initialize.sql=use glue.wap
```
The root cause is that hive defines a use:database, which will initialize access to the database default.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4952 from dev-lpq/glue_database.
Closes#4952
8f0951d4d [pengqli] add Glue database
90e47712f [pengqli] enhance AWS Glue database
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4881
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4886 from rhh777/jdbcengine.
Closes#4881
e0439f8e6 [haorenhui] [KYUUBI #4881] update settings.md
d667d8fe8 [haorenhui] [KYUUBI #4881] update conf/docs
cba06b4b9 [haorenhui] [KYUUBI #4881] simplify code
80be4d27c [haorenhui] [KYUUBI #4881] fix style
4f0fa3ab2 [haorenhui] [KYUUBI #4881] JDBCEngine performs initialization sql
Authored-by: haorenhui <haorenhui@kingsoft.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
For the operation getNextRowSet method, we shall add lock for it.
For example, for spark operation, the result iterator is not thread-safe, it might throw exception(if the jdbc client to kyuubi server connection socket timeout).
For incremental collect mode, the fetchResult might trigger a spark task to collect the incremental result(`self.next().toIterator`).
The jdbc client to kyuubi gateway timeout, but the fetchResult request has been sent to engine.
Then the jdbc client re-send the fetchResult request.
And the getNextResultSet in spark engine side concurrent execute.
And the result iterator is not thread-safe and might cause NPE.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4949 from turboFei/lock_next_rowset.
Closes#4949
8f18f3236 [fwang12] getNextRowSetInternal and withLockRequired
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- fetch versions for NodeJS and Pnpm from Maven properties
- enabled npm dependency caching for global packages and web-ui module, as the step above takes extra time in builds.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4948 from bowenliang123/pnpm-version.
Closes#4948
80b0f6934 [liangbowen] Reuse versions of NodeJS and Pnpm from maven properties
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- The current lockfileVersion of the `pnpm` lock file has been bumped from 6.0 to 6.1 in #4931 (passed CI as it uses latest 8.x pnpm), which is incompatible with the currently `pnpm` version 8.4.1 in maven's frontend plugin 8.4.1
- Bump pnpm version to solve the conflicts
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4947 from bowenliang123/pnpm-8.6.1.
Closes#4947
074dcca1f [liangbowen] Bump pnpm version to 8.6.1 for lockfileVersion 6.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4847
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4848 from huangzhir/kyuubisession_leak.
Closes#4847
37e58ce66 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala
170c044f5 [huangzhir] Add some logging and modify some code style.
0c393cd9d [huangzhir] fix sytle
d0183298e [huangzhir] "Use the ENGINE_ALIVE_PROBE_ENABLED configuration to enable the Kyuubi session engine alive probe. The default value of ENGINE_ALIVE_PROBE_ENABLED is false. Use the ENGINE_ALIVE_TIMEOUT configuration to determine the duration for checking the engine's alive status. The engineAliveMaxFailCount configuration controls the maximum number of failures allowed during engine alive checks."
b716dd8f6 [huangzhir] fix kyuubi session leak caused by engine stop
Lead-authored-by: huangzhir <306824224@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#4917
- support extracting table owner from `ResolvedIdentifier`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
<img width="1266" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/e3066d0e-7a14-41da-96f6-032a5c53780f">
Closes#4941 from zhouyifan279/drop-table.
Closes#4917
b2207ed17 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
bc4661a13 [zhouyifan279] [KYUUBI #4917][Bug][AUTHZ] Table owner undefied in Iceberg 1.3.0 on Spark 3.4
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Update doc to mark the spark plugin's config `spark.sql.optimizer.insertRepartitionNum` used for Spark 3.1 only
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4933 from bowenliang123/insert-num.
Closes#4933
5ed6e2867 [liangbowen] comment and style
280a6af03 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
7f01cf3b6 [liangbowen] spark.sql.optimizer.insertRepartitionNum only available for Spark 3.1.x
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4936 from zhouyifan279/kill-executor.
Closes#4936
2aaa84cb1 [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Bump vite from 3.0.0 to 4.2.3
- Bump vitest from 0.22.0 to 0.32.0
- change package vitest/coverage-c8 to vitest/coverage-v8 as required by vitest (https://github.com/vitest-dev/vitest/pull/3339)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4931 from bowenliang123/vite4.
Closes#4931
4221e2ab0 [liangbowen] bump vite from 3.0.0 to 4.2.3 and vitest from 0.22.0 to 0.32.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- apply the usage of `ReflectUtils` and `Dyn*` to the modules of engines and plugins (eg. Spark engine, Authz plugin, lineage plugin, beeline)
- remove similar redundant methods for calling reflected methods or getting field values
- unified reflection helper methods with type casting support, as `getField[T]` for getting field values from `getFields`, `invokeAs[T]` for invoking methods in `getMethods`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4879 from bowenliang123/reflect-use.
Closes#4879
c685fb67d [liangbowen] bug fix for "Cannot bind static field options" when executing "bin/beeline"
fc1fdf1de [liangbowen] import
59c3dd032 [liangbowen] comment
c435c131d [liangbowen] reflect util usage
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Kyuubi Server Proxy Engine UI
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4795 from zwangsheng/KYUUBI_3420.
Closes#3420
079dc1c60 [zwangsheng] fix frontend unit test case
6e71b4518 [Cheng Pan] fix
cf7ca5145 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala
9a91d62a0 [Cheng Pan] polish
a5dcfae18 [zwangsheng] fix
5d4a8c239 [zwangsheng] Rebase
71d22fc9a [zwangsheng] fix
3b0152f33 [zwangsheng] [KYUUBI #3420][UI] Proxy Engnie UI
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4732
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4813 from huangzhir/operation_log.
Closes#4813
39dcab61d [huangzhir] remove unrelated code
545efaee6 [huangzhir] remove object lock
f0d090f9a [huangzhir] throw exception if user requests FETCH_PRIOR
162e008a1 [huangzhir] remove FETCH_PRIOR test
3b40f6bac [huangzhir] fix style
78e49698c [huangzhir] add extra log test
68154fecb [huangzhir] Merge remote-tracking branch 'origin/master' into operation_log
21c46c06c [huangzhir] code rewritten ,fetch log only support FETCH_NEXT and FETCH_FIRST
cbd714a2b [huangzhir] Add the operationHandle parameter to the /v1/operations/:operationId/log interface.
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
to adapt Spark 3.5, the new conf `spark.sql.execution.arrow.useLargeVarType` was introduced in https://github.com/apache/spark/pull/39572
the signature of function `ArrowUtils#toArrowSchema` before
```scala
def toArrowSchema(
schema: StructType,
timeZoneId: String,
errorOnDuplicatedFieldNames: Boolean): Schema
```
after
```scala
def toArrowSchema(
schema: StructType,
timeZoneId: String,
errorOnDuplicatedFieldNames: Boolean,
largeVarTypes: Boolean = false): Schema
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4923 from cfmcgrady/arrow-toArrowSchema.
Closes#4923
3806494a5 [Fu Chen] Update Arguments of ArrowUtils#toArrowSchema Function
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
