Commit Graph

2811 Commits

Author SHA1 Message Date
Cheng Pan
161dd3fa8d [KYUUBI #4317] Upload kyuubi-spark-batch-submit.log.* on CI failure
### _Why are the changes needed?_

```
...
07:43:26.520 KyuubiSessionManager-exec-pool: Thread-779 INFO ProcBuilder: Logging to /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/runner/kyuubi-spark-batch-submit.log.0
...
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4317 from pan3793/ci-log.

Closes #4317

dbde5306 [Cheng Pan] Upload kyuubi-spark-batch-submit.log on CI failure

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-02-13 19:37:50 +08:00
odone
41f08059f0 [KYUUBI #3935] Support use Trino client to submit SQL
### _Why are the changes needed?_

Close #3935

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4232 from iodone/kyuubi-3935.

Closes #3935

936ea1f8 [odone] address
e7bd01a1 [odone] support trino client connect kyuubi trino server
9ea8b6af [odone] [WIP] trion request/response implementation

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-02-13 19:28:14 +08:00
liangbowen
3b0137ae78 [KYUUBI #4308] [DOCS] Make README more welcoming
### _Why are the changes needed?_
 

- Make Readme more welcoming to readers, in
  - Elegancy: clear, center aligned Kyuubi logo in front
  - Maturity: always good to see numbers of contributors, lines and PRs
  - Guidance: quick starting point to project homepage and documentation for next steps
  - Sociability: showing star number of the project

Preview link: https://github.com/bowenliang123/incubator-kyuubi/blob/readme-welcome/README.md

Before:
<img width="917" alt="image" src="https://user-images.githubusercontent.com/1935105/218124450-1cb099bb-87fb-41ad-8048-0a8c4653c754.png">

After:
<img width="913" alt="image" src="https://user-images.githubusercontent.com/1935105/218263680-8237f838-b0b9-42c4-b814-b909651aaef2.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4308 from bowenliang123/readme-welcome.

Closes #4308

35fafffb [Bowen Liang] add links for badges
55721d17 [liangbowen] remove licence badge and add total lines badge
42971575 [Bowen Liang] Update README.md
5909b1d6 [liangbowen] make Readme more welcoming

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-13 13:15:18 +08:00
dnskr
68b70ca1d2
[KYUUBI #4314] [DOCS][HELM] Refine helm chart docs
### _Why are the changes needed?_
The changes are needed to make doc page clearer and a bit nicer, and reflect latest changes in the chart.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>

Closes #4314 from dnskr/update_helm_chart_doc_page.

Closes #4314

d7c18fd53 [dnskr] [DOCS][HELM] Refine helm chart docs

Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-13 00:40:52 +08:00
hongkunyoo
b6ebd5e8c5
[KYUUBI #4307] Small missing kubectl
Maybe `kubectl` command is missing?

Closes #4307 from hongkunyoo/patch-1.

Closes #4307

549357af7 [hongkunyoo] Small missing `kubectl`

Authored-by: hongkunyoo <hongkunyoo@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-13 00:37:51 +08:00
dnskr
7f86611b31
[KYUUBI #4313] [K8S][HELM] Comment license in helm chart NOTES
### _Why are the changes needed?_
The changes are needed to hide license text printed after chart installed.
Before changes:
```
$ helm install kyuubi ${KYUUBI_HOME}/charts/kyuubi -n kyuubi --create-namespace
NAME: kyuubi
LAST DEPLOYED: Sat Feb 11 20:35:52 2023
NAMESPACE: kyuubi
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

The chart has been installed!

In order to check the release status, use:
  helm status kyuubi -n kyuubi
    or for more detailed info
  helm get all kyuubi -n kyuubi

************************
******* Services *******
************************
THRIFT_BINARY:
- To access kyuubi-thrift-binary service within the cluster, use the following URL:
    kyuubi-thrift-binary.kyuubi.svc.cluster.local
- To access kyuubi-thrift-binary service from outside the cluster for debugging, run the following command:
    kubectl port-forward svc/kyuubi-thrift-binary 10009:10009 -n kyuubi
  and use 127.0.0.1:10009
```

After changes:
```
$ helm install kyuubi ${KYUUBI_HOME}/charts/kyuubi -n kyuubi --create-namespace
NAME: kyuubi
LAST DEPLOYED: Sat Feb 11 20:37:45 2023
NAMESPACE: kyuubi
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The chart has been installed!

In order to check the release status, use:
  helm status kyuubi -n kyuubi
    or for more detailed info
  helm get all kyuubi -n kyuubi

************************
******* Services *******
************************
THRIFT_BINARY:
- To access kyuubi-thrift-binary service within the cluster, use the following URL:
    kyuubi-thrift-binary.kyuubi.svc.cluster.local
- To access kyuubi-thrift-binary service from outside the cluster for debugging, run the following command:
    kubectl port-forward svc/kyuubi-thrift-binary 10009:10009 -n kyuubi
  and use 127.0.0.1:10009
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4313 from dnskr/comment_license_in_helm_chart_notes.

Closes #4313

7cc76639b [dnskr] [K8S][HELM] Comment license in helm chart NOTES

Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-12 18:42:36 +08:00
runzhliu
de15dbe28a [KYUUBI #4299] Bump MySQL JDBC driver to 8.0.32
### _Why are the changes needed?_

bump Mysql jdbc driver to 8.0.32
release: https://github.com/mysql/mysql-connector-j/releases/tag/8.0.32
Mysql jdbc driver used in test scope only
nothing new after running build/dependency.sh --replace

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4309 from runzhliu/patch-7.

Closes #4299

a9e43e91 [runzhliu] Merge branch 'apache:master' into patch-7
53ce3a37 [runzhliu] Merge branch 'apache:master' into patch-7
d3f02cd6 [runzhliu] Bump Mysql jdbc driver from 8.0.31 to 8.0.32

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-11 09:07:05 +08:00
runzhliu
e545df5048 [KYUUBI #4299] Bump JUnit to 4.13.2
### _Why are the changes needed?_

bump JUnit4 from 4.13.1 to 4.13.2
release note: https://github.com/junit-team/junit4/blob/HEAD/doc/ReleaseNotes4.13.2.md
nothing new after running `build/dependency.sh --replace`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4310 from runzhliu/patch-6.

Closes #4299

8857d254 [runzhliu] Merge branch 'apache:master' into patch-6
c2893632 [runzhliu] Bump Junit4 from 4.13.1 to 4.13.2

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-11 08:54:07 +08:00
Cheng Pan
7cd206642c
[KYUUBI #4302] Change log level from debug to info for creating work dir
### _Why are the changes needed?_

The message is not noisy and is helpful when failing to create, e.g. no permission.

```
2023-02-10 14:30:32.731 INFO org.apache.kyuubi.operation.LaunchEngine: Processing simba's query[d51a4def-d3d7-454b-a63d-28e27a1e3c3f]: RUNNING_STATE -> ERROR_STATE, time taken: 0.1 seconds
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: java.io.IOException: 权限不够
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1012)
	at org.apache.kyuubi.engine.ProcBuilder.$anonfun$engineLog$6(ProcBuilder.scala:187)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.kyuubi.engine.ProcBuilder.engineLog(ProcBuilder.scala:184)
	at org.apache.kyuubi.engine.ProcBuilder.engineLog$(ProcBuilder.scala:159)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.engineLog$lzycompute(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.engineLog(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.ProcBuilder.processBuilder(ProcBuilder.scala:141)
	at org.apache.kyuubi.engine.ProcBuilder.processBuilder$(ProcBuilder.scala:135)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.processBuilder$lzycompute(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.processBuilder(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198)
	at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194)
	at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:174)
	at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160)
	at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165)
	at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:260)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:260)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:120)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
	at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
	at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:75)
	at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:56)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:51)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: 权限不够
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1012)
	at org.apache.kyuubi.engine.ProcBuilder.$anonfun$engineLog$6(ProcBuilder.scala:187)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.kyuubi.engine.ProcBuilder.engineLog(ProcBuilder.scala:184)
	at org.apache.kyuubi.engine.ProcBuilder.engineLog$(ProcBuilder.scala:159)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.engineLog$lzycompute(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.engineLog(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.ProcBuilder.processBuilder(ProcBuilder.scala:141)
	at org.apache.kyuubi.engine.ProcBuilder.processBuilder$(ProcBuilder.scala:135)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.processBuilder$lzycompute(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.processBuilder(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:198)
	at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:197)
	at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:36)
	at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:194)
	at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:174)
	at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:160)
	at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:165)
	at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:260)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:260)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:120)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
	at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:49)
	... 5 more (state=,code=0)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4302 from pan3793/info.

Closes #4302

24c8ab39d [Cheng Pan] Change log level from debug to info for creating work dir

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 16:22:19 +08:00
liangbowen
d862272645
[KYUUBI #4274] [INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check
### _Why are the changes needed?_

#### Motivation:
- speed up maven building and testing with `mvnd` (https://github.com/apache/maven-mvnd). `mvnd` itself embeds exactly the same distribution of maven with more improvements and features on the client side and compilation server in daemon mode.
- also inspired by custom github action for mvnd installing in Apache Camel (https://github.com/apache/camel/blob/main/.github/actions/install-mvnd/action.yml)

#### Changes in this PR:
- introducing `mvnd` by adding `build/mvnd` script
- use `mvnd` version `0.9.0` which embeds  `maven` 3.8.7. The maven version embedded in `mvnd` is also check and guaranteed  the same as maven version required in pom by `build/mvnd` script.
- use mvnd in CI jobs of Depedency Check and Style Check

#### Comparision
- CI jobs
(both with maven dependencies cached in local repo)

Job | with `build/mvn` | with `build/mvnd`
--- | --- | ---
Dependency Check | 9min1sec;[see log](https://github.com/apache/kyuubi/actions/runs/3966175262/jobs/6796688187) | 6min 46 sec ; [see log](https://github.com/apache/kyuubi/actions/runs/4124518481/jobs/7123867015)
Style Check | 10min32sec ; [see log](https://github.com/apache/kyuubi/actions/runs/4122613274/jobs/7119603428)  | 7min50s ; [see log](https://github.com/apache/kyuubi/actions/runs/4125013599/jobs/7125006946)
Licence Check | 32s ; [see log](https://github.com/apache/kyuubi/actions/runs/4130799867/jobs/7137835062) | 21s ; [see log](https://github.com/apache/kyuubi/actions/runs/4130923126/jobs/7138088114)

- building entire maven project on local machine (skipping test running and style checking with `fast` profile)

`build/mvnd clean install -Pfast`
<img width="645" alt="Pasted Graphic" src="https://user-images.githubusercontent.com/1935105/217001594-52dfd19d-6c57-4bc4-8336-90bb0c030bd4.png">

`build/mvn clean install -Pfast`
<img width="643" alt="Pasted Graphic 1" src="https://user-images.githubusercontent.com/1935105/217001619-c2ebe7e3-bd14-4f5c-bb02-2ae4ae99cf7f.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4274 from bowenliang123/ci-mvnd.

Closes #4274

380c1c9ea [liangbowen] update cache key
a93bba7b7 [liangbowen] style
bc8da133e [Bowen Liang] Merge branch 'master' into ci-mvnd
97becc035 [liangbowen] add `build/maven-mvnd-*/**` to rat-excludes
5b61f50b9 [liangbowen] apply mvnd to license check
7be181d0f [liangbowen] typo in build/mvnd
c52861df8 [liangbowen] update
f6d0eb28c [liangbowen] introduce mvnd

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 16:19:45 +08:00
Cheng Pan
4e226ac3cc
Bump 1.8.0-SNAPSHOT 2023-02-10 15:25:49 +08:00
zwangsheng
708d982eb9
[KYUUBI #4184][SUBTASK-1] Refactor docker-image-tool.sh
### _Why are the changes needed?_

See more in  #4184

This subtask remove build kyuubi docker image from dev mode(source code).

In follow subtask, will add feature build kyuubi docker image from source code.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4217 from zwangsheng/tool/docker-image-tool.

Closes #4184

34c866025 [zwangsheng] Add +x premission for docker-image-tool
97b404db7 [zwangsheng] Copy download spark binary
6f267e221 [zwangsheng] Refactor docker-image-tool.sh

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 14:28:14 +08:00
Fu Chen
255b110392
[KYUUBI #4296] Bump Spark in spark-master profile from 3.4.0-SNAPSHOT to 3.5.0-SNAPSHOT
### _Why are the changes needed?_

https://github.com/apache/spark/pull/39733

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4296 from cfmcgrady/spark-3.5.0-SNAPSHOT.

Closes #4296

e16e3f1b0 [Fu Chen] bump spark from 3.4.0-SNAPSHOT to 3.5.0-SNAPSHOT

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 13:05:17 +08:00
Cheng Pan
547e5ca617
[KYUUBI #4145] Change lock and polling seq_num path on service discovery
### _Why are the changes needed?_

This PR proposes to change the paths of distributed lock and seq_num(used for POLLING engine pool select policy) on the Service Discovery component. The reason is that namespace `${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/` should be dedicated to engine registration, we'd better use the separated namespace for other functionalities.

- lock path
```
# before
${serverSpace}_${shareLevel}_${engineType}/lock/${user}/${subdomain}

# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_lock/${user}/${subdomain}
```

- seq_num
```
# before
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/seq_num/${user}/${poolName}

# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_seqNum/${user}/${poolName}
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4145 from pan3793/namespace.

Closes #4145

c912b3f66 [Cheng Pan] name
3326b9b95 [Cheng Pan] name
10083db0b [Cheng Pan] Change lock and polloing seq_num path on service discovery

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 13:04:24 +08:00
Cheng Pan
5318585550
[KYUUBI #4288] Use eclipse-temurin:8-jdk-focal as default base image
### _Why are the changes needed?_

eclipse-temurin is the successor for openjdk, see https://github.com/apache/spark/pull/37705

"The core change is: the OS of base image changes debian-bullseye to ubuntu-focal (based on debian bullseye)."

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4288 from pan3793/image.

Closes #4288

b7b619018 [Cheng Pan] Use eclipse-temurin:8-jdk-focal as default base image

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 12:25:57 +08:00
Cheng Pan
8d269b83d3
[KYUUBI #4297] Daily publish master snapshot docker image
### _Why are the changes needed?_

This PR proposes to change the publish `master-snapshot` image from every commit of the master branch to the daily scheduled, it saves GA resources.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4297 from pan3793/docker-nightly.

Closes #4297

1424287cb [Cheng Pan] nit
32cffe8ab [Cheng Pan] Daily publish master snapshot docker image

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 12:23:48 +08:00
runzhliu
673e8e1ce7 [KYUUBI #4286] Bump Jackson from 2.14.1 to 2.14.2
### _Why are the changes needed?_

- Jackson 2.14.2 release note: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.14.2

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4286 from runzhliu/patch-4.

Closes #4286

6902e2ec [runzhliu] Update dependencyList
3cde2004 [runzhliu] Merge branch 'apache:master' into patch-4
8ac6eccd [runzhliu] bump the jackson maven dependecy version to 2.14.2

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-10 07:39:22 +08:00
Cheng Pan
afc912c25d
[KYUUBI #4253] Change default Flink version to 1.16
### _Why are the changes needed?_

This PR updates `pom.xml` and CI workflow to change the default Flink version to 1.16, close #4253

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4289 from pan3793/flink-1.16.

Closes #4253

dc3885a38 [Cheng Pan] [KYUUBI #4253] Change default Flink version to 1.16

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 05:16:45 +08:00
Cheng Pan
61df9e2cc4
[KYUUBI #4292] Pin threeten 1.7.0
### _Why are the changes needed?_

Pin `org.threeten:threeten-extra:1.7.0` for spark master as a workaround for https://github.com/ThreeTen/threeten-extra/issues/226

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4292 from pan3793/threeten.

Closes #4292

8c1a24fd5 [Cheng Pan] Pin threeten 1.7.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 22:42:50 +08:00
Cheng Pan
363e798b7c [KYUUBI #4283] [TEST] Fix flaky test in SessionsResourceSuite
### _Why are the changes needed?_

This PR proposes to fix the flaky test
```
- post session exception if failed to open engine session *** FAILED ***
```
which was introduced in #4185

https://github.com/apache/kyuubi/actions/runs/4131523933/jobs/7139249142

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4283 from pan3793/flaky.

Closes #4283

f5c70077 [Cheng Pan] [TEST] Fix flaky test in SessionsResourceSuite

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 13:18:42 +00:00
Kent Yao
1c7a66ff84 [KYUUBI #4269] Add a GA Job For Documentation Verification
### _Why are the changes needed?_

Add a GA Job For Documentation Verification

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4269 from yaooqinn/docga.

Closes #4269

c9b39de6 [Cheng Pan] Update .github/workflows/docs.yml
db2d37f8 [Kent Yao] Update .github/workflows/docs.yml
c5bc8518 [Kent Yao] trigger ga
6340129b [Kent Yao] trigger ga
fafbf847 [Kent Yao] trigger ga
52f23240 [Kent Yao] trigger ga
1e0000a7 [Kent Yao] Add a GA Job For Documentaion Verification

Lead-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 12:53:47 +00:00
runzhliu
0bdcfb8216 [KYUUBI #4248] Modify the GitHub Action of setting up the Minikube cluster
### _Why are the changes needed?_

bump the minikube version to 1.29.0 and the kubernetes version to 1.26.1.

After ran some tests for the used action manusa/actions-setup-minikubev2.7.2, it turns out that it would have some problems when I bump the minikube version to 1.29.0, please refer to [this link](https://github.com/runzhliu/yum-with-browser/actions/runs/4113082204/jobs/7098775474) for details.

In addition, I have tried another minikube action medyagh/setup-minikubemaster, it's good to bump the version for both minikube  and Kubernetes, please refer to [this link](https://github.com/runzhliu/yum-with-browser/actions/runs/4112397435/jobs/7097254604#step:5:8).

However, I would prefer to [this way](https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L945-L966) to setup a minikube cluster for the rest Kubernetes related tests, it's also good to set up the cluster as well, please refer to [this link](https://github.com/runzhliu/yum-with-browser/actions/runs/4113018527/jobs/7098637193#step:4:2).

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4248 from runzhliu/patch-1.

Closes #4248

7d12e0e8 [runzhliu] Update master.yml
52448ae0 [runzhliu] Update master.yml
5fb0b9ec [runzhliu] Update master.yml
3491b33a [runzhliu] Merge branch 'apache:master' into patch-1
64101d95 [runzhliu] bump the minikube version to 1.29.0 and the kubernetes version to 1.26.1
4b278697 [runzhliu] Merge branch 'apache:master' into patch-1
b19e0983 [runzhliu] Update master.yml
f6bc18ee [runzhliu] Update master.yml
bb133312 [runzhliu] Update master.yml
b44ad030 [runzhliu] Update master.yml
7f22810f [runzhliu] Update master.yml
09a6dadf [runzhliu] Update master.yml
60623715 [runzhliu] Update master.yml
0d2c6568 [runzhliu] Update master.yml
5c8b0cc6 [runzhliu] bump the minikube version to 1.29.0 and the kubernetes version to 1.26.1

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 20:52:11 +08:00
Cheng Pan
cc6f54ed0c
[KYUUBI #4247][FOLLOWUP] Fix build/mvn version comparison
### _Why are the changes needed?_

`build/mvn` should download the maven when the local `mvn` version does not match, this corrects the change in #4261

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4291 from pan3793/mvn-version.

Closes #4247

42717a73c [Cheng Pan] [KYUUBI #4247][FOLLOWUP] Fix build/mvn version comparison

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 20:45:02 +08:00
liangbowen
68cc0e4097 [KYUUBI #4262] [AUTHZ] Change from DROP to ALTER as required privilege of the source table in AlterTableRenameCommand
### _Why are the changes needed?_

to close #4262 .
- change the required privilege of the source table from DROP to ALTER
- skip privilege checks of the new target table, as the same way in Ranger's Hive plugin

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4281 from bowenliang123/4262-renametable.

Closes #4262

e7117de9 [liangbowen] change required privilege of the source table from DROP to ALTER

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 20:12:52 +08:00
df_liu
b47f0890e1
[KYUUBI #4250] Bump Flink to 1.16.1
### _Why are the changes needed?_
close #4250

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4275 from df-Liu/fix_4250.

Closes #4250

723c4b745 [df_liu] [KYUUBI #4250] fix 1.16.1
0e151a408 [df_liu] [KYUUBI #4250] Bump Flink 1.16.1

Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 20:10:13 +08:00
df_liu
5cd33a0e3c
[KYUUBI #4252][FOLLOWUP] Bump Flink 1.14.6
### _Why are the changes needed?_

updated `.github/workflows/master.yml#L161`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4284 from df-Liu/follow_4252.

Closes #4252

3b0069418 [df_liu] [KYUUBI #4252][FOLLOWUP] Bump Flink 1.14.6

Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 20:09:06 +08:00
Cheng Pan
3bb283ecd7
[KYUUBI #4287] [INFRA] Remove Travis
### _Why are the changes needed?_

As mentioned in https://github.com/apache/kyuubi/issues/4139, we are not able to use Travis for AMR testing in the future.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4287 from pan3793/rm-travis.

Closes #4287

efe2a4854 [Cheng Pan] nit
c72a8b00c [Cheng Pan] Remove travis

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 19:50:07 +08:00
Cheng Pan
a31d4490c8
[KYUUBI #4271] Clarify incremental collect mode ignore kyuubi.operation.result.max.rows
### _Why are the changes needed?_

`kyuubi.operation.result.max.rows` does not take effect on incremental collect mode because of performance concerns, this PR updates the configuration docs to mention that.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4271 from pan3793/maxrow.

Closes #4271

29b290a3e [Cheng Pan] nit
3d2872352 [Cheng Pan] log
277ebb5ff [Cheng Pan] ifx
091511a91 [Cheng Pan] nit
f15fb2270 [Cheng Pan] nit
1a1259ef5 [Cheng Pan] nit
ba57a2660 [Cheng Pan] fix
0b58d8b1a [Cheng Pan] ut
74b59dcee [Cheng Pan] nit
230defbff [Cheng Pan] Increamental collect mode should respect kyuubi.operation.result.max.rows

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 19:30:26 +08:00
Cheng Pan
fec6ee6110
[KYUUBI #4285] Do not cancel CI jobs on stable branch
### _Why are the changes needed?_

Since https://github.com/apache/kyuubi/pull/2175, the previous running CI jobs will be canceled on the same branch or PR to save CI resources, this PR keeps the behavior for PR but doesn't cancel the running CI jobs when commit to the stable branch.

Ref: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-a-fallback-value

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

I pushed an empty commit to verify the behavior does not change in PR, and I need to merge this to master first, then merge another PR immediately to verify the rest behaviors.

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4285 from pan3793/ci-con.

Closes #4285

3aa7f427e [Cheng Pan] empty
4156eb196 [Cheng Pan] Do not cancel CI jobs on stable branch

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 19:27:42 +08:00
liangbowen
9531552720 [KYUUBI #4266] [INFRA] Add Dependency Review step to prevent introducing vulnerable dependencies
### _Why are the changes needed?_

- Add actions/dependency-review-action (https://github.com/actions/dependency-review-action) as new step in the dependency workflow, which fails when future PR is bringing vulnerable dependencies.
- configuring `fail-on-severity` to `moderate` level

<img width="760" alt="image" src="https://user-images.githubusercontent.com/1935105/217551476-e2858c04-ebcb-4028-a209-e547690b5d79.png">

### _How was this patch tested?_
- [x] Pass the Dependency Check CI job

Closes #4266 from bowenliang123/dependency-review.

Closes #4266

63a84b22 [liangbowen] add actions/dependency-review-action to dependency workflow

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 14:14:55 +08:00
df_liu
3f5121ab2d [KYUUBI #4251] Bump Flink to 1.15.3
### _Why are the changes needed?_

close #4251

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4276 from df-Liu/fix_4251.

Closes #4251

eb5565cb [df_liu] [KYUUBI #4251] Bump Flink 1.15.3

Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 08:35:43 +08:00
df_liu
afcfe9afb5 [KYUUBI #4252] Bump Flink to 1.14.6
### _Why are the changes needed?_

close #4252

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4277 from df-Liu/fix_4252.

Closes #4252

a5969d01 [df_liu] [KYUUBI #4252] Bump Flink 1.14.6

Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 08:34:47 +08:00
Thomas Prelle
e5743c8309 [KYUUBI #4255][AUTHZ] Add authz for describe relation
### _Why are the changes needed?_
During a describe table call, if the table it's a v2 table, it's a DescribeRelation a not a DescribeTableCommand.

So DescribeRelation should have same authorization than DescribeTableCommand.

Fix https://github.com/apache/kyuubi/issues/4255

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4257 from tprelle/fixDescribeRelationAuthz.

Closes #4255

3ffdbc9d [Thomas Prelle] [AUTHZ] Add authz for describe relation

Authored-by: Thomas Prelle <thomas.prelle@ubisoft.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-09 00:06:07 +08:00
runzhliu
4070d78fe7 [KYUUBI #4268] Bump Arrow version to 11.0.0
### _Why are the changes needed?_

to close #4268 .

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4273 from runzhliu/patch-2.

Closes #4268

5329a842 [runzhliu] bump Arrow version to 11.0.0

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-08 23:43:52 +08:00
liangbowen
0eff3cec53 [KYUUBI #4270] Register shutdown hook for plugin cleanup
### _Why are the changes needed?_

to fix #4270.
- Register shutdown hook for plugin cleanup

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4272 from bowenliang123/authz-shutdown-hook.

Closes #4270

3f376f43 [liangbowen] add log
6a2018f6 [liangbowen] nit
8b7ed86c [liangbowen] register plugin's cleanup shutdownHook

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-08 22:56:43 +08:00
liangbowen
31077b5763 [KYUUBI #4265] Bump maven-surefire-plugin to 3.0.0-M8
### _Why are the changes needed?_

- bump maven-surefire-plugin 3.0.0-M7 (released in June 2022) to 3.0.0-M8 (release in Jan 2023), release notes: https://blogs.apache.org/maven/entry/apache-maven-surefire-failsafe-plugin1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4265 from bowenliang123/surefire-3.0.0-M8.

Closes #4265

60f9b2da [liangbowen] bump maven-surefire-plugin to 3.0.0-M8

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-08 00:26:53 +08:00
edddddy
95318d5782
[KYUUBI #4247] build/mvn should use the exact version defined in pom.xml
### _Why are the changes needed?_

As mentioned in #4247, it seems to be better that using the same maven version with master. Thus, modify the version check logic in `build/mvn` from equal or greater to equal.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4261 from edddddy/mvn_ver_cp.

Closes #4247

11f19eca1 [1456173400] change the if condition to equal

Lead-authored-by: edddddy <rmrfall@qq.com>
Co-authored-by: 1456173400 <1456173400@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-07 19:04:10 +08:00
liangbowen
301185c650 [KYUUBI #4256] [DOCS] Add PYTHON option to kyuubi.operation.language config
### _Why are the changes needed?_

- as pyspark is supported in #3780, exposing PYTHON option for `kyuubi.operation.language` as an experimental feature

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4256 from bowenliang123/doc-language-python.

Closes #4256

d10c0281 [liangbowen] update description
d05cedda [liangbowen] add PYTHON option in kyuubi.operation.language

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-07 12:27:45 +08:00
yehere
1b92d80678 [KYUUBI #3934] Compatiable with Trino rest dto
### _Why are the changes needed?_

close #3934

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4182 from yehere/kyuubi-3934.

Closes #3934

ced64e2c [yehere] [KYUUBI #3934] Add more result types support
3ef85230 [yehere] [KYUUBI #3934] Optimization for code review
69e1f442 [yehere] [KYUUBI #3934] Merge the test class to TrinoContextSuite
4f0a0152 [yehere] [KYUUBI #3934] Merge the class to TrinoContext
7c9473f6 [yehere] [KYUUBI #3934] Format style, with Copyright  Profiles
2023f3ce [yehere] [KYUUBI #3934] Format and add test case
a2243b46 [yehere] [KYUUBI #3934] Compatiable with Trino rest dto

Authored-by: yehere <867171931@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-02-07 11:08:48 +08:00
fwang12
f62a7ac587 [KYUUBI #3968][FOLLOWUP] Get schema url with correct version ordering
### _Why are the changes needed?_

Get schema url with correct version ordering.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4254 from turboFei/version_order.

Closes #3968

57474e65c [fwang12] refactor
561a1cde9 [fwang12] refactor
0a3275aa7 [fwang12] save
64236740d [fwang12] save
a723a4023 [fwang12] save
c31564865 [fwang12] save
13b3d29e0 [fwang12] save
347081d77 [fwang12] save

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-02-07 10:51:51 +08:00
Cheng Pan
82a1883561 [KYUUBI #4258] Bump testcontainers-scala 0.40.12
### _Why are the changes needed?_

Keep deps update-to-date

https://mvnrepository.com/artifact/com.dimafeng/testcontainers-scala

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4258 from pan3793/testcontainers-scala.

Closes #4258

1352f013 [Cheng Pan] Bump testcontainers-scala 0.40.12

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-06 23:11:15 +08:00
liangbowen
534fc9f20f [KYUUBI #4209] [Authz] Set logging level to DEBUG in PrivilegesBuilder for multiple specs
### _Why are the changes needed?_

- change logging level from `warn` to `debug` in Authz
- to eliminate duplicated unuseful messages (as screenshot below) with multiple specs in extracting resources with lower Spark versions
![WechatIMG7517](https://user-images.githubusercontent.com/1935105/215329357-ad242cf4-9c98-44b0-978a-f37304a85ef0.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4209 from bowenliang123/authz-debug.

Closes #4209

f3b949bc [liangbowen] change logging level from `warn` to `debug` in multi extractor rule in Authz serde

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-06 15:49:56 +08:00
liangbowen
5b70b41a01
[KYUUBI #4244] Improvement in auto-generated Markdown docs with MarkdownBuilder
### _Why are the changes needed?_

Note:
- No output changes to existed generated docs as in `settings.md` and `functions.md`.

Improvement:
- readability in Scala code for doc auto-generation, and easy to maintain docs in group of sections
- reline on markdown linting from `flexmark`, and reducing un-meaningful blank lines or alignment
- declaration over operation by replacing repeated usages of `newOutput` itself by elegantly wrapped `.line`,`.lines`and etc.
- less fragile and more handy for changing in a long single line, as now using auto margin stripping in `.line()` method
- reusable extracted licence and auto-generation hints
- more elegant and safer way to read and appending file content
- possible less memory footprint by apply operators to Stream instead of to ArrayBuffer

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4244 from bowenliang123/config-regen.

Closes #4244

3d90ad304 [liangbowen] make buffer private in MarkdownBuilder
8250a55f3 [liangbowen] remove licence.md
a4c7baf78 [liangbowen] add MarkdownBuilder.apply
248a046a4 [liangbowen] Improvement in auto-generated Markdown docs with MarkdownBuilder

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-06 14:51:17 +08:00
ulysses-you
8760eeef0e [KYUUBI #4241] Only force close engine ref when open session failed
### _Why are the changes needed?_

We do not need force close engine builder and application if open session succeded. The engine itself will shutdown in connection level.

This pr tries to fix regression of https://github.com/apache/kyuubi/pull/2482

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4241 from ulysses-you/skip-close.

Closes #4241

0b280411 [ulysses-you] style
16369123 [ulysses-you] Only force close engine ref when open session failed

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-02-06 11:12:00 +08:00
runzhliu
b84cb2f2d7
[KYUUBI #4234] Keep the same version for Integration Test on Kubernetes
### _Why are the changes needed?_

Keep the same version for  Integration Tests on Kubernetes and Minikube.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4234 from runzhliu/master.

Closes #4234

94345659d [runzhliu] Keep the same version for  Integration test on Kubernetes

Authored-by: runzhliu <runzhliu@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-06 10:27:23 +08:00
Alex Wiss-Wolferding
2b958c69e1
[KYUUBI #4218] Using DB and table name when checking Delta table schema.
### _Why are the changes needed?_

To close #4218 .
This change ensures BI tools can list columns on Delta Lake tables in all schemas.

<img width="312" alt="image" src="https://user-images.githubusercontent.com/89149767/215793967-722eb5f9-ffe4-4ffb-b7f9-1ded06c146d7.png">

<img width="725" alt="image" src="https://user-images.githubusercontent.com/89149767/215794036-871f005f-1494-487d-90aa-1f99891177c2.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4219 from nousot-cloud-guy/feature/delta-db-schema.

Closes #4218

569843213 [Alex Wiss-Wolferding] Reversing match order in getColumnsByCatalog.
a6d973a3e [Alex Wiss-Wolferding] Revert "[KYUUBI #1458] Delta lake table columns won't show up in DBeaver."
20337dc96 [Alex Wiss-Wolferding] Revert "Using DB and table name when checking Delta table schema."
f7e4675a7 [Alex Wiss-Wolferding] Using DB and table name when checking Delta table schema.

Authored-by: Alex Wiss-Wolferding <alex@nousot.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-06 10:20:23 +08:00
dnskr
8a99750a20
[KYUUBI #4147] [K8S][HELM] Add configuration support for multiple frontends to helm chart
### _Why are the changes needed?_
The changes allow to configure multiple frontends in the helm chart.
Notes:
 - Removed unused code
 - `server.confDir` renamed to `kyuubiConfDir`
 - `server.conf` renamed to `kyuubiConf`
 - `livenessProbe` and `readinessProbe` changed to execute `bin/kyuubi status` which actually prints `Kyuubi is not running`. The issue needs to be reviewed. Also `livenessProbe` and `readinessProbe` should be revisited in the next PR.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4147 from dnskr/helm_frontend_configs.

Closes #4147

ab10dd15b [dnskr] [K8S][HELM] Address PR comments
65e1f593e [dnskr] [K8S][HELM] Address some PR comments
36b5fc64f [dnskr] [K8S][HELM] Add configuration support for multiple frontends to helm chart

Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-06 10:11:00 +08:00
zwangsheng
4dd00c1499
[KYUUBI #4247] Bump maven from 3.8.6 to 3.6.7
### _Why are the changes needed?_

For `maven version` in CI is 3.8.7(get info by adding `/usr/bin/mvn --version` in `.github/workflows/dep.yaml`),
![popo_2023-02-06  09-34-07](https://user-images.githubusercontent.com/52876270/216862168-beda949f-a77b-44db-ad11-b959e27627ef.jpg)

We should bump maven version property in parent pom.yaml 3.8.6 => 3.8.7.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4247 from zwangsheng/bump/build_mvn_version.

Closes #4247

431c248e8 [zwangsheng] [Bump] Maven Version for build/mvn

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-06 09:50:23 +08:00
liangbowen
f6778487ea [KYUUBI #4239] Remove duplicate buildConf methods and use KyuubiConf.buildConf directly
### _Why are the changes needed?_

- Remove duplicate buildConf methods in a series configs , change to import and use `KyuubiConf.buildConf` directly

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4239 from bowenliang123/duplicate-buildconf.

Closes #4239

04495e69 [liangbowen] remove duplciate buildConf methods

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-02-03 16:57:12 +08:00
fwang12
265aaf2854 [KYUUBI #3658][FOLLOWUP] Show ssl enabled protocols
### _Why are the changes needed?_

Before:
```
2023-02-02 22:21:35.507 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: SSL Server Socket enabled protocols: [Ljava.lang.String;2fca3eb5
```

Seems `enabledProtocols` is a java array, the output is not friendly.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4238 from turboFei/ssl_followup.

Closes #3658

999ef90e6 [fwang12] Show enabled ssl protocols

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-02-03 16:17:39 +08:00