Commit Graph

57 Commits

Author SHA1 Message Date
Cheng Pan
4e226ac3cc
Bump 1.8.0-SNAPSHOT 2023-02-10 15:25:49 +08:00
fwang12
b420243a1f [KYUUBI #4155] Reduce the application info call for batch
### _Why are the changes needed?_

Reduce the application info call for batch.
- If terminated and applicationInfo is defined, return applicationInfo directly.
- For batch report, return the existing applicationInfo directly.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4155 from turboFei/terminate_state.

Closes #4155

9d7e16121 [fwang12] comment
a0d70a633 [fwang12] Fix style
d9814c5b4 [fwang12] get or fetch
e547ff071 [fwang12] refine the variable
f9130e30e [fwang12] refactor code
5913d2419 [fwang12] fix ut
3b2772672 [fwang12] reduce app info call
a001dd9c4 [fwang12] do not call yarn for batch report
beaa54b32 [fwang12] if terminated, do not call

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-01-16 07:44:15 +08:00
liangbowen
48e93c9cf2
[KYUUBI #4110] Update MySQL connector dependency name to mysql-connector-j in 8.0.31
### _Why are the changes needed?_

- bump MySQL connector version from `8.0.27` to `8.0.31`
- From `8.0.31`, MySQL Java connector changed the dependency name from `mysql:mysql-connector-java` to `com.mysql:mysql-connector-j`, refer to docs: https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-installing-maven.html, and maven repo https://mvnrepository.com/artifact/com.mysql/mysql-connector-j/8.0.31
- connector version release note: https://github.com/mysql/mysql-connector-j/blob/release/8.0/CHANGES
- MySQL connector is only used in test scope and not part of distribution

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4110 from bowenliang123/mysql-connector-j-8.0.31.

Closes #4110

c3682674 [liangbowen] bump mysql connector version to 8.0.31 and change to use new dependency name in com.mysql:mysql-connector-j

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-01-06 18:17:18 +08:00
毕博
32b06c648a
[KYUUBI #4000] [FEATURE] Support Apache Phoenix engine base on jdbc engine
refer to https://github.com/apache/incubator-kyuubi/issues/3999
### _Why are the changes needed?_

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

- config Apache Phoenix jdbc engine in kyuubi-defaults.conf
<img width="882" alt="图片" src="https://user-images.githubusercontent.com/33744252/208430375-b52fc40d-3404-41c6-b526-db9a000cb702.png">
- use beeline connected
<img width="1399" alt="图片" src="https://user-images.githubusercontent.com/33744252/208430859-44069ecd-b9f6-4977-a771-70b8986daadd.png">
- query phoenix in kyuubi
<img width="1435" alt="图片" src="https://user-images.githubusercontent.com/33744252/208432332-fb1bc939-78bc-4ed1-88b1-21eb343e4fc0.png">

Closes #4000 from 531651225/add_phonix_jdbc-engin.

Closes #4000

2b947fdd [毕博] fix PhoenixDialect fetchSize and pom dependency alphabetically
1fd13dcd [毕博] Merge branch 'master' into add_phonix_jdbc-engin
812e4a0e [毕博] add integration test
c666f355 [毕博] add integration test
df18b4be [毕博] using the origin artifact/group org.apache.phoenix in pom
89feb691 [毕博] [FEATURE] Support Apache Phoenix engine base on jdbc engine

Authored-by: 毕博 <bibo@mafengwo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-01-05 21:57:26 +08:00
liangbowen
7459fac3b6 [KYUUBI #4082] Bump Jersey from 2.36 to 2.38
### _Why are the changes needed?_

Release notes for Jersey:
- https://github.com/eclipse-ee4j/jersey/releases/tag/2.38
- https://github.com/eclipse-ee4j/jersey/releases/tag/2.37

Changes in this PR:
- `jakarta.activation-api` is now removed from `dependencyList`
- `jakarta.activation-api` added to `kyuubi-kubernetes-it` module's dependency in test scope for satisfying  `hadoop-client-minicluster`

Facts:
- Before : with jersey 2.36,
	- `kyuubi-kubernetes-it` relies on `hadoop-client-minicluster` in test scope
	- `hadoop-client-minicluster` uses `DataSource` and other classes in package `javax.activation` without depending on the depencency
	- `javax.activation` package is actually provied by `jakarta.activation-api` included by `jersey-media-json-jackson`

- Investigation: with jersey 2.38
	- `jakarta.activation-api` was removed in Jersey 2.38 compared to 2.36 (https://github.com/eclipse-ee4j/jersey/compare/2.36...2.38)
	- from the blame history, it's clear that `jakarta.activation-api` was explictly excluded from `jersey-media-json-jackson`
	(https://github.com/eclipse-ee4j/jersey/blob/2.38/media/json-jackson/pom.xml), which in order to apply changes from `maven-enforcer-plugin` in (7e7d339281 (diff-f47eb7c28b888c017334e057492d4d2eab5be3e0e02bea1ff10765542bf9b9c8R101))

- Solution:
	- added `jakarta.activation-api` to `kyuubi-kubernetes-it` module's dependency in test scope statisfying `hadoop-client-minicluster`
	- confirm `jakarta.activation-api` is no longer directly or indirectly dependent in maven runtime scope

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4082 from bowenliang123/jersey-2.38.

Closes #4082

88c7cd80 [liangbowen] move jakarta.activation-api dependency closer to its dependant hadoop-client-minicluster
ce6df817 [liangbowen] add jakarta.activation-api dependency in test scope to kubernetes-integration-tests_2.1 module
8422dc2f [Bowen Liang] Merge branch 'master' into jersey-2.38
606cad0a [liangbowen] Bump Jersey from 2.36 to 2.38

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-01-05 01:48:36 +00:00
Tianlin Liao
8e8d8be8d2
[KYUUBI #4020] remove incubating from kyuubi source code
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4069 from lightning-L/kyuubi-4020.

Closes #4020

97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2023-01-04 09:43:20 +08:00
zwangsheng
b4e0a97044
[KYUUBI #3953] [IT][Test][K8S] Fix Kyuubi with Spark On Kubernetes Flaky Test
### _Why are the changes needed?_

To close #3952 & #3176

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3953 from zwangsheng/test/fix_spark_on_kubernetes_flaky_test.

Closes #3953

d884f0db [zwangsheng] fix sty;e
9923407c [zwangsheng] using toApplicationState format
e2194010 [zwangsheng] ifx
471745d8 [zwangsheng] Test

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-12-12 17:00:56 +08:00
zwangsheng
c9eb3cc589
[KYUUBI #3914] [K8S][SPARK] Set spark driver host to ip instead of pod name when kyuubi on k8s submit spark with client deploy-mode
### _Why are the changes needed?_

Set `spark.driver.host` to ip instead of pod name when kyuubi on k8s submit spark with client deploy-mode.

When Kyuubi On Kubernetes submit spark with client deploy mode, spark driver will using kyuubi pod name as `spark.driver.host`, which can't be recognized by executors(Exclude kyuubi deployed as Statefulset + serivce case).

We have done this in #1596

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3914 from zwangsheng/k8s/set_driver_host_if_client.

Closes #3914

8fb89657 [zwangsheng] fix style
fc38143f [Binjie Yang] Merge branch 'master' into k8s/set_driver_host_if_client
4a313655 [zwangsheng] fix style
c61b7b8f [zwangsheng] fix

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-12-12 15:55:07 +08:00
zwangsheng
46ffc7b603
[KYUUBI #3385] Set executor pod name prefix if missing in spark on k8s case
### _Why are the changes needed?_

See more in #3590
For #3590 this PR is reverted, and author not reply for long time.
Fix this issue #3385 here.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3913 from zwangsheng/k8s/set_executor_pod_name_prefix.

Closes #3385

64b2c6b7 [zwangsheng] fix for review
b58ff3f5 [zwangsheng] add unit test
cb0ad9f1 [zwangsheng] fix
7a36292b [zwangsheng] init

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-12-12 10:23:37 +08:00
zwangsheng
a0fc33c6af
[KYUUBI #3869] [K8S][IT][BUG] Fix the issue that connect conf is not used in the jdbc connection string
### _Why are the changes needed?_

Fix the issue that connect conf is not used in the Kyuubi On Kubernetes IT's jdbc connection string.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

- [x] Wait for IT ci

Closes #3869 from zwangsheng/bugfix/kyuubi_on_k8s_it_connect_conf.

Closes #3869

3cecd5f4 [zwangsheng] fix
63025a28 [zwangsheng] fix
924949f4 [zwangsheng] fix
21e93298 [zwangsheng] fix
fc5794ef [zwangsheng] fix
6dca96cd [zwangsheng] test
c2c81bb4 [zwangsheng] test
b8bb820b [zwangsheng] add unit test
dad4c739 [zwangsheng] fix bind
406f1de5 [zwangsheng] proxy for 185
c7d6ee6d [zwangsheng] merge
2290a24f [Binjie Yang] Update master.yml
9ffcb498 [Binjie Yang] Update KyuubiOnKubernetesTestsSuite.scala
714b340d [Binjie Yang] Update KyuubiOnKubernetesTestsSuite.scala
fa7fc542 [Binjie Yang] Update KyuubiOnKubernetesTestsSuite.scala
af4b9881 [zwangsheng] set 777 for /
49f705eb [zwangsheng] set 777 for test
fc66843d [zwangsheng] stop ci
e2ba0bcf [zwangsheng] add test name
7db4eab1 [zwangsheng] fast test cluster
33d490d1 [zwangsheng] add unit test
e2e12f4e [zwangsheng] fast test cluster
e8251011 [zwangsheng] test
b66468f5 [zwangsheng] test
633d99e4 [zwangsheng] change host
40ba5740 [zwangsheng] test
e393f9a5 [zwangsheng] test
532cd7df [zwangsheng] merge
4597572e [zwangsheng] test
b8fc86a1 [Binjie Yang] Update KyuubiOnKubernetesTestsSuite.scala
34be2761 [zwangsheng] TEST
a3c60e45 [zwangsheng] Changes
19e3bc22 [zwangsheng] for fast test
3ad2337f [zwangsheng] try cluster
39df2c40 [zwangsheng] try cluster
ed8f8baa [zwangsheng] fix client
7f711acb [zwangsheng] fix
b034731e [zwangsheng] fix
d646f4ac [zwangsheng] fix
2b9591c4 [zwangsheng] debug
a977d907 [zwangsheng] fix
0c3486fa [zwangsheng] debug
f0a0304b [zwangsheng] Add serviceAccount
eb3424ab [zwangsheng] fix user
cac7e69d [zwangsheng] proxy user
2886520f [zwangsheng] debug
25a677c6 [zwangsheng] debug
9f201d89 [zwangsheng] debug
e533664d [zwangsheng] fix it test
d9bf9173 [zwangsheng] fix it test

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-12-06 10:24:52 +08:00
sychen
e49f775431
[KYUUBI #3847] Add jdbc-shaded profile to support IDE debug
### _Why are the changes needed?_
1. `mvn antrun:runbuild-info -pl kyuubi-common`
2. debug engine suite

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3847 from cxzl25/profile_jdbc_debug.

Closes #3847

e1c32067 [Cheng Pan] Merge branch 'master' into profile_jdbc_debug
f41d3cd9 [sychen] use jdbc-shaded
25b54b18 [sychen] use jdbc-shaded
dd23f864 [sychen] use kyuubi-hive-jdbc as default
79b0ffcb [sychen] add jdbc profile

Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-30 22:34:48 +08:00
liangbowen
2ac10f91d5
[KYUUBI #3842] [Improvement] Support maven pom.xml code style check with spotless plugin
### _Why are the changes needed?_

Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3843 from bowenliang123/spotless-pom.

Closes #3842

3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-23 22:08:00 +08:00
Cheng Pan
689cbe2c18 [KYUUBI #3715] Remove unexpected mysql connector jar
### _Why are the changes needed?_

Fix #3715.

It was added in https://github.com/apache/incubator-kyuubi/pull/3240, and I didn't notice a binary Jar was shipped in the patch, and the worse thing is it was included in the source release tarball.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3716 from pan3793/mysql.

Closes #3715

0fdcac98 [Cheng Pan] fix
833d4679 [Cheng Pan] nit
5d098397 [Cheng Pan] [KYUUBI #3715] Remove unexpected mysql connector jar

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-29 10:22:58 +00:00
zwangsheng
9827c60aa4
[KYUUBI #3663] [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
### _Why are the changes needed?_

Current kyuubi build kubernetes client for `KyuubiApplicationOperator` to find or kill spark engine on kubernetes.

This pr aims to extract the code for building kubernetes client into `KubernetesUtils` and to do auto build following the logic in Spark.

We prefer to building with user-configured parameters; When missing, the kubernetes client logic will build one from environment variables and kubeconfig.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3663 from zwangsheng/feature/perfer_build_from_pod_env.

Closes #3663

0a4c8c79 [zwangsheng] clear properties
ce408db0 [zwangsheng] fix test
fb28ea7a [zwangsheng] add ci test
943c68dd [zwangsheng] add unit test
ddc9a2b2 [zwangsheng] fix scala
75380d13 [zwangsheng] comment
e6826609 [zwangsheng] master
683cc260 [zwangsheng] context provided master
7b42133c [zwangsheng] add doc
b56829c5 [zwangsheng] fix helm
5884475d [zwangsheng] rebuild
a7f0b551 [zwangsheng] ad
e7f9ec05 [zwangsheng] add setting
b77bbd9b [zwangsheng] add doc
3f3253d3 [zwangsheng] init

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-25 14:21:41 +08:00
Bowen Liang
fe431b2a36
[KYUUBI #3635] Delete temp directories on exit in test suites with Utils.createTempDir
### _Why are the changes needed?_

to close #3635 .
- change test suits to  to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3660 from bowenliang123/3635-deletetmp.

Closes #3635

8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-18 22:19:07 +08:00
Cheng Pan
84297ea466
Revert "[KYUUBI #3635] Delete files on exit in test suites with Utils.createTempDir"
This reverts commit 1de59bf6ec.
2022-10-17 22:37:21 +08:00
Bowen Liang
1de59bf6ec
[KYUUBI #3635] Delete files on exit in test suites with Utils.createTempDir
### _Why are the changes needed?_

to close #3635 .
- change test suits to  to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3636 from bowenliang123/3635-tempdir.

Closes #3635

5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang]  delete files on exit in test suits with Utils.createTempDir

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-17 10:00:49 +08:00
Cheng Pan
2cb34c4d00
[KYUUBI #3597] Engine should prefer ip for registering on K8s cluster mode
### _Why are the changes needed?_

When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.

Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266

To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.

Close #3578

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3597 from pan3793/k8s.

Closes #3597

8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-10 12:04:41 +08:00
df_liu
d06c656cc6
[KYUUBI #3560] Flink SQL engine supports run DDL across versions
### _Why are the changes needed?_

Followup #3230

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3560 from df-Liu/flink_ddl.

Closes #3560

0dbdfb3f [df_liu] flink ddl

Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2022-09-27 15:28:37 +08:00
Cheng Pan
c24a192dcb
[KYUUBI #2040] [TEST] Add Zookeeper 3.4/3.5/3.6/3.7 integration test
### _Why are the changes needed?_

This PR proposes to add Testcontainers based Zookeeper 3.4/3.5/3.6/3.7 integration test, we need it to verify the compatibility if we upgrade Zookeeper client to the new version. See details in #1941

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2040 from pan3793/zk-3-4-it.

Closes #2040

929540f6 [Cheng Pan] [TEST] Add Zookeeper 3.4/3.5/3.6 integration test

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-13 09:32:16 +08:00
Cheng Pan
6a90b3e824 [KYUUBI #3122] GetInfo supports return server/engine info
### _Why are the changes needed?_

Workaround for #3032, close #3323

There are known some ODBC drivers e.g. [Databricks ODBC driver](https://www.databricks.com/spark/odbc-drivers-download) depending on `TGetInfoType.CLI_DBMS_VER` and `TGetInfoType.CLI_DBMS_NAME` to check server compatibilities and abilities.

This PR proposes to introduce a new configuration `kyuubi.server.info.provider=SERVER/ENGINE` to make GetInfo support return either server or engine information.

Since beeline will call GetInfo in the initialization phase, to make sure the beeline fast open experience, in async launch mode, when the engine is not ready, return server info regardless `kyuubi.server.info.provider`.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

Testing w/ PowerBI

![image](https://user-images.githubusercontent.com/26535726/188945975-0d0fc95c-f989-4025-ad7d-c024e23ec328.png)

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3122 from pan3793/info.

Closes #3122

742bdbe3 [Cheng Pan] nit
bb85d2ba [Cheng Pan] style
fd75238c [Cheng Pan] fix
9ddb2afd [Cheng Pan] nit
fd8f7979 [Cheng Pan] fix ut
840205e5 [Cheng Pan] nit
f9996d53 [Cheng Pan] GetInfo supports returning engine info

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-08 02:14:38 +00:00
yikf
a54daf27c2
[KYUUBI #3298] Unify the approach of get classLoader in Kyuubi
### _Why are the changes needed?_

fix https://github.com/apache/incubator-kyuubi/issues/3298

Curently, a serval snippets use `getClass.getClassLoader` to get the classLoader in Apache Kyuubi, this pr aims to unify this behavior in `Utils.getContextOrKyuubiClassLoader`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3299 from yikf/unify-calssloader.

Closes #3298

ae081ad9 [yikf] Unify the approach of get classLoader in Kyuubi

Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-23 13:50:10 +08:00
SteNicholas
77b036f3a8
[KYUUBI #3264] [RELEASE] Bump 1.7.0-SNAPSHOT
### _Why are the changes needed?_

Preparing v1.7.0-SNAPSHOT with branch-1.6 cut

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3264 from SteNicholas/prepare-1.7.0-snapshot.

Closes #3264

374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-18 11:23:54 +08:00
Min
3cf5a20b7a
[KYUUBI #3239] [Subtask] DorisSQLEngine - Add integration tests
### _Why are the changes needed?_

close https://github.com/apache/incubator-kyuubi/issues/3239#issue-1338893480
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3240 from zhaomin1423/jdbc_it.

Closes #3239

8da64f29 [Min] fix it
24cceba2 [Min] add dependency
b477326c [Min] add server to dependency
e74aa93f [Min] fix getJdbcUrl conflict
01e6a302 [Min] [KYUUBI #3239] [Subtask] DorisSQLEngine - Add integration tests

Authored-by: Min <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-17 14:14:31 +08:00
wangjunbo
b3723392db
[KYUUBI #833] Check if spark.kubernetes.executor.podNamePrefix is invalid
### _Why are the changes needed?_
close #833

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3164 from Kwafoor/master.

Closes #833

63995b5c [wangjunbo] [kyuubi-833]Check if spark.kubernetes.executor.podNamePrefix is invalid
ef6663dd [wangjunbo] [kyuubi-833]Check if spark.kubernetes.executor.podNamePrefix is invalid

Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-08-11 09:51:47 +08:00
ulysses-you
24b93840f6
[KYUUBI #3136] Change Map to a case class ApplicationInfo as the application info holder
### _Why are the changes needed?_

closes https://github.com/apache/incubator-kyuubi/issues/3136

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3137 from ulysses-you/application-info.

Closes #3136

d600414a [ulysses-you] style
89c68d06 [ulysses-you] style
f6b885f6 [ulysses-you] address comment
1f469c3e [ulysses-you] unknown
b0af99f8 [ulysses-you] unknown
6f1b67e3 [ulysses-you] address comment
1f9047e9 [ulysses-you] nit
e0a02e4b [ulysses-you] fix k8s
1ba0559d [ulysses-you] fix test
61788562 [ulysses-you] fix test
51eeca98 [ulysses-you] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into application-info
c4871134 [ulysses-you] fix
7a876f06 [ulysses-you] fix
18524785 [ulysses-you] correct application state mapping
41234566 [ulysses-you] status compatibility
01f63852 [ulysses-you] Make application info stable

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-07-29 14:51:27 +08:00
Fei Wang
6b414083c9 [KYUUBI #3119] [TEST] Using more light-weight SparkPi for batch related tests
### _Why are the changes needed?_

Using more light-weight SparkPi example to prevent the GA OOM issue.

BTW, it should resolve #3114

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3119 from turboFei/batch_pi.

Closes #3119

58270f86 [Fei Wang] fix flaky test
fdf3bb23 [Fei Wang] use pi

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-07-27 10:31:18 +08:00
Fu Chen
0e91019794
[KYUUBI #3121] [CI] Fix GA oom issue
### _Why are the changes needed?_

This PR aims to fix the OOM issue that happened on Github Action

### _How was this patch tested?_

Pass CI.

Closes #3121 from cfmcgrady/fix-ga-oom.

Closes #3121

48138494 [Fu Chen] engine_name length
22b2323e [Fu Chen] address comment
ffed26ff [Fu Chen] enabled rest frontend
a6df03ea [Fu Chen] fix ga oom issue

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-25 16:58:26 +08:00
Cheng Pan
c5d29260a3
[KYUUBI #3037] Handles configuring the JUL -> SLF4J bridge
### _Why are the changes needed?_

Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3037 from pan3793/jul.

Closes #3037

fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-11 17:54:39 +08:00
zwangsheng
3bc299d01e
[KYUUBI #2868] [K8S] Add KubernetesApplicationOperation
### _Why are the changes needed?_

close #2868
Add Kubernetes Application Operation.
Help Tag Spark Driver with Engine Ref Id.

1) Find taged Engine:
`KyuubiApplicationManager => KubernetesApplicationOperation =(doesn't found taged pod)=> KubernetesApplicationOperation#JpsApplicationOperation`

2) `SparkBatchProcessBuilder` add logic to help tag Kubernetes Engine Tag.

3) Add Unit Test help to check `KubernetesApplicationOperation` find & kill  Spark Engine On Kubernetes with both client mode and cluster mode.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2869 from zwangsheng/feature/k8s-application-operation.

Closes #2868

04c97210 [zwangsheng] remove
fc151603 [zwangsheng] fix kyuubi on kubernetes integration test
29a3da52 [zwangsheng] fix
f43f0989 [zwangsheng] fix
2f2c9581 [zwangsheng] test
209dd65d [zwangsheng] fix jps
e12c681c [zwangsheng] fix tag
80971f98 [zwangsheng] fix
0760b63d [zwangsheng] fix
087eadb1 [zwangsheng] fix
d4989b09 [zwangsheng] try
93d83f81 [zwangsheng] add test
19378da4 [zwangsheng] fix style
63007261 [zwangsheng] fix test
fd4bf714 [zwangsheng] fix
66fc902b [zwangsheng] Using jps for client
2a1b797f [zwangsheng] fix setting
4827c459 [zwangsheng] fix
9ee90da6 [zwangsheng] decoupling spark
6b5831da [zwangsheng] useless driver label
f9ca898c [zwangsheng] fix
2add72fe [zwangsheng] fix setting
aaa081f8 [zwangsheng] fix style & dependency list
e102ac73 [zwangsheng] add kubernetesApplicationOperation

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-07-01 12:15:12 +08:00
zwangsheng
caa3ed2aac
[KYUUBI #2943][Bug][K8S] Remove Start Local Kyuubi Server For Kyuubi On K8S Test
### _Why are the changes needed?_

With #2943
close #2943

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2944 from zwangsheng/fix/remove_useless_server.

Closes #2943

6656d0ca [zwangsheng] fix test
faac8ff9 [zwangsheng] fix
57677ea5 [zwangsheng] remove

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-06-27 11:59:09 +08:00
Cheng Pan
2fd4e3a858
[KYUUBI #2827] [BUILD][TEST] Decouple integration tests from kyuubi-server
### _Why are the changes needed?_

This PR moves engine integration tests except to spark engine from `kyuubi-server` module to dedicated IT modules e.g. `kyuubi-hive-it`, then we can spilt different engine test into jobs to speed up CI tes

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

before
<img width="836" alt="image" src="https://user-images.githubusercontent.com/26535726/173265839-5d0f1ce6-e133-4d60-9a95-c95aa16f817f.png">

after
<img width="825" alt="image" src="https://user-images.githubusercontent.com/26535726/173265903-fb01e910-8f78-4091-91da-dca25be4357d.png">

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2827 from pan3793/hive-it.

Closes #2827

df3798f1 [Cheng Pan] Remove Java 11 test for Hive
e1645338 [Cheng Pan] nit
66524226 [Cheng Pan] nit
3aef2cef [Cheng Pan] review
500c9cff [Cheng Pan] MVN_OPT
3b4ad142 [Cheng Pan] nit
daa6e0ab [Cheng Pan] review
3383d4c1 [Cheng Pan] [BUILD][TEST] Decouple integration tests from kyuubi-server

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-06-18 00:01:34 +08:00
Fu Chen
0acf9717d0
[KYUUBI #2247] Change log4j2 properties to xml
### _Why are the changes needed?_

- change log4j2-test.properties to log4j2-test.xml
- add the unit test log4j2.xml for spark relative submodule, and remove the log4j.properties

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2850 from cfmcgrady/kyuubi-2247.

Closes #2247

a33d4d80 [Fu Chen] style
f99dadac [Fu Chen] fix style
49c99dea [Fu Chen] add log4j2.xml for spark relative submodule
a8a38561 [Fu Chen] change log4j2-test.properties to log4j2-test.xml

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-06-10 18:57:25 +08:00
Cheng Pan
8539a568f5
[KYUUBI #2822] [GA] Set log level to info
### _Why are the changes needed?_

Have seen run out of disk spaces in GitHub Action, set log level to info to reduce log size

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2822 from pan3793/log.

Closes #2822

b355452a [Cheng Pan] followup
541ca5fe [Cheng Pan] remove unsed filter
41f656ca [Cheng Pan] [GA] Set log level to info

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-06-07 22:33:41 +08:00
benjobs
c84ea87c81
[KYUUBI #2730] [WIP][KYUUBI #2238] Support Flink 1.15
### Why are the changes needed?

Flink 1.15.0 is out, Kyuubi needs to support it.

Please clarify why the changes are needed. For instance

1. new add flink 1.15 support
2. new add `flink-1.14` and `flink-1.15` maven profiles to support both Flink 1.14 & 1.15

flink 1.14  all functions and codes are kept unchanged without any impact

flink 1.15 support part of the work is still in progress, At present, 32 test cases have passed and 5 have failed, You can review, put forward suggestions for improvement, I will continue to improve.

### how to build with flink 1.14

```
mvn clean install -DskipTests -Pflink-1.14
```

### how to build with flink 1.15

```
mvn clean install -DskipTests -Pflink-1.15
```

### how to run test cases with flink 1.14

enable `flink-1.14` maven profile  and run the test cases

### how to run test cases with flink 1.15

enable `flink-1.15` maven profile  and run the test cases

### _How was this patch tested?_

- [x]  Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2730 from wolfboys/master.

Closes #2730

Closes #2238

ba9716f8 [benjobs] Improve check flink version
0616e74d [benjobs] [KYUUBI #2238] Support Flink 1.15

Authored-by: benjobs <benjobs@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-25 11:31:19 +08:00
hongdongdong
33c816240d [KYUUBI #2348] Add it test for trino engine
### _Why are the changes needed?_

Add it test for trino engine

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2638 from hddong/trino-it.

Closes #2348

cf6b8d5d [hongdongdong] Add IT tests for trino engine

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
2022-05-19 16:33:53 +08:00
odone
1ab68974b7
[KYUUBI #2471] Fix the bug of dynamically loading external packages
### _Why are the changes needed?_

In scala code mode implementation, we need to call `repl.addUrlsToClassPath` to set the classpath of the current thread to repl before entering `repl`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2475 from iodone/dev-1.

Closes #2471

21b92cc4 [odone] [KYUUBI #2471] add dynamic jar generator
3542786e [odone] [KYUUBI #2471] Fixed
3db2a7df [odone] [KYUUBI #2471] Fixed
65303683 [odone] [KYUUBI #2471] Fixed

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-05-12 16:57:02 +08:00
jiaoqingbo
6b6da1f4d0
[KYUUBI #2333][KYUUBI #2554] Configuring Flink Engine heap memory and java opts
### _Why are the changes needed?_

fix #2554

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2579 from jiaoqingbo/kyuubi2554.

Closes #2333

Closes #2554

f0365c91 [jiaoqingbo] code review
1700aab9 [jiaoqingbo] code review
1ca10a65 [jiaoqingbo] fix ut failed
b53dcdd4 [jiaoqingbo] code review
f9ceb72c [jiaoqingbo] [KYUUBI #2554] Configuring Flink Engine heap memory and java opts

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-10 12:31:01 +08:00
jiaoqingbo
3b04d9942a
[KYUUBI #2346] [Improvement] Simplify FlinkProcessBuilder with java executable
…xecutable

### _Why are the changes needed?_

fix #2346

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2418 from jiaoqingbo/kyuubi2346.

Closes #2346

eda5d383 [jiaoqingbo] after code review
720fede3 [jiaoqingbo] execute source config.sh and fix ut failed and delete debug log
1b53a9ba [jiaoqingbo] Merge branch 'master' into kyuubi2346
0c7eb5e7 [jiaoqingbo] add hadoop classpath to UT
ff2bb14c [jiaoqingbo] Merge branch 'master' into kyuubi2346
c9e8019d [jiaoqingbo] change ut
308ae5fd [jiaoqingbo] code review
f3eb068b [jiaoqingbo] fix ui failed
4e6d168d [jiaoqingbo] spoltless apply
9eae5576 [jiaoqingbo] fix ut failed
bc00e690 [jiaoqingbo] delete flink-sql-engine.sh
d6b87b9c [jiaoqingbo] delete childProcEnv
3aa8738a [jiaoqingbo] [KYUUBI #2346] [Improvement] Simplify FlinkProcessBuilder with java executable

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-29 18:23:34 +08:00
zwangsheng
7fa04947c4
[KYUUBI #2344] [Improvement] Add Kyuubi Server on Kubernetes with Spark Cluster mode integration test
### _Why are the changes needed?_

Add Kyuubi Server on Kubernetes with Spark Cluster mode integration test

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2409 from zwangsheng/improve/add-kyuubi-k8s-spark-cluster-test.

Closes #2344

7d20425d [zwangsheng] fix scala
bf2fafef [zwangsheng] fix scala
551aa7d7 [zwangsheng] fix java & class name
bd8a7a44 [zwangsheng] fix scala
42ba9fd0 [zwangsheng] fix scala
de4135b5 [zwangsheng] fix scala
511af18e [zwangsheng] try

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-21 18:15:24 +08:00
zwangsheng
62db92f72b
[KYUUBI #2381] [Test] Add Kyuubi on k8s With Spark on k8s client deploy-mode unit test
### _Why are the changes needed?_

Add Kyuubi on k8s With Spark on k8s client deploy-mode unit test

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2381 from zwangsheng/improve/add-kyuubi-k8s-spark-client.

Closes #2381

4f4d72f5 [zwangsheng] fix npe
623b3568 [zwangsheng] fix style
03ab628f [zwangsheng] add kyuubi on k8s spark client test

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-18 10:59:29 +08:00
zwangsheng
65a272f6e4
[KYUUBI #2359] [Test] Build WithKyuubiServerOnKuberntes
### _Why are the changes needed?_

Refactor KyuubiOnKubernetesTestsSuite
Build WithKyuubiServerOnKuberntes like WithKyuubiServerOnYarn

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2359 from zwangsheng/test/kyuubi_on_kubernetes.

Closes #2359

c1048c6a [zwangsheng] fix
81c98cd2 [zwangsheng] unused import
976f9091 [zwangsheng] fix
bad6720b [zwangsheng] fix
98d8995c [zwangsheng] refactor
628de557 [zwangsheng] style check
9dc6675a [zwangsheng] Build WithKyuubiServerOnKubernetes

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-15 14:16:54 +08:00
zwangsheng
a4c521c69b
[KYUUBI #2216] [Test] [K8s] Add Spark Cluster mode on Kubernetes integration test
### _Why are the changes needed?_

Close #2216

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2258 from zwangsheng/feature/add_cluster_k8s_test.

Closes #2216

7eea16b0 [zwangsheng] add linces
8effcf07 [Binjie Yang] Spark On K8s With Cluster
9cd03ad9 [ulysses-you] config
ce7143ee [zwangsheng] fix unused import
0a06521f [zwangsheng] fix
f9293925 [zwangsheng] test
88ec89f9 [zwangsheng] test
0f780f37 [zwangsheng] test
9b27bfa6 [zwangsheng] Test
df43cd76 [zwangsheng] Using Kerberos Submit Spark
99893a64 [zwangsheng] call beforall help set kerbores
0a732d94 [zwangsheng] Spark On K

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: ulysses-you <ulyssesyou18@gmail.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-12 21:54:04 +08:00
Kent Yao
19f1d41178
[KYUUBI #2287] Revamp Flink IT by random port and merge tests
### _Why are the changes needed?_

Flink IT shall use random-available port for kyuubi server
we shall merge ITs to prevent creating duplicated servers and engines

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2287 from yaooqinn/flink.

Closes #2287

a0acce20 [Kent Yao] Revamp Flink IT by random port and merge tests
ad5ae0f4 [Kent Yao] Revamp Flink IT by random port and merge tests

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-07 09:57:24 +08:00
ulysses-you
4f0323d5a9
[KYUUBI #2214] Add Spark Engine on Kubernetes integration test
### _Why are the changes needed?_

Rename `kyuubi-deployment-kubernetes-it` to `kyuubi-kubernetes-it` which include:
- Kyuubi Server on Kubernetes it
- Kyuubi Spark Engine on Kubernetes it

Now, we use the package name to distingush the scope of it:
- deployment package is for Kyuubi Server
- spark package is for Spark Engine

So it's easy to add Flink Engine on kubernetes it in future.

Note that, this PR only supports Spark client mode on Kubernetes it.

Since Spark only have two tags of docker image https://hub.docker.com/r/apache/spark/tags, this PR only test 3.2.1 version

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2214 from ulysses-you/spark-on-k8s-client.

Closes #2214

f6dda432 [ulysses-you] Add Spark Engine on Kubernetes integration test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-25 15:53:12 +08:00
Ada
c09cd654bb
[KYUUBI #2203][FLINK] Support flink conf set by kyuubi conf file
### _Why are the changes needed?_

Support setting Flink configuration by setting `kyuubi-default.conf`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2206 from deadwind4/KYUUBI-2203.

Closes #2203

eeeb91cd [Ada] fix IT case
5eefba57 [Ada] add IT case
86fc57e1 [Ada] [KYUUBI #2203][engine/flink] Support flink conf set by kyuubi conf file

Authored-by: Ada <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-24 23:35:02 +08:00
Cheng Pan
f8efcb7125
[KYUUBI #2103] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_

This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093

We need more time to evaluate it and add more test cases to ensure stability before the next release.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2103 from pan3793/revert.

Closes #2103

Closes #1948

575dfe8d [Cheng Pan] properties order
4c94481e [Cheng Pan] fix deps
bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test
bad4e90e [Cheng Pan] surefire
f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2"
cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf"
52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes"

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-12 21:05:39 +08:00
Kent Yao
abf2026816
[KYUUBI #2045] Preparing v1.6.0-SNAPSHOT
### _Why are the changes needed?_

Preparing v1.6.0-SNAPSHOT with branch-1.5 cut

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2045 from yaooqinn/pv.

Closes #2045

6a506a8f [Kent Yao] Preparing v1.6.0-SNAPSHOT

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-06 22:21:45 +08:00
Cheng Pan
7887a47e02
[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
### _Why are the changes needed?_

[THRIFT-4805](https://issues.apache.org/jira/browse/THRIFT-4805) has been fixed in thrift 0.13.0, we can remove the workaround of log4j2 configurations because we use thrift 0.16.0 now, the change also removes the following warnings.

```
2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
```

`maven-surefire-plugin` is only used in `kyuubi-hive-jdbc`, we should disable it in other modules.

```
[INFO] --- maven-surefire-plugin:2.22.0:test (default-test)  kyuubi-ctl_2.12 ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
[INFO] Running org.apache.kyuubi.ctl.ServiceControlCliSuite
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in org.apache.kyuubi.ctl.ServiceControlCliSuite
[INFO] Running org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1996 from pan3793/log.

Closes #1996

291441c0 [Cheng Pan] nit
f2c94246 [Cheng Pan] Tune surefire plugin in conf
e1ee6402 [Cheng Pan] Clean up log4j2 conf

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-03 12:58:38 +08:00
Kent Yao
f25e5c93b8
[KYUUBI #1949] [INFA] Add homepage for every module in maven central
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Take spark as an example.

![image](https://user-images.githubusercontent.com/8326978/154938094-87e53b38-5264-4aca-a7e2-269f9bb3c2d1.png)

[link](https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.2.1)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1949 from yaooqinn/url.

Closes #1949

f88c0598 [Kent Yao] [INFA] Add homepage for every module in maven cental
684f588f [Kent Yao] [INFA] Add homepage for every module in maven cental

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-21 21:09:29 +08:00