Commit Graph

349 Commits

Author SHA1 Message Date
Min Zhao
26d52faa7a
[KYUUBI #2360] [Subtask] Configuring Hive engine heap memory and java opts
### _Why are the changes needed?_

Configuring Hive engine heap memory and java opts

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2371 from zhaomin1423/2360.

Closes #2360

fc372571 [Min Zhao] [KYUUBI #2360] [Subtask] Configuring Hive engine heap memory and java opts
52547d9c [Min Zhao] add unit tests

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-17 20:31:45 +08:00
Fei Wang
4b42c735f7 [KYUUBI #2308][SUB-TASK][KPIP-4] Batch job configuration ignore list and pre-defined configuration in server-side
### _Why are the changes needed?_

To close #2308

Support to ignore some batch configuration items and pre-define some configuration in server side.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2386 from turboFei/KYUUBI_2308_kpip4_batch_conf.

Closes #2308

2e517ec4 [Fei Wang] [KYUUBI #2308][SUB-TASK][KPIP-4] Batch job configuration ignore list and server predefined configurations

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-04-17 01:27:27 +08:00
Min Zhao
1a58aaf78c
[KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
### _Why are the changes needed?_

update developer.md to describe what append descriptions of new configurations to settings.md

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2372 from zhaomin1423/2369.

Closes #2369

a044c156 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
56f878a9 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
581894b7 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-16 22:33:25 +08:00
Min Zhao
1a68b866ce
[KYUUBI #2361] [Improvement] Configuring Trino Engine heap memory and java opts
### _Why are the changes needed?_

Configuring Trino Engine heap memory and java opts

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2387 from zhaomin1423/2361.

Closes #2361

e8ef7f6f [Min Zhao] [KYUUBI #2361] [Improvement] Configuring Trino Engine heap memory and java opts

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-16 22:03:17 +08:00
Min Zhao
7c6789a674
[KYUUBI #2338] [DOCS] Upgrade sphinx dependencies for documentation build
### _Why are the changes needed?_

Upgrade sphinx dependencies for documentation build

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2388 from zhaomin1423/2338.

Closes #2338

d68d682e [Min Zhao] [KYUUBI #2338] [DOCS] Upgrade sphinx dependencies for documentation build

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-16 21:59:13 +08:00
Fei Wang
b6d5c64cb4 [KYUUBI #2353] [SUB-TASK][KPIP-4] Implement BatchJobSubmission operation and basic KyuubiBatchSessionImpl
### _Why are the changes needed?_

To close #2306 and close #2307

In this PR, I implement BatchJobSubmission operation and  introduce basic `KyuubiBatchSessionImpl`.
TODO:
- Normalize/validate the batch request
  -  batch request fields
  - merge with server pre-defined batch conf

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2353 from turboFei/KPIP_4_batch_submission_op.

Closes #2353

9bc6050c [Fei Wang] remove unused conf
ef8e962c [Fei Wang] check application in current thread
8738660b [Fei Wang] dedup code
27f22008 [Fei Wang] use static secret id instead of conf
6794ff7a [Fei Wang] Use Seq instead of java.util.List
2f4f9b15 [Fei Wang] Remove BatchType enumaration
7d380800 [Fei Wang] remove dead code
a94a9e6a [Fei Wang] remove jars,files fileds on BatchRequest
6021a1e7 [Fei Wang] add ut for result set
07a939c9 [Fei Wang] refactor long line
a918a496 [Fei Wang] address comments
73229e70 [Fei Wang] set engine max life time
bbe3f1f4 [Fei Wang] unique the application cehcker thread pool
9643e42c [Fei Wang] refactor
11dd71f7 [Fei Wang] add KyuubiBatchYarnClusterSuite
12169910 [Fei Wang] add ut for batch session
47da8c1a [Fei Wang] add open batch session api
6dcf60d9 [Fei Wang] add ut for static batch secret id
a212e62b [Fei Wang] [SUB-TASK][KPIP-4] Implement BatchJobSubmission operation and basic KyuubiBatchSessionImpl

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-04-15 16:42:36 +08:00
Ada Wang
d0a4697f90
[KYUUBI #2248][DOCS] Add a flink on yarn kerberos doc
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2363 from deadwind4/flink-krb-doc.

Closes #2248

50f191fa [Ada Wang] [KYUUBI #2248][DOCS] Add a flink on yarn kerberos doc

Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-14 16:44:20 +08:00
zwangsheng
659d981c66
[KYUUBI #2337] [DOCS] Access Kyuubi with Kyuubi JDBC Driver
### _Why are the changes needed?_

Access Kyuubi with Kyuubi-Hive-JDBC Documentation is missing
Close #2337

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2339 from zwangsheng/ISSUE/#2337.

Closes #2337

d4154e6d [zwangsheng] fix mvn url
510708ce [zwangsheng] fix
c4bb9366 [zwangsheng] KYUUBI-2337

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-13 18:39:29 +08:00
ulysses-you
12416a7831
[KYUUBI #2345] [DOC] Hot Upgrade Kyuubi Server
### _Why are the changes needed?_

Help user upgrade easily.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2345 from ulysses-you/hot-upgrade.

Closes #2345

a5898711 [ulysses-you] typo
84bdf4bc [ulysses-you] address comment
e32fd3dc [ulysses-you] enrich
3c92545c [ulysses-you] hot-upgrade

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-13 18:06:51 +08:00
Tianlin Liao
4bc14657bf [KYUUBI #2250] Support to limit the spark engine max running time
### _Why are the changes needed?_

To close #2250

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2295 from lightning-L/kyuubi-2250.

Closes #2250

71851bf8 [Tianlin Liao] [KYUUBI #2250] limit the spark engine max running time
7314df84 [Tianlin Liao] [KYUUBI #2250] add method to shutdown threadpool executor in ThreadUtils

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-04-08 22:48:39 +08:00
ulysses-you
82a024a983
[KYUUBI #2292] Unify spark extension location
### _Why are the changes needed?_

Move `dev/kyuubi-spark-extension*` to `extension/spark/`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2291 from ulysses-you/unify-spark-extension.

Closes #2292

4e6f4ca6 [ulysses-you] fix
c2cc018d [ulysses-you] unify
2375af0d [ulysses-you] unify

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-07 13:58:58 +08:00
Kent Yao
0ed865f704
[KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
### _Why are the changes needed?_

In this PR, we narrow the scope of 8f15622d9b to support arbitrary parameters which configured in kyuubi-defaults.conf only.

We shall avoid propagating all `sys.props` of Kyuubi server to engine side, which is a dangerous behavior

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2283 from yaooqinn/2084.

Closes #2084

434db31c [Kent Yao] fix flink
bdd983d4 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
78bada0b [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
55e6fdf1 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
b2239bc7 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-07 12:16:41 +08:00
jiaoqingbo
c1a68a7cf3
[KYUUBI #2266] The default value of frontend.connection.url.use.hostname should be set to true to be consistent with previous versions
### _Why are the changes needed?_

fix #2266
in 1.5.0 version,the seriviceUri consists of ip:port
<img width="576" alt="e3c7a03dbcbeeaedce52139755fe227" src="https://user-images.githubusercontent.com/14961757/161362120-61e45f93-d4fe-491b-ab06-9f02e78e70f3.png">
in 1.4.1 version ,the seriviceUri  consists of hostname:port

<img width="591" alt="88cf334a0a425227ed6be7637fc10bd" src="https://user-images.githubusercontent.com/14961757/161364285-d8841387-6393-4280-9565-7d3ce419aa81.png">
using following command to connect would failed
./bin/beeline -u 'jdbc:hive2://ocdp63.asiainfo.com:2008,ocdp172.asiainfo.com:2008,ocdnfo.com:2008/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=spark/_HOSTocdp;'

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2269 from jiaoqingbo/2266.

Closes #2266

b77b19d8 [jiaoqingbo] modify doc
e87d9670 [jiaoqingbo] [KYUUBI #2266] The default value of frontend.connection.url.use.hostname should be set to true to be consistent with previous versions
25063a34 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
0ad3b764 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
f9d68c7f [jiaoqingbo] [KYUUBI #2244] load-kyuubi-env.sh should print SPARK_ENGINE_HOME for consistency

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-02 21:10:05 +08:00
hongdongdong
a18a3a956c
[KYUUBI #2275][DOCS] Fix missing prefix in trino engine quick start
### _Why are the changes needed?_

Fix missing prefix in trino engine quick start

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2276 from hddong/trino-quick-start-fix.

Closes #2275

156ce2ef [hongdongdong] [KYUUBI #2275][DOCS] Fix missing prefix in trino engine quick start

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-02 17:48:54 +08:00
hongdongdong
0ee5371081
[KYUUBI #2243][DOCS] Add quick start for trino engine
### _Why are the changes needed?_

Add quick start for trino engine

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2265 from hddong/trino-quick-start.

Closes #2243

fc4c2743 [hongdongdong] [KYUUBI #2243] Add quick start for trino engine

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-01 22:55:35 +08:00
Ada Wang
5733452b29 [KYUUBI #2209] Add detail usage documents of Flink engine
### _Why are the changes needed?_

Add flink usage detail document to show how to use in standalone and how to set flink option etc.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2215 from deadwind4/KYUUBI-2209.

Closes #2209

408c332a [Ada Wang] delete flink basics.md
246e550a [Ada Wang] update flink doc links
8eacc137 [Ada Wang] fix docs
5b7911f6 [Ada] [KYUUBI #2209][FLINK][DOCS] Add detail usage

Lead-authored-by: Ada Wang <wang4luning@gmail.com>
Co-authored-by: Ada <wang4luning@gmail.com>
Signed-off-by: KenjiFujima <thanosxnicholas@gmail.com>
2022-03-31 23:47:47 +08:00
Tianlin Liao
e16c728dc4
[KYUUBI #1021] Expire CredentialsRef in a proper time to reduce memor…
…y usage

### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2210 from lightning-L/kyuubi-1021.

Closes #1021

6a2654ab [Tianlin Liao] [KYUUBI #1021] Expire CredentialsRef in a proper time to reduce memory usage

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-29 18:34:59 +08:00
KenjiFujima
f80d1a8be3
[KYUUBI #2241] Remove unused deployment documents of Spark
### _Why are the changes needed?_

Remove unused deployment documents of Spark.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2242 from KenjiFujima/KYUUBI-2241.

Closes #2241

b9264590 [KenjiFujima] [KYUUBI #2241] Remove unused deployment documents of Spark

Authored-by: KenjiFujima <thanosxnicholas@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-29 12:30:43 +08:00
Fei Wang
9797ff0d3a
[KYUUBI #2225] Support to set result max rows for spark engine
### _Why are the changes needed?_

Support to set result max rows for spark engine.
And it will only take affect if incremental mode is not enabled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2225 from turboFei/max_res.

Closes #2225

bb2acd8b [Fei Wang] nit
5d827d11 [Fei Wang] address comments
3193d22e [Fei Wang] address comments
51983c35 [Fei Wang] address comments
2904b4ce [Fei Wang] address comments
f2a8aa62 [Fei Wang] fix docs
232f975c [Fei Wang] add ut
508b3560 [Fei Wang] Add engine spark max rows

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-28 10:07:27 +08:00
Fei Wang
e8445b7f28 [KYUUBI #2102] Support to retry the internal thrift request call and add engine liveness probe to enable fast fail before retry
### _Why are the changes needed?_

To close #2102

Support to retry all the internal thrift request calls(except RenewDelegationToken now), and fast fail if the remote engine is not stable or not alive.

In this PR, it supports engine liveness probe.

If it is enabled, a companion thrift client will be created and open a liveness probe session when opening remote engine session.
It will send some simple thrift request(GetInfo) to check whether the remote engine is alive, and fast fail before retry if remote engine is not connectable.

#### Why not use the same thrift client to check engine liveness before retry?
I tried that, but met `out of resp sequence` error.
For example:
1. send getOperationStatus request
2.  read time out
3. send GetInfoType request
4. receive getOperationStatus response (out of resp sequence)

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2122 from turboFei/retry_rpc.

Closes #2102

3926ba04 [Fei Wang] adress comments
ade4ede6 [Fei Wang] add timeout
1b7a64f9 [Fei Wang] Only check remote engine alive before retry
98e03f8e [Fei Wang] refactor
fac388cf [Fei Wang] remove unused import
9c6d8737 [Fei Wang] add ut
9b595650 [Fei Wang] Support to retry the thrift request and engine alive probe

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-03-26 00:23:44 +08:00
sychen
e7c4201226
[KYUUBI #2119][FOLLOWUP] Support output progress bar in Spark engine
### _Why are the changes needed?_

Based on this https://github.com/apache/incubator-kyuubi/pull/2162
Add several configuration items to control the output of the progress bar.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2168 from cxzl25/KYUUBI-2119-FOLLOWUP.

Closes #2119

36db8c5f [sychen] Add configuration progress bar

Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-21 11:01:42 +08:00
Ada Wong
3c2463b107
[KYUUBI #2172][BUILD] Bump Flink 1.14.4 version
### _Why are the changes needed?_

Bump Flink 1.14.4 version.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2173 from deadwind4/flink-1.14.4.

Closes #2172

1be8ed91 [Ada Wong] [KYUUBI #2172][BUILD] Bump Flink 1.14.4 version

Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-19 18:42:46 +08:00
Min Zhao
2911ac26c7
[KYUUBI #2024] Hive Backend Engine - ProcBuilder for HiveEngine
### _Why are the changes needed?_

Add ProcBuilder for HiveEngine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2146 from zhaomin1423/hive_process_builder.

Closes #2024

1baf1397 [Min Zhao] [KYUUBI #2024] Hive Backend Engine - ProcBuilder for HiveEngine
23fdc3b7 [Min Zhao] [KYUUBI #2024] Hive Backend Engine - ProcBuilder for HiveEngine

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-19 14:56:57 +08:00
jiaoqingbo
62f685ff69
[KYUUBI #2156][FOLLOWUP] Fix configuration format in document
### _Why are the changes needed?_

fix #2157 cofiguration format error

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2165 from jiaoqingbo/2156-followup.

Closes #2156

9862ddb2 [jiaoqingbo] [KYUUBI #2156][FOLLOWUP] Fix configuration format in document
47157c5c [jiaoqingbo] Merge branch 'master' into 2156-followup
9c89c081 [jiaoqingbo] [KYUUBI #2156][FOLLOWUP]fix cofiguration format
46084328 [jiaoqingbo] Merge branch 'master' into 2156
96a22e59 [jiaoqingbo] add link to doc
dc11a214 [jiaoqingbo] [KYUUBI #2156] Change log to reflect exactly why getting token failed

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-18 16:44:24 +08:00
jiaoqingbo
31be7a30e8
[KYUUBI #2156] Change log to reflect exactly why getting token failed
### _Why are the changes needed?_

fix #2156

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2157 from jiaoqingbo/2156.

Closes #2156

96a22e59 [jiaoqingbo] add link to doc
dc11a214 [jiaoqingbo] [KYUUBI #2156] Change log to reflect exactly why getting token failed

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-17 19:12:54 +08:00
Ada Wong
36507f8efc
[KYUUBI #2148][DOCS] Add dev/reformat usage
### _Why are the changes needed?_

Add a `dev/reformat` usage in `developer.md`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2149 from deadwind4/reformat-doc.

Closes #2148

0457d879 [Ada Wong] fix typo
67af7e4b [Ada Wong] [KYUUBI #2148][DOCS] Add dev/reformat usage

Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-16 22:10:39 +08:00
liuzy04
95fc6da90f
[KYUUBI #2150] [DOCS] Fix Getting Started With Kyuubi on Kubernetes
### _Why are the changes needed?_

helm安装kyuubi服务完成后会提示:
```script
NAME: kyuubi-helm
LAST DEPLOYED: Wed Mar 16 13:14:32 2022
NAMESPACE: kyuubi
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
......

Get kyuubi expose URL by running these commands:
  **export NODE_PORT=$(kubectl get --namespace kyuubi -o jsonpath="{.spec.ports[0].nodePort}" services kyuubi-helm-nodeport)**
  export NODE_IP=$(kubectl get nodes --namespace kyuubi -o jsonpath="{.items[0].status.addresses[0].address}")
  echo $NODE_IP:$NODE_PORT
```

使用命令
```
export NODE_PORT=$(kubectl get --namespace kyuubi -o jsonpath="{.spec.ports[0].nodePort}" services kyuubi-helm-nodeport)
```
会得到一个报错:
```
Error from server (NotFound): services "kyuubi-helm-nodeport" not found
```
使用`kubectl get svc -A`,查询到namespace:kyuubi下只有一个service:"kyuubi-svc",运行
```
export NODE_PORT=$(kubectl get --namespace kyuubi -o jsonpath="{.spec.ports[0].nodePort}" services kyuubi-svc)
```
成功。so这里给的serviceName是否是错的?

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2150 from HYBG-1126/HYBG-1126.

Closes #2150

ea152f33 [liuzy04] helm安装kyuubi服务完成后会提示: NAME: kyuubi-helm LAST DEPLOYED: Wed Mar 16 13:14:32 2022 NAMESPACE: kyuubi STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: # # Licensed to the Apache Software Foundation (ASF) under one or more ......

Authored-by: liuzy04 <liuzy04@mail.jj.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-16 15:37:13 +08:00
Ada Wong
2225b9a175
[KYUUBI #2143] [KYUBBI #2142][DOCS] Add IDEA setup guide
### _Why are the changes needed?_

Add IDEA setup guide for developers.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2143 from deadwind4/KYUUBI-2142.

Closes #2143

f0450c05 [Ada Wong] Update formatter for Java
be77cfe5 [Ada Wong] [KYUBBI #2142][DOCS] Add IDEA setup guide

Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-16 14:43:21 +08:00
Tianlin Liao
eb4d289068
[KYUUBI #1936][FOLLOWUP] Send credentials when opening session and wait for completion
### _Why are the changes needed?_

Follow up #1936

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2092 from lightning-L/kyuubi-1936.

Closes #1936

2bb2c10d [Tianlin Liao] [KYUUBI #1936][FOLLOWUP] Send credentials when opening session and wait for completion

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-16 09:32:53 +08:00
jiaoqingbo
8f15622d9b
[KYUUBI #2084] Support arbitrary parameters for KyuubiConf
### _Why are the changes needed?_

fix #2084

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2091 from jiaoqingbo/2084.

Closes #2084

624216fa [Cheng Pan] permission
c45a71ce [jiaoqingbo] delete empty line
29987bf1 [jiaoqingbo] fix ut failed
2e7f0acc [jiaoqingbo] Merge branch 'master' into 2084
04c0f8e1 [jiaoqingbo] Merge branch 'master' into 2084
1267d195 [jiaoqingbo] Merge branch '2084' of https://github.com/jiaoqingbo/incubator-kyuubi into 2084
1bb2b6d6 [jiaoqingbo] Merge branch 'master' into 2084
db4a4aac [jiaoqingbo] Merge branch 'master' into 2084
82097388 [jiaoqingbo] ignore java/sun prefix
d4640ec3 [jiaoqingbo] add  log
d5af758f [jiaoqingbo] modify ut
0f3af6c6 [jiaoqingbo] modify ut case
9ffb4177 [jiaoqingbo] Merge branch 'master' into 2084
52dd5090 [jiaoqingbo] modify ut
3f881748 [jiaoqingbo] modify test conf
705f94af [jiaoqingbo] modify test conf
096fba1d [jiaoqingbo] modify test conf
83f1aba9 [jiaoqingbo] add more example configuration
d2581534 [jiaoqingbo] fix compile problem
65c3198a [jiaoqingbo] [KYUUBI #2084] Support arbitrary parameters for KyuubiConf

Lead-authored-by: jiaoqingbo <1178404354@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-15 09:37:16 +08:00
SteNicholas
a83cd49e1b
[KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
### _Why are the changes needed?_

 Add `Deploy Kyuubi Flink engine on Yarn`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2131 from SteNicholas/KYUUBI-1866.

Closes #1866

ba639f15 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn
cc6f4d44 [SteNicholas] [KYUUBI #1866][FOLLOWUP] Add Deploy Kyuubi Flink engine on Yarn

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-14 14:45:44 +08:00
Cheng Pan
f8efcb7125
[KYUUBI #2103] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_

This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093

We need more time to evaluate it and add more test cases to ensure stability before the next release.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2103 from pan3793/revert.

Closes #2103

Closes #1948

575dfe8d [Cheng Pan] properties order
4c94481e [Cheng Pan] fix deps
bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test
bad4e90e [Cheng Pan] surefire
f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2"
cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf"
52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes"

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-12 21:05:39 +08:00
SteNicholas
b8389dae40
[KYUUBI #1866][FOLLOWUP] Add logging of Flink SQL Engine
### _Why are the changes needed?_

Add logging of Flink SQL Engine in `logging.md`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2111 from SteNicholas/KYUUBI-1866.

Closes #1866

7a0bdbb8 [SteNicholas] [KYUUBI #1866] Add logging of Flink SQL Engine

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-12 20:44:29 +08:00
Ada Wong
8f7b2c6640
[KYUUBI #1866][DOCS] Add flink sql engine quick start
### _Why are the changes needed?_

Add quick start documents of the Flink SQL Engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2106 from deadwind4/KYUUBI-1866-quickstart.

Closes #1866

2533aafd [Ada Wong] remove Yarn section
6aa4db8a [Ada Wong] compress png
ff6bff72 [Ada Wong] [KYUUBI #1866][DOCS] Add flink sql engine quick start

Authored-by: Ada Wong <rsl4@foxmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-12 17:07:35 +08:00
Min Zhao
b7a5cfcf78
[KYUUBI #2108] Add description about trino in the config of engine.type
### _Why are the changes needed?_

Add description about trino in the config of engine.type.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2109 from zhaomin1423/fix_engine_type_doc.

Closes #2108

adf898c0 [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type
8f11514e [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-12 16:32:08 +08:00
SteNicholas
2486c5dff3
[KYUUBI #2089] Add debugging instructions for Flink engine
### _Why are the changes needed?_

Add debugging instructions for Flink engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2098 from SteNicholas/KYUUBI-2089.

Closes #2089

08189b85 [SteNicholas] [KYUUBI #2089] Add debugging instructions for flink engine

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-11 10:21:20 +08:00
Fei Wang
1e23e7a93c [KYUUBI #1978] Support NEGOTIATE/BASIC authorization for restful frontend service
### _Why are the changes needed?_

to close #1978

Support both NEGOTIATE and BASIC authentication for restful frontend service.

At first, I involve two auth schemes, FYI [hadoop/HttpConstants.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FHttpConstants.java)
- BASIC: mapping to NOSASL, NONE, LDAP, CUSTOM authentication
- NEGOTIATE: mapping to KERBEROS authentication

BTW, hadoop also supports `Digest` auth scheme.

Two authentication handlers:
- BasicAuthenticationHandler(reuse existing passwdAuthenticationProvider(LDAP/CUSTOM))
- KerberosAuthenticationHandler(refer [hadoop/KerberosAuthenticationHandler.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Fserver%2FKerberosAuthenticationHandler.java) and [hadoop/KerberosUtil.java](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project%2Fhadoop-auth%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fsecurity%2Fauthentication%2Futil%2FKerberosUtil.java))

#### AuthenticationFilter
- redirect the http request according to the authentication scheme specified in `Authorization` header.

For basic authentication
The value of Authorization header is `BASIC ${encodeBase64(user + ":" + password)}`.
Then using the passwdAuthenticationProvider to authenticate `user` and `password`.

For SPNEGO(kerberos) authentication.
The value of Authorization header is `NEGOTIATE ${encodeBase64(clientToken)}`.
Then checking whether the clientToken is valid.

And currently, the AuthenticationFilter take affect for all the requests with pathSpec `/api/*`.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2049 from turboFei/kyuubi_1978_kerberos.

Closes #1978

c8b6362b [Fei Wang] refactor
cedec70f [Fei Wang] add spnego unit test
c1d45cde [Fei Wang] rename package to http.authentication
3d0b220b [Fei Wang] address comments
296f181e [Fei Wang] fix ut
f9371e14 [Fei Wang] spnego and basic auth

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-03-10 21:04:20 +08:00
SteNicholas
a71c511ba3
[KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
### _Why are the changes needed?_

YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2082 from SteNicholas/KYUUBI-2081.

Closes #2081

c6f76c5c [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR
fee5c21f [SteNicholas] [KYUUBI #2081] YARN_CONF_DIR shall be added to kyuubi server classpath as HADOOP_CONF_DIR

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-10 15:03:11 +08:00
yanghua
a882c4bf4e
[KYUUBI #2079] Update kyuubi layer source file to add flink and trino…
… engine

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2080 from yanghua/KYUUBI-2079.

Closes #2079

2bf98720 [yanghua] [KYUUBI #2079] Update kyuubi layer source file to add flink and trino engine

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-10 09:39:15 +08:00
Cheng Pan
54dfb4bbc0
[KYUUBI #1215][DOC] Document incremental collection
### _Why are the changes needed?_

Document incremental collection, close #1215

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

<img width="1914" alt="1" src="https://user-images.githubusercontent.com/26535726/157191008-451bee2a-eb6b-4bb6-869b-5b0f75a21448.png">
<img width="1919" alt="2" src="https://user-images.githubusercontent.com/26535726/157191016-e183bbf5-aa4a-496d-a250-5d14cf04101d.png">
<img width="1920" alt="3" src="https://user-images.githubusercontent.com/26535726/157191026-343a39d7-0ab8-4886-9a51-3670394ef6be.png">

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2057 from pan3793/doc.

Closes #1215

82677a03 [Cheng Pan] grammar
3be0e3f1 [Cheng Pan] fix
b467e975 [Cheng Pan] Update
d256cbaf [Cheng Pan] compress picture
b3c5fb64 [Cheng Pan] Fix
4c06307c [Cheng Pan] [KYUUBI #1215][DOC] Document incremental collection

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-08 20:49:51 +08:00
HYBG-1126
85d515dbd2
[KYUUBI #1999] Update quick_start_with_helm.md
cretate ns kyuubi ---> create ns kyuubi

### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1999 from HYBG-1126/patch-1.

Closes #1999

f68f73e9 [HYBG] Update quick_start_with_helm.md

Lead-authored-by: HYBG-1126 <hybg1126@outlook.com>
Co-authored-by: HYBG <44338121+HYBG-1126@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-03 23:29:27 +08:00
Kent Yao
114fefefa6
[KYUUBI #2004] Sync contents for CONTRIBUTING & COMMUNITY between web and main repo
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add CONTRIBUTING.md to list COMMUNITY resources that I contributor can use
delete contributions and improvement-proposal in main repo

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2004 from yaooqinn/CONTRIBUTING.

Closes #2004

ddbcd521 [Kent Yao] CONTRIBUTING
f81c14b3 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into CONTRIBUTING
3bdbdc9b [Kent Yao] CONTRIBUTING
574ceff2 [Kent Yao] CONTRIBUTING
f0a6b4c4 [Kent Yao] CONTRIBUTING
c55c6a02 [Kent Yao] CONTRIBUTING

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-03-03 20:40:33 +08:00
Cheng Pan
7887a47e02
[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
### _Why are the changes needed?_

[THRIFT-4805](https://issues.apache.org/jira/browse/THRIFT-4805) has been fixed in thrift 0.13.0, we can remove the workaround of log4j2 configurations because we use thrift 0.16.0 now, the change also removes the following warnings.

```
2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
```

`maven-surefire-plugin` is only used in `kyuubi-hive-jdbc`, we should disable it in other modules.

```
[INFO] --- maven-surefire-plugin:2.22.0:test (default-test)  kyuubi-ctl_2.12 ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch"
[INFO] Running org.apache.kyuubi.ctl.ServiceControlCliSuite
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in org.apache.kyuubi.ctl.ServiceControlCliSuite
[INFO] Running org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1996 from pan3793/log.

Closes #1996

291441c0 [Cheng Pan] nit
f2c94246 [Cheng Pan] Tune surefire plugin in conf
e1ee6402 [Cheng Pan] Clean up log4j2 conf

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-03 12:58:38 +08:00
SteNicholas
6d757a3df9
[KYUUBI #1963] Make plan only mode skippable for configuable plans
### _Why are the changes needed?_

Make plan only mode skippable for configuable plans.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1967 from SteNicholas/KYUUBI-1963.

Closes #1963

e253aec6 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans
b5cff405 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-24 12:46:34 +08:00
Kent Yao
644b917a30
[KYUUBI #1970] Replace mirror-cn profile with Apache officially suggested dlcdn
### _Why are the changes needed?_

Replace `mirror-cn` profile with Apache officially suggested dlcdn.

mirror-cn is only suite for CN, while dlcdn is much wiser.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1970 from yaooqinn/cdn.

Closes #1970

a325f828 [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn
a0756b4f [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-24 12:07:46 +08:00
SteNicholas
770499ca54
[KYUUBI #1948] Upgrade thrift version to 0.16.0
### _Why are the changes needed?_

Upgrade libthrift to 0.16.0 due to [CVE-2020-13949](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949) and the coming upstream change of Spark https://github.com/apache/spark/pull/34362

### _What changed in this PR?_

- Upgrade libthrift to 0.16.0

- Shade and relocate `thrift` and `hive-service-rpc` classes in `kyuubi-spark-engine`, it's necessary to avoide conflicting with old thrift libs bundled in Spark binary releases.

- Due to thrift change the method signature, the subclasses those interfaces in Kyuubi also need to modify to pass compile.
We rely on Hive 2.3.9 jars in some components, e.g. `kyuubi-hive-jdbc`, `LocalMetaServer` in `kyuubi-server` test classes.

Some classes in Hive jars compiled against old thrift interfaces which are not compatible with thrift 0.16.0, it causes runtime link error, we found the following classes which breaks the test and copied them with necessary modification to make it work with thrift 0.16.0.

    - `TFramedTransport`
    - `TFilterTransport`
    - `TUGIAssumingTransport`
    - `TUGIContainingTransport`

- Next Steps, I think it's worth to do in separated PRs.

    - Recover the `HiveDelegationTokenProviderSuite`, one approach is use an isolate classloader to load HMS classes and thrift 0.9.3 classes from Maven, this approach can also be used for the planed Zoopkeeper upgrading to help us verficating the compatibility of Zookeeper Server 3.4.x.
    - Rewrite `kyuubi-hive-jdbc` to make it decouple with Hive jars, because there maybe other places which may not work with thrift 0.16.0 but the UTs does not cover.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1953 from SteNicholas/KYUUBI-1948.

Closes #1948

de5d1ea2 [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0
898effcd [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0
803e270c [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-23 20:51:59 +08:00
Paul Lin
e54d431024
[KYUUBI #1883] Support max result rows for Flink queries
### _Why are the changes needed?_

Currently, Flink engine would pull all result rows into memory before returning it to the client. This would be problematic for large result sets and infinite result sets.

This is a sub-task of KPIP-2 https://github.com/apache/incubator-kyuubi/issues/1322.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1938 from link3280/feature/FLINK-1883.

Closes #1883

80020cee [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala
1b958221 [Paul Lin] [KYUUBI #1883] Avoid allocating too much buffer space
5be7535c [Paul Lin] [KYUUBI #1883] Support max result rows for Flink queries

Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-22 16:49:37 +08:00
Wang Zhen
1d64bb279e
[KYUUBI #1796] Destroy the process after the engine starts
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Destroy the process after the engine starts. #1796

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate
add conf: `kyuubi.session.engine.startup.waitCompletion  false`
![image](https://user-images.githubusercontent.com/17894939/150467332-6d30a316-494a-4d1a-8c79-a7c9479b16cb.png)

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1815 from wForget/KYUUBI-1796.

Closes #1796

12aea19a [Wang Zhen] fix
27cb602a [Wang Zhen] fix
e70e7071 [Wang Zhen] add log
ecaf828c [Wang Zhen] regenerate settings.md
a5f6c5de [Wang Zhen] fix desc
09812beb [Wang Zhen] Merge remote-tracking branch 'origin/master' into KYUUBI-1796
27fab5eb [Wang Zhen] [KYUUBI-1796] Destroy the process after the engine starts

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-02-18 18:55:42 +08:00
SteNicholas
6c6238c2ae
[KYUUBI #1919] Add more enum values for OperationModes
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add more enum values for `OperationModes`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1933 from SteNicholas/KYUUBI-1919.

Closes #1919

adcc7f5e [SteNicholas] [KYUUBI #1919] Add more enum values for OperationModes

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-02-18 15:48:15 +08:00
zhenjiaguo
2501f9ad83
[KYUUBI #1914] Update metric doc
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Update metric doc when metric change.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![kyuubi-metric](https://user-images.githubusercontent.com/29809822/154208648-7c8741ab-4dd1-4132-b716-ac093d7ef90f.png)

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1916 from zhenjiaguo/update_metric_doc.

Closes #1914

e8ea7f6c [zhenjiaguo] update metric doc
a5d730ce [zhenjiaguo] update metric doc

Authored-by: zhenjiaguo <zhenjiaguo@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-02-17 10:38:54 +08:00