Commit Graph

234 Commits

Author SHA1 Message Date
zwangsheng
9827c60aa4
[KYUUBI #3663] [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
### _Why are the changes needed?_

Current kyuubi build kubernetes client for `KyuubiApplicationOperator` to find or kill spark engine on kubernetes.

This pr aims to extract the code for building kubernetes client into `KubernetesUtils` and to do auto build following the logic in Spark.

We prefer to building with user-configured parameters; When missing, the kubernetes client logic will build one from environment variables and kubeconfig.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3663 from zwangsheng/feature/perfer_build_from_pod_env.

Closes #3663

0a4c8c79 [zwangsheng] clear properties
ce408db0 [zwangsheng] fix test
fb28ea7a [zwangsheng] add ci test
943c68dd [zwangsheng] add unit test
ddc9a2b2 [zwangsheng] fix scala
75380d13 [zwangsheng] comment
e6826609 [zwangsheng] master
683cc260 [zwangsheng] context provided master
7b42133c [zwangsheng] add doc
b56829c5 [zwangsheng] fix helm
5884475d [zwangsheng] rebuild
a7f0b551 [zwangsheng] ad
e7f9ec05 [zwangsheng] add setting
b77bbd9b [zwangsheng] add doc
3f3253d3 [zwangsheng] init

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-25 14:21:41 +08:00
Fei Wang
7bcb522d6e [KYUUBI #3658] Support SSL for Kyuubi thrift binary connection
### _Why are the changes needed?_

Support SSL for kyuubi thrift binary conenction.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate
Will add screenshot for IT.

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3658 from turboFei/ssl_binary.

Closes #3658

ffd603ec [Fei Wang] comment
ce652b7e [Fei Wang] revert UT
52e24abb [Fei Wang] ut
f6cae357 [Fei Wang] save
c6b664f1 [Fei Wang] exclude passwords
d34a4ca2 [Fei Wang] comments
11bfa9ad [Fei Wang] refactor
e2ca77ea [Fei Wang] align
86aaf87c [Fei Wang] with alternative
d17e0d11 [Fei Wang] style
eb4ba52a [Fei Wang] Support SSL for Kyuubi thrift binary connection

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-10-24 21:39:31 +08:00
yuqi
c25961ea02 [KYUUBI #2887] Add a POLLING balance policy for spark engine pool
### _Why are the changes needed?_

As described in #2887, random policy may cause task-hot-issues or production accident, so a POLLING balance policy is added to avoid this problem, we can "set kyuubi.engine.pool.balance.policy = POLLING" to use it.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3662 from ychris78/epbp_1018.

Closes #2887

90834c1f [yuqi] fix bug
c0ce8a7e [yuqi] Fix typos
bec4ca13 [yuqi] Fix typos
dc4e3637 [yuqi] please add test case to cover the added methods
837c920b [yuqi] rename method getAndInc to getAndIncrement and modify the doc
02a8311f [yuqi] add getAndInc method and refer it
9436937d [yuqi] Improve configuration documentation
21b5d7d1 [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool Improve configuration documentation and delete useless comments
f0699b3c [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool

Authored-by: yuqi <1450636878@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-10-24 14:17:59 +08:00
Fu Chen
393d32a30c
[KYUUBI #3615] Retry opening the engine when encountering a special error
### _Why are the changes needed?_

to close #3615

steps to reproduce

1. start kyuubi server
2. connect to the kyuubi server with client beeline and run query `select 1`
3. force kill the spark application
4. connect to the kyuubi server with another client **A** immediately

```
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:466)
        at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:482)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
        at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
        at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
        at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.aache.thrift.transport.TSocket.open(TSocket.java:221)
        ... 14 more (state=,code=0)
```

If the engine exits without calling the shutdown hook, the ZNODE with EPHEMERAL_SEQUENTIAL mode will be deleted upon the client's disconnect, and note that the engine will be marked as disconnected after `kyuubi.zookeeper.embedded.max.session.timeout` (the default value is 1 minutes)

This PR retries opening the engine when encountering a specialty exception (java.net.ConnectException). Before this PR, client **A** will throw an exception, after this PR, client **A** will work properly.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3618 from cfmcgrady/kyuubi-3615.

Closes #3615

96acd66b [Fu Chen] address comment
a1395a60 [Fu Chen] fix style
870216d1 [Fu Chen] Retry opening the engine when encountering a special error

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-20 14:31:35 +08:00
jiaoqingbo
17df2428e6
[KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark
### _Why are the changes needed?_

fix #3640

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3641 from jiaoqingbo/kyuubi3640.

Closes #3640

c1b2d4a6 [jiaoqingbo] code review
a65bc132 [jiaoqingbo] code review
ab04f135 [jiaoqingbo] [KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-19 17:32:00 +08:00
Cheng Pan
2cb34c4d00
[KYUUBI #3597] Engine should prefer ip for registering on K8s cluster mode
### _Why are the changes needed?_

When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.

Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266

To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.

Close #3578

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3597 from pan3793/k8s.

Closes #3597

8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-10 12:04:41 +08:00
yikf
12a85fe337
[KYUUBI #3512] Use dedicated ExecutionContext for EventBus async execution
### _Why are the changes needed?_

Fix https://github.com/apache/incubator-kyuubi/issues/3512

Currently, Kyuubi EventBus uses `scala.concurrent.ExecutionContext.Implicits.global` to execute async event handler. Generally, it's discouraged to use that global ec, instead, we should create dedicated ec for each workload, this pr aims to use dedicated ExecutionContext for EventBus async execution

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3524 from Yikf/ec.

Closes #3512

9757372d [yikf] Use dedicated ExecutionContext for EventBus async execution

Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-28 13:56:48 +08:00
HYBG
78f5865599
[KYUUBI #3502] [DOCS] Fix typo and use beeline provided by Kyuubi
1. Fix typo ${SPAAK_HOME}
2. Change ${SPAAK_HOME} to ${KYUUBI_HOME}

Closes #3502 from HYBG-1126/master.

Closes #3502

acf43e08 [海洋饼干] 更新文档docs/deployment/kyuubi_on_kubernetes.md 1、61行${SPAAK_HOME}为${SPARK_HOME} 2、91、96行,${SPAAK_HOME}应为${KYUUBI_HOME}

Lead-authored-by: HYBG <hybg1126@outlook.com>
Co-authored-by: 海洋饼干 <hybg1126@outlook.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-16 19:41:04 +08:00
yikf
4de12cdb6f
[KYUUBI #3449] Change default server info provider to ENGINE
### _Why are the changes needed?_

close https://github.com/apache/incubator-kyuubi/issues/3449, this pr aims to change `kyuubi.server.info.provider` default value to "ENGINE", and modify corresponding tests.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3476 from Yikf/default-server-info.

Closes #3449

bc474d3b [Cheng Pan] Update docs/deployment/migration-guide.md
23e7690e [yikf] Change default server info provider to ENGINE

Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-14 10:32:53 +08:00
Min
9440d1b708
[KYUUBI #3309] Add configs of engine event logger for each engine
### _Why are the changes needed?_

The default value of kyuubi.engine.event.loggers is SPARK, but it can't be used outside the spark engine, so we add the config for each engines.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3309 from zhaomin1423/add_engine_logger_conf.

Closes #3309

37b3c05f [Xiao Zhao] fix
b22356f2 [Min] fix typo
431bb02b [Min] update settings.md
44475f46 [Min] Add configs of engine event logger for each engine

Lead-authored-by: Min <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-09-09 09:58:50 +08:00
Cheng Pan
29caadb2ba Revert "[KYUUBI #3020] kyuubi ldap add new config property kyuubi.authentication.ldap.bindpw and kyuubi.authentication.ldap.attrs"
This reverts commit 5d88c7b207.
2022-09-08 08:59:50 +00:00
Cheng Pan
6a90b3e824 [KYUUBI #3122] GetInfo supports return server/engine info
### _Why are the changes needed?_

Workaround for #3032, close #3323

There are known some ODBC drivers e.g. [Databricks ODBC driver](https://www.databricks.com/spark/odbc-drivers-download) depending on `TGetInfoType.CLI_DBMS_VER` and `TGetInfoType.CLI_DBMS_NAME` to check server compatibilities and abilities.

This PR proposes to introduce a new configuration `kyuubi.server.info.provider=SERVER/ENGINE` to make GetInfo support return either server or engine information.

Since beeline will call GetInfo in the initialization phase, to make sure the beeline fast open experience, in async launch mode, when the engine is not ready, return server info regardless `kyuubi.server.info.provider`.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

Testing w/ PowerBI

![image](https://user-images.githubusercontent.com/26535726/188945975-0d0fc95c-f989-4025-ad7d-c024e23ec328.png)

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3122 from pan3793/info.

Closes #3122

742bdbe3 [Cheng Pan] nit
bb85d2ba [Cheng Pan] style
fd75238c [Cheng Pan] fix
9ddb2afd [Cheng Pan] nit
fd8f7979 [Cheng Pan] fix ut
840205e5 [Cheng Pan] nit
f9996d53 [Cheng Pan] GetInfo supports returning engine info

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-08 02:14:38 +00:00
yikf
331f1a3bdf
[KYUUBI #3414] Tidy up plan only related modes and styles
### _Why are the changes needed?_

Fix https://github.com/apache/incubator-kyuubi/issues/3414

Currently, Kyuubi supports output of planOnly in different modes in different styles, As more and more modes and styles are supported, defining enumerations in kyuubiConf is a bit messy and not easy to extend, so this pr aims to tidy up these

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3415 from Yikf/tidy-up.

Closes #3414

e81142e5 [yikf] Tidy up plan only related modes and styles

Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-09-06 13:51:42 +08:00
yikf
cc60a15363
[KYUUBI #3376] PlanOnly supports output in different styles
### _Why are the changes needed?_

Fix https://github.com/apache/incubator-kyuubi/issues/3376

This pr aims support planOnly output in different styles (Users can perform SQL scheduling based on the JSON description of TreeNode)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3377 from Yikf/planOnly-support-json.

Closes #3376

b63ae718 [yikf] Support json_description for PlanOnlyStatement

Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-09-05 15:01:31 +08:00
yikf
8f57c431b8
[KYUUBI #3357]Improve doc description for configuration kyuubi.operation.plan.only.mode
### _Why are the changes needed?_

Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3358 from Yikf/planOnly-doc.

Closes #3357

571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`

Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-08-29 11:19:40 +08:00
wangjunbo
8870183ed7
[KYUUBI #3128] Support CostMode for PlanOnlyStatement
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3304 from Kwafoor/master.

Closes #3128

88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement

Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-08-24 14:06:40 +08:00
liangbowen
2b67ab6e69
[KYUUBI #3222][FOLLOWUP] Fixing placeholder and config of user in JDBC Authentication Provider
### _Why are the changes needed?_

To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,

1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3288 from bowenliang123/jdbc-auth-config-update.

Closes #3222

502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of  kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-22 13:36:32 +08:00
liangbowen
97b14f8bc3
[KYUUBI #3275] [KYUUBI 3269] [DOCS] Doc for JDBC authentication provider
### _Why are the changes needed?_

fix https://github.com/apache/incubator-kyuubi/issues/3269

Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3275 from bowenliang123/doc-jdbc-auth.

Closes #3275

e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
2022-08-19 17:38:05 +08:00
pengqli
5d88c7b207 [KYUUBI #3020] kyuubi ldap add new config property kyuubi.authentication.ldap.bindpw and kyuubi.authentication.ldap.attrs
### _Why are the changes needed?_

Among the kyuubi ldap configuration parameters, there is currently "kyuubi.authentication.ldap.base.dn=", but there is no place corresponding to the CN password setting. similar to hive
"hive.server2.authentication.ldap.binddn",
"hive.server2.authentication.ldap.bindpw".

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3213 from dev-lpq/branch-ldap-1.6.

Closes #3020

31817cc6 [pengqli] upgrade ldap to support ldap before 1.6.0
31094030 [pengqli] upgrade ldap to support ldap before 1.6.0
f62e4c96 [pengqli] upgrade ldap to support ldap before 1.6.0
e03e036e [pengqli] add AUTHENTICATION_LDAP_GUIDKEY deprecated
6f680dfc [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
19f8e6ce [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
07df56b8 [pengqli]  upgrade LdapAuthenticationProviderImpl
1a48f418 [pengqli] upgrade settings.md
8e9b5721 [pengqli] upgrade ldap ldapBindpw
5793a811 [pengqli] upgrade ldap ldapBindpw
822d4c30 [pengqli] upgrade docs/deployment/settings.md
c2b20418 [pengqli] upgrade ldap WithLdapServer UTs
5d7a301c [pengqli] add AUTHENTICATION_LDAP_PASSWORD into serverOnlyConfEntries
10b42b1b [pengqli] upgrade AuthenticationProviderFactorySuite UTs
490bfae8 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
5e071252 [pengqli] upgrafe config kyuubi.authentication.ldap.attrs
4e771ec6 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs

Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-08-18 20:29:12 +08:00
ulysses-you
a3b6c67567
[KYUUBI #3255] Add miss engine type config docs
### _Why are the changes needed?_

master branch has supported 5 engine types. the docs of `kyuubi.engine.type` missed hive and jdbc.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3255 from ulysses-you/engine-type.

Closes #3255

a1f3c320 [ulysses-you] add config

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-08-17 17:02:49 +08:00
liangbowen
a63587b1e3
[KYUUBI #3222] JDBC Authentication Provider for server
### _Why are the changes needed?_

Add JDBC authentication provider as implementation of PasswdAuthenticationProvider out of box.

Compared to currently support authentication methods like Kerberos and LDAP, JDBC source is a much easy and stable source for practical deployment.

The solution should provide:

easy to use and config jdbc connection details
handy to customize query statements for authentication

Adding JDBC to AuthMethods enum and JDBCPasswdAuthenticationProvider into package org.apache.kyuubi.service.authentication with listed features:

- specify JDBC driver name and load the driver class
- configs of JDBC url, username and password
- select query with placeholders for checking user and password , like `SELECT 1 from user_pass_hash where username = ${user} and password = MD5(${password})`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3235 from bowenliang123/feature-jdbc-auth-provider.

Closes #3222

17403b33 [liangbowen] cleanup docs
d5f43e0e [liangbowen] remove unuseful logs for unrecognized placeholder error
e9af0966 [liangbowen] use clone instead of repeatly generating configs
6fc42bf3 [liangbowen] code styling
a9404fa3 [liangbowen] use {} for intercept
77f5f86b [liangbowen] remove unuseful comment
6765affb [liangbowen] changed to use in-memory derby db for test
543c66cb [liangbowen] prefer scala style string usage
a4fe582f [liangbowen] refactor connection creation on using HikariDataSource in HikariCP. add HikariCP dependencies to kyuubi-common
3a4d5fe9 [liangbowen] update checkConfigs() signature
5a0ac495 [liangbowen] output password length only in checkConfigs
1c956dfb [liangbowen] update KyuubiAuthenticationFactorySuite
4ebe12e2 [liangbowen] add JDBC value to AuthTypes enum
9885f813 [liangbowen] add JDBC condition for getValidPasswordAuthMethod
b9ffac3c [liangbowen] Merge branch 'master' into feature-jdbc-auth-provider
aeb19ce5 [liangbowen] update doc
653bc126 [liangbowen] add more checks for query sql
cdec2066 [liangbowen] more test cases
36729197 [liangbowen] fix ddl statement and remove truncate statement in test
30974d16 [liangbowen] update format
575301ca [liangbowen] update options usage
1dc4187b [liangbowen] update settings.md config doc
cd2c7c2b [liangbowen] update settings.md config doc
15176b2b [liangbowen] fix import orders
46cc1dda [liangbowen] add config docs in docs/deployment/settings.md
7025330b [liangbowen] fix derby startup error in test
df4be568 [liangbowen] update code style
49c18c25 [liangbowen] update
996f7969 [liangbowen] add unit test in JdbcAuthenticationProviderImplSuite
0e7f0ad0 [liangbowen] refactor config and init process.remove unused import.
0dc75fee [liangbowen] implement JDBC Authentication Method

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-17 10:47:55 +08:00
Xiao Zhao
33b70cfeb8
[KYUUBI #3107] [Subtask] DorisSQLEngine - Add process builder (#3123)
* fix conflict

* redact password

* add jars to cp

* update javaOptions
2022-08-11 23:30:44 +08:00
hongdongdong
c17829bf10
[KYUUBI #3104] Support SSL for Etcd
### _Why are the changes needed?_

Support SSL for Etcd

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3105 from hddong/support-etcd-ssl.

Closes #3104

49aadb9c [hongdongdong] change enable to enabled
87fa6269 [hongdongdong] [KYUUBI #3104] Support SSL for Etcd

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-29 16:24:50 +08:00
Fei Wang
d75f48ea70 [KYUUBI #3052][FOLLOWUP] Do not use the ip in proxy http header for authentication to prevent CVE
### _Why are the changes needed?_

Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3078 from turboFei/http_real_ip_cve.

Closes #3052

e7c41ea9 [Fei Wang] prevent cve

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-07-15 18:34:56 +08:00
hongdongdong
ce72a502a3
[KYUUBI #3054] Add description of the discovery client in the conf doc
### _Why are the changes needed?_

Add description of the discovery client in the conf doc.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3054 from hddong/conf-to-etcd.

Closes #3054

d364d099 [hongdongdong] Add description of the discovery client in the conf doc

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-07-14 14:31:17 +08:00
Fei Wang
a3973a0b3d [KYUUBI #3052] Support to get the real client ip address for http connection when using VIP as kyuubi server load balancer
### _Why are the changes needed?_

To close #3052

If your server is behind a load balancer or other proxy, the server will see this  load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3053 from turboFei/http_real_ip.

Closes #3052

b375d02a [Fei Wang] add ut
23ef1c9c [Fei Wang] refactor
907c61c0 [Fei Wang] also take affect for http
a958c50f [Fei Wang] add conf

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-07-13 14:50:06 +08:00
hongdongdong
32970ce685
[KYUUBI #2644] Add etcd discovery client for HA
### _Why are the changes needed?_

Add etcd discovery client for HA

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2767 from hddong/etcd-support.

Closes #2644

6fa37578 [hongdongdong] Rebase and remove from rat-excludes
fc23a2b4 [hongdongdong] upgrade grpc-1.47.0
61c77d27 [hongdongdong] Fix tests
28abd13e [hongdongdong] fix comments
9e81a6a3 [hongdongdong] Rename HA_ZK_ENGINE_REF_ID to HA_ENGINE_REF_ID
aa2b4260 [hongdongdong] [KYUUBI #2644][WIP] Add etcd discovery client for HA

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-07-12 11:04:35 +08:00
Fei Wang
13e618cf9c [KYUUBI #3012] Remove unused thrift request max attempts and related ut
### _Why are the changes needed?_

After #2948 and #2953, the OPERATION_THRIFT_CLIENT_REQUEST_MAX_ATTEMPTS is not used.

In this pr, I remove the attempts config entry and related ut that based on the request attempts.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3012 from turboFei/follow_up_engine_alive.

Closes #3012

3d15e44a [Fei Wang] remove deprecated docs
89c9f2cb [Fei Wang] remove docs
edb6ff2e [Fei Wang] remove ut
696bd853 [Fei Wang] saev
f91c61bf [Fei Wang] remove unused attempt

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-07-05 17:50:06 +08:00
hongdongdong
66a45f3e6c [KYUUBI #2895] Show final info in trino engine
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![trino-process](https://user-images.githubusercontent.com/17537134/174044438-cfdda31f-a644-4753-81ef-7c62ea0a078c.png)

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2896 from hddong/trino-result-info.

Closes #2895

757019cf [hongdongdong] fix scala style
f40b1c53 [hongdongdong] Fix
3edbdfb3 [hongdongdong] [KYUUBI #2895] Show final info in trino engine

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
2022-07-05 09:17:18 +08:00
Cheng Pan
c7e2b3229b
[KYUUBI #2850][FOLLOWUP] Replace log4j2.properties by log4j2.xml
### _Why are the changes needed?_

Replace log4j2.properties by log4j2.xml

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3001 from pan3793/log4j.

Closes #2850

e9396017 [Cheng Pan] fix
4e06c89c [Cheng Pan] [KYUUBI #2850][FOLLOWUP] Replace log4j2.properties by log4j2.xml

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-04 16:48:02 +08:00
Fei Wang
b585fb4257 [KYUUBI #2801] Add local dir allow list and check the application access path URI
### _Why are the changes needed?_

To close #2801

Customer might set some parameters to upload the local files, such as spark.files=/etc/security/keytabs/kyuubi.keytab. It is not security for kyuubi server.

In this pr, I introduce a local dir allow list and customer can only upload the files under these dirs.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2991 from turboFei/local_white_list.

Closes #2801

6315067d [Fei Wang] comments
deb9fa8a [Fei Wang] remove unused
3330fa98 [Fei Wang] refactor
6eca2f8b [Fei Wang] move to KyuubiApplicationManager and add ut
1290d9e2 [Fei Wang] add more
f14ec7d4 [Fei Wang] add more path configs and add comment
82357183 [Fei Wang] refactor
f201f577 [Fei Wang] disable relative path
f806e69a [Fei Wang] refactor
4f2faa6a [Fei Wang] save

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-07-04 14:50:02 +08:00
zwangsheng
3bc299d01e
[KYUUBI #2868] [K8S] Add KubernetesApplicationOperation
### _Why are the changes needed?_

close #2868
Add Kubernetes Application Operation.
Help Tag Spark Driver with Engine Ref Id.

1) Find taged Engine:
`KyuubiApplicationManager => KubernetesApplicationOperation =(doesn't found taged pod)=> KubernetesApplicationOperation#JpsApplicationOperation`

2) `SparkBatchProcessBuilder` add logic to help tag Kubernetes Engine Tag.

3) Add Unit Test help to check `KubernetesApplicationOperation` find & kill  Spark Engine On Kubernetes with both client mode and cluster mode.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2869 from zwangsheng/feature/k8s-application-operation.

Closes #2868

04c97210 [zwangsheng] remove
fc151603 [zwangsheng] fix kyuubi on kubernetes integration test
29a3da52 [zwangsheng] fix
f43f0989 [zwangsheng] fix
2f2c9581 [zwangsheng] test
209dd65d [zwangsheng] fix jps
e12c681c [zwangsheng] fix tag
80971f98 [zwangsheng] fix
0760b63d [zwangsheng] fix
087eadb1 [zwangsheng] fix
d4989b09 [zwangsheng] try
93d83f81 [zwangsheng] add test
19378da4 [zwangsheng] fix style
63007261 [zwangsheng] fix test
fd4bf714 [zwangsheng] fix
66fc902b [zwangsheng] Using jps for client
2a1b797f [zwangsheng] fix setting
4827c459 [zwangsheng] fix
9ee90da6 [zwangsheng] decoupling spark
6b5831da [zwangsheng] useless driver label
f9ca898c [zwangsheng] fix
2add72fe [zwangsheng] fix setting
aaa081f8 [zwangsheng] fix style & dependency list
e102ac73 [zwangsheng] add kubernetesApplicationOperation

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-07-01 12:15:12 +08:00
Tianlin Liao
6e4b5582da [KYUUBI #2956] Support to config the connect/socket timeout of rest client for kyuubi-ctl
### _Why are the changes needed?_

Close #2956

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2958 from lightning-L/kyuubi-2956.

Closes #2956

a88de508 [Tianlin Liao] allow 0 timeout for rest client; increase default timeout
94a57cce [Tianlin Liao] update
d569abe8 [Tianlin Liao] [KYUUBI #2956] Support to config the connection/socket timeout of rest client for kyuubi-ctl

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-29 18:47:58 +08:00
Fei Wang
6a9d5ff244 [KYUUBI #2948] Remove thrift request timeout for KyuubiSyncThriftClient
### _Why are the changes needed?_

Remove thrift request timeout for KyuubiSyncThriftClient

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2948 from turboFei/remove_request_timeout.

Closes #2948

a9c5c084 [Fei Wang] remove request timeout

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-27 14:45:52 +08:00
Tianlin Liao
825c70db7b [KYUUBI #2919] Fix typo and wording for JDBCMetadataStoreConf
### _Why are the changes needed?_

Fix typo and wording for JDBCMetadataStoreConf.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2919 from lightning-L/typo-1.

Closes #2919

f02ffb8a [Tianlin Liao] fix typo and wording for JDBCMetadataStoreConf

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-21 12:46:49 +08:00
Tianlin Liao
c664c84f80 [KYUUBI #2628][FOLLOWUP] Support waitCompletion for submit batch
### _Why are the changes needed?_

Support waitCompletion for submit batch

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2909 from lightning-L/kyuubi-2628-1.

Closes #2628

23fe761f [Tianlin Liao] minor fix
06cd66ff [Tianlin Liao] support waitUntilComplete for submit batch; use kyuubi instance url for log batch/submit batch

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-20 11:56:36 +08:00
Fei Wang
7baf98954d [KYUUBI #2834] [SUB-TASK][KPIP-4] Support to retry the metadata requests on transient issue and unblock main thread
### _Why are the changes needed?_

We need support to retry the jdbc requests.

Because we add new jdbc dependency for batch HA, and the database might be in maintenance window.

And there might be master-slave switch, and the database might be unavailable for few minutes.

We need tolerant this issue and unblock the main thread.

BTW, this pr refactor the metadata store naming.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2834 from turboFei/retrying_jdbc_event.

Closes #2834

109a63e70 [Fei Wang] fix docs
a186484be [Fei Wang] refactor the jdbc pool name
cfac9da53 [Fei Wang] rename metadata table name
bb8710ba3 [Fei Wang] comments
1197a098b [Fei Wang] add private package scope
a95694ba9 [Fei Wang] comments
e2f09e784 [Fei Wang] refactor
057eb73c5 [Fei Wang] refactor
88f3ffb6f [Fei Wang] refactor
e0595bdee [Fei Wang] comments
48270e869 [Fei Wang] refactor conf keys
e4f190c5c [Fei Wang] update docs
ed5e90e37 [Fei Wang] add max queues limitation
c5a03c9e2 [Fei Wang] remove getOrCreate
3e39ca94c [Fei Wang] remove
b79fb0954 [Fei Wang] comments
1d309cee0 [Fei Wang] comments
9108a3f0e [Fei Wang] fix docs
5cdffe710 [Fei Wang] fix rat
2e7228fc7 [Fei Wang] fix ut
520236d7b [Fei Wang] rename more
f66d68bcc [Fei Wang] merge into metadata manager
2f70a2f12 [Fei Wang] rename
80e6100fb [Fei Wang] refactor
09cad092a [Fei Wang] add ut
ea4eea74f [Fei Wang] refactor
6b00eb0b3 [Fei Wang] refactor to MetadataRequest
58f1c0587 [Fei Wang] commements
198feb53f [Fei Wang] add more docs
670a23561 [Fei Wang] refactor
a20d54df2 [Fei Wang] waitStateStoreRetryCompletion after super.close closes operation
97a763aef [Fei Wang] refactor RetryingStateStoreRequest
3e136cf34 [Fei Wang] [SUB-TASK][KPIP-4] Support to retry the session state operations if meet transient jdbc issue

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-16 12:20:49 +08:00
Mahmoud Bahaa
1ea245d254
[KYUUBI #886] Add HTTP transport mode support to KYUUBI - no Kerberos support
Same as https://github.com/apache/incubator-kyuubi/pull/2815 but with Kerberos auth removed as was not complete nor tested. so thought to make a separated PR with no Kerberos support then we can add Kerberos support later on

### _Why are the changes needed?_

Add http transport protocol support to Kyuubi
main code was taken from:

1. 25ac0731d7
2. https://github.com/apache/hive/blob/branch-3.1/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
2. https://github.com/apache/hive/blob/branch-3.1/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2833 from mahmoudbahaa/transport-mode-http-no-kerberos.

Closes #886

3a24abec [Mahmoud Bahaa] fix test cases
ce982af7 [Mahmoud Bahaa] [KYUUBI #886] Add HTTP transport mode support to KYUUBI - no kerberos support
b6371859 [SteNicholas] [KYUUBI #886] Add HTTP transport mode support to KYUUBI
36fb0760 [Mahmoud Bahaa] update .gitignore to include conf non template files

Lead-authored-by: Mahmoud Bahaa <mahmoud.bahaa@incorta.com>
Co-authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-06-16 10:33:00 +08:00
Fei Wang
57e3733413
[KYUUBI #2859][SUB-TASK][KPIP-4] Support --conf for kyuubi-ctl
### _Why are the changes needed?_

To close #2859

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2875 from turboFei/kyuubi_2859_retry_delete.

Closes #2859

7908d827 [Fei Wang] revert commands related
23d357dc [Fei Wang] fix ut
76730014 [Fei Wang] print close resp first
76449dad [Fei Wang] support --conf

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-06-16 09:09:23 +08:00
Tianlin Liao
cb48338591 [KYUUBI #2628][SUB-TASK][KPIP-4] Implement kyuubi-ctl for batch job operation
### _Why are the changes needed?_

To close #2628
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2823 from lightning-L/kyuubi-2628.

Closes #2628

3af66099 [Tianlin Liao] refactor
19ca40fe [Tianlin Liao] fix list/log batch
d0fc0a9d [Tianlin Liao] refactor RestClientFactory
617732c8 [Tianlin Liao] refactor command
897afd38 [Tianlin Liao] rename ClientFactory to RestClientFactory
9ed61166 [Tianlin Liao] add list batch option
953e7169 [Tianlin Liao] refactor kyuubi-ctl conf
ad2a9b6f [Tianlin Liao] fix license and style
60943eb2 [Tianlin Liao] fix delete/list operation after rebase
97cbe9df [Tianlin Liao] add kyuubi rest config: hostUrl, spnegoHost, authSchema
c46e8f7a [Tianlin Liao] implement create/get/delete/list/log/submit batch command
71c9cb9a [Tianlin Liao] reorganize cmd folder and rename command class
cf7478bb [Tianlin Liao] use reflect to generate command object
e985c7bb [Tianlin Liao] rename ServiceControlXXX to ControlXXX

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-06-14 11:19:18 +08:00
Fei Wang
e725725155
[KYUUBI #2373][SUB-TASK][KPIP-4] Support to recovery batch session on Kyuubi instances restart
### _Why are the changes needed?_

To close #2373

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2790 from turboFei/recovery_batch.

Closes #2373

81632191 [Fei Wang] trigger test
6ac6f9bd [Fei Wang] async recovery
6d9edf4e [Fei Wang] comments
daa6719f [Fei Wang] refactor
0c3a2e14 [Fei Wang] remove waitAppCompletion
04e139ee [Fei Wang] comments
b5d11d8f [Fei Wang] comment for method name
321dfa95 [Fei Wang] refactor
90be2df7 [Fei Wang] address comments
14409582 [Fei Wang] batch recovery

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-06-08 10:31:50 +08:00
Fei Wang
73c6b1b14a [KYUUBI #2717] [SUB-TASK][KPIP-4] Introduce jdbc session state store for batch session multiple HA
### _Why are the changes needed?_

Close #2715
Close #2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.

This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and  refactor it during implementation.

In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.

The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information

For JDBCStateStore:
- support to specify data base type:
  - 1. DERBY - it will use derby jdbc driver by default
  - 2. MYSQL - it will use mysql jdbc by default
  - 3. CUSTOM - the user need specify the jdbc driver
- support to init schema

By default, the jdbc url is a derby in-memory database url for single kyuubi instance.

To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.

The datasource is managed by HiKariCP.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2717 from turboFei/kyuubi_2715_hikaricp_v2.

Closes #2717

367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-05-31 15:06:39 +08:00
Kent Yao
414d1a8620
[KYUUBI #2764] [DOCS] Fix tables in docs being coverd by right toc sidebar
### _Why are the changes needed?_

make table width fit browsers with 100% width, and 1st column where contains the kyuubi conf fit the table with 25% width.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

<img width="1716" alt="image" src="https://user-images.githubusercontent.com/8326978/170655840-2309574e-4098-4b0f-b91d-50a30033b9ac.png">

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2764 from yaooqinn/layout.

Closes #2764

9391a638 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
33817406 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-27 17:21:42 +08:00
Kent Yao
e1921fc8c0
[KYUUBI #2751] [DOC] Replace sphinx_rtd_theme with sphinx_book_theme
### _Why are the changes needed?_

make documentation style to fit official web site

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/8326978/170434595-d5ec5068-ab2d-484e-9371-9331311cc0f1.png)
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2751 from yaooqinn/doc2.

Closes #2751

b8123525 [Kent Yao] Switch

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-27 02:22:36 +08:00
sychen
9b5023070a
[KYUUBI #2721] Implement dedicated set/get catalog/database operators
### _Why are the changes needed?_
close #2721

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2728 from cxzl25/KYUUBI-2721.

Closes #2721

67051644 [sychen] use placeholders and confOverlay
2acf0a45 [sychen] unused import
8bc40b9f [sychen] fix UT
3495b280 [sychen] fix jdbc
e55bb2e2 [sychen] style
ff9078ab [sychen] fix UT
a0247c7e [sychen] Merge branch 'master' into KYUUBI-2721
f62461ce [sychen] use placeholders
698fbbdd [sychen] set catalog/schema empty resultset
45caf552 [sychen] fix import
7f13ad8d [sychen] fix server spark catalog UT
4982ec71 [sychen] hive engine 1.8
9877038f [sychen] add UT
21e57870 [sychen] set/get catalog/database operators

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-26 18:06:30 +08:00
Min Zhao
7f94501740
[KYUUBI #2658] [Subtask] DorisSQLEngine with execute statement support
### _Why are the changes needed?_

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2595 from zhaomin1423/jdbc_engine.

Closes #2658

90bdf97c [Min Zhao] add jdbc dialect
d2bfd60c [Min Zhao] [KYUUBI #2486] [FEATURE] Support Apache Doris engine

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-05-25 15:10:46 +08:00
hongdongdong
3b81a4953f [KYUUBI #2631] Rename high availability config key to support multi discovery client
### _Why are the changes needed?_

Rename high availability config key to support muti discovery client

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2633 from hddong/muti-discovery-config-support.

Closes #2631

bc434ac9 [hongdongdong] [KYUUBI #2631] Rename high availability config key to support muti discovery client

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
2022-05-25 10:01:46 +08:00
Fei Wang
f1cf95fe96 [KYUUBI #2719] [SUB-TASK][KPIP-4] Support internal rest request authentication to enable http request redirection across kyuubi instances
### _Why are the changes needed?_

Support internal rest request authentication to enable redirect http request across kyuubi instances

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2719 from turboFei/internal_access.

Closes #2719

b9e150e9 [Fei Wang] revert config key change
aa378727 [Fei Wang] make it internal
ce96d922 [Fei Wang] comments
6fe8523b [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-05-23 15:44:48 +08:00
sychen
7584e3abb5
[KYUUBI #2714] Log4j2 layout pattern add date
### _Why are the changes needed?_
Because `kyuubi-server` is a long-running service, its logs should preferably have dates.
When Log4j1 was used in previous versions, the default log and template had dates.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2714 from cxzl25/KYUUBI-log4j-layout-pattern.

Closes #2714

b249c296 [sychen] change layout pattern

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-23 03:15:44 +08:00
Fu Chen
85d68b20ca
[KYUUBI #2619] Add profile spark-3.3
### _Why are the changes needed?_

to close #2619

add maven profile `spark-3.3`, part of #2620

### _How was this patch tested?_

Pass CI.

Closes #2692 from cfmcgrady/kyuubi-2619-1.

Closes #2619

fbd336fe [Fu Chen] refine SparkProcessBuilderSuite - `engine log truncation`
039ef828 [Fu Chen] move module kyuubi-spark-connector-kudu to spark-3.3
03d799ec [Fu Chen] update kyuubi.operation.plan.only.excludes
724fc7a2 [Fu Chen] GA
53b97a18 [Fu Chen] add profile `spark-3.3`

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-05-19 17:07:04 +08:00