Commit Graph

679 Commits

Author SHA1 Message Date
hezhao2
0ce697eefa [KYUUBI #5654] Introduce new config to forcibly rewrite pod name
### _Why are the changes needed?_
close #5654
This pr is to introduce a config for rewriting pod name, which can help users forcibly rewrite driver or executor pod name prefix.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5672 from zhaohehuhu/dev-1111.

Closes #5654

6a450e444 [hezhao2] introduce new config to force rewrite pod name

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 10:21:05 +08:00
GianlucaPrincipini
4be584cabe [KYUUBI #5599] Correct the parameter name of fetching op log in RESTful API
### _Why are the changes needed?_

It corrects the fetching op log Rest API documentation to match request parameter case in code, solving issue #5599

### _How was this patch tested?_
No tests needed.

### _Was this patch authored or co-authored using generative AI tooling?_

No AI tool has been used for this patch.

Closes #5673 from GianlucaPrincipini/fix-fetching-op-log-param-doc.

Closes #5599

f134af050 [GianlucaPrincipini] Change maxRows "param" to "maxrows" in documentation

Authored-by: GianlucaPrincipini <gianluca.principini@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-13 10:10:07 +08:00
fwang12
c149809fb9 [KYUUBI #5564] Support to config ticket cache location in JDBC URL
### _Why are the changes needed?_

Support to config customized ticket cache location in kyuubi connection URL.

Before, we can only set it via environment variable `KRB5CCNAME`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5564 from turboFei/ticket_cache.

Closes #5564

140ae6ec2 [fwang12] since 1.8.0
caaf33a44 [fwang12] check is blank
0fae9f9c5 [fwang12] doc
05b9e9df8 [fwang12] ticket cache

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-03 00:35:27 +08:00
Cheng Pan
8980e0752d
Revert "[KYUUBI #4994][REST] Support listing all engines"
This reverts commit 16ae852891.
2023-11-03 00:25:18 +08:00
Cheng Pan
9bf234473a [KYUUBI #5568] Check administrator only when security is enabled
### _Why are the changes needed?_

This PR aims to improve out-of-box UX.

Currently, administrators' checking is enforced even when security is disabled, and we are going to deliver the preview of Web UI in 1.8, it may confuse users who use the default configuration to launch Kyuubi sever and explore Kyuubi WebUI (because the user "anonymous" can not visit all users' sessions/engines without being configured as an administrator).

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5568 from pan3793/admin.

Closes #5568

2eccf659f [Cheng Pan] code dedup and fix test
76f4fd4a3 [Cheng Pan] fix test
d7ca94650 [Cheng Pan] Check administrator only when security is enabled

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-02 10:09:01 +08:00
zhouyifan279
d9a26c3b09 [KYUUBI #5480] Support setting kyuubi hive jdbc client protocol version
### _Why are the changes needed?_
When using Kyuubi Hive JDBC to Hive Server2, TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10 is used and can not be changed.

When we connected to Hive Server2 with version lower than 2.2.0, we got the following error:
```
org.apache.kyuubi.shade.org.apache.thrift.TApplicationException:
Required field 'client_protocol' is unset!
Struct:TOpenSessionReq(client_protocol:null, configuration:{kyuubi.client.version=1.7.3, set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, kyuubi.client.ipAddress=172.16.19.113, use:database=default})
```

In this PR, we introduced a session conf `clientProtocolVersion`.
By adding `clientProtocolVersion=8` to jdbc url, the error got fixed.

Changes of `kyuubi_jdbc.rst`

<img width="867" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/8f98edf9-15c4-4d1b-9299-83b24136352b">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Manual tests against Hive Server2 version 2.1.1-cdh6.3.0

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No.

Closes #5480 from zhouyifan279/protocol-version.

Closes #5480

0ee7b1f68 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
61b8038d3 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
ed4c29f2b [Cheng Pan] Update docs/deployment/migration-guide.md
dc16a05fc [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
1a6bfd88c [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
cb00edc37 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version
c99fc4802 [zhouyifan279] Support setting kyuubi hive jdbc client protocol version

Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-30 23:51:54 +08:00
senmiaoliu
c24e984a3e
[KYUUBI #5423] Support chaining SessionConfAdvisors
### _Why are the changes needed?_

close #5423

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5485 from lsm1/branch-5423.

Closes #5423

0bd3b0233 [senmiaoliu] mention in the migration guide
dddafeb9b [senmiaoliu] support multi session conf advisor

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-10-30 09:48:25 +08:00
lawulu
30a127a1ba [KYUUBI #5412] Resolve the relative zk configuration dir based on KYUUBI_HOME
### _Why are the changes needed?_
Both   the default  value of `kyuubi.zookeeper.embedded.data.log.dir` and `kyuubi.zookeeper.embedded.data.dir` are `embedded_zookeeper`. It is used based on the current dir.
If the configuration dir is a relative path , it is resolved relative to `KYUUBI_HOME`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5412 from biangjuang/master.

Closes #5412

17e502c03 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
e0aa01c82 [Cheng Pan] Update kyuubi-zookeeper/src/main/scala/org/apache/kyuubi/zookeeper/ZookeeperConf.scala
bb6af93df [Cheng Pan] Update docs/deployment/migration-guide.md
53bd61bc3 [lawulu] Resolve the relative zk configuration dir based on KYUUBI_HOME

Lead-authored-by: lawulu <biangjuang@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-27 10:48:22 +08:00
Cheng Pan
cb93d6fd1a [KYUUBI #5523] [DOC] Update the Kyuubi supported components version
### _Why are the changes needed?_

Kyuubi 1.8 added the support for JDK 17 and Flink 1.18.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

<img width="1139" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/2316fce3-20fc-49dc-a967-a40f9ea8f706">

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5523 from pan3793/doc-ver.

Closes #5523

d9024ccc7 [Cheng Pan] Update the Kyuubi supported components version

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-25 22:12:08 +08:00
liupeiyue
9a409956a1 [KYUUBI #5282] Support configure Trino session conf in kyuubi-default.conf
### _Why are the changes needed?_

Refer to spark and flink settings conf,  support configure Trino session conf in kyuubi-default.conf
issue : https://github.com/apache/kyuubi/issues/5282

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5283 from ASiegeLion/kyuubi-master-trino.

Closes #5282

87a3f57b4 [Cheng Pan] Apply suggestions from code review
effdd79f4 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
399a200f7 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml
7462b32c2 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml--Update documentation
5295f5f94 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml

Lead-authored-by: liupeiyue <liupeiyue@yy.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-24 16:19:44 +08:00
yangming
289549f7f3 [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine
### _Why are the changes needed?_

As https://github.com/apache/kyuubi/pull/4000 adds the Apache Phoenix support for the JDBC engine without upgrading the configuration's description.

This PR aims to update the outdated configuration description to reflect the capabilities of the Kyuubi JDBC engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5294 from ymZhao1001/doc0915.

Closes #5294

8d61c36f9 [yangming] update jdbc doc
d082b4d34 [yangming] run dev/reformat and dev/gen/gen_all_config_docs.sh
5b814d86b [zhaoyangming] add jdbc Phoenix dialect

Lead-authored-by: yangming <261635393@qq.com>
Co-authored-by: zhaoyangming <zhaoyangming@deepexi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-24 16:16:29 +08:00
Kent Yao
1224b0d2c9
[KYUUBI #5500] Add Kyuubi Code Program to Doc
### _Why are the changes needed?_

Add Kyuubi Code Program to Doc for long-term management and sponsorship.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

no

Closes #5500 from yaooqinn/program.

Closes #5500

14f144747 [Kent Yao] Add Kyuubi Code Program to Doc

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2023-10-23 17:39:37 +08:00
yikaifei
59c25b9851
[KYUUBI #5427] [AUTHZ] Shade spark authz plugin
### _Why are the changes needed?_

This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5427 from Yikf/shade-authz.

Closes #5427

d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin

Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
2023-10-20 20:10:34 +08:00
senmiaoliu
16752164a1
[KYUUBI #5408] MetadataManager tries MySQL 8 driver class first
### _Why are the changes needed?_

close #5408

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

NO

Closes #5433 from lsm1/branch-kyuubi-5408.

Closes #5408

8c6cc223d [senmiaoliu] fix style
20415dc44 [senmiaoliu] use com.mysql.cj.jdbc.Driver first

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-10-18 12:55:00 +08:00
zwangsheng
b24d94e74f
[KYUUBI #5328] Batch supports priority scheduling
### _Why are the changes needed?_

Follow #5329 and close #5328:

1. Add new config `kyuubi.metadata.store.jdbc.priority.enabled` to control whether enable priority scheduling, due to users may experience performance issues when using MySQL5.7 as metastore backend and enabling kyuubi batch v2 priority feature.
2. When priority scheduling is enabled, `KyuubiBatchService` picks metadata job with `ORDER BY priority DESC, create_time ASC`.
3. Insert metadata with priority field, default priority value is `10`.
4. Add new config `kyuubi.batch.priority` for each batch priority.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5352 from zwangsheng/KYUUBI#5328.

Closes #5328

687ed1ed6 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
58621b557 [zwangsheng] fix comments
1bf81e75c [zwangsheng] fix style
7ed2551b3 [zwangsheng] update default priority desc & improve UT
21ceccb01 [zwangsheng] fix doc
27fc5e825 [zwangsheng] enrich desc
c0bbc0dfd [zwangsheng] fix style
6b8d0f091 [zwangsheng] fix comment
67eb2524d [zwangsheng] fix comment
e1705c34d [zwangsheng] Add config to control whether pick order by priority or not
129a46729 [zwangsheng] Add unit test for pickBatchForSubmitting
fcaf85d92 [zwangsheng] Fix unit test
f7ca2219e [zwangsheng] Fix unit test
8d4b276ff [wangsheng] fix code style
4c6b99090 [wangsheng] fix comments
654ad843a [zwangsheng] [KYUUBI #5328][V2] Kyuubi Server Pick Metadata job with priority

Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: wangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-16 22:43:02 +08:00
hezhao2
4bb67bdf3b
[KYUUBI #5381] Change the default metrics reporter to Prometheus
### _Why are the changes needed?_

Close #5381

change default metrics reporter to prometheus since Kyuubi 1.8

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5344 from zhaohehuhu/Improvement-0928.

Closes #5381

84f4c8208 [hezhao2] reset METRICS_REPORTERS for test case
b9ee5f711 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
86165a6fe [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
a3605b626 [hezhao2] set METRICS_PROMETHEUS_PORT to 0 for test cases
f1a4d2861 [hezhao2] restore version number for kyuubi.metrics.reporters in doc
dae40e1a2 [hezhao2] change default metrics reporter to prometheus

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-16 11:46:56 +08:00
Bowen Liang
74e52f6924 [KYUUBI #5409] [DOCS] Update config docs regeneratoin hints in developer guide
### _Why are the changes needed?_

- Update the regeneration hits for `setting.md` in `develop.md`, as the config doc`settings.md` is checked and able to regenearted by `dev/gen/gen_all_config_docs.sh` script.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5409 from bowenliang123/regen-settings.

Closes #5409

985f39e51 [Bowen Liang] update docs

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-10-12 15:11:18 +08:00
ITzhangqiang
e51095edaa
[KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
### _Why are the changes needed?_

The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.

This PR to fix the performance issue on our side.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No.

Closes #5400 from ITzhangqiang/KYUUBI_5365.

Closes #5365

dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations

Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-11 21:41:22 +08:00
Angerszhuuuu
79b147ad3a [KYUUBI #5362] Remove Spark 3.0 support for Authz
### _Why are the changes needed?_
To close #5362 .

Considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects.
Drop support EOLs of Spark 3 for the coming Spark 4.x era in kyuubi v1.9.
We will still do bugfix release for these spark3.0.x users.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5363 from AngersZhuuuu/KYUUBI-5362.

Closes #5362

d34cd6e2b [Angerszhuuuu] Update build.md
99f414bd4 [Angerszhuuuu] Update build.md
a5129e4f6 [Angerszhuuuu] Update build.md
6ee008cc5 [Angerszhuuuu] Update README.md
af792cc42 [Angerszhuuuu] Update master.yml
69b333161 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5362
528554e9c [Angerszhuuuu] Update IcebergCatalogPrivilegesBuilderSuite.scala
427ebd48d [Angerszhuuuu] Update DataMaskingForJDBCV2Suite.scala
64809a54f [Angerszhuuuu] update
f7d89fd9b [Angerszhuuuu] [KYUUBI-5362] Kyuubi remove Authz test for spark3.0.3

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-11 10:18:50 +08:00
mrtisttt
42f9c7c921 [KYUUBI #5347] [DOCS] Fix java code errors in quick_start_with_jdbc
### _Why are the changes needed?_

There were obvious errors in the case code, so these cases were fixed.

### _How was this patch tested?_

Just fix case code, no test need. Of course, I have run the case code and have confirmed that there is no problem.

Closes #5347 from mrtisttt/fix-hive-jdbc-example-code.

Closes #5347

ec6ecf758 [mrtisttt] [DOCS] Fix java code errors in quick_start_with_jdbc

Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-10-06 11:18:28 +08:00
mrtisttt
57b6773c92 [KYUUBI #5345] [DOCS] Fix non-kerberized code description in quick_start_with_jdbc
### _Why are the changes needed?_

In the `Getting Started with Hive JDBC`documentation, section`Connect to non-kerberized Kyuubi Server`, the code description is incorrect:
```
The below java code is using a keytab file to login and connect to Kyuubi server by JDBC.
```
Obviously, the sections and code here are describing how to connect to a kyuubi service without kerberos authentication.

### _How was this patch tested?_
Just fix code description, no test need.

Closes #5345 from mrtisttt/fix-non-kerberized-code-description.

Closes #5345

fa0418f00 [mrtisttt] [DOCS] Fix non-kerberized code description in quick_start_with_jdbc

Authored-by: mrtisttt <mrtisttt@126.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-30 00:16:00 +08:00
senmiaoliu
16ae852891
[KYUUBI #4994][REST] Support listing all engines
### _Why are the changes needed?_

close #4994

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5157 from lsm1/branch-kyuubi_4994.

Closes #4994

02e7eb5fa [senmiaoliu] use abbr a
6d001e519 [senmiaoliu] replace 'allengine' with 'all'
f9c548299 [senmiaoliu] fix style
df64e773e [senmiaoliu] fix style
6a7d40e63 [senmiaoliu] remove allenginecommand
6306dd8a2 [senmiaoliu] support list all engine

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-09-23 23:18:32 +08:00
Cheng Pan
cd325b48ae
[KYUUBI #5306] YarnApplicationOperation supports proxy user
### _Why are the changes needed?_

For the secured YARN cluster, the Kyuubi Server's user typically has no permission to kill the application. Proxy user or admin should be used instead.

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_acl.html#concept_yarn_app_acls__section_killing_an_app

> For YARN, the following three groups of users are allowed to kill a running application:
> - The application owner
> - A cluster administrator defined in yarn.admin.acl
> - A queue administrator defined in aclAdministerApps for the queue in which the application is running

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

Verified ADMIN mode in internal deployment. (output message is formatted for readable)
```
Error: Batch e351185f-1ed8-437a-91bf-da2174e611e2 failed:
{
    "id":"e351185f-1ed8-437a-91bf-da2174e611e2",
    "user":"da_music",
    "batchType":"SPARK",
    "name":"SparkPi",
    "appStartTime":0,
    "appId":"application_1694730881181_58306",
    "appUrl":"http://xxxx-rm-2.xxxx:8088/cluster/app/application_1694730881181_58306",
    "appState":"KILLED",
    "appDiagnostic":"Application application_1694730881181_58306 was killed by user yarn at 10.49.59.149",
    "kyuubiInstance":"kyuubi-1.kyuubi-headless.spark.svc.cluster.local:10099",
    "state":"CANCELED",
    "createTime":1695102138188,
    "endTime":1695102163341,
    "batchInfo":{}
}
```

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5306 from pan3793/kill-proxy-user.

Closes #5306

2b2e54307 [Cheng Pan] address comments
e7e9a9c57 [Cheng Pan] nit
9cf2afc61 [Cheng Pan] polish
ff82d1230 [Cheng Pan] polish
bf0057b41 [Cheng Pan] ApplicationManager supports proxy user

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-20 11:34:13 +08:00
Cheng Pan
e27ed44385
[KYUUBI #5273] Update release docs
### _Why are the changes needed?_

As title

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5273 from pan3793/doc-release.

Closes #5273

c4f96c69f [Cheng Pan] 1
2266af358 [Cheng Pan] nit
1d52c1a86 [Cheng Pan] Bump version for web-ui and update release docs

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-12 10:49:59 +08:00
Hanna Liashchuk
55feb2b58c [KYUUBI #5227] [DOCS] Added deploy mode info
### _Why are the changes needed?_

<img width="1080" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/cc118574-c752-449a-9b20-04d2cc59b118">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5227 from hanna-liashchuk/patch-2.

Closes #5227

baf391ba9 [Hanna Liashchuk]  fix DOC style
e46dfddc3 [Cheng Pan] Update docs/deployment/engine_on_kubernetes.md
0746a350a [hanna-liashchuk] upd
233bba00b [hanna-liashchuk] Added deploy mode info

Lead-authored-by: Hanna Liashchuk <g.liashchuk@temabit.com>
Co-authored-by: hanna-liashchuk <47921651+hanna-liashchuk@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-11 11:46:18 +00:00
Paul Lin
1f0901b5de [KYUUBI #5265] [DOCS][FLINK] Add notice about Jobmanager HA to Flink deployment docs
### _Why are the changes needed?_
As titled.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No.

Closes #5265 from link3280/docs_jm_ha.

Closes #5265

947854a5a [Paul Lin] Improve language
f7ea8dc51 [Paul Lin] Improve language
c40b94bf1 [Paul Lin] Add notice about Jobmanager HA to Flink deployment docs

Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
2023-09-09 01:01:03 +08:00
senmiaoliu
9ea0a1b35d
[KYUUBI #5244][Improvement] Make engineAliveMaxFailCount configurable
### _Why are the changes needed?_

close #5244
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5251 from lsm1/branch-kyuubi-5244.

Closes #5244

bcadaa53a [senmiaoliu] rename
a6c92773d [senmiaoliu] fix style
1f38fa711 [senmiaoliu] fix style
3ff57ff8b [senmiaoliu] Make engineAliveMaxFailCount configurable

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-09-08 10:52:32 +08:00
hezhao2
5ec2c2ee3f
[KYUUBI #5118][DOC] Add Flink KDF doc
### _Why are the changes needed?_
Add Flink KDF doc to close the issue #5118

<img width="1094" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/23cd886e-c94a-406b-81a2-eb2caf4c1775">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5233 from zhaohehuhu/Improvement-0901.

Closes #5118

fcd45b360 [hezhao2] code reformat
46bb9700c [hezhao2] Add Flink KDF doc

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
2023-09-07 14:49:57 +08:00
odone
8c512f467b
[KYUUBI #3444] Support the planOnly mode of kyuubi spark engine support SQL lineage
Closes #3444

### _Why are the changes needed?_

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3558 from iodone/kyuubi-3444.

Closes #3444

acaa72afe [odone] remove plugin dependency from kyuubi spark engine
739f7dd5b [odone] remove plugin dependency from kyuubi spark engine
1146eb6e0 [odone] kyuubi-3444

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-09-01 18:02:31 +08:00
Fantasy-Jay
19d54ec739
[KYUUBI #4540] Support to deny some users to make connection
### _Why are the changes needed?_

Improvement: https://github.com/apache/kyuubi/issues/4540

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5182 from zhuyaogai/issue-4540.

Closes #4540

73652ae6d [Fantasy-Jay] Merge remote-tracking branch 'origin/issue-4540' into issue-4540
ae7a3b5bb [Fantasy-Jay] Support to deny some user to make connection.
46ee2f158 [Fantasy-Jay] Support to limit some user to make connection.
3d7e5e534 [Fantasy-Jay] Support to limit some user to make connection.
1f173ffe4 [Fantasy-Jay] Merge branch 'master' into issue-4540
1d3a3d47f [Fantasy-Jay] Support to limit some user to make connection.
e4a1b3eb3 [Fantasy-Jay] Support to limit some user to make connection.
1c120a992 [Fantasy-Jay] Support to limit some user to make connection.
6b657aa99 [Fantasy-Jay] Support to limit some user to make connection.
506c4751b [Fantasy-Jay] Support to limit some user to make connection.
f99072ef5 [Fantasy-Jay] Support to limit some user to make connection.

Authored-by: Fantasy-Jay <13631435453@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-29 14:26:46 +08:00
liangbowen
beea6d5fce [KYUUBI #5185] [Improvement] Use Set collection for order-insensitive configs
### _Why are the changes needed?_

- Use `Set` collection for order-insensitive configs instead of `Seq`
  - kyuubi.frontend.thrift.binary.ssl.disallowed.protocols
  - kyuubi.authentication
  - kyuubi.authentication.ldap.groupFilter
  - kyuubi.authentication.ldap.userFilter
  - kyuubi.kubernetes.context.allow.list
  - kyuubi.kubernetes.namespace.allow.list
  - kyuubi.session.conf.ignore.list
  - kyuubi.session.conf.restrict.list
  - kyuubi.session.local.dir.allow.list
  - kyuubi.batch.conf.ignore.list
  - kyuubi.engine.deregister.exception.classes
  - kyuubi.engine.deregister.exception.messages
  - kyuubi.operation.plan.only.excludes
  - kyuubi.server.limit.connections.user.unlimited.list
  - kyuubi.server.administrators
  - kyuubi.metrics.reporters
- Support skipping blank elements

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5185 from bowenliang123/conf-toset.

Closes #5185

c656af78a [liangbowen] conf to set

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-08-25 10:28:15 +08:00
Paul Lin
e9ca8272b0 [KYUUBI #4806][FLINK] Support time-out incremental result fetch for Flink engine
### _Why are the changes needed?_
As titled.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5134 from link3280/KYUUBI-4806.

Closes #4806

a1b74783c [Paul Lin] Optimize code style
546cfdf5b [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperation.scala
b6eb7af4f [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/ResultSet.scala
1563fa98b [Paul Lin] Remove explicit StartRowOffset for Flink
4e61a348c [Paul Lin] Add comments
c93294650 [Paul Lin] Improve code style
6bd0c8e69 [Paul Lin] Use dedicated thread pool
15412db3a [Paul Lin] Improve logging
d6a2a9cff [Paul Lin] [KYUUBI #4806][FLINK] Implement incremental result fetching

Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
2023-08-24 11:58:08 +08:00
odone
fdfb8f6d66
[KYUUBI #5170] Identifier compatible with spark3.4
### _Why are the changes needed?_

close #5170

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5171 from iodone/kyuubi-5170.

Closes #5170

7cd0e2cbf [odone] identifier compatible with sprk3.4

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-16 16:07:54 +08:00
senmiaoliu
564ee509f0
[KYUUBI #5162][DOC] Add statements about contributions co-authored with AIGC in developer guide page
### _Why are the changes needed?_

close #5162

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

 No

Closes #5163 from lsm1/features/kyuubi-5162.

Closes #5162

c5bba8c20 [senmiaoliu] fix
af5e816fb [Kent Yao] Update docs/contributing/code/developer.md
ceb44358d [senmiaoliu] Add statements about contributions co-authored with AIGC in developer guide page

Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-16 11:58:39 +08:00
Cheng Pan
4e4a4e1153
[KYUUBI #5158] Allow embedded Zookeeper binding IP address
### _Why are the changes needed?_

This PR provides a new configuration `kyuubi.zookeeper.embedded.client.use.hostname` to control whether use hostname or IP address on binding, to improve the out-of-box experience of K8s case, similar to `kyuubi.frontend.connection.url.use.hostname`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5158 from pan3793/zk-ip.

Closes #5158

260dae987 [Cheng Pan] nit
3ac0c430e [Cheng Pan] doc
de50f65e8 [Cheng Pan] false
890b13bd9 [Cheng Pan] Allow embedded Zookeeper binding IP address

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-16 01:53:51 +08:00
hezhao2
9dcb61e00d
[KYUUBI #5153] [DOC] Minor change w/ to with
### _Why are the changes needed?_

Correct a typo in this doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5153 from zhaohehuhu/Improvement-0810.

Closes #5153

b2b593d94 [hezhao2] Update quick start hive jdbc instruction

Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-14 17:26:55 +08:00
liangbowen
ab1c46d431 [KYUUBI #5154] [Doc] Move configuration docs to the top level
### _Why are the changes needed?_

- Move the configuration docs to the top level of docs, which is most commonly used and referenced
- update relevant doc links

![image](https://github.com/apache/kyuubi/assets/1935105/a1dd35cc-d37f-4294-9fed-b275956c2cc5)

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5154 from bowenliang123/config-doc-first.

Closes #5154

b49ed3f8b [liangbowen] nit
db7f0d14d [liangbowen] update doc links
f8fd697a2 [liangbowen] move config docs to the top level
7448e4487 [liangbowen] change title of settings doc
40214ddd8 [liangbowen] move config doc in the front of deployment

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-08-11 18:23:08 +08:00
liangbowen
9a001c8d7c [KYUUBI #5120] [JDBC] Allow using session's user and password for connecting database in JDBC engine
### _Why are the changes needed?_

- allow using the session's user/password to connect database in the JDBC engine
- it is allowed to be applied on any share level of engines, since every kyuubi session maintains a dedicated JDBC connection in the current implementation

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5120 from bowenliang123/jdbc-user.

Closes #5120

7b8ebd137 [liangbowen] Use session's user and password to connect to database in JDBC engine

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-08-07 13:33:50 +08:00
Cheng Pan
ba99744b09
[KYUUBI #3374] Support password authentication for Trino engine
### _Why are the changes needed?_

Support SSL for trino engine.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3374 from hddong/support-trino-password.

Closes #3374

f39daaf78 [Cheng Pan] improve
6308c4cf7 [hongdongdong] Support SSL for trino engine

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-04 09:50:56 +08:00
Cheng Pan
7bc0dbf932
[KYUUBI #5130] Allow setting Zookeeper SPN using Kyuubi configuration
### _Why are the changes needed?_

One of the basic ideas of Kerberos authentication is that the Client MUST provide the Server Principal Name (SPN).

[ZOOKEEPER-1467](https://issues.apache.org/jira/browse/ZOOKEEPER-1467) (fixed in 3.6.0, 3.5.7) allows to configure Zookeeper SPN at the client side.

Previously, there is no way but only `-Dzookeeper.server.principal=zookeeper/_HOSTREALM` to set ZK SPN, this PR allows the user to set it via Kyuubi configuration `kyuubi.ha.zookeeper.auth.serverPrincipal`.

In default, if `zookeeper.server.principal` is not configured, it assumes that the Zookeeper server uses `zookeeper/_HOST<SAME_REALM_WITH_CLIENT_PRINCIPAL>`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5130 from pan3793/zk-spn.

Closes #5130

6f0a1bbdc [Cheng Pan] doc
290e172aa [Cheng Pan] docs
a43b0c67a [Cheng Pan] nit
9a33446b7 [Cheng Pan] Allow set Zookeeper server principal

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-03 19:29:02 +08:00
senmiaoliu
224ae7cc4f [KYUUBI #5122][DOC] Hive KDF usage
### _Why are the changes needed?_

close #5122

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5125 from lsm1/features/kyuubi_5122.

Closes #5122

02d0769cc [senmiaoliu] add hive kdf docs

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-08-03 09:39:49 +08:00
hezhao2
156628cf72
[KYUUBI #5083] [DOC] Add LDAP document
### _Why are the changes needed?_

I'd like to update LDAP doc to guide users for setup LDAP authentication in Kyuubi.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

<img width="1395" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/6925a8e3-dfaf-48ad-a442-bb635fe75830">

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5083 from zhaohehuhu/Improvement-0721.

Closes #5083

8c0e149dd [Cheng Pan] polish
22f8d3aa6 [Cheng Pan] nit
822fa66b3 [hezhao2] sync
78ae12345 [hezhao2] further explanation for LDAP filters
7ebc61acf [Cheng Pan] Update docs/security/ldap.md
bb06810f7 [Cheng Pan] Update docs/security/ldap.md
8d19fdf31 [Cheng Pan] Update docs/security/ldap.md
c2fa2806e [Cheng Pan] Update docs/security/ldap.md
2acbb87db [hezhao2] update LDAP doc
22027e1f2 [hezhao2] update LDAP doc

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-31 20:32:16 +08:00
senmiaoliu
3162050566
[KYUUBI #5076] Add KDF engine_url
### _Why are the changes needed?_

close #5076

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5102 from lsm1/features/kyuubi_5076.

Closes #5076

ce7cfe678 [senmiaoliu] kdf support engine url

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-31 13:52:21 +08:00
senmiaoliu
998b5d5349
[KYUUBI #5099] Document link failure
### _Why are the changes needed?_

close #5099

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5103 from lsm1/features/kyuubi_5099.

Closes #5099

84a1ecad0 [senmiaoliu] fix doc

Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-28 10:46:53 +08:00
pengqli
bbc3fdf5e5
[KYUUBI #5094] Add Python JayDeBeApi doc
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5094 from dev-lpq/add_python_doc.

Closes #5094

c7d50d75a [pengqli] upgrade Python-JayDeBeApi doc
41f96fc1b [pengqli] upgrade Python-JayDeBeApi doc
dd0f91bd6 [pengqli] upgrade Python-JayDeBeApi doc
ae1b7bc63 [pengqli] upgrade Python-JayDeBeApi doc
189d7c835 [pengqli] upgrade Python-JayDeBeApi doc
2e1e7b418 [pengqli] upgrade Python-JayDeBeApi doc
362a43296 [pengqli] add Python-JayDeBeApi doc

Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-26 18:10:27 +08:00
mans2singh
b04829a2fc
[KYUUBI #5085] Update config section of deploy on kubernetes
### _Why are the changes needed?_
Updated [kyuubi on kubernetes config section](https://kyuubi.readthedocs.io/en/master/deployment/kyuubi_on_kubernetes.html#config) to state <code> Kyuubi **does** not recommend using this way on Kubernetes</code>

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5086 from mans2singh/ISSUE-5085.

Closes #5086

5faf0df2e [mans2singh] [KYUUBI # 5085] Update config section based on review comments
df9f62f36 [mans2singh] [KYUUBI # 5085] Update config section of deploy on kubernetes

Authored-by: mans2singh <mans2singh@yahoo.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-24 22:38:03 +08:00
liupeiyue
14818cfb43 [KYUUBI #5065] Call destroy first on killing Spark startup process to allows it release temp files
### _Why are the changes needed?_

fix bug https://github.com/apache/kyuubi/issues/5065

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5066 from ASiegeLion/master.

Closes #5065

08d1ac077 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
bf908f5af [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
9144582f9 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
f1c95e409 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
907123a93 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
f30a9fc39 [liupeiyue] [KYUUBI-#5065] Call destroy first on killing Spark startup process to allows it release temp files
449be44d7 [文艺攻城狮] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala
987ffc7fe [文艺攻城狮] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
995386f98 [文艺攻城狮] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
ad3d11191 [liupeiyue] [KYUUBI-#5065]destroy the spark engine release the submitted temp files

Lead-authored-by: liupeiyue <liupeiyue@yy.com>
Co-authored-by: 文艺攻城狮 <945076608@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-20 17:58:40 +08:00
toshihiko.uchida
02fb525e11
[KYUUBI #5010] Make Kyuubi server's connection URL configurable
### _Why are the changes needed?_

This PR resolves https://github.com/apache/kyuubi/issues/5010, by introducing a new configuration property that makes it possible to override the hostname or ip of the Kyuubi server's thrift frontend service that will be registered for the service discovery ensemble.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

Before applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0`, the znode becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=0.0.0.0:10009;version=1.8.0-SNAPSHOT;sequence=0000000007]
```
After applying the patch, when we set `kyuubi.frontend.bind.host=0.0.0.0` and `kyuubi.frontend.advertised.host=some_kyuubi_host`, it becomes as follows.
```
[zk: some_zk_host:some_zk_port(CONNECTED) 0] ls /some_kyuubi_zk_namespace
[serviceUri=some_kyuubi_host:10009;version=1.8.0-SNAPSHOT;sequence=0000000009]
```

- [X] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

`./build/mvn clean install -pl kyuubi-common` succeeded.

Closes #5015 from touchida/issue-5010-connection-url.

Closes #5010

4598c2baf [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
bf53a8a13 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
0ddf09f0f [toshihiko.uchida] Use pattern matching instead of if-else
34c61d90d [toshihiko.uchida] Rename to kyuubi.frontend.advertised.host and apply to all frontends
7fd980baf [toshihiko.uchida] [KYUUBI #5010] Make Kyuubi server's thrift frontend connection URL configurable

Lead-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-17 11:47:29 +08:00
Xieming LI
ffd8852b60
[KYUUBI #5002] Fail the engine fast when no incoming connection in CONNECTION mode
### _Why are the changes needed?_
Please refer to #4997

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
1. connect to KyuubiServer with beeline
2. Confirm the Application is ACCEPTed in ResourceManager, Restart KyuubiServer
3. Confirmed that Engine was terminated shortly
```
23/06/28 10:44:59 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Current open session is 0
23/06/28 10:45:00 ERROR spark.SparkSQLEngine: Spark engine has been terminated because no incoming connection for more than 60000 ms, deregistering from engine discovery space.
23/06/28 10:45:00 WARN zookeeper.ZookeeperDiscoveryClient: This Kyuubi instance lniuhpi1616.nhnjp.ism:46588 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/06/28 10:45:00 INFO spark.SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
```

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5002 from risyomei/feature/failfast.

Closes #5002

402d6c01f [Xieming LI] Changed runInNewThread based on comment
58f11e157 [Xieming LI] Changed runInNewThread to non-blocking
c6bb02d6a [Xieming LI] Fixed Unit Test
168d996d0 [Xieming LI] Start countdown after engine is started
48ee819f2 [Xieming LI] Fixed a typo
a8d305942 [Xieming LI] Using runInNewThread ported from Spark
21f0671df [Xieming LI] Updated document
a7d5d1082 [Xieming LI] Changed the default value to turn off this feature
437be512d [Xieming LI] Trigger CI to test agagin
42a847e84 [Xieming LI] Added Configuration for timeout, changed to ThreadPoolExecutor
639bd5239 [Xieming LI] Fail the engine fast when no incoming connection in CONNECTION mode

Authored-by: Xieming LI <risyomei@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-16 23:00:16 +08:00
sychen
66e460dfe2 [KYUUBI #5049] [DOCS] PyHive Kerberos usage doc
### _Why are the changes needed?_

<img width="795" alt="image" src="https://github.com/apache/kyuubi/assets/3898450/2888dd28-a27f-491e-8552-72f5646cdb5f">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5049 from cxzl25/pyhive_kerberos.

Closes #5049

6c8b0e62b [sychen] pyhive kerberos usage doc

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-13 13:46:31 +08:00