Commit Graph

2964 Commits

Author SHA1 Message Date
Cheng Pan
dbb741fc5b
[KYUUBI #4557][HELM] Kyuubi server should bind Pod IP by default
### _Why are the changes needed?_

Fix #4557.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

After changing, the Kyuubi Server correctly binds the Pod IP.
```
KyuubiTBinaryFrontendService#64 - Starting and exposing JDBC connection at: jdbc:hive2://10.88.224.214:10009/
```

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4561 from pan3793/helm-host.

Closes #4557

018a3a067 [Cheng Pan] [KYUUBI #4557][HELM] Kyuubi server should bind Pod IP by default

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-23 19:07:57 +08:00
fwang12
351bab3676 [KYUUBI #4584] Post the session exception to EventBus in time
### _Why are the changes needed?_

When session exception thrown, post it to event bus in time.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4584 from turboFei/handle_post.

Closes #4584

920ee29a4 [fwang12] post exception in time

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-23 18:11:03 +08:00
liangbowen
b942656f00
[KYUUBI #4572] Bump Iceberg from 1.1.0 to 1.2.0
### _Why are the changes needed?_

- Iceberg 1.2.0 release notes: https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.2.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4572 from bowenliang123/iceberg-1.2.0.

Closes #4572

f4bb2a1d3 [liangbowen] bump iceberg from 1.1.0 to 1.2.0

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-23 18:08:51 +08:00
zwangsheng
cb962304e5 [KYUUBI #4563] [Improvement] Format error log output in case of metrics json file not found
### _Why are the changes needed?_

Before
```log
2023-03-20 10:02:04.718 ERROR org.apache.kyuubi.metrics.JsonReporterService: Error writing metrics to json file/pathi/metrics/report.json
```

After
```log
2023-03-20 10:02:04.718 ERROR org.apache.kyuubi.metrics.JsonReporterService: Error writing metrics to json file: /path/metrics/report.json
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4563 from zwangsheng/fix/format_json_file_not_found_error.

Closes #4563

7b37616a3 [Binjie Yang] Update kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/JsonReporterService.scala
daace3598 [zwangsheng] [Improvement] Format error log output in case of metrics json file not found

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-23 18:01:16 +08:00
fwang12
e39bddab91 [KYUUBI #4541] Support to customize the attributes to expose for Spark engine
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4541 from turboFei/expose.

Closes #4541

f882b4dda [fwang12] engine register attributes

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-23 13:00:49 +08:00
guanhua.lgh
3771dc9bf2
[KYUUBI #4583] [DOCS] Rename Flink Table Store to Apache Paimon (Incubating)
… `Connectors for Spark SQL Query Engine `

### _Why are the changes needed?_

To update docs.
Update Flink Table Store to Apache Paimon (Incubating) in docs `Connectors for Spark SQL Query Engine `

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4583 from huage1994/docss-3067.

Closes #4583

5eaf76b84 [guanhua.lgh] [DOC] Update Flink Table Store  to Apache Paimon (Incubating) in docs `Connectors for Spark SQL Query Engine `

Authored-by: guanhua.lgh <guanhua.lgh@alibaba-inc.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-23 13:00:08 +08:00
Fu Chen
acdfa6ce1d [KYUUBI #4581] [TEST] Add KyuubiCommands parse python snippets unit test
### _Why are the changes needed?_

to resolve https://github.com/apache/kyuubi/pull/4333#issuecomment-1442793799

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4581 from cfmcgrady/beeline-parser-python-ut.

Closes #4581

9bffa033b [Fu Chen] add KyuubiCommands parse python snippets unit test

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-03-22 23:09:24 +08:00
Alex
06f3884627 [KYUUBI #4575] Fix the empty last line may causes the session to exit directly
### _Why are the changes needed?_
If the last line of the file is a blank line, there may be a cmd whose content is empty but the length is not 0 after split by ";". When running to this cmd, it will return false, causing the session to exit directly.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
vim test.sql and write :
<img width="527" alt="image" src="https://user-images.githubusercontent.com/43542750/226647817-855d866e-836c-4c60-bc81-0bb192d3d6ff.png">
then start beeline and source this file successfully and session won't exit:
<img width="745" alt="image" src="https://user-images.githubusercontent.com/43542750/226652451-823891f3-2768-4165-8816-c052461d1f9c.png">
<img width="598" alt="image" src="https://user-images.githubusercontent.com/43542750/226650769-25a8becf-4655-41ec-bf91-7ae7e28b7d34.png">

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4575 from Kiss736921/fix_file_empty_last_line.

Closes #4575

ff3fcb189 [Alex] fix the empty last line may causes the session to exit directly

Authored-by: Alex <zoulimin@kanzhun.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-22 19:36:43 +08:00
odone
f9701b03d7 [KYUUBI #4579] Fix the unstable situation of test cases for Trino-Fronted-Service
### _Why are the changes needed?_

In the cancellation implementation of Trino frontend service, closing the operation will also close the session. There is no need to check the status of the operation here.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4579 from iodone/trino-fronted-test-fix.

Closes #4579

75e02dd1 [odone] Fix the unstable situation of test cases.

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-22 17:27:11 +08:00
phyyou
da000c6054
[KYUUBI #4576] Multi arch build for ARM support in Docker Hub
Close #4576

* Setup QEMU and Buildx.
* Use action docker/build-push-actionv4.
* build multi platform (amd64, arm64)
* build cache for faster build 

### _Why are the changes needed?_

Based on This [Disscussion](https://github.com/apache/kyuubi/discussions/4487) and [Issue](https://github.com/apache/kyuubi/issues/4576)

Update publish-snapshot-docker workflow

### _How was this patch tested?_
now test in progress...

![image](https://user-images.githubusercontent.com/34825352/226776684-4d57dd0e-141a-44a0-a023-8eb43b57bdba.png)

https://github.com/phyyou/_kyuubi/actions/runs/4485366649/jobs/7886838765

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4577 from phyyou/feat/buildx-at-docker-hub-snapshot.

Closes #4576

b75b05e9d [phyyou] chore(build): Chore on workflow yml
11353e7df [phyyou] build: Update publish snapshot docker workflow

Authored-by: phyyou <gydudwls@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-22 16:31:31 +08:00
Tianlin Liao
141919308e [KYUUBI #4523] support close engine session gracefully
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4545 from lightning-L/kyuubi-4523.

Closes #4523

2ee634cd5 [Tianlin Liao] modify conf description; add test case
12b930456 [Tianlin Liao] refer to HIVE-13415
c8a9db6a2 [Tianlin Liao] minor fix
9942d2515 [Tianlin Liao] move logic to SessionManager.closeSession
62b990b3e [Tianlin Liao] [KYUUBI #4523] support close engine session gracefully

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-22 13:42:03 +08:00
wForget
c4f3195bd6 [KYUUBI #3929] Refactor lineage plugin to add LineageDispatcher
### _Why are the changes needed?_

Refactor lineage plugin to add LineageDispatcher.

close #3929

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3919 from wForget/dev-lineage-dispatcher.

Closes #3929

5df2aa2f [wforget] add doc
98683ebc [wforget] fix
7b97b2e0 [wForget] rebase
4b046868 [wForget] separate LineageDispatcherType class file
e14cf838 [wForget] Refactor lineage plugin to add LineageDispatcher

Lead-authored-by: wForget <643348094@qq.com>
Co-authored-by: wforget <643348094@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-22 09:46:14 +08:00
zwangsheng
824aba6070
[KYUUBI #3646][Improvement][UI] Front-end style should bracket same line
### _Why are the changes needed?_

Should bracket same line.

According to https://github.com/npetruzzelli/prettier-config-standard, should set true to bracketSameLine and also drop jsxBracketSameLine, which been deprecated.

Close #3646

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

- [x] CI

Closes #4571 from zwangsheng/KYUUBI_4570.

Closes #3646

2b61d1173 [zwangsheng] [KYUUBI #3646] bracket same line

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-21 21:05:57 +08:00
Alex
e00f6b5bf4
[KYUUBI #4568] Make kyuubi-ctl doc enable variable 'release' automatic substitution
### _Why are the changes needed?_
Kyuubi-ctl doc kyuubi version not convert automatic. The version of doc is 1.8.0-SNAPSHOT, but  kyuubi version is 1.6.0-SNAPSHOT.

<img width="1911" alt="image" src="https://user-images.githubusercontent.com/43542750/226536654-23a28284-76ef-4c03-bb47-24146a92a94f.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1920" alt="image" src="https://user-images.githubusercontent.com/43542750/226536803-08a6d89e-e6fd-4134-b438-052d87fdb0a3.png">

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4568 from Kiss736921/optimize_ctl_doc.

Closes #4568

eb10a9e3b [Alex] Update docs/tools/kyuubi-ctl.rst
2006a59b3 [Alex] Update docs/tools/kyuubi-ctl.rst
2501e119f [Alex] convert kyuubi-ctl.md to kyuubi-ctl.rst to enable variable 'release' automatic substitution

Lead-authored-by: Alex <zoulimin@kanzhun.com>
Co-authored-by: Alex <43542750+Kiss736921@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-21 20:58:01 +08:00
Binjie Yang
7727a6267d [KYUUBI #4565] [UI] Fix out-date README.md about Development Project
### _Why are the changes needed?_

 Fix out-date `README.md` about `Development Project`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4565 from zwangsheng/doc/fix_web_ui_readme.

Closes #4565

a08698845 [Binjie Yang] Update kyuubi-server/web-ui/README.md
5b3e25313 [Binjie Yang] Update kyuubi-server/web-ui/README.md
10adf17fb [Binjie Yang] Update kyuubi-server/web-ui/README.md
a7ee5a4fe [zwangsheng] [UI] Fix out-date README.md about run dev
b4d4f6076 [zwangsheng] [UI] Fix out-date README.md about run dev

Lead-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Co-authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-21 20:49:50 +08:00
xu.guo
465e23a0f6
[KYUUBI #4508][BEELINE] Beeline should reset stdin after consuming init SQL file
### _Why are the changes needed?_

To fix #4508 .

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4510 from thomasg19930417/master.

Closes #4508

f8405eb93 [Cheng Pan] Update kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/KyuubiCommands.java
3ad46c1d6 [Cheng Pan] Update kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/KyuubiCommands.java
31fa80771 [xu.guo] Merge branch 'master' of https://github.com/thomasg19930417/kyuubi
41e090d66 [thomasgx] Merge branch 'apache:master' into master
ba82229a8 [xu.guo] Move initializeConsoleReader to KyuubiCommands#connect
e06927b78 [xu.guo] Replace create new  consoleReader instance with  call initializeConsoleReader
86f2e5078 [xu.guo] Fix code style
b0c472229 [xu.guo] Fix Beeline with -i run sql faild
042987aa6 [xu.guo] Fix Beeline with -i run sql faild

Lead-authored-by: xu.guo <xu.guo@brgroup.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: thomasgx <570736711@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-20 17:13:28 +08:00
Cheng Pan
391be269c3
[KYUUBI #4559] [CHAT] Pretty chat engine launch command
### _Why are the changes needed?_

```
2023-03-20 03:31:12.571 INFO org.apache.kyuubi.engine.EngineRef: Launching engine:
/Users/chengpan/.sdkman/candidates/java/current/bin/java \
	-Xmx1g \
	-cp /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-chat-engine/target/kyuubi-chat-engine_2.12-1.8.0-SNAPSHOT.jar:/Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-chat-engine/target/scala-2.12/jars/* \
	org.apache.kyuubi.engine.chat.ChatEngine \
	--conf kyuubi.session.user=chengpan \
	--conf hive.server2.thrift.resultset.default.fetch.size=1000 \
	--conf kyuubi.client.ipAddress=10.221.96.184 \
	--conf kyuubi.engine.submit.time=1679254587454 \
	--conf kyuubi.engine.type=CHAT \
	--conf kyuubi.ha.addresses=10.221.96.184:2181 \
	--conf kyuubi.ha.engine.ref.id=1a19e1e5-d8d3-49eb-bee0-3c806212134b \
	--conf kyuubi.ha.namespace=/kyuubi_1.8.0-SNAPSHOT_USER_CHAT/chengpan/default \
	--conf kyuubi.ha.zookeeper.auth.type=NONE \
	--conf kyuubi.server.ipAddress=10.221.96.184 \
	--conf kyuubi.session.connection.url=10.221.96.184:10009 \
	--conf kyuubi.session.real.user=chengpan
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4559 from pan3793/gpt-cmd.

Closes #4559

1ba70f02d [Cheng Pan] nit
34aecc1b0 [Cheng Pan] nit
f4815866d [Cheng Pan] nit
526c79f15 [Cheng Pan] nit
28bd9d9fa [Cheng Pan] [CHAT] Pretty chat engine launch command

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-20 17:06:54 +08:00
liangbowen
301a05af59 [KYUUBI #4558] [CHAT] Make ChatGPT model ID configurable
### _Why are the changes needed?_

- Make ChatGPT model ID configurable

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4558 from bowenliang123/chatgpt-model.

Closes #4558

63f8ee30d [liangbowen] nit
3012ccaaa [liangbowen] make chatgpt model configurable

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-19 22:27:58 +08:00
liangbowen
24e87ef21a [KYUUBI #4556] [CHAT] Refactor ChatGPTProvider to use openai-java client
### _Why are the changes needed?_

- use Java SDK `openai-java` for ChatGPT which is popular and listed in official website, https://github.com/TheoKanning/openai-java
- Focus on lifecycle in ChatGPTProvider, and prevent handling lower-level concepts in details, like POJO mapping, HTTP request handling.
- follow the changes from upstream changes from OpenAI

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4556 from bowenliang123/chatgpt-third.

Closes #4556

ecf1e2cf6 [liangbowen] manually add `openai-gpt3-java:*` and its dependency to LICENSE-binary
53b8375a5 [liangbowen] refactor ChatGPTProvider to use `openai-java` SDK

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-19 19:24:41 +08:00
Chao Chen
c03664b8d8
[KYUUBI #4517] [FLINK] Fix multiple executions lead to abnormal results on Flink 1.14
Fix bug in flink 1.14 version, multiple executions lead to abnormal results

### _Why are the changes needed?_

Fix bug in flink 1.14 version, multiple executions lead to abnormal results

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4517 from waywtdcc/flink1.14_result_ok_error.

Closes #4517

96ce6129c [Cheng Pan] ut
1bd9d1e2f [Cheng Pan] nit
5e5bccc91 [Cheng Pan] Migrate Flink engine Java code to Scala
4afb02064 [chenchao4] Fix bug in flink 1.14 version, multiple executions lead to abnormal results
3d5dc64c5 [chenchao4] Fix bug in flink 1.14 version, multiple executions lead to abnormal results
c084864bd [chenchao4] Fix bug in flink 1.14 version, multiple executions lead to abnormal results
954d76062 [chenchao4] Fix bug in flink 1.14 version, multiple executions lead to abnormal results
d63ec55f2 [chenchao4] Fix bug in flink 1.14 version, multiple executions lead to abnormal results

Lead-authored-by: Chao Chen <chenchao4@grgbanking.com>
Co-authored-by: chenchao4 <Chenchao123>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-19 17:12:07 +08:00
liangbowen
06e69ddb2b [KYUUBI #4295] Introduce super-linter action for linting JSON, XML, ENV files and bash_exec
### _Why are the changes needed?_

- introduce `action/super-linter` with multiple linters, (https://github.com/marketplace/actions/super-linter)
- using the smaller size version of image `github/super-linter/slim` for faster image pulling
- enable linting for `bash_exec` for checking executable of bash scripts, with fixed `tools/spark-block-cleaner/kubernetes/docker/entrypoint.sh`
- enable  integrity checks for JSON files in JSONC style
- enable integrity checks for XML files
- enable linting for ENV files

### _How was this patch tested?_
- [x] Pass CI jobs

Closes #4295 from bowenliang123/superlinter-action.

Closes #4295

321d24912 [liangbowen] update
497bf801c [liangbowen] use JSONC instead of JSON to support comments
43787e233 [liangbowen] use superlinter

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-18 21:54:45 +08:00
liangbowen
067c6010a6
[KYUUBI #4554] [CHAT] Code improvement in ChatGPTProvider
### _Why are the changes needed?_

- set authentication as default header in client construction instead of  request construction
- handle response's status code in scala style
- transforming config's long value to int with `.intValue` instead of `asInstanceOf` casting
- fix var name to `response`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4554 from bowenliang123/chatgpt-http.

Closes #4554

114484a4d [liangbowen] httpclient improvement in ChatGPTProvider

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-18 15:34:16 +08:00
sychen
6ded07974e
[KYUUBI #4548] Kyuubi Chat Engine supports Chinese questions and HTTP proxy
### _Why are the changes needed?_

- Support Chinese question
- Support proxy settings
- Support setting timeout

<img width="1228" alt="image" src="https://user-images.githubusercontent.com/3898450/225851246-8762a451-9743-4c1d-8a33-cc49a926dfec.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4548 from cxzl25/chatgpt_followup.

Closes #4548

1d5715442 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
7add6a733 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
55974f298 [sychen] fix
2d360e102 [sychen] typo
19b5d0814 [sychen] doc
bdf8e29b6 [sychen] 1.utf8;2.proxy;timeout

Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-17 21:48:58 +08:00
Cheng Pan
f1eb449c42
[KYUUBI #4544] Initial implement Kyuubi Chat Engine
### _Why are the changes needed?_

Introduce a brand new CHAT engine, it's supposed to support different backends, e.g. ChatGPT, 文心一言, etc.

This PR implements the following providers:

- ECHO, simply replies a welcome message.
- GPT: a.k.a ChatGPT, powered by OpenAI, which requires a API key for authentication. https://platform.openai.com/account/api-keys

Add the following configurations in `kyuubi-defaults.conf`
```
kyuubi.engine.chat.provider=[ECHO|GPT]
kyuubi.engine.chat.gpt.apiKey=<chat-gpt-api-key>
```

Open an ECHO beeline chat engine.
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=ECHO'
```

```
Connecting to jdbc:hive2://localhost:10009/
Connected to: Kyuubi Chat Engine (version 1.8.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Client (version 1.7.0)
Beeline version 1.7.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10009/> Hello, Kyuubi!;
+----------------------------------------+
|                 reply                  |
+----------------------------------------+
| This is ChatKyuubi, nice to meet you!  |
+----------------------------------------+
1 row selected (0.397 seconds)
```

Open a ChatGPT beeline chat engine. (make sure your network can connect the open API and configure the API key)
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=GPT'
```

<img width="1109" alt="image" src="https://user-images.githubusercontent.com/26535726/225813625-a002e6e2-3b0d-4194-b061-2e215d58ba94.png">

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4544 from pan3793/chatgpt.

Closes #4544

87bdebb6d [Cheng Pan] nit
f7dee18f3 [Cheng Pan] Update docs
9beb55162 [cxzl25] chat api (#1)
af38bdc7c [Cheng Pan] update docs
9aa6d83a6 [Cheng Pan] Initial implement Kyuubi Chat Engine

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: cxzl25 <cxzl25@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-17 13:39:30 +08:00
liangbowen
7709cd18a7 [KYUUBI #4504] [Authz] [Bug] Fix source table privilege requirement when querying permanent view in Spark 3.1 and below
### _Why are the changes needed?_

To fix #4504.
- add ut for querying the second column of permenant view with multi-column source table (in purpose for reproducing cases for Spark 3.1 and below)
- make user RuleAuthorization check only once, by marking `KYUUBI_AUTHZ_TAG` tag on plan's children and itself , to prevent penetration checks to source table of perm view

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4529 from bowenliang123/4504-authz-permview.

Closes #4504

b5f9bedcf [liangbowen] minor update
3efba2d23 [liangbowen] update ut and policy
a7584d9d9 [liangbowen] update
c0c11558e [liangbowen] update
ee208d71d [liangbowen] update
3c7804e76 [liangbowen] fix ut
5ab0d1a1f [liangbowen] check RuleAuthorization run only once

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-17 13:16:13 +08:00
odone
04155f2d69 [KYUUBI #4331] [KYUUBI #4431] Lineage supports Union and MergeInto
close #4331
close #4431

### _Why are the changes needed?_

Because `optimizedPlan` has been replaced by `analyzedPlan`, iceberg tables do not need to be additionally processed when new logical plans are generated during some optimization stages. They can be treated as regular tables.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4543 from iodone/fix-4431.

Closes #4331

Closes #4431

2adddcc5 [odone] fix `Union` and `MergeInto` bug

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-17 13:13:16 +08:00
zwangsheng
4d67dac437
[KYUUBI #4546] Fully exclude metrics dir from git
### _Why are the changes needed?_

Follow our attitude towards other excluded folders, exclude the entire `metrics` folder.
Like in `.gitignore`
```
embedded_zookeeper/
```

`metrics` dir will be created in source code when run `KyuubiServer` in local IDE.

Close #4546

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4547 from zwangsheng/KYUUBI_4546.

Closes #4546

8f13670e6 [zwangsheng] [KYUUBI #4546] Ignore full metrics dir in git version control

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-17 12:31:36 +08:00
zwangsheng
58a4f58a1f
[KYUUBI #3420][SPARK] Expose UI url on registering engine service
### _Why are the changes needed?_

see detail in KYUUBI #3436

Splitting it out of a giant PR https://github.com/apache/kyuubi/pull/4002 to perfect

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4459 from zwangsheng/3420.

Closes #3420

2fd9b3131 [zwangsheng] [KYUUBI #3420]Fouce on spark web ui
3fbee02b7 [zwangsheng] [KYUUBI #3420] With inter config and expose explicit conf
09841d17b [zwangsheng] [KYUUBI #3420] Fix style
b47c3e5d1 [zwangsheng] [KYUUBI #3420] Fix unit test
b3ab3e23b [zwangsheng] [KYUUBI #3420] Expose more spark engine info
69292abf4 [zwangsheng] [KYUUBI #3420] Using common test case
cf889b09b [zwangsheng] [KYUUBI #3420] Fix unit test
1a0ac582f [zwangsheng] [KYUUBI #3420] Fix style
daf41de2d [zwangsheng] [KYUUBI #4453] Add etcd test
ab1038369 [zwangsheng] [KYUUBI #3420 Add Spark Web UI Url in zk node]

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-16 22:11:11 +08:00
He Zhao
b7290dc6a9
[KYUUBI #4537][UI] Enable Vite proxy server for web ui development
### _Why are the changes needed?_

As title and close #4537

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

![popo_2023-03-16  16-03-22](https://user-images.githubusercontent.com/52876270/225552935-525d80d6-520f-4570-9aaf-adc4f97dea44.jpg)

Closes #4538 from zwangsheng/KYUUBI_4537.

Closes #4537

2cae68044 [zwangsheng] [KYUUBI #4537] fix
19fc9bd8b [zwangsheng] [KYUUBI #4537] Set 0.0.0.0 for mac & windows
e8cff7853 [zwangsheng] [KYUUBI #4537] Set 0.0.0.0 for mac & windows
8af446ab0 [zwangsheng] [KYUUBI #4537] Remove test code
2897749ef [zwangsheng] [KYUUBI #4537] Remove test code
e714637e2 [zwangsheng] [KYUUBI #4537][Improvement][UI] Enabled Vite Server Proxy for developer work in local with front and backend server

Lead-authored-by: He Zhao <hezhao2@cisco.com>
Co-authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-16 22:02:46 +08:00
maruilei
427df68312
[KYUUBI #4526][FOLLOWUP] Defer evaluation for GroupProvider
### _Why are the changes needed?_

As described in the discussion: https://github.com/apache/kyuubi/discussions/4512

Since the pr https://github.com/apache/kyuubi/pull/3897 , we introduced the feature of `Supplying pluggable GroupProvider`.

But now when we create a new EngineRef, no matter which engine share level is used, getGroupNames will always be called.

This may cause users to encounter some confusion stack trace when they do not use the GROUP engine share level.

So we solve the problem through moving the `sessionManager.groupProvider.primaryGroup` to the inside of `EngineRef.appUser`, to ensure that the feature of `Supplying pluggable GroupProvider` is only valid for the GROUP engine share level.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4527 from merrily01/branch-1.7-groupProvider.

Closes #4526

0a858a921 [maruilei] Address comments.
c63a0f97b [maruilei] [KYUUBI #4526][FOLLOWUP] Ensure that the feature of `Supplying pluggable GroupProvider` is only valid for the GROUP engine share level.

Authored-by: maruilei <maruilei@58.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-16 18:15:32 +08:00
fwang12
47376c9ff2 [KYUUBI #3739][FOLLOWUP][REST] Remove unused parameters in SessionsResource rest docs
### _Why are the changes needed?_

as title.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4521 from turboFei/rest_docs.

Closes #3739

589237ca6 [fwang12] save
8e7d9bec6 [fwang12] final
35b67ba1e [fwang12] doc
40163311f [fwang12] save
5711d3d72 [fwang12] fix
bab07902f [fwang12] fix ut
d58c2753a [fwang12] save
989eaffd5 [fwang12] remove
2960602bc [fwang12] remove outdate docs

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-16 17:04:02 +08:00
Yikf
709899f866 [KYUUBI #4532] [AUTHZ] Displays the columns involved in extracting the aggregation operator
### _Why are the changes needed?_

This PR aims to display the columns involved in extracting the aggregation operator.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4532 from Yikf/agg-authZ.

Closes #4532

6a3468f11 [Yikf] Displays the columns involved in extracting the aggregation operator

Authored-by: Yikf <yikaifei@apache.org>
Signed-off-by: Yikf <yikaifei@apache.org>
2023-03-16 14:26:20 +08:00
odone
25e7b22553 [KYUUBI #4330] Non-temporary views do not resolve to a specific real table
close #4330

### _Why are the changes needed?_

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4503 from iodone/kyuubi-4330.

Closes #4330

d2c48e7a [odone] Instead of `optimizedPlan` with `analyzedPlan`
12614d19 [odone] add skip permenent view support

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-16 12:58:41 +08:00
Yikf
41e9505722
[KYUUBI #4525][KSHC] Partitioning predicates should take effect to filter data
### _Why are the changes needed?_

This PR aims to close https://github.com/apache/kyuubi/issues/4525.

The root cause of this problem is that Apache Spark does predicate push-down in `V2ScanRelationPushDown`, but the spark-hive-connector does not apply push-down predicates for data filtering.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4528 from Yikf/KYUUBI-4525.

Closes #4525

a65a1873f [Yikf] Partitioning predicates should take effect to filter data

Authored-by: Yikf <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-16 10:12:44 +08:00
Alex
0f45f26a35
[KYUUBI #4492] Correct engine subdomain calculation in kyuubi-ctl
### _Why are the changes needed?_

The CONNECTION share level engines always use a UUID as the subdomain in registering path, kyuubi-ctl's wrong subdomain calculation causes a failure on listing engines.
```
./kyuubi-ctl list engine --zk-quorum xxx --namespace kyuubi --user xxx
```
```
2023-03-10 13:53:32.939 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2023-03-10 13:53:32.945 ERROR org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Failed to get service node info
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kyuubi_1.7.0_CONNECTION_SPARK_SQL/xxx/default
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1659) ~[zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230) ~[curator-framework-2.12.0.jar:?]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[curator-framework-2.12.0.jar:?]
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.12.0.jar:?]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216) ~[curator-framework-2.12.0.jar:?]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[curator-framework-2.12.0.jar:?]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[curator-framework-2.12.0.jar:?]
	at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.getServiceNodesInfo(ZookeeperDiscoveryClient.scala:214) ~[kyuubi-ha_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.util.CtlUtils$.getServiceNodes(CtlUtils.scala:63) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.util.CtlUtils$.$anonfun$listZkServerNodes$1(CtlUtils.scala:86) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.util.CtlUtils$.$anonfun$listZkServerNodes$1$adapted(CtlUtils.scala:80) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.util.CtlUtils$.listZkServerNodes(CtlUtils.scala:80) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cmd.list.ListCommand.doRun(ListCommand.scala:32) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cmd.list.ListCommand.doRun(ListCommand.scala:24) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
	at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala) ~[kyuubi-ctl_2.12-1.7.0.jar:1.7.0]
2023-03-10 13:53:32.949 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2023-03-10 13:53:32.952 INFO org.apache.zookeeper.ZooKeeper: Session: 0x500011010ecc76e closed
2023-03-10 13:53:32.953 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x500011010ecc76e
```
### _How was this patch tested?_

- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  buid model kyuubi-ctl and deploy to my environment and run : kyuubi-ctl list engine -zk xxx:2181 -n kyuubi -u xxx
  get correct result.

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4492 from Kiss736921/fix_list_engine_error.

Closes #4492

29b87ed57 [Alex] change param name engine to engineNode
9e87b0853 [Alex] optimize get cmd architecture and complete delete engine for all share level
20544c296 [Cheng Pan] ctl should handle server and engine nodes seperately
9000129df [Alex] fix list engine no node exception

Lead-authored-by: Alex <zoulimin@kanzhun.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-16 01:35:46 +08:00
Cheng Pan
6aac3e66fe
[KYUUBI #4513] Bump Kyuubi 1.7.0 in Playground
### _Why are the changes needed?_

Upgrade Kyuubi 1.7.0 and enable REST protocol

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4513 from pan3793/playground.

Closes #4513

96d25daf8 [Cheng Pan] expose 10099
d163353a9 [Cheng Pan] Bump Kyuubi 1.7.0 in Playground

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-15 20:57:31 +08:00
liangbowen
38cf59d47b [KYUUBI #4502] Reduce build concurency mvnd.minThreads in CI builds
### _Why are the changes needed?_

- to fix mvnd failure in `Install` step of `Sytle Check` job in (https://github.com/apache/kyuubi/actions/runs/4403008223/jobs/7711886893#step:7:1012)
- by lowering `mvnd.minThreads` to reduce memory pressure in concurrent builds

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4502 from bowenliang123/sytle-mvn.

Closes #4502

a3ddb6270 [liangbowen] lower `mvnd.minThreads`

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-15 17:08:59 +08:00
Cheng Pan
7b8ca1235c
[KYUUBI #4519] Update metadata when batch application state changed
### _Why are the changes needed?_

Kyuubi Server should update the metadata store when the batch application state changes, otherwise the peers see the outdated application state.

The following inconsistent happened on a dual Kyuubi Server deployment
```
23/03/15 11:15:36 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: null
23/03/15 11:15:41 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: null, appStatus: null
23/03/15 11:15:46 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:15:51 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:15:56 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:01 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:06 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:11 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:16 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:21 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING
23/03/15 11:16:26 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:31 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING
23/03/15 11:16:37 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:42 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING
23/03/15 11:16:47 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: PENDING
23/03/15 11:16:52 batchId: e0b06788-46c0-484e-8648-deb08fc5dbd2, appId: spark-934bd65c1de14fceb22411986dc5fe99, appStatus: RUNNING
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4519 from pan3793/batch-metadata.

Closes #4519

6b9b3e706 [Cheng Pan] Update metadata when batch application state changed

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-15 15:08:11 +08:00
Tianlin Liao
a027b6e0a1 [KYUUBI #4498] list sessions/operations with conditions
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4507 from lightning-L/kyuubi-4498.

Closes #4498

6e3101288 [Tianlin Liao] [KYUUBI #4498] list sessions/operations with conditions

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-15 14:04:14 +08:00
fwang12
ba0eac70ab [KYUUBI #4518] Return engine id in SessionData and post kyuubi instance in KyuubiOperationEvent
### _Why are the changes needed?_

As title.

BTW, for KyuubiSessionEvents, you can get kyuubiInstance from serverIP.
```
 * param serverIP A unique Kyuubi server id, e.g. kyuubi server ip address and port,
 *                 it is useful if has multi-instance Kyuubi Server
 ```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4518 from turboFei/instance_follow.

Closes #4518

2ad4c1646 [fwang12] event

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-15 14:03:32 +08:00
fwang12
e53da12419 [KYUUBI #4516] Return kyuubi instance in SessionData and OperationData
### _Why are the changes needed?_

It is helpful, because now kyuubi gateway does not support multiple nodes ha.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4516 from turboFei/kyuubi_instance.

Closes #4516

a679dc524 [fwang12] return kyuubi instance

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-15 11:13:07 +08:00
zwangsheng
b23c87c318
[KYUUBI #4467][K8S][BATCH] Tolerate Driver Pod ephemerally invisible after submitting
### _Why are the changes needed?_

The following discussion assumes using Spark cluster mode w/ `waitCompletion=false`.

In Spark on Yarn, the application is visible immediately after `spark-submit` is returned, but things are different in Spark on K8s, Driver Pod is ephemerally invisible after submitting, so NOT_FOUND is returned instead of UNKNOWN or PENDING.

To tolerate the above case, `kyuubi.engine.submit.timeout` is introduced, ApplicationManager will report UNKNOWN instead of NOT_FOUND during the Driver Pod scheduling period.

More detail in #4467
1. Remove `KubernetesApplicationOperation`'s `JpsApplicationOperation` for handle Client Deploy Mode(`YarnApplicationOperation` doesn't handle this either)
2. Add engine submit timeout for `KubernetesApplicationOperation` to return Unknown status when not found driver pod in time range.
3. GetApplicationInfo with it's submit time

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4469 from zwangsheng/4467.

Closes #4467

562b67463 [zwangsheng] [KYUUBI #4467] Fix Setting.md
362c43d1b [zwangsheng] [KYUUBI #4467] Fix Setting.md
ac69f4d81 [zwangsheng] [KYUUBI #4467] Add Config Desc
d2b9fb660 [zwangsheng] [KYUUBI #4467] save tab
eac880fcf [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
7a20b97a4 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
aa4c7716a [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
c5bd888ab [zwangsheng] [KYUUBI #4467] note it test
a86dcefba [zwangsheng] [KYUUBI #4467] Using default none
aed7f8794 [Cheng Pan] Update docs/deployment/settings.md
490df7dc0 [zwangsheng] [KYUUBI #4467] fix complie
33f3a5be8 [zwangsheng] [KYUUBI #4467] fix comments
4745790cf [zwangsheng] [KYUUBI #4467] Fix IT Test
924cfe38e [zwangsheng] [KYUUBI #4467] Fix Setting.md
5f8aeaacc [zwangsheng] [KYUUBI #4467] KubernetesApplicationOperation Wait if not fount driver pod in limit time range

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-14 23:01:43 +08:00
Kent Yao
8267108e7f
[KYUUBI #4497] [TEST] Add RowFilteringTestBase to improve the test coverage
### _Why are the changes needed?_

Add RowFilteringTestBase to improve the test coverage

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4497 from yaooqinn/rf.

Closes #4497

4d1834b71 [Kent Yao] [TEST] Add RowFilteringTestBase to improve the test coverage
f661fc902 [Kent Yao] [TEST] Add RowFilteringTestBase to improve the test coverage

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-14 14:21:17 +08:00
fwang12
d626c9fbd8 [KYUUBI #4501] Allow administrator to specify the user for engine admin
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4501 from turboFei/engine_admin.

Closes #4501

82a0b9687 [fwang12] get username

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-14 14:12:16 +08:00
minyk
5c93a8c199
[KYUUBI #4496] Call super.clusterManager() when batchConf does not have spark.master
### _Why are the changes needed?_

`defaultMaster` finds `spark.master` config from the `spark-defaults.conf` file directly, so it bypasses spark configurations from the `kyuubi-defaults.conf`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4505 from minyk/kyuubi-4496.

Closes #4496

7bfc6f844 [minyk] call super.clusterManager() instead of defaultMaster

Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-14 14:06:53 +08:00
fwang12
316c9fd537 [KYUUBI #4506] Return sessionType in session data
### _Why are the changes needed?_

As title.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4506 from turboFei/session_data_type.

Closes #4506

9617ee68c [fwang12] Session type

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2023-03-14 14:01:49 +08:00
miaowang
f420238f4f
[KYUUBI #4494] bin/kyuubi should use exec to run Kyuubi server
### _Why are the changes needed?_

The difference between w/ and w/o `exec` in shell script

https://www.baeldung.com/linux/exec-command-in-shell-script

> When using exec, however, the command following exec replaces the current shell. This means no subshell is created and the current process is replaced with this new command.

We must do this change to make `bin/kyuubi` suitable for Cloudera Manager.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4494 from wangmiao1002/master.

Closes #4494

ed9320447 [miaowang] Modify script run_kyuubi mode

Authored-by: miaowang <miaowang@gaojihealth.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-11 00:19:04 +08:00
Kent Yao
1bef628ab3 [KYUUBI #4493] [AUTHZ][TEST] Enable Tests for Spark 3.1 with iceberg tables
### _Why are the changes needed?_

Enable Tests for Spark 3.1 with iceberg tables

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4493 from yaooqinn/it.

Closes #4493

b40e9de05 [Kent Yao] [AUTHZ][TEST] Enable Tests for Spark 3.1 with iceberg tables
29a94addd [Kent Yao] [AUTHZ][TEST] Enable Tests for Spark 3.1 with iceberg tables

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-10 23:04:04 +08:00
Alex
454aed67d7 [KYUUBI #4491] Fix Trino typo
### _Why are the changes needed?_
fix trino typos

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4491 from Kiss736921/fix_trino_typos.

Closes #4491

73eb20f3a [Alex] fix trino typos

Authored-by: Alex <zoulimin@kanzhun.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-10 09:16:21 +00:00
liangbowen
61a6096043 [KYUUBI #4489] [REST] Fix missing org.apache.commons.logging by including jcl-over-slf4j dependency in REST client
### _Why are the changes needed?_

- To fix missing package `org.apache.commons.logging` used by  `org.apache.httpcomponents:httpclient`in `kyuubi-rest-client`

![image](https://user-images.githubusercontent.com/1935105/224203940-246db855-0ffa-469e-8a67-58143e6e99e3.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4489 from bowenliang123/httpclient-commonlogging.

Closes #4489

ccaff9b62 [liangbowen] Include `jcl-over-slf4j` dependency
740e56158 [liangbowen] Revert "skip excluding commons-logging in rest-client"
4d9ad5dcf [liangbowen] skip excluding commons-logging in rest-client

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-10 11:05:32 +08:00