### Why are the changes needed?
1. Persist the kubernetes application terminate info into metastore to prevent the event lose.
2. If it can not get the application info from informer application info store, fallback to get the application info from metastore instead of return NOT_FOUND directly.
3. It is critical because if we return false application state, it might cause data quality issue.
### How was this patch tested?
UT and IT.
<img width="1917" alt="image" src="https://github.com/user-attachments/assets/306f417c-5037-4869-904d-dcf657ff8f60" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#7029 from turboFei/kubernetes_state.
Closes#7028
9f2badef3 [Wang, Fei] generic dialect
186cc690d [Wang, Fei] nit
82ea62669 [Wang, Fei] Add pod name
4c59bebb5 [Wang, Fei] Refine
327a0d594 [Wang, Fei] Remove create_time from k8s engine info
12c24b1d0 [Wang, Fei] do not use MYSQL deprecated VALUES(col)
becf9d1a7 [Wang, Fei] insert or replace
d167623c1 [Wang, Fei] migration
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
This is the follow-up of #5686, renaming `./pyhive` to `./python`, and also adding `**/python/*` to RAT exclusion list temporarily.
"PyHive" may not be a suitable name after being part of Apache Kyuubi, let's use a generic dir name `python`, and discuss the official name later(we probably keep the code at `./python` eventually).
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Recover RAT checked.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6279 from pan3793/pyhive-1.
Closes#5686
42d338e71 [Cheng Pan] [KYUUBI #5686][FOLLOWUP] Rename pyhive to python
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This is the first step of #6146, to gain a clear commit history, this PR just simply copied the `hive-beeline` module from Apache Hive 3.1.3, with minimal change to pass the tests and manually test basic functionalities, following PRs are going to remove other Hive deps gradually.
## Describe Your Solution 🔧
- Copy source code and test case from Apache Hive 3.1.3
- Drop `org.apache.hive:hive-beeline:3.1.3`
- Backport HIVE-21584 to support JDK 9+
- Drop `HiveCli`, `HiveSchemaTool` and `BeelineInPlaceUpdateStream`, and the corresponding test cases
- Temporary ignore(will fix later) `TestClientCommandHookFactory#testConnectHook` because of error `NoClassDefFound org/apache/curator/RetryPolicy`
- Tune testing code to pass UT
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Minimal changes to pass the unit tests.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6109 from pan3793/fork-beeline.
Closes#6109
885f9fe2b [Cheng Pan] NOTICE
a2efa1c41 [Cheng Pan] fix
5bb1cc9f6 [Cheng Pan] Copy from Apache Hive 3.1.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close https://github.com/apache/kyuubi/issues/5484
Kyuubi provides a basic new Web UI which is built on top of Vue3, we can remove the legacy dummy Web UI in 1.8.
The new UI hosts at `http://<host>:<port>/ui/` and the legacy UI hosts at `http://<host>:<port>/`, we should
1. Remove the legacy UI routing from Jetty
2. Remove all files related to legacy UI
3. Redirect `http://<host>:<port>/` to `http://<host>:<port>/ui/`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
building with the command `build/dist --web-ui`, then `cd dist` and perform `bin/kyuubi run`
access http://0.0.0.0:10099 could correctly redirect to http://0.0.0.0:10099/ui/
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1e8a67f6-e4db-415e-8a47-dd7c41b487cf">
swagger is render correctly too.
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1cb4ba31-9965-4468-b7c3-b0319ba959e6">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5516 from pan3793/5484.
Closes#5484
9d58ef758 [Cheng Pan] address comment and fix test
6d4c0981b [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- To avoid the generated web-ui's coverage report files triggering Rat plugin checks on Maven
```
[WARNING] Files with unapproved licenses:
kyuubi-server/web-ui/coverage/index.html
kyuubi-server/web-ui/coverage/locales/en_US/index.html
kyuubi-server/web-ui/coverage/locales/zh_CN/index.html
...
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.15:check (default) on project kyuubi-parent: Too many files with unapproved license: 21 See RAT report in: /kyuubi/target/rat.txt ->
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5135 from bowenliang123/rat-exclude.
Closes#5135
1ec4411cb [liangbowen] add web-ui/coverage/* to rat exclusions
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
```
Usage:
+----------------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--web-ui] [--flink-provided] [--hive-provided] |
| [--spark-provided] [--mvn <maven_executable>] <maven build options> |
+----------------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
web-ui: - whether to include web ui
flink-provided: - whether to make a package without Flink binary
hive-provided: - whether to make a package without Hive binary
spark-provided: - whether to make a package without Spark binary
mvn: - external maven executable location
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Create binary artifacts using `build/dist --tgz --web-ui` and run, then open `http://0.0.0.0:10099/ui`
<img width="1401" alt="image" src="https://user-images.githubusercontent.com/26535726/220753103-ce801f12-f394-4ece-92a4-1902e93c62c7.png">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4397 from pan3793/webui-build.
Closes#4397
97901d63e [Cheng Pan] doc
37d5e2ad3 [Cheng Pan] mirror-cdn
c5751dd5b [Cheng Pan] remove unused dep
d308defb7 [Cheng Pan] nit
9abca4705 [Cheng Pan] nit
c1d184afd [Cheng Pan] nit
7091d5bf5 [Cheng Pan] regex
f0ac16b3c [Cheng Pan] [BUILD] `build/dist` support --web-ui
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4022 from lightning-L/kyuubi-3968.
Closes#3968
8560a2f57 [lightning_L] Merge pull request #2 from turboFei/kyuubi-3968
5f76107dd [fwang12] follow up
7f6cb1bce [lightning_L] Merge pull request #1 from turboFei/kyuubi-3968
cc0d6cbb8 [lightning_L] Merge branch 'apache:master' into kyuubi-3968
46ea82eff [fwang12] nit
11b1f8cf9 [fwang12] follow up
54fa3df89 [Tianlin Liao] fix NPE when folder does not exist
0353e708f [Tianlin Liao] update regex
a475f7b76 [Tianlin Liao] list all metadata store sql files and use the one with the largest version number
043b43b61 [Tianlin Liao] fix license
871b60e47 [Tianlin Liao] fix
ece1f6010 [Tianlin Liao] fix
40831c59f [Tianlin Liao] [KYUUBI #3968] Upgrading and migration script for Jdbc
Lead-authored-by: Tianlin Liao <tiliao@ebay.com>
Co-authored-by: lightning_L <tianlinliao@163.com>
Co-authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support Trino get schemas statement. We convert statement to operation at Trino frontend.
Note, this pr does not support the whole code path from Trino frontend. This pr aims to add an example how to make Trino frontend work with existed Kyuubi server framework.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4030 from ulysses-you/trino-get-schemas.
Closes#4029
f43408c2 [ulysses-you] address comments
a29bcdb8 [ulysses-you] fix
6fddbe6e [ulysses-you] support trino get schemas
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#3337
excludes `node_modules` folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3338 from zwangsheng/fix/linese_check_exclude_node_modules.
Closes#3338
80f45440 [zwangsheng] fix
4f310f70 [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is a pr about init frontend framework using vue3.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3250 from tongwl/KYUUBI-3248.
Closes#3250
2c0d7744 [William Tong] add licenses for files
4ab298fb [William Tong] transfer element-plus auto import to full import
4a9478c2 [William Tong] code change from as commented
a1aaba03 [William Tong] Init frontend framework
Authored-by: William Tong <weitong@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We need support to retry the jdbc requests.
Because we add new jdbc dependency for batch HA, and the database might be in maintenance window.
And there might be master-slave switch, and the database might be unavailable for few minutes.
We need tolerant this issue and unblock the main thread.
BTW, this pr refactor the metadata store naming.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2834 from turboFei/retrying_jdbc_event.
Closes#2834
109a63e70 [Fei Wang] fix docs
a186484be [Fei Wang] refactor the jdbc pool name
cfac9da53 [Fei Wang] rename metadata table name
bb8710ba3 [Fei Wang] comments
1197a098b [Fei Wang] add private package scope
a95694ba9 [Fei Wang] comments
e2f09e784 [Fei Wang] refactor
057eb73c5 [Fei Wang] refactor
88f3ffb6f [Fei Wang] refactor
e0595bdee [Fei Wang] comments
48270e869 [Fei Wang] refactor conf keys
e4f190c5c [Fei Wang] update docs
ed5e90e37 [Fei Wang] add max queues limitation
c5a03c9e2 [Fei Wang] remove getOrCreate
3e39ca94c [Fei Wang] remove
b79fb0954 [Fei Wang] comments
1d309cee0 [Fei Wang] comments
9108a3f0e [Fei Wang] fix docs
5cdffe710 [Fei Wang] fix rat
2e7228fc7 [Fei Wang] fix ut
520236d7b [Fei Wang] rename more
f66d68bcc [Fei Wang] merge into metadata manager
2f70a2f12 [Fei Wang] rename
80e6100fb [Fei Wang] refactor
09cad092a [Fei Wang] add ut
ea4eea74f [Fei Wang] refactor
6b00eb0b3 [Fei Wang] refactor to MetadataRequest
58f1c0587 [Fei Wang] commements
198feb53f [Fei Wang] add more docs
670a23561 [Fei Wang] refactor
a20d54df2 [Fei Wang] waitStateStoreRetryCompletion after super.close closes operation
97a763aef [Fei Wang] refactor RetryingStateStoreRequest
3e136cf34 [Fei Wang] [SUB-TASK][KPIP-4] Support to retry the session state operations if meet transient jdbc issue
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
thanks to the pr original author yaooqinn https://github.com/apache/incubator-kyuubi/pull/1670
After this pr, the endpoint of OpenAPI will change from`http://${ip}:${port}/openapi.json` to `http://${ip}:${port}/api/openapi.json`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2061 from cfmcgrady/pr-1670.
Closes#2061
969b7d3d [Fu Chen] update dev/dependencyList
ff4b6bc2 [Fu Chen] fix ut
207553b6 [Fu Chen] addressed comment
a3baefe0 [Fu Chen] new line at end of file
ff36f957 [Fu Chen] fix
bb6abb3f [Fu Chen] language
b22be2ec [Fu Chen] style
f0c9562f [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
d4ac4cba [Fu Chen] fix
cd5ee2b8 [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
1b108a04 [Kent Yao] ui
85e31519 [Kent Yao] ui
0c20fcff [Kent Yao] ui
c2abde93 [Kent Yao] ui
b4dff513 [Kent Yao] ui
43dffab8 [Kent Yao] Merge branch 'master' into web
7cd273a3 [Kent Yao] swagger ui enabled
518882e2 [Kent Yao] swagger ui enabled
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Implement KyuubiHistoryServerPlugin. #1816
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
**SparkUI:**
**SparkHistoryServer:**
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1820 from wForget/KYUUBI-1816.
Closes#1816
a3256920 [Wang Zhen] fix test
256363d8 [Wang Zhen] fix
0aecc305 [Wang Zhen] load kyuubiConf from sparkConf
86608fa5 [wForget] fix tests
2481721b [wForget] fix
d4f5825d [wForget] Merge remote-tracking branch 'origin/master' into KYUUBI-1816
18c18458 [wForget] fix checkstyle
c21f8b10 [wForget] [KYUUBI-1816] add AppHistoryServerPlugin services
0adf3634 [Wang Zhen] [KYUUBI-1816] Add KyuubiHistoryServerPlugin
8f5b1196 [Wang Zhen] [KYUUBI-1816] Refactor EngineEventsStore to get Events through ElementTrackingStore.
678396d1 [Wang Zhen] [KYUUBI-1816] Add SparkSQLEngineEventListener
fcc1a925 [Wang Zhen] [KYUUBI-1816] Add the @KVIndex property to the event, which will be used for KvStore
Lead-authored-by: Wang Zhen <wangzhen07@qiyi.com>
Co-authored-by: wForget <643348094@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Extract rat configurations from pom to file
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1717 from yaooqinn/rat.
Closes#1717
3f6836fe [Kent Yao] Extract rat configuation from pom to file
cb267b37 [Kent Yao] Extract rat configuation from pom to file
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
