### _Why are the changes needed?_
This PR adds a basic Grafana Dashboard template, also updates the metrics docs to guide users to use Prometheus and Grafana to monitor the Kyuubi server.
The Grafana Dashboard template is exported from the Grafana OSS v11.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1484" alt="image" src="https://github.com/user-attachments/assets/417b35fa-cd12-4e51-b73f-2955282aa187" />
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5147 from zhaohehuhu/Improvement-0809.
Closes#5834
f6fc2d71e [Cheng Pan] fix style
465f0546a [Cheng Pan] update dashboard
3fa2d237e [hezhao2] add status chart
4b2bd3dbc [hezhao2] add status chart
185f2cccf [hezhao2] make it compatible with kyuubi 1.8
457085be5 [hezhao2] add REAMDE.md to guide users
45e3ba3e5 [hezhao2] add docker file build a grafana image and load dashboards available
dbc22108b [hezhao2] Add Grafana dashboard template
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Apache Iceberg 1.6.1 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1
## Describe Your Solution 🔧
In the project POM file, I have updated the Apache Iceberg version from 1.6.0 to 1.6.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6659 from pionCham/upgrate-iceberg-version.
Closes#6659
923019440 [chengpeiming] Update the docs for Spark connector
433981e66 [chengpeiming] Supplement other configurations
1617e36fe [chengpeiming] Upgrate iceberg.version in pom.xml
Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
# 🔍 Description
Upgrade to latest Apache Iceberg 1.6.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass existing GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6558 from pan3793/iceberg-1.6.0.
Closes#6558
75e42938f [Cheng Pan] nit
ba508b6bb [Cheng Pan] nit
0180521aa [Cheng Pan] Bump Iceberg 1.6.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6323
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6333 from lsm1/branch-bump-spark-3.4.3.
Closes#6323
cc324fd77 [senmiaoliu] bump spark 3.4.3
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
# 🔍 Description
## Issue References 🔗
## Describe Your Solution 🔧
Update Helm Chart and playground to use 1.9.0 after release.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6195 from zwangsheng/update_helm_playground.
Closes#6195
c553d4c21 [Binjie Yang] Update Helm Chart and playground to use 1.9.0
Authored-by: Binjie Yang <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Promote the latest release version 1.8.1
## Describe Your Solution 🔧
Update Helm Chart and playground to use 1.8.1
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6076 from pan3793/r-181.
Closes#6076
aada476c2 [Cheng Pan] Update Helm Chart and playground to use 1.8.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5944
## Describe Your Solution 🔧
Add prometheus and grafana docker image to docker compose yaml, and by defaults, prometheus will get kyuubi server's metrics and grafana set prometheus as default datasource.
Help developers quickly build a dashboard development environment.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
In local test
- [x] all container start success
- [x] By defaults, prometheus get kyuubi's metrics
- [x] Grafana take the prometheus as default datasource
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#5945 from zwangsheng/KYUUBI#5944.
Closes#5944
031deedeb [Binjie Yang] Update docker/playground/README.md
dbf41e37d [zwangsheng] declare metrics related config
e6efe3071 [zwangsheng] make grafana as anonymous
404dd42ff [zwangsheng] fix comments
f7ca4ae1c [zwangsheng] fix prometheus
089f8d08c [zwangsheng] eof
68720788e [zwangsheng] add license header
cfa6edfef [zwangsheng] [KYUUBI #5944][Improvement] Introduce Prometheus and Grafana for Kyuubi Playground
Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: Binjie Yang <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#5856
## Describe Your Solution 🔧
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5859 from zml1206/KYUUBI-5856.
Closes#5856
872fd06d2 [zml1206] Revert changes in SparkProcessBuilderSuite
bc4996f90 [zml1206] Bump spark 3.4.2
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Promote [1.8.0](https://kyuubi.apache.org/release/1.8.0.html), the docker images already uploaded to DockerHub
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Go to `docker/playground`, and use `docker compose up -d` to run compose services as daemon
```
docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/tpcds/tiny'
```
```
0: jdbc:hive2://0.0.0.0:10009/tpcds/tiny> select kyuubi_version();
2023-11-07 06:12:57.332 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[41fe38d5-dc3f-4fc3-8a25-3e3c4200ce31]: PENDING_STATE -> RUNNING_STATE, statement:
select kyuubi_version()
23/11/07 06:12:57 INFO ExecuteStatement: Processing anonymous's query[41fe38d5-dc3f-4fc3-8a25-3e3c4200ce31]: PENDING_STATE -> RUNNING_STATE, statement:
select kyuubi_version()
23/11/07 06:12:57 INFO ExecuteStatement:
Spark application name: kyuubi_USER_SPARK_SQL_anonymous_default_1d81daf7-9136-4b0d-97f6-2cd71bcb74fb
application ID: local-1699337554479
application web UI: http://0.0.0.0:4040
master: local[*]
deploy mode: client
version: 3.4.1
Start time: 2023-11-07T06:12:33.491
User: anonymous
23/11/07 06:12:57 INFO ExecuteStatement: Execute in incremental collect mode
23/11/07 06:12:57 INFO CodeGenerator: Code generated in 14.080013 ms
23/11/07 06:12:57 INFO ExecuteStatement: Processing anonymous's query[41fe38d5-dc3f-4fc3-8a25-3e3c4200ce31]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.126 seconds
2023-11-07 06:12:57.461 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[41fe38d5-dc3f-4fc3-8a25-3e3c4200ce31] in FINISHED_STATE
2023-11-07 06:12:57.461 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[41fe38d5-dc3f-4fc3-8a25-3e3c4200ce31]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.129 seconds
+-------------------+
| kyuubi_version() |
+-------------------+
| 1.8.0 |
+-------------------+
1 row selected (0.329 seconds)
0: jdbc:hive2://0.0.0.0:10009/tpcds/tiny>
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5640 from pan3793/play-180.
Closes#5640
f8c6cf19d [Cheng Pan] Upgrade playground to Kyuubi 1.8.0 and Spark 3.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Follow [Kyuubi Release Guide](https://kyuubi.readthedocs.io/en/master/community/release.html) to update versions for playground and helm charts
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5324 from wForget/dev.
Closes#5324
7110e56bd [wforget] Bump latest 1.7.3 in playground and helm charts
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi v1.7.2 is available now, follow the [release guide](https://kyuubi.readthedocs.io/en/master/community/release.html#keep-other-artifacts-up-to-date) to update version in playground and helm charts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5298 from pan3793/172.
Closes#5298
8af52636e [Cheng Pan] Bump latest 1.7.2 in playgound and helm charts
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Upgrade to latest Kyuubi in playground.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
Successfully published docker image via `BUILDX=1 docker/playground/build-image.sh`
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5266 from pan3793/playground-171.
Closes#5266
17015e4d9 [Cheng Pan] remove iceberg catalog
a060f7067 [Cheng Pan] spark 3.3.3
c3176049b [Cheng Pan] fix
b11d0bb0f [Cheng Pan] nit
dc6a2b905 [Cheng Pan] Upgrade playground to Kyuubi 1.7.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Hadoop 3.3.5 brings back ARM64 support.
https://www.mail-archive.com/common-devhadoop.apache.org/msg39131.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4601 from pan3793/playground-hadoop.
Closes#4601
765def7a5 [Cheng Pan] Bump Hadoop 3.3.5 for playground
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Bump Iceberg 1.2.0 for playground
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4600 from pan3793/playground-iceberg.
Closes#4600
064828102 [Cheng Pan] Bump Iceberg 1.2.0 for playground
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Upgrade Kyuubi 1.7.0 and enable REST protocol
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4513 from pan3793/playground.
Closes#4513
96d25daf8 [Cheng Pan] expose 10099
d163353a9 [Cheng Pan] Bump Kyuubi 1.7.0 in Playground
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4338 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4339 from a49a/bump-spark332.
Closes#4338
6c741d82 [Luning Wang] [KYUUBI #4338] Bump Spark from 3.3.1 to 3.3.2
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR upgrades Kyuubi playground components to latest versions, including Spark 3.3.1, Kyuubi 1.6.1-incubating, Iceberg 1.1.0, and reorganizes the files in catalog folders.
The images has been uploaded to Docker Hub.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
cd docker/playground
docker compose up -d
```
```
docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/tpcds/tiny'
```
<img width="1416" alt="image" src="https://user-images.githubusercontent.com/26535726/204845474-b0dff173-d335-43bf-b61f-d8c34ee26ff8.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3881 from pan3793/playground.
Closes#3881
a3d42377 [Cheng Pan] Upgrade playground components to latest versions
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add [Docker Compose](https://docs.docker.com/compose/)-based Kyuubi Playground, w/ basic docs.
- Only support Spark engine for now.
- The images are not part of the official apache binary release, so we don't have license issue.
- The images will be published to the docker hub under `nekyuubi` namespace, which is managed by one of the PPMC members, but not the whole PPMC.
- The images support both x86 & arm platforms.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1408" alt="image" src="https://user-images.githubusercontent.com/26535726/189531044-50812bc6-43d1-43ab-8ec7-fb160ad805b3.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3473 from pan3793/play.
Closes#3473
47b6c753 [Cheng Pan] doc
0325b646 [Cheng Pan] license header
89b07907 [Cheng Pan] incremental
76dc08ec [Cheng Pan] nit
a0a58915 [Cheng Pan] docs
763a2240 [Cheng Pan] arg
25a9bfd1 [Cheng Pan] nit
11d92c16 [Cheng Pan] Docker Compose-based Kyuubi Playground
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
