# 🔍 Description
## Issue References 🔗
To close#5739
This pull request fixes #
## Describe Your Solution 🔧
Fix a spark extension import issue
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested
**Be nice. Be informative.**
Closes#5740 from AngersZhuuuu/KYUUBI-5739.
Closes#5739
cdb824c70 [Angerszhuuuu] [KYUUBI #5739][Bug] Fix spark extension pom import issue
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Bumps [io.netty:netty-codec-http2](https://github.com/netty/netty) from 4.1.93.Final to 4.1.100.Final.
<details>
<summary>Commits</summary>
<ul>
<li><a href="58df783eb4"><code>58df783</code></a> [maven-release-plugin] prepare release netty-4.1.100.Final</li>
<li><a href="58f75f665a"><code>58f75f6</code></a> Merge pull request from GHSA-xpw8-rcwv-8f8p</li>
<li><a href="491144865a"><code>4911448</code></a> Do not fail when compressing empty HttpContent (<a href="https://redirect.github.com/netty/netty/issues/13655">#13655</a>)</li>
<li><a href="caca5e5a1e"><code>caca5e5</code></a> When read PoolSubpage's variant fields, it should lock on PoolSubpage's head ...</li>
<li><a href="d97f2a5606"><code>d97f2a5</code></a> Update checkout action to latest version (<a href="https://redirect.github.com/netty/netty/issues/13649">#13649</a>)</li>
<li><a href="275341f01c"><code>275341f</code></a> Fix issue with unrecognized JVM option while running with Java 11 (<a href="https://redirect.github.com/netty/netty/issues/13648">#13648</a>)</li>
<li><a href="5db037beed"><code>5db037b</code></a> Speedup max direct memory estimation via Unsafe (<a href="https://redirect.github.com/netty/netty/issues/13643">#13643</a>)</li>
<li><a href="ce5c78cec1"><code>ce5c78c</code></a> Update actions to the latest version (<a href="https://redirect.github.com/netty/netty/issues/13644">#13644</a>)</li>
<li><a href="d7a8169f1b"><code>d7a8169</code></a> [maven-release-plugin] prepare for next development iteration</li>
<li><a href="8e3fe28425"><code>8e3fe28</code></a> [maven-release-plugin] prepare release netty-4.1.99.Final</li>
<li>Additional commits viewable in <a href="https://github.com/netty/netty/compare/netty-4.1.93.Final...netty-4.1.100.Final">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.
Closes#5395 from dependabot[bot]/dependabot/maven/io.netty-netty-codec-http2-4.1.100.Final.
Closes#5395
51028b289 [Cheng Pan] remove obsolete comments
a8f916c0f [Bowen Liang] update dependencyList
6813ccb4e [dependabot[bot]] Bump io.netty:netty-codec-http2 from 4.1.93.Final to 4.1.100.Final
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Apache parent pom version 31 release note: https://github.com/apache/maven-apache-parent/releases/tag/apache-31
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5676 from bowenliang123/apache-parent-31.
Closes#5676
064fc9da8 [Bowen Liang] bump apache parent pom version from 30 to 31
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Close#5375
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5416 from ymZhao1001/jdbc-pg-dialect.
Closes#5375
d81988d84 [zhaoyangming] postgreSQL
915f9fb0a [yangming] change to like
d8da12af5 [yangming] reformat
29c63e38e [zhaoyangming] add postgresql dependency
ec328ad93 [zhaoyangming] add postgresql dependency
a8944fed5 [zhaoyangming] update postgresql to postgreSQL
cf7b69107 [zhaoyangming] Merge remote-tracking branch 'origin/jdbc-pg-dialect' into jdbc-pg-dialect
c127aa3d3 [zhaoyangming] update postgresql to postgreSQL
a693d6c34 [yangming] reformat
0d12a6ceb [zhaoyangming] add postgresql dependency
c7d3fa3da [yangming] fix conflict
dde1564b6 [zhaoyangming] add test info
2a49b338a [zhaoyangming] style
c8ce15f29 [zhaoyangming] StringBuilder is redundant.
5d70173cf [yangming] JDBC Engine supports PostgreSQL
Lead-authored-by: zhaoyangming <zhaoyangming@deepexi.com>
Co-authored-by: yangming <261635393@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
resolve: #5505
Show available commands when users type 'HELP;' in the beeline.
#### Solution:
- Using `ExtendedParser` parse statement and return Operation of Flink engine.
- Check whether the operation is HelpOperation or not.
- dependency on `flink-table-planner-loader.jar`.
#### **Why not using Flink SQL Client Parser(SqlCommandParserImpl) to obtain the Command enum?**
Flink 1.16's approach:
```
val opt:Optional[Operation] = org.apache.flink.table.client.cli.SqlCommandParserImpl.parseCommand()
check opt.get() instance of HelpOperation or not
if yes return CliStrings.MESSAGE_HELP
```
Flink 1.17 & 1.18
```
val opt: Optional[Command] = org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement()
check opt.get() is Command.HELP or not
if yes return CliStrings.MESSAGE_HELP
```
The `Command` class is added from Flink1.17;
The `SqlCommandParserImpl` package is changed, and the method name is changed from Flink1.17;
This approach requires distinguishing between different Flink versions and maintaining both implementations.
It's more complicated, so abandoned.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5585 from YesOrNo828/help.
Closes#5505
e73b15e43 [Xianxun Ye] [KYUUBI #5505] [FLINK] Add job,update,delete,truncate and call statements
5943dd072 [Xianxun Ye] [KYUUBI #5505] [FLINK] Add help messages
fdc2db6ab [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command, fixed Pan's comments
a728048fc [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command, fixed Pan's comments
6323cfb85 [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command
Authored-by: Xianxun Ye <yesorno828423@gmail.com>
Signed-off-by: Paul Lin <paullin3280@gmail.com>
### _Why are the changes needed?_
Update codecov token to fix stale code coverage action
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
no
Closes#5501 from yaooqinn/token.
Closes#5501
1f4053231 [Kent Yao] java-17
fcd8391e7 [Kent Yao] java-17
5ccead77d [Kent Yao] upgrade jacoco
5cb4a8374 [Kent Yao] Revert "rm"
45d7a2134 [Kent Yao] rm
0a3dd2f42 [Kent Yao] Update codecov token
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Updated the download maven plugin from 1.6.8 to 1.7.1 to solve #5508
- release note: https://github.com/maven-download-plugin/maven-download-plugin/releases/tag/1.7.1
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes#5525 from rojin254/patch-1.
Closes#5508
2a628b40e [Rojin Raju] Update download maven plugin version to 1.7.1 in pom.xml
Lead-authored-by: Rojin Raju <rojinraju7@gmail.com>
Co-authored-by: Rojin Raju <46747837+rojin254@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Bump Kafka client version from 3.4.0 to 3.5.1
- Bump Docker image for Kafka 3 to 3.5.0 (equivalent to Confluent Platform 7.5.x)
- https://docs.confluent.io/platform/current/installation/versions-interoperability.html#cp-and-apache-ak-compatibility
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5486 from bowenliang123/bump-kafka.
Closes#5486
017ea860d [Bowen Liang] update dependencyList
20ad988ed [liangbowen] bump kafka client version 3.5.1
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Resolve: #5405 Support the Flink 1.18
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5465 from YesOrNo828/flink-1.18.
Closes#5405
a0010ca14 [Xianxun Ye] [KYUUBI #5405] [FLINK] Remove flink1.18 rc repo
2a4ae365c [Xianxun Ye] Update .github/workflows/master.yml
d4d458dc7 [Xianxun Ye] [KYUUBI #5405] [FLINK] Update the flink1.18-rc3 repo
99172e3da [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage
4c0cf887b [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage
c74f5c31b [Xianxun Ye] [KYUUBI #5405] [FLINK] fixed Pan's comments.
1933ebadd [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18
Authored-by: Xianxun Ye <yesorno828423@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close https://github.com/apache/kyuubi/issues/5484
Kyuubi provides a basic new Web UI which is built on top of Vue3, we can remove the legacy dummy Web UI in 1.8.
The new UI hosts at `http://<host>:<port>/ui/` and the legacy UI hosts at `http://<host>:<port>/`, we should
1. Remove the legacy UI routing from Jetty
2. Remove all files related to legacy UI
3. Redirect `http://<host>:<port>/` to `http://<host>:<port>/ui/`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
building with the command `build/dist --web-ui`, then `cd dist` and perform `bin/kyuubi run`
access http://0.0.0.0:10099 could correctly redirect to http://0.0.0.0:10099/ui/
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1e8a67f6-e4db-415e-8a47-dd7c41b487cf">
swagger is render correctly too.
<img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1cb4ba31-9965-4468-b7c3-b0319ba959e6">
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5516 from pan3793/5484.
Closes#5484
9d58ef758 [Cheng Pan] address comment and fix test
6d4c0981b [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5427 from Yikf/shade-authz.
Closes#5427
d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin
Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
### _Why are the changes needed?_
Apache Paimon is an incubating Apache project of data lake platform for high-speed data ingestion, changelog tracking and efficient real-time analytics.
- Initial support for Paimon tables in Authz plugin
- Create Table Command: https://paimon.apache.org/docs/master/engines/spark3/#create-table
- Paimon `0.5.0-incubating` supports Spark 3.1/3.2/3.3/3.4.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5462 from bowenliang123/authz-paimon-createtable.
Closes#5462
8393f6ecc [liangbowen] remove spark-authz-paimon-test profile, use 3.4 as spark binary version for Paimon
a4e01a2a8 [liangbowen] update workflow inclusions
f372882aa [liangbowen] update workflow exclusions
7785dec52 [liangbowen] update workflow include
64ee20106 [liangbowen] update
73edea38f [liangbowen] add spark-authz-paimon-test profile and excluded for Spark 3.5
de58605bb [liangbowen] nit
81a4a4370 [liangbowen] nit
eb2de8359 [Bowen Liang] exclude paimon tests for spark-3.5
4f58d7727 [Bowen Liang] fix typo
cf360e6da [Bowen Liang] drop database
af8ffda14 [Bowen Liang] drop database
dc3ee9462 [Bowen Liang] support Paimon create table
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
close [#5449](https://github.com/apache/kyuubi/issues/5449).
Unlike the initial preview release, Delta Spark 3.0.0 is now built on top of Apache Spark™ 3.5.
Delta Spark maven artifact has been renamed from delta-core to delta-spark.
https://github.com/delta-io/delta/releases/tag/v3.0.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5450 from zml1206/5449.
Closes#5449
a7969ed6a [zml1206] bump Delta Lake 3.0.0
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Iceberg 1.3.0 use Spark's `ReplaceData` since 3.4 as Logical plan for "DELETE FROM", instead of Iceberg's `DeleteFromTable` in Spark3.4
- Requiring select privilege for input table, even it's the same with output table
- compatible Spark 3.4 and 3.5, subquies in the plan
- enable iceberg test for authz plugin on Spark 3.4 and 3.5
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4916 from bowenliang123/authz-replacedata.
Closes#4916
658917752 [Bowen Liang] Merge branch 'master' into authz-replacedata
17f5d84dd [liangbowen] update
25009908a [Bowen Liang] ut of user with select only for DELETE FROM
fa77cea15 [Bowen Liang] ut
fd9bb8f3a [Bowen Liang] update
f9cfb98a9 [Bowen Liang] assertion for Spark 3.5
f574e0da3 [Bowen Liang] assertion for Spark 3.5
2449c27de [Bowen Liang] bring back single call ut for someone
78786988d [Bowen Liang] bring back single call ut for someone
b8e4a6319 [wforget] fix style
0e26c08b4 [wforget] fix IcebergCatalogRangerSparkExtensionSuite with spark-3.5
02781781f [wforget] enable iceberg tests in spark-3.5
215e1b861 [wforget] fix TableCommands
d019b1632 [wforget] followup
ae17e076b [wforget] Merge remote-tracking branch 'origin/master' into authz-replacedata
b88f77355 [Bowen Liang] update
febcb3ee5 [Bowen Liang] isSparkV34OrGreater
91d41b438 [Bowen Liang] replace data
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: wforget <643348094@qq.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To close#5284.
Hudi also is a common used data format, since kyuubi already support iceberg and delta, we should also support hudi.
In this pr we support hoodie sql about ALTER COMMAND in authz
In this PR we use default Hudi version 0.14.0.
We support from spark 3.1 to spark 3.4, since Hudi don't support spark 3.5 yet
- [x] spark 3.1
- [x] spark 3.2
- [x] spark 3.3
- [x] spark 3.4
- [ ] spark 3.5
Also since Hudi only supports Scala 2.12, I also made Hudi as a separate profile to avoid importing Hudi when enable Scala 2.13
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5287 from AngersZhuuuu/KYUUBI-5284.
Closes#5284
f171e11af [Angerszhuuuu] Update pom.xml
3f57a3dc5 [Angerszhuuuu] follow comment
f6c764028 [Angerszhuuuu] follow comment
51797e25c [Angerszhuuuu] trigger
b3c059af9 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
3510e7601 [liangbowen] remove scope in dependencyManagement
14ea0d498 [liangbowen] change to use `spark.binary.version` for hudi dependency by default
354260eb0 [liangbowen] remove the abbreviation tite
658bddbab [liangbowen] remove clarification and use ALTERTABLE_PROPERTIES for opType
150edcd40 [Angerszhuuuu] update
30c417b19 [Angerszhuuuu] trigger
56e5cb17b [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
fe9b75270 [Angerszhuuuu] update
888943831 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
db749a277 [Angerszhuuuu] update
9b09e78c2 [Angerszhuuuu] Update HudiCommands.scala
87de62e52 [Angerszhuuuu] follow comment
2d551d112 [Angerszhuuuu] Update master.yml
89082e06b [Angerszhuuuu] Update master.yml
7c7846378 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
d32ca9839 [Angerszhuuuu] Update master.yml
ec43e2a7b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
b3611fd3e [Angerszhuuuu] update
2a0dfa74f [Angerszhuuuu] Update AuthZUtils.scala
45ee9e251 [Angerszhuuuu] update
0560a5e14 [Angerszhuuuu] Update pom.xml
97c50f622 [Angerszhuuuu] update
f57ee0093 [Angerszhuuuu] Update table_command_spec.json
fb72197e6 [Angerszhuuuu] update
2154cf928 [Angerszhuuuu] trigger
44469359f [Angerszhuuuu] trigger
b0e768cb8 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
83795ed63 [Angerszhuuuu] Update pom.xml
eed190f92 [Angerszhuuuu] update
361660145 [Angerszhuuuu] update
1ed1f3ab6 [Angerszhuuuu] Update
7ee3c7dd5 [Angerszhuuuu] Merge branch 'KYUUBI-5284' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5284
ee0916f63 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
010260fa4 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5284
c11d02def [Angerszhuuuu] update
b84f91f65 [Angerszhuuuu] update
42fbb0ffa [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
c1346adb1 [Angerszhuuuu] update
2ec63ae94 [Angerszhuuuu] Update pom.xml
39bce7468 [Angerszhuuuu] update
c70b0ea2f [Angerszhuuuu] Update pom.xml
e1d85ff77 [Angerszhuuuu] Update pom.xml
59012ac25 [Angerszhuuuu] Update pom.xml
a46de65b5 [Angerszhuuuu] Update HoodieTest.java
b8173b893 [Angerszhuuuu] update
055713329 [Angerszhuuuu] Update table_command_spec.json
d7b21e820 [Angerszhuuuu] Update HoodieCatalogRangerSparkExtensionSuite.scala
0a93ff794 [Angerszhuuuu] [KYUUBI #5284] Kyuubi authz support Hoodie Alter Table Command
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar in the following order:
1. `SPARK_SCALA_VERSION` system env
2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
outputTables(List(v2_catalog.db.target_t))
columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5196 from bowenliang123/scala213-test.
Closes#5196
97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
It is basically copied from `kyuubi-extension-spark-3-4`.
### _How was this patch tested?_
Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5336 from wForget/dev_spark_3_5.
Closes#5336
7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.4.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5358 from wForget/bump_iceberg_1_4.
Closes#5358
e03d8d672 [wforget] comment
db0b6b978 [wforget] Bump iceberg 1.4.0
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The current maven-source-plugin version(3.3.0) inherits from `org.apache:parent:pom:30`, unfortunately, MSOURCES-121 breaks shaded modules to create source artifacts.
```
[ERROR] We have duplicated artifacts attached.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-source-plugin:3.3.0:jar-no-fork (attach-sources) on project kyuubi-hive-jdbc-shaded: Presumably you have configured maven-source-plugn to execute twice times in your build. You have to configure a classifier for at least on of them. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-source-plugin:3.3.0:jar-no-fork (attach-sources) on project kyuubi-hive-jdbc-shaded: Presumably you have configured maven-source-plugn to execute twice times in your build. You have to configure a classifier for at least on of them.
```
We should skip upgrading until MSOURCES-141 gets fixed.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5318 from pan3793/mvn-source.
Closes#5318
7d12d12e3 [Cheng Pan] [KYUUBI #5318] Pin maven-source-plugin 3.2.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
upgrade snakeyaml from 1.33 to 2.2 reducing direct CVE vulnerabilities, see (https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes)
[CVE-2022-1471](https://nvd.nist.gov/vuln/detail/CVE-2022-1471)
SnakeYaml's Constructor() class does not restrict types which can be instantiated during deserialization. Deserializing yaml content provided by an attacker can lead to remote code execution. We recommend using SnakeYaml's SafeConsturctor when parsing untrusted content to restrict deserialization. We recommend upgrading to version 2.0 and beyond.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5293 from dev-lpq/snakeyaml_critical.
Closes#5293
5b2412d8e [pengqli] upgrade snakeyaml from 1.33 to 2.2
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5253
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes#5254 from lsm1/branch-kyuubi-5253.
Closes#5253
b5c407671 [senmiaoliu] bump testcontainers 0.41.0
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 6158599c4f)
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#5253
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes#5254 from lsm1/branch-kyuubi-5253.
Closes#5253
b5c407671 [senmiaoliu] bump testcontainers 0.41.0
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.3.1
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5250 from yabola/auto_ci.
Closes#5250
85f2625a5 [chenliang.lu] [minor] update iceberg version
Authored-by: chenliang.lu <chenlianglu@tencent.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Apache Commons Lang 3.13.0 release note: https://commons.apache.org/proper/commons-lang/changes-report.html#a3.13.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5241 from bowenliang123/commonslang3-3.13.
Closes#5241
99e5ffbd9 [Bowen Liang] update dependencyList
f2bf9c554 [Bowen Liang] bump commons lang3 to 3.13.0
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Remove the provided dependency `flink-table-planner_${scala.binary.version}` which provides a legacy table planner API on Scala, but is never used in Kyuubi's source code or in runtime directly.
- `** The legacy planner is deprecated and will be dropped in Flink 1.14.**` according to [Flink 1.13's doc of Legacy Planner](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/table/legacy_planner/)
- Kyuubi has dropped support for Flink 1.14 and before in #4588
- Remove the unused provided dependency `flink-sql-parser`
- All tests on Scala 2.12 work fine without them, as `flink-table-runtime` dependency provides enough Java API for usage.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5222 from bowenliang123/flink-remove-planner.
Closes#5222
716ec06e9 [liangbowen] remove flink-sql-parser dependency
0922ba5af [liangbowen] Remove unnecessary dependency flink-table-planner
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- adding source code in `src/main/scala-2.12/**/*.scala` and `src/main/scala-2.13/**/*.scala` to including paths for Scala spotless styling
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5209 from bowenliang123/scala-reformat.
Closes#5209
84b4205e9 [liangbowen] use wildcard for scala versions
0c32c582d [liangbowen] reformat scala-2.12 specific code
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Kyuubi fully supports Spark 3.4 now, it's time to move forward.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5202 from pan3793/default-3.4.
Closes#5202
a0efccdbf [Cheng Pan] nit
30456dbb9 [Cheng Pan] nit
1cc83c871 [Cheng Pan] enable lineage test
d8ca7c7d8 [Cheng Pan] Switch to Spark 3.4 by default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://spark.apache.org/news/spark-3-3-3-released.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5150 from pan3793/spark-3.3.3.
Closes#5150
61583609b [Cheng Pan] image
3021dd80b [Cheng Pan] remove staging repo
71b8aa843 [Cheng Pan] Revert "tgz"
d9125e63e [Cheng Pan] tgz
ebe3107c9 [Cheng Pan] Bump Spark 3.3.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- introducing `scala-2.13` for development use only
- adapt the common modules compilable on Scala 2.13, including
- kyuubi-common
- kyuubi-ctl
- kyuubi-ha
- use separate compile options for Scala 2.13 , skipping deprecated scala compiler option `-Yno-adapted-args`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
Closes#5183 from bowenliang123/scala213-common.
Closes#5183
1e65a5d03 [Bowen Liang] Align scala 2.13 version with Spark to 2.13.8
0003c8bed [liangbowen] duplicate scala-maven-plugin configs in profile
8b448a5f0 [liangbowen] 2.13.11
36d2284c4 [liangbowen] comment
28b515a0c [liangbowen] comment
865c5e55b [liangbowen] update
52e381200 [liangbowen] ensure kyuubi-ctl module compilable on scala 2.13
70dbc7ae2 [liangbowen] ensure kyuubi-ha module compilable on scala 2.13
84c1e4b4d [liangbowen] ensure kyuubi-common module compilable on scala 2.13
e116d77f5 [liangbowen] skip deprecated compiler option "-Yno-adapted-args" in 2.13
26c27a2ee [liangbowen] introduce new profile scala-2.13
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
https://hadoop.apache.org/release/3.3.6.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5116 from pan3793/hadoop-3.3.6.
Closes#5116
c3717e7fb [Cheng Pan] Bump Hadoop 3.3.6
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It was planned but actually delayed, remove this dummy module to save CI and avoid confusing users and release managers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5113 from pan3793/remove-kudu.
Closes#5113
ff8fd2e6a [Cheng Pan] Remove Spark Kudu connector
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Welcome to Spark 4.0
https://github.com/apache/spark/pull/42048
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5068 from cfmcgrady/spark-4.0.
Closes#5068
42cf63e2d [Fu Chen] bump spark 4.0.0-SNAPSHOT
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
### _Why are the changes needed?_
Now the kubernetes client version in apache/spark is 6.7.2(https://github.com/apache/spark/pull/41490), I think it is better to align with that.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5059 from turboFei/k8s_client_version.
Closes#5059
79955d17a [fwang12] deps
4a68cc820 [fwang12] bump
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The main change is copy code from `kyuubi-extension-spark-common_2.12` and `kyuubi-extension-spark-3-3`. The reason copy `kyuubi-extension-spark-common_2.12` is that, Spark3.4 v1 writes does not have ctas...
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5018 from ulysses-you/extension.
Closes#5018
1ca4951e1 [ulysses-you] fix
fb20bb1f0 [ulysses-you] fix
338a3120a [ulysses-you] rebase
241a436fd [ulysses-you] fix
70618f368 [ulysses-you] fix
5f5e2b7c8 [ulysses-you] fix
301ae5c04 [ulysses-you] fix
bb9416811 [ulysses-you] fix
a38595d0a [ulysses-you] copy
9dec27d6c [ulysses-you] copy
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
- Scalafmt 3.7.5 release note: https://github.com/scalameta/scalafmt/releases/tag/v3.7.5
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5007 from bowenliang123/scalafmt-3.7.5.
Closes#5007
f3f7163a4 [liangbowen] Bump Scalafmt from 3.7.4 to 3.7.5
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Bump Scala to 2.12.18
- release note: https://github.com/scala/scala/releases/tag/v2.12.18
- Bump silencer-plugin from 1.7.10 to 1.7.13 for compatibility
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5001 from bowenliang123/scala-2.12.18.
Closes#5001
e4a5b9b0d [liangbowen] update dependencyList
e847bbced [liangbowen] Bump silencer-plugin from 1.7.10 to 1.7.13
daceece66 [liangbowen] Bump scala from 2.12.17 to 2.12.18
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final
Currently, we can not upgrade to 4.1.94.Final or above because of https://github.com/netty/netty/pull/13408
```
java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4992 from pan3793/netty.
Closes#4992
9dd8f9ad0 [Cheng Pan] nit
e6953144e [Cheng Pan] Bump Netty 4.1.93.Final
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://spark.apache.org/releases/spark-release-3-4-1.html
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#4990 from pan3793/spark-3.4.1.
Closes#4990
68203b2c0 [Cheng Pan] Bump Spark 3.4.1
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Apache parent POM version 30 release note: https://github.com/apache/maven-apache-parent/releases/tag/apache-30
- remove `maven.plugin.surefire.version` property and bump the surefire plugin from 3.0.0-M8 to 3.1.2
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4966 from bowenliang123/parent-pom-30.
Closes#4966
970e48584 [liangbowen] Bump Apache parent pom from 29 to 30
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Improve log on executing init schema DDL.
- Add `IF NOT EXISTS` on `CREATE INDEX` statement.
- Fix `kyuubi.metadata.store.jdbc.url` configuration description.
- Delete SQLite files on performing `build/mvn clean`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4953 from pan3793/sqlite-followup.
Closes#4950
a0b44f37c [Cheng Pan] [KYUUBI #4950][FOLLOWUP] Misc improvement for usage of SQLite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- The current lockfileVersion of the `pnpm` lock file has been bumped from 6.0 to 6.1 in #4931 (passed CI as it uses latest 8.x pnpm), which is incompatible with the currently `pnpm` version 8.4.1 in maven's frontend plugin 8.4.1
- Bump pnpm version to solve the conflicts
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4947 from bowenliang123/pnpm-8.6.1.
Closes#4947
074dcca1f [liangbowen] Bump pnpm version to 8.6.1 for lockfileVersion 6.0
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
