c8f01e69cb
3913 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
lllong33
|
c8f01e69cb
|
[KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6394 from lllong33/patch-1.
Closes #6394
eb79375aa [lllong33] Update tpcds.rst, The order table removed
Lead-authored-by: lllong33 <704669594@qq.com>
Co-authored-by: lllong33 <29864358+lllong33@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
bd33b7b176
|
[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above
# 🔍 Description
Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118, which breaks the Kyuubi UI tab.
## Describe Your Solution 🔧
Temporarily disable UI Tab for Spark 4.0 and above.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Build Kyuubi with Spark 3.5 and successfully launch the Spark engine with Spark 4.0.0 preview1.
<img width="1339" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/34a001e5-5c70-4928-bf28-743825c0ead3">
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6390 from pan3793/ui-4.0.
Closes #6390
2abad8c35 [Cheng Pan] Temporarily disable UI Tab for Spark 4.0 and above
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
PorterZhang2021
|
854e35efb0
|
[KYUUBI #6305][FOLLOWUP] Improve package Spark SQL engine both Scala 2.12 and 2.13
# 🔍 Description ## Issue References 🔗 This pull request fixes #6305 ## Describe Your Solution 🔧 ### Solution 1 use `<profile>` - Inappropriate I found a way to use <profiles>, roughly as follows: ```xml <profile> <id>scala-2.12</id> <properties> <scala.binary.version>2.12</scala.binary.version> </properties> </profile> ``` After specifying, I attempted to use 1. `build/mvn install -DskipTests -Dmaven.javadoc.skip=true -Dmaven.scaladoc.skip=true - Dmaven.source.skip -Pscala-2.12 -Pscala-2.13 -pl: kyuubi-spark-sql-engine -am` 2. `build/mvn install -DskipTests -Dmaven.javadoc.skip=true -Dmaven.scaladoc.skip=true - Dmaven.source.skip -Pscala-2.13 -Pscala-2.12 -pl: kyuubi-spark-sql-engine -am` #### Problem But in the end, it was found that if both '-Pscala-2.12' and '-Pscala-2.13' are used at the same time, '-Pscala2.13' will be selected by default, which may not be a good solution to this problem. ### Solution2 Later, I thought about whether it was possible to filter the parameter '$' internally. It is effective. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 1. scala2.12   2. scala2.13   --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6330 from PorterZhang2021/followup-6297. Closes #6305 5524b13f5 [Cheng Pan] nit cccb94d98 [PorterZhang2021] [# 6297] imporve Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up fb03d60c2 [Porter Zhang] Merge branch 'apache:master' into followup-6297 393435ea9 [PorterZhang2021] [# 6297] imporve Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up 0b49f60e6 [Porter Zhang] Merge branch 'apache:master' into followup-6297 f7c7a65c8 [PorterZhang2021] [# 6297] Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up 3d2926afc [Porter Zhang] Merge branch 'apache:master' into followup-6297 eb6406148 [PorterZhang2021] [followup-issue6297] improve issue6297 956ac4955 [PorterZhang2021] [# 6297] Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up Lead-authored-by: PorterZhang2021 <PorterZhang2021@outlook.com> Co-authored-by: Porter Zhang <96274454+PorterZhang2021@users.noreply.github.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: PorterZhang2021 <porterzhang2021@outlook.com> Co-authored-by: Porter Zhang <porterzhang2021@outlook.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
PorterZhang2021
|
43ec939ad7
|
[KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
# 🔍 Description ## Issue References 🔗 This pull request fixes #6297 ## Describe Your Solution 🔧 Here are my main modifications: 1. In the `build/dist` file, I added an additional packaging command that detects the version of `SCALA_VERSION`. If it is version 2.12, then it will additionally package version 2.13, otherwise it will package version 2.12 2. In the section of `# Copy spark engines`, the Jar packages of versions 2.12 and 2.13 will be moved to `$DISTDIR/externals/engines/spark/` This is my first time submitting a PR, so there may be many unfamiliar areas, and I don't know if my changes meet the needs of the community. More guidance may be needed. When testing locally, it was able to be packaged successfully ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪   --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6305 from PorterZhang2021/issue-6297. Closes #6297 efc864373 [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297 cb9310bee [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297 b6c907509 [PorterZhang2021] Finished task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297 Authored-by: PorterZhang2021 <porterzhang@aliyun.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wforget
|
6d0c874a70 |
[KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
# 🔍 Description
## Issue References 🔗
This pull request fixes #6344
`FlinkProcessBuilder` specifies `yarn.ship-files`, `yarn.application.name` and `yarn.tags` configurations of kyuubi platform. Sometimes we also need to customize these configurations, so we should prioritize these user configurations.
## Describe Your Solution 🔧
FlinkProcessBuilder prioritizes user configurations.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
added new unit test
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6342 from wForget/hotfix2.
Closes #6344
feca972ca [wforget] address comment
17df0844d [wforget] fix test and add flink constant
ece91cc0c [wforget] FlinkProcessBuilder prioritizes user configurations
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
zml1206
|
e3cad1e007
|
[KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
# 🔍 Description
## Issue References 🔗
[Delta 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0) is available, which is built on top of Spark 3.5.
## Describe Your Solution 🔧
Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6382 from zml1206/delta-3.2.
Closes #6382
6de96c6c7 [zml1206] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
Authored-by: zml1206 <zhuml1206@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
pengbei
|
0c744a9284
|
[KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
# 🔍 Description
## Issue References 🔗
This pull request fixes #6348
## Describe Your Solution 🔧
In the project POM file, I have updated the Apache Iceberg version from 1.5.0 to 1.5.2
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6349 from mrpengbei/KYUUBI6348.
Closes #6348
930389810 [Peng Bei] Update pom.xml
119bb785d [pengbei] [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.1 #6348
Lead-authored-by: pengbei <pengbei@corp.netease.com>
Co-authored-by: Peng Bei <101689518+mrpengbei@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
1edd31d400
|
[KYUUBI #6328] Bump Hive 2.3.10
# 🔍 Description
Hive 2.3.10 is available now, and it will be the latest version of the 2.3 serial.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6328 from pan3793/hive-2310.
Closes #6328
82a9fb70c [Cheng Pan] Update .github/workflows/master.yml
10af5e13b [Cheng Pan] hive-2.3.10-rc-1
2dbb2b54e [Cheng Pan] test Hive 2.3.10 RC0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Wang, Fei
|
d17efd1102 |
[KYUUBI #4847][FOLLOWUP] Exclude the alive probe sessions in terminating checker
# 🔍 Description
## Issue References 🔗
This pull request fixes #
follow up of #4847
Address comments: https://github.com/apache/kyuubi/issues/4847#issuecomment-2072945805
## Describe Your Solution 🔧
In this pr, when checking the engine terminating, it will ignore the alive probe sessions.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6355 from turboFei/engine_idle.
Closes #4847
a8e26e71d [Wang, Fei] comments
418d0b41c [Wang, Fei] val
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
wangjunbo
|
494c65bcc5 |
[KYUUBI #6172][TASK][EASY] Support to interrupt the thrift request immediately after marking the engine not alive
Support to interrupt the thrift request immediately after marking the engine not alive # 🔍 Description ## Issue References 🔗 This pull request fixes #6172 ## Describe Your Solution 🔧 |
||
|
wforget
|
6c8d4631b6 |
[KYUUBI #6373] Avoid NPE in get operation log api
# 🔍 Description
## Issue References 🔗
This pull request fixes #6373
## Describe Your Solution 🔧
`ThriftUtils#EMPTY_ROW_SET` is not suitable for OperationLog , so I add a new `LOG_EMPTY_ROW_SET`
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6378 from wForget/KYUUBI-6373.
Closes #6373
f25aa593f [wforget] revert
ef87b3f4c [wforget] revert and fix npe
7219d35de [wforget] add test
06135fe0e [wforget] fix test
c0bd29078 [wforget] [KYUUBI #6373] Add empty RowSet for OpertationLog
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
89a2c0c550 |
[KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
# 🔍 Description
## Issue References 🔗
This pull request fixes #
I found that, with saveToFile enabled with the default min size threshold, even I run a simple `set` command, It also save the result to file.
<img width="1718" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/5bcc0da1-201a-453a-8568-d1bfadd7adef">
I think we need to skip this kind of queries.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6377 from turboFei/check_is_DQL.
Closes #6377
da9c2a921 [Wang, Fei] ut
04e20db5f [Wang, Fei] conf
8f20ed84b [Wang, Fei] refine the check
f558dcca5 [Wang, Fei] ut
c81340333 [Wang, Fei] DQL
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
e277353396 |
[KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
# 🔍 Description
## Issue References 🔗
This pull request fixes #6370
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6372 from turboFei/kyuubi_6370.
Closes #6370
d0c6a9236 [Wang, Fei] fix
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Cheng Pan
|
f149af7f2b
|
[KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
# 🔍 Description
Unify the words and provide the tag for the waiting submission case.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6366 from pan3793/improve-log.
Closes #6366
30e3fdee0 [Cheng Pan] fix style
a3b36ea33 [Cheng Pan] Improve log message for YARN getApplicationInfoByTag
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
b749df2418 |
[KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
# 🔍 Description
## Issue References 🔗
This pull request fixes #6346
## Describe Your Solution 🔧
Change Kyuubi JDBC Driver default port and make it effective.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6347 from wForget/KYUUBI-6346.
Closes #6346
0a3085205 [wforget] [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
dependabot[bot]
|
df346eae5e |
[KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
Bumps [org.bouncycastle:bcprov-jdk18on](https://github.com/bcgit/bc-java) from 1.77 to 1.78.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html">org.bouncycastle:bcprov-jdk18on's changelog</a>.</em></p>
<blockquote>
<p>2.1.1 VersionDate: 2024, TBD.</p>
<p>2.2.1 VersionDate: 2024, 18th April.</p>
<p>2.3.1 VersionRelease: 1.78Date: 2024, 7th April.</p>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a href="https://github.com/bcgit/bc-java/commits">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes #6357 from dependabot[bot]/dependabot/maven/org.bouncycastle-bcprov-jdk18on-1.78.
Closes #6357
91c1ad3e6 [dependabot[bot]] ⬆️ Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
9fa4f78d52 |
[KYUUBI #6363] [INFRA] Increase CI concurrency
# 🔍 Description
Previously, we were asked to limit the job concurrency to 10, while the new policy is more relaxed.
> All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20.
https://infra.apache.org/github-actions-policy.html
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Monitor CI
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6363 from pan3793/ci-con.
Closes #6363
101121aa9 [Cheng Pan] [INFRA] Increase CI concurrency
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wangjunbo
|
7523a4ec86
|
[KYUUBI #6244][TEST] Fix test logs upload error
# 🔍 Description
## Issue References 🔗
This pull request fixes #6244
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6351 from beryllw/kyuubi-6244.
Closes #6244
603431087 [wangjunbo] [KYUUBI #6244] Fix test logs upload error
460b25985 [wangjunbo] [KYUUBI #6244] Fix test logs upload error
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
5942686395 |
[KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
# 🔍 Description
## Issue References 🔗
This pull request fixes #6338
## Describe Your Solution 🔧
Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6340 from wForget/KYUUBI-6338.
Closes #6338
acd73a16b [wforget] fix
5e64ed3a5 [wforget] comments
48b8ec6cd [wforget] fix style and address comments
ca653492c [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
Fei Wang
|
031306db0a |
[KYUUBI #6049] Support to filter sessions/operations with session type
# 🔍 Description
Support to filter sessions/operations with session type.
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6049 from turboFei/batch_interactive.
Closes #6049
68390c774 [Wang, Fei] add ut
bfc2cb8c2 [Fei Wang] save
c1979c7ea [Fei Wang] saev
Lead-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
9fe210c8ed
|
[KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
# 🔍 Description
## Issue References 🔗
This pull request fixes #
We found that some pods were not deleted if kyuubi server restarted
The root cause is that:
com.google.common.cache::cleanup will not trigger removalListener action, we shall use `invalidateAll` instead.
Testing:
Nothing print for below code:
```
import com.google.common.cache.{Cache, CacheBuilder, RemovalNotification}
var removed = 0
val cache: Cache[String, String] = CacheBuilder.newBuilder()
.removalListener((notification: RemovalNotification[String, String]) => {
removed += 1
println("removed: " + removed)
}).build()
(0 until 1000).foreach { i =>
cache.put(i.toString, i.toString)
}
cache.cleanup()
```
Replacing `cache.cleanup()` with `cache.invalidateAll()`, the output is expected.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6319 from turboFei/app_k8s_leak.
Closes #6320
ff2adfc60 [Wang, Fei] invalidate all
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
hezhao2
|
238c893f39
|
[KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
# 🔍 Description
## Issue References 🔗
Now, MaxScanStrategy can be adopted to limit max scan file size in some datasources, such as Hive. Hopefully we can enhance MaxScanStrategy to include support for the datasourcev2.
## Describe Your Solution 🔧
get the statistics about files scanned through datasourcev2 API
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes #5852 from zhaohehuhu/dev-1213.
Closes #6315
3c5b0c276 [hezhao2] reformat
fb113d625 [hezhao2] disable the rule that checks the maxPartitions for dsv2
acc358732 [hezhao2] disable the rule that checks the maxPartitions for dsv2
c8399a021 [hezhao2] fix header
70c845bee [hezhao2] add UTs
3a0739686 [hezhao2] add ut
4d26ce131 [hezhao2] reformat
f87cb072c [hezhao2] reformat
b307022b8 [hezhao2] move code to Spark 3.5
73258c2ae [hezhao2] fix unused import
cf893a0e1 [hezhao2] drop reflection for loading iceberg class
dc128bc8e [hezhao2] refactor code
661834cce [hezhao2] revert code
6061f42ab [hezhao2] delete IcebergSparkPlanHelper
5f1c3c082 [hezhao2] fix
b15652f05 [hezhao2] remove iceberg dependency
fe620ca92 [hezhao2] enable MaxScanStrategy when accessing iceberg datasource
Authored-by: hezhao2 <hezhao2@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
PorterZhang2021
|
88952c568e
|
[KYUUBI #6294] Simplify Netty and gRPC dependency management
# 🔍 Description
## Issue References 🔗
This pull request fixes #6294
## Describe Your Solution 🔧
The netty jars come from `grpc-netty`, `arrow-memory-netty`, use `netty-bom` and `grpc-bom` to simplify the dependency management.
Run `./build/dependency.sh --replace`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6310 from PorterZhang2021/issue-6294.
Closes #6294
327030170 [Cheng Pan] simplify netty dependency management
cdb435d01 [PorterZhang2021] [# 6294] Prune unused Netty libraries
e0676ed5a [PorterZhang2021] Finished Prune unused Netty libraries [#6294]
Lead-authored-by: PorterZhang2021 <PorterZhang2021@outlook.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Harry
|
f0d1633e03
|
[KYUUBI #6298][HELM] Make ServiceMonitor use selector labels
# 🔍 Description
## Issue References 🔗
This pull request fixes #6298
## Describe Your Solution 🔧
Change matchlabels field to use template `selectorLabels`
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
ServiceMonitor is not able to match kyuubi service
#### Related Unit Tests
Ran `helm template` and produced yaml files as expected
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6301 from sudohainguyen/helmchart.
Closes #6298
cdc669852 [Harry] Include template consistently
bd82fb426 [Harry] [KYUUBI #6298][HELM] Make ServiceMonitor use selector labels to discover metric service
Authored-by: Harry <quanghai.ng1512@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
dupeng
|
75bdcc4ad1
|
[KYUUBI #6251] Improve kyuubi-beeline help message
This pull request fixes #6251
As `kyuubi-beeline` is derived from Hive `beeline`, I haven't made extensive modifications to its help message, only adjusting some formatting to enhance its appearance. Below are the specific changes:
1. Replace `jdbc:hive2//` with `jdbc:kyuubi//`.
2. Replace `beeline` with `kyuubi-beeline`.
3. Capitalize the first letter.
4. If the comment for a parameter spans multiple lines, add an additional `\n` at the end of the comment to separate it from the following line.
5. Append the help information for `--python-mode` to the front of `--help`.
6. Add some examples.
7. Improved letter indentation.
Here is the whole help message:
```
$ ./bin/kyuubi-beeline --help
Usage: kyuubi-beeline <options>.
Options:
-u <database url> The JDBC URL to connect to.
-c <named url> The named JDBC URL to connect to,
which should be present in beeline-site.xml
as the value of beeline.kyuubi.jdbc.url.<namedUrl>.
-r Reconnect to last saved connect url (in conjunction with !save).
-n <username> The username to connect as.
-p <password> The password to connect as.
-d <driver class> The driver class to use.
-i <init file> Script file for initialization.
-e <query> Query that should be executed.
-f <exec file> Script file that should be executed.
-w, --password-file <file> The password file to read password from.
--hiveconf property=value Use value for given property.
--hivevar name=value Hive variable name and value.
This is Hive specific settings in which variables
can be set at session level and referenced in Hive
commands or queries.
--property-file=<property file> The file to read connection properties (url, driver, user, password) from.
--color=[true|false] Control whether color is used for display.
--showHeader=[true|false] Show column names in query results.
--escapeCRLF=[true|false] Show carriage return and line feeds in query results as escaped \r and \n.
--headerInterval=ROWS; The interval between which heades are displayed.
--fastConnect=[true|false] Skip building table/column list for tab-completion.
--autoCommit=[true|false] Enable/disable automatic transaction commit.
--verbose=[true|false] Show verbose error messages and debug info.
--showWarnings=[true|false] Display connection warnings.
--showDbInPrompt=[true|false] Display the current database name in the prompt.
--showNestedErrs=[true|false] Display nested errors.
--numberFormat=[pattern] Format numbers using DecimalFormat pattern.
--force=[true|false] Continue running script even after errors.
--maxWidth=MAXWIDTH The maximum width of the terminal.
--maxColumnWidth=MAXCOLWIDTH The maximum width to use when displaying columns.
--silent=[true|false] Be more silent.
--autosave=[true|false] Automatically save preferences.
--outputformat=<format mode> Format mode for result display.
The available options ars [table|vertical|csv2|tsv2|dsv|csv|tsv|json|jsonfile].
Note that csv, and tsv are deprecated, use csv2, tsv2 instead.
--incremental=[true|false] Defaults to false. When set to false, the entire result set
is fetched and buffered before being displayed, yielding optimal
display column sizing. When set to true, result rows are displayed
immediately as they are fetched, yielding lower latency and
memory usage at the price of extra display column padding.
Setting --incremental=true is recommended if you encounter an OutOfMemory
on the client side (due to the fetched result set size being large).
Only applicable if --outputformat=table.
--incrementalBufferRows=NUMROWS The number of rows to buffer when printing rows on stdout,
defaults to 1000; only applicable if --incremental=true
and --outputformat=table.
--truncateTable=[true|false] Truncate table column when it exceeds length.
--delimiterForDSV=DELIMITER Specify the delimiter for delimiter-separated values output format (default: |).
--isolation=LEVEL Set the transaction isolation level.
--nullemptystring=[true|false] Set to true to get historic behavior of printing null as empty string.
--maxHistoryRows=MAXHISTORYROWS The maximum number of rows to store beeline history.
--delimiter=DELIMITER Set the query delimiter; multi-char delimiters are allowed, but quotation
marks, slashes, and -- are not allowed (default: ;).
--convertBinaryArrayToString=[true|false]
Display binary column data as string or as byte array.
--python-mode Execute python code/script.
-h, --help Display this message.
Examples:
1. Connect using simple authentication to Kyuubi Server on localhost:10009.
$ kyuubi-beeline -u jdbc:kyuubi://localhost:10009 -n username
2. Connect using simple authentication to Kyuubi Server on kyuubi.local:10009 using -n for username and -p for password.
$ kyuubi-beeline -n username -p password -u jdbc:kyuubi://kyuubi.local:10009
3. Connect using Kerberos authentication with kyuubi/localhostmydomain.com as Kyuubi Server principal(kinit is required before connection).
$ kyuubi-beeline -u "jdbc:kyuubi://kyuubi.local:10009/default;kyuubiServerPrincipal=kyuubi/localhostmydomain.com"
4. Connect using Kerberos authentication using principal and keytab directly.
$ kyuubi-beeline -u "jdbc:kyuubi://kyuubi.local:10009/default;kyuubiClientPrincipal=usermydomain.com;kyuubiClientKeytab=/local/path/client.keytab;kyuubiServerPrincipal=kyuubi/localhostmydomain.com"
5. Connect using SSL connection to Kyuubi Server on localhost:10009.
$ kyuubi-beeline -u "jdbc:kyuubi://localhost:10009/default;ssl=true;sslTrustStore=/usr/local/truststore;trustStorePassword=mytruststorepassword"
6. Connect using LDAP authentication.
$ kyuubi-beeline -u jdbc:kyuubi://kyuubi.local:10009/default -n ldap-username -p ldap-password
7. Connect using the ZooKeeper address to Kyuubi HA cluster.
$ kyuubi-beeline -u "jdbc:kyuubi://zk1:2181,zk2:2181,zk3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi" -n username
```
If you have any suggested revisions, I will promptly respond and make the necessary adjustments.
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6300 from dupen01/issue-6251.
Closes #6251
69f9b3dc9 [dupeng] deleted usage() from KyuubiBeeLine.java;updated some examples
463bcfb85 [dupeng] update --delimiter description
95e13c0b6 [dupeng] update some example message
6625f99b5 [dupeng] update cmd-usage
6570ec255 [dupeng] update cmd-usage
59ea1a495 [dupeng] add kyuubi-beeline help message
Authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
58843eda2e
|
[KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
# 🔍 Description
## Issue References 🔗
This pull request fixes #6290
## Describe Your Solution 🔧
Serializing the `SparkException` object may cause NPE, so I referred to SparkListenerEvent serialization methods to implement `ExceptionDeserializer` and `ExceptionSerializer`.
error detail:
```
(was java.lang.NullPointerException) (through reference chain: org.apache.kyuubi.engine.spark.events.SparkOperationEvent["exception"]->org.apache.spark.SparkException["internalError"])
com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.apache.kyuubi.engine.spark.events.SparkOperationEvent["exception"]->org.apache.spark.SparkException["internalError"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:323)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:780)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at com.fasterxml.jackson.databind.ser.std.ReferenceTypeSerializer.serialize(ReferenceTypeSerializer.java:386)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:772)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:655)
at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:479)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:318)
at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3303)
at org.apache.spark.util.JsonProtocol$.writeSparkEventToJson(JsonProtocol.scala:110)
at org.apache.spark.util.JsonProtocol$.$anonfun$sparkEventToJsonString$1(JsonProtocol.scala:63)
at org.apache.spark.util.JsonProtocol$.$anonfun$sparkEventToJsonString$1$adapted(JsonProtocol.scala:62)
at org.apache.spark.util.JsonUtils.toJsonString(JsonUtils.scala:36)
at org.apache.spark.util.JsonUtils.toJsonString$(JsonUtils.scala:33)
at org.apache.spark.util.JsonProtocol$.toJsonString(JsonProtocol.scala:54)
at org.apache.spark.util.JsonProtocol$.sparkEventToJsonString(JsonProtocol.scala:62)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.$anonfun$new$1(KyuubiSparkEventSuite.scala:44)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.apache.kyuubi.KyuubiFunSuite.withFixture(KyuubiFunSuite.scala:65)
at org.apache.kyuubi.KyuubiFunSuite.withFixture$(KyuubiFunSuite.scala:59)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.withFixture(KyuubiSparkEventSuite.scala:25)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(KyuubiSparkEventSuite.scala:25)
at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.runTest(KyuubiSparkEventSuite.scala:25)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.org$scalatest$BeforeAndAfterAll$$super$run(KyuubiSparkEventSuite.scala:25)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.kyuubi.KyuubiSparkEventSuite.run(KyuubiSparkEventSuite.scala:25)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
Caused by: java.lang.NullPointerException
at org.apache.spark.SparkThrowableHelper$.isInternalError(SparkThrowableHelper.scala:64)
at org.apache.spark.SparkThrowableHelper.isInternalError(SparkThrowableHelper.scala)
at org.apache.spark.SparkThrowable.isInternalError(SparkThrowable.java:50)
at org.apache.spark.SparkException.isInternalError(SparkException.scala:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:772)
... 69 more
```
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
new unit test
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6289 from wForget/hotfix2.
Closes #6290
cf701f97e [wforget] fix test
41df6ce9b [wforget] Add Exception Serialization for SparkOperationEvent
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wangjunbo
|
8b4a649d34
|
[KYUUBI #6079] Web UI support Basic authN
# 🔍 Description
## Issue References 🔗
This pull request fixes #6079
## Describe Your Solution 🔧
<img width="1160" alt="image" src="https://github.com/apache/kyuubi/assets/25627922/eef26a0e-478a-4e65-a0a4-629e0afa9a52">
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6258 from beryllw/kyuubi-6079.
Closes #6079
7a90286a7 [wangjunbo] support input valid
6e2093e66 [wangjunbo] format code
a781c0ae7 [wangjunbo] add License
788bdfad0 [wangjunbo] [KYUUBI #6079] Web UI support Basic authN
97772e595 [wangjunbo] [KYUUBI #6079] Web UI support Basic authN
5560d4f6c [wangjunbo] [KYUUBI #6079] Web UI support Basic authN
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
hezhao2
|
7d93267727
|
[KYUUBI #6291] Avoid NPE in MySQLErrPacket
# 🔍 Description
as title
## Issue References 🔗
This pull request fixes #6291
## Describe Your Solution 🔧
Add a rule to check if it's null
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6292 from zhaohehuhu/dev-0411.
Closes #6291
a58eed609 [hezhao2] refactor
40ad5e0fa [hezhao2] Avoid NPE in MySQLErrPacket
e02be17b4 [dupeng] [KYUUBI #6288] [KYUUBI #6283] Improve the message of distribution built without enabling web ui
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
dupeng
|
7db47f141c
|
[KYUUBI #6288] [KYUUBI #6283] Improve the message of distribution built without enabling web ui
# 🔍 Description Improve message of distribution built without enabling web ui. ## Issue References 🔗 This pull request fixes #6283 ## Describe Your Solution 🔧 Here is my implement: <img width="1512" alt="image" src="https://github.com/apache/kyuubi/assets/34719039/fc5e859e-1c92-44a1-8e27-316986c5d9c3"> ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6288 from dupen01/issue-6283. Closes #6288 a5886e271 [dupeng] revise the message of page bad14a748 [dupeng] improve the message of distribution built without enabling web ui Authored-by: dupeng <dunett@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit e02be17b4eb1def27fc66036e9f49ad80c6f4221) Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wforget
|
242256be9c |
[KYUUBI #6287] Avoid missing matche
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Avoid matching error caused by failed stage.
error detail:
```
24/04/09 02:11:41 ERROR AsyncEventQueue: Listener SQLOperationListener threw an exception
scala.MatchError: failed (of class java.lang.String)
at org.apache.spark.kyuubi.SQLOperationListener.onStageCompleted(SQLOperationListener.scala:147)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:35)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1387)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
```
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6287 from wForget/hotfix2.
Closes #6287
01e96f051 [wforget] Avoid missing matche
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
hezhao2
|
7b8f7108c8
|
[KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
# 🔍 Description
## Issue References 🔗
close #6252
## Describe Your Solution 🔧
Upgrade hive-service-rpc 4.0.0
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass existing tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6262 from zhaohehuhu/dev-0407.
Closes #6252
d7df86d93 [Cheng Pan] address comments
9bdf04eb2 [hezhao2] delete code
fd7b231c1 [hezhao2] delete code
d52468754 [hezhao2] reformat
88c004474 [hezhao2] throws UnsupportedOperationException for UploadData and DownloadData method in TFrontendService
62c5b89fb [Cheng Pan] Update pom.xml
1ac087c4c [Cheng Pan] Update pom.xml
78bca3a0d [hezhao2] Upgrade hive-service-rpc 4.0.0
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
17c7e3e046 |
[KYUUBI #6268] Specify logDir for RollingFile filePattern
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
We should specify `sys:logDir` for RollingFile filePattern, otherwise the log will be rolled to kyuubi work directory.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6268 from wForget/hotfix2.
Closes #6268
b599583be [wforget] Specify logDir for RollingFile filePattern
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
amanraj2520
|
de505c3d96
|
[KYUUBI #6212] Added audit handler shutdown to the shutdown hook
# 🔍 Description
This pull request fixes #6212
When Kyuubi cleans up Ranger related threads like PolicyRefresher, it should also shutdown the audit threads that include SolrZkClient. Otherwise Spark Driver keeps on running since SolrZkClient is a non-daemon thread. Added the cleanup as part of the shutdown hook that Kyuubi registers.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6233 from amanraj2520/auditShutdown.
Closes #6212
e663d466c [amanraj2520] Refactored code
ed293a9a4 [amanraj2520] Removed unused import
95a6814ad [amanraj2520] Added audit handler shutdown to the shutdown hook
Authored-by: amanraj2520 <rajaman@microsoft.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
0f9f6cb56f
|
[KYUUBI #6267] Remove unused dependency management in POM
# 🔍 Description
This pull request removes unused dependency management in POM
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6267 from pan3793/clean-pom.
Closes #6267
d19f719bf [Cheng Pan] Remove usued dependency management in POM
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
liuxiao
|
4ec70f2c5d
|
[KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
# 🔍 Description
## Issue References 🔗
This pull request fixes #6271
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6274 from liuxiaocs7/issue-6271.
Closes #6271
36215436d [liuxiao] Upgrade dep list
13c41d150 [liuxiao] Upgrade kafka-clients from 3.5.1 to 3.5.2
Authored-by: liuxiao <liuxiao2103@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
dupeng
|
8ed1043189
|
[KYUUBI #6248] Web SQL Editor supports run SQL snippets
# 🔍 Description
when some text is selected, only send the selected text.
## Issue References 🔗
This pull request fixes #6248
## Describe Your Solution 🔧
The Monaco Editor API is called to get the selected SQL statement, and without any other dependency.
<img width="581" alt="image" src="https://github.com/apache/kyuubi/assets/34719039/ce6ac5ae-b1fc-4778-a57e-dfda9ad51275">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6272 from dupen01/issue-6248.
Closes #6248
67d47cd4e [dupeng] Improved nonempty judgment
e0767bb64 [dupeng] Finished task "Web SQL Editor supports run SQL snippets #6248"
Authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
e7ce500921
|
[KYUUBI #6260] Clean up and improve comments for spark extensions
# 🔍 Description
This pull request
- improves comments for SPARK-33832
- removes unused `spark.sql.analyzer.classification.enabled` (I didn't update the migration rules because this configuration seems never to work properly)
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Review
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6260 from pan3793/nit.
Closes #6260
d762d30e9 [Cheng Pan] update comment
4ebaa04ea [Cheng Pan] nit
b303f05bb [Cheng Pan] remove spark.sql.analyzer.classification.enabled
b021cbc0a [Cheng Pan] Improve docs for SPARK-33832
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
63b67a56f1 |
[KYUUBI #6216] Support to deny some client ips to make connection
# 🔍 Description
## Issue References 🔗
This pull request fixes #6216
## Describe Your Solution 🔧
Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
SessionLimiterSuite.test("test session limiter with ip deny list")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6217 from wForget/KYUUBI-6216.
Closes #6216
e0a058c1d [wforget] update ctl
f66426eda [wforget] typo
5c5308ad8 [wforget] typo
959242e9b [wforget] Support to deny some client ips to make connection
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
Cheng Pan
|
19786b0d22
|
[KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 4.5.2 to 4.5.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vitejs/vite/blob/v4.5.3/packages/vite/CHANGELOG.md">vite's changelog</a>.</em></p> <blockquote> <h2>4.5.3 (2024-03-24)</h2> <ul> <li>fix: <code>fs.deny</code> with globs with directories (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16250">#16250</a>) (<a href="https://github.com/vitejs/vite/commit/96a7f3a">96a7f3a</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16250">#16250</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Wang, Fei
|
d2f57338b9 |
[KYUUBI #5767][FOLLOWUP] Fix spark batch conf not convert issue
# 🔍 Description
## Issue References 🔗
Convert the spark batch conf key if not start with `spark.`, it is impacted by #5767
```
protected def convertConfigKey(key: String): String = {
if (key.startsWith("spark.")) {
key
} else if (key.startsWith("hadoop.")) {
"spark.hadoop." + key
} else {
"spark." + key
}
}
```
## Describe Your Solution 🔧
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6256 from turboFei/convert_key.
Closes #5767
d76655e05 [Wang, Fei] Fix break
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
1618b2191c
|
[KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
# 🔍 Description
## Issue References 🔗
This pull request fixes #6223
Even the user specify `spark.jars`, but they can not access the classes in jars with Scala code.
## Describe Your Solution 🔧
Add the jars into repl classpath.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
UT.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6235 from turboFei/scala_repl_urls.
Closes #6223
344502660 [Wang, Fei] scala 2.13
cc6e28989 [Wang, Fei] todo
a8b373167 [Wang, Fei] refine
65b438ccf [Wang, Fei] remove scala reflect check
eb257c7a8 [Wang, Fei] using -classpath
e1c6f0e11 [Wang, Fei] revert 2.13
15d37662d [Wang, Fei] repl
41ebe1011 [Wang, Fei] fix ut
ed5d344f8 [Wang, Fei] info
1cdd82ab4 [Wang, Fei] comment
aa4292dac [Wang, Fei] fix
a10cfa5e0 [Wang, Fei] ut
63fdb8877 [Wang, Fei] Use global.classPath.asURLs instead of class loader urls
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
senmiaoliu
|
a86063ae9b
|
[KYUUBI #6243] Add missing Impala alias for JDBC engine
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
---
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6238 from lsm1/branch-conf-add-Impala.
Closes #6243
de3093d6a [senmiaoliu] add Impala conf
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
Cheng Pan
|
08d398d45c
|
[KYUUBI #6234] Fix invalid code-block syntax in docs
# 🔍 Description
## Issue References 🔗
This pull request fixes docs issue
## Describe Your Solution 🔧
Correct syntax
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
<img width="858" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/66a74736-48d9-464d-b8cf-c3886aa1d125">
<img width="855" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/fd6cd267-2fa6-493c-9976-61600ae427a0">
#### Behavior With This Pull Request 🎉
<img width="869" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/adc7b7e0-29bf-4ba3-b96b-fddb14f6053a">
<img width="877" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1aac5c46-9064-4e34-95a0-16d64e7ef32e">
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6234 from pan3793/docs-fix.
Closes #6234
801b5d980 [Cheng Pan] Fix invalid code-block syntax in docs
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
jialiang
|
bb7a8ab2e1
|
[KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6228
issue Description:
Using the branch kyuubi 1.9.0
1. The kyuubi sql editor frontend dropdown menu cannot select the submission type.
Solution: Set disabled to false.
Issue 2: After making the change in step one, the sql editor can select different types of sql type, but regardless of the selected type, the backend always initiates a spark engine. Upon investigation, it was found that the frontend parameter settings were inappropriate. The key in the JSON and the constructor parameter names in the backend request object did not match, leading to parsing failure, and the parameters were always null.
Solution: Modify the parameter name.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
The kyuubi sql editor frontend dropdown menu cannot select the submission type.
regardless of the selected type, the backend always initiates a spark engine
#### Behavior With This Pull Request 🎉
Manual testing. After modification, the backend request can obtain the parameters normally and initiate the corresponding type of engine.
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6227 from JiaLiangC/fix_sqleditor_opensession_parameters.
Closes #6228
e24b2d975 [jialiang] Fix Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor
Authored-by: jialiang <jialiangcaimd@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
db0dd01869 |
[KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Improve DropIgnoreNonexistent rule for spark 3.5
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
DropIgnoreNonexistentSuite
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6215 from wForget/hotfix2.
Closes #6215
cb1d34de1 [wforget] Improve DropIgnoreNonexistent rule for spark 3.5
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
tatian
|
676f111a25
|
[KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
# 🔍 Description
## Issue References 🔗
This pull request fixes #6221
## Describe Your Solution 🔧
Fix the static function that split the sql by parameter placeholder '?', add the handling of cases when the placeholder is within the double quote or comment.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
org.apache.kyuubi.jdbc.hive.UtilsTest.testSplitSqlStatement
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6222 from TakawaAkirayo/fix_sql_split_issue.
Closes #6221
a501c38ea [tatian] [KYUUBI #6221] Let UT cover another case
3b9714569 [tatian] [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
Authored-by: tatian <tatian@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|
||
|
wforget
|
01f7dc07f9 |
[KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
We should check `spark.memory.offHeap.enabled` when applying for `executorOffHeapMemory`.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6211 from wForget/hotfix.
Closes #6211
1c7c8cd75 [wforget] Check memory offHeap enabled for CustomResourceProfileExec
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
69be6cff46 |
[KYUUBI #2782][FOLLOWUP] Add missed UNION_TYPE back
# 🔍 Description
## Issue References 🔗
This pull request fixes #
Fix https://github.com/apache/kyuubi/pull/6207#discussion_r1538539729
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6209 from turboFei/fix_union_type.
Closes #2782
622c838cd [Wang, Fei] miss
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Wang, Fei
|
ab82c6782e |
[KYUUBI #6207] Support to retrieve Spark UserDefinedType result
# 🔍 Description
## Issue References 🔗
This pull request fixes #
To fix below issue:
```
24/03/25 00:47:10 ERROR SparkTBinaryFrontendService: Error getting result set metadata:
java.lang.IllegalArgumentException: Unrecognized type name: struct<type:tinyint,size:int,indices:array<int>,values:array<double>>
```
<img width="1567" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/9067d2d2-06d9-4937-b328-71434def34fd">
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
After this pr:
<img width="1728" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/2d9f4f0b-9ac4-48e9-9e6a-4c0f1616edf9">
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6207 from turboFei/udt.
Closes #6207
b492568e3 [Wang, Fei] Revert "miss union type"
39ac1c42f [Wang, Fei] miss union type
8c32f54af [Wang, Fei] comment
00a469855 [Wang, Fei] getColumnTypeName
d7d291652 [Wang, Fei] ut
edce5cf1f [Wang, Fei] support udt
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
(cherry picked from commit
|
||
|
Cheng Pan
|
208c9ec04e
|
[KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
# 🔍 Description
The function is forked from Apache Spark, and we should align behavior with upstream timely.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6205 from pan3793/quoteIfNeeded.
Closes #6205
2e44fe757 [Cheng Pan] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit
|