Bumps [ws](https://github.com/websockets/ws) from 8.8.1 to 8.17.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p>
<blockquote>
<h2>8.17.1</h2>
<h1>Bug fixes</h1>
<ul>
<li>Fixed a DoS vulnerability (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>).</li>
</ul>
<p>A request with a number of headers exceeding the[<code>server.maxHeadersCount</code>][]
threshold could be used to crash a ws server.</p>
<pre lang="js"><code>const http = require('http');
const WebSocket = require('ws');
<p>const wss = new WebSocket.Server({ port: 0 }, function () {
const chars = "!#$%&'*+-.0123456789abcdefghijklmnopqrstuvwxyz^_`|~".split('');
const headers = {};
let count = 0;</p>
<p>for (let i = 0; i < chars.length; i++) {
if (count === 2000) break;</p>
<pre><code>for (let j = 0; j &lt; chars.length; j++) {
const key = chars[i] + chars[j];
headers[key] = 'x';
if (++count === 2000) break;
}
</code></pre>
<p>}</p>
<p>headers.Connection = 'Upgrade';
headers.Upgrade = 'websocket';
headers['Sec-WebSocket-Key'] = 'dGhlIHNhbXBsZSBub25jZQ==';
headers['Sec-WebSocket-Version'] = '13';</p>
<p>const request = http.request({
headers: headers,
host: '127.0.0.1',
port: wss.address().port
});</p>
<p>request.end();
});
</code></pre></p>
<p>The vulnerability was reported by <a href="https://github.com/rrlapointe">Ryan LaPointe</a> in <a href="https://redirect.github.com/websockets/ws/issues/2230">websockets/ws#2230</a>.</p>
<p>In vulnerable versions of ws, the issue can be mitigated in the following ways:</p>
<ol>
<li>Reduce the maximum allowed length of the request headers using the
[<code>--max-http-header-size=size</code>][] and/or the [<code>maxHeaderSize</code>][] options so
that no more headers than the <code>server.maxHeadersCount</code> limit can be sent.</li>
</ol>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="3c56601092"><code>3c56601</code></a> [dist] 8.17.1</li>
<li><a href="e55e5106f1"><code>e55e510</code></a> [security] Fix crash when the Upgrade header cannot be read (<a href="https://redirect.github.com/websockets/ws/issues/2231">#2231</a>)</li>
<li><a href="6a00029edd"><code>6a00029</code></a> [test] Increase code coverage</li>
<li><a href="ddfe4a804d"><code>ddfe4a8</code></a> [perf] Reduce the amount of <code>crypto.randomFillSync()</code> calls</li>
<li><a href="b73b11828d"><code>b73b118</code></a> [dist] 8.17.0</li>
<li><a href="29694a5905"><code>29694a5</code></a> [test] Use the <code>highWaterMark</code> variable</li>
<li><a href="934c9d6b93"><code>934c9d6</code></a> [ci] Test on node 22</li>
<li><a href="1817bac06e"><code>1817bac</code></a> [ci] Do not test on node 21</li>
<li><a href="96c9b3dedd"><code>96c9b3d</code></a> [major] Flip the default value of <code>allowSynchronousEvents</code> (<a href="https://redirect.github.com/websockets/ws/issues/2221">#2221</a>)</li>
<li><a href="e5f32c7e1e"><code>e5f32c7</code></a> [fix] Emit at most one event per event loop iteration (<a href="https://redirect.github.com/websockets/ws/issues/2218">#2218</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.8.1...8.17.1">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6484 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/ws-8.17.1.
Closes#6484
58a7da865 [dependabot[bot]] ⬆️ Bump ws from 8.8.1 to 8.17.1 in /kyuubi-server/web-ui
Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Address comments https://github.com/apache/kyuubi/issues/4847#issuecomment-2114284381
Now, for `checkEngineConnectionAlive`, it use the client to send `TGetInfoType` to engine and cause the user session never idle for timeout.
## Describe Your Solution 🔧
We shall reuse the alive probe client.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
Pass the current UT.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6468 from turboFei/engine_alive_check.
Closes#4847
e2368b206 [Wang, Fei] reuse
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
The concurrency limit for the engine startup process is mainly used to avoid overload on the machine(or container) of the Kyuubi server, the current implementation holds startupProcessSemaphore until the session is established successfully. While for Spark on YARN cluster mode, some YARN queue resource insufficiency may block the subsequent Spark application submissions to other queues, significantly affecting the Kyuubi server's resource utilization.
## Describe Your Solution 🔧
We should immediately release the `startupProcessSemaphore` after the engine startup process exits (i.e., after the `spark-submit` process exits) as the load has already disappeared.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70%
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6463 from ic4y/master-p003.
Closes#6463
f7de68ce3 [ic4y] Improve code quality
d8b0248df [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency Limit
Authored-by: ic4y <ic4y@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
[`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues.
The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6458 from pan3793/commons-logging.
Closes#6458
114ec766a [Cheng Pan] fix
79d4121a1 [Cheng Pan] fix
6633e83ee [Cheng Pan] fix
21127ed0b [Cheng Pan] always perform install on dep check
98b13dfcf [Cheng Pan] Remove commons-logging from binary release
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Improve data lake dependency management by extracting the following Maven properties:
- `delta.artifact`
- `hudi.artifact`
- `iceberg.artifact`
- `paimon.artifact`
It often takes a while for the downstream data lakes to support the new Spark versions, extracting those properties makes it easy to override in the new profile on the Kyuubi project's `pom.xml` to workaround before data lakes jars are available.
One use case is a19bb7c18e
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6427 from pan3793/datalake-dep.
Closes#6427
74a9300e0 [Cheng Pan] Improve datalake dependency management
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
The `kyuubi-util-scala_2.12-<version>-tests.jar` accidentally leaked to the compile scope but should be in the test scope.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Run `build/dist` and check `dist/jars`
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6439 from pan3793/util-scala-test.
Closes#6439
0576248f5 [Cheng Pan] fix
2bf2408f5 [Cheng Pan] fix
f7151dfc6 [Cheng Pan] kyuubi-util-scala test jar leaked to compile scope
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6428
## Describe Your Solution 🔧
Remove axios request timeout limitation.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Ref to #6428 description.
#### Behavior With This Pull Request 🎉
Get result after a period of time.
#### Related Unit Tests
kyuubi-server/web-ui: `pnpm test` all passed.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6429 from zRains/remove-axios-timeout.
Closes#6428
e947deabe [zRain] fix: remove timeout
Authored-by: zRain <2296342883@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR fixes tests in spark engine and kyuubi server modules with Spark 4.0.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Since Spark 4.0.0-preview1 is still under voting phase, this PR does not add CI, the change was tested in https://github.com/apache/kyuubi/pull/6407 with Spark 4.0.0-preview1 RC1
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6425 from pan3793/spark-4.
Closes#6425
101986416 [Cheng Pan] Fix tests in spark engine and kyuubi server modules with Spark 4.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Merge standard error to output.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6422 from wForget/redirect.
Closes#6422
277d688d5 [wforget] Merge standard error to output
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
We can not manage Jersey dependencies globally because SPARK-47118 (4.0.0) upgrades Jersey from 2 to 3, using an unmatched Jersey version in modules like Spark SQL engine would fail Spark to start thus abort UT.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Tested with Spark 4.0
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6415 from pan3793/server-jersey.
Closes#6415
804d6367b [Cheng Pan] nit
fcb6ee573 [Cheng Pan] manage jersey in kyuubi-server module
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# Description
When `SPARK_HOME` is not set explicitly, the Kyuubi server supports detecting it based on Scala versions, while the rules are not applicable for Spark 4.
This PR enhances the SPARK_HOME detection logic to make it support both Spark 3 and Spark 4.
The above logic is mainly used for testing purposes, the change does not affect users who configure `SPARK_HOME` in `kyuubi-env.sh`.
## Types of changes
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan
#### Related Unit Tests
- `SparkProcessBuilderSuite`
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6413 from pan3793/spark4-home.
Closes#6413
20e71fd7d [Cheng Pan] SPARK_HOME detection supports Spark 4
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6410
## Describe Your Solution 🔧
Check if asyncRequestExecutor is initialized before shutdown to avoid NPE.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6414 from wForget/KYUUBI-6410.
Closes#6410
1b7d35efe [wforget] comment
b83ea22cf [wforget] Check if asyncRequestExecutor is initialized before shutdown
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
`Base64.getMimeEncoder`(RFC2045) might generate newline when encoded chars exceed 76, so I changed it to `Base64.getEncoder`(RFC4648).
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Use the changed BeeLine to connect to Kyuubi Server 1.9.0, everything goes well.
```
$ bin/beeline -u 'jdbc:kyuubi://0.0.0.0:10009/'
...
Connected to: Spark SQL (version 3.4.1)
Driver: Kyuubi Project Hive JDBC Client (version 1.10.0-SNAPSHOT)
Beeline version 1.10.0-SNAPSHOT by Apache Kyuubi
0: jdbc:kyuubi://0.0.0.0:10009/>
```
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6408 from wForget/base64.
Closes#6408
a19f6f64f [wforget] change to Base64.getEncoder/getDecoder
7052a9346 [wforget] change to Base64.getEncoder/getDecoder
1b94ad991 [wforget] Change to Base64 urlEncoder/urlDecoder for handle guid and credential
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11)
Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6392 from pan3793/servlet.
Closes#6392
27d412599 [Cheng Pan] fix
9f1e72272 [Cheng Pan] other spark modules
f4545dc76 [Cheng Pan] fix
313826fa7 [Cheng Pan] exclude
7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6344
`FlinkProcessBuilder` specifies `yarn.ship-files`, `yarn.application.name` and `yarn.tags` configurations of kyuubi platform. Sometimes we also need to customize these configurations, so we should prioritize these user configurations.
## Describe Your Solution 🔧
FlinkProcessBuilder prioritizes user configurations.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
added new unit test
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6342 from wForget/hotfix2.
Closes#6344
feca972ca [wforget] address comment
17df0844d [wforget] fix test and add flink constant
ece91cc0c [wforget] FlinkProcessBuilder prioritizes user configurations
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
# 🔍 Description
## Issue References 🔗
In this pr, engineName and engineUrl will return within KyuubiSessionEvent/SessionData, these information are helpful to get the session info straight forward.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6379 from turboFei/app_name_session_event.
Closes#6379
f9c4b0dfe [Wang, Fei] refine
15b4cfc9d [Wang, Fei] engine info
4662878e6 [Wang, Fei] id name url
9e1d72b9c [Wang, Fei] id name url
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
follow up of #4847
Address comments: https://github.com/apache/kyuubi/issues/4847#issuecomment-2072945805
## Describe Your Solution 🔧
In this pr, when checking the engine terminating, it will ignore the alive probe sessions.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6355 from turboFei/engine_idle.
Closes#4847
a8e26e71d [Wang, Fei] comments
418d0b41c [Wang, Fei] val
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
Support to interrupt the thrift request immediately after marking the engine not alive
# 🔍 Description
## Issue References 🔗
This pull request fixes#6172
## Describe Your Solution 🔧12c5568c9b/kyuubi-server/src/main/scala/org/apache/kyuubi/client/KyuubiSyncThriftClient.scala (L103-L110)
When probe fails and exceeds engineAliveTimeout, not interrupt the thrift request immediately, only marked `remoteEngineBroken` and wait next `engineAliveProbeInterval` to interrupt.
Unit test `KyuubiOperationPerConnectionSuite` assert timeout 3s.
12c5568c9b/kyuubi-server/src/test/scala/org/apache/kyuubi/operation/KyuubiOperationPerConnectionSuite.scala (L344-L346)
Exception log
```
03:25:15.125 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 WARN KyuubiSyncThriftClient: The engine[local-1714879506640] alive probe fails
org.apache.kyuubi.shaded.thrift.transport.TTransportException: Socket is closed by peer.
...
03:25:16.126 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 WARN KyuubiSyncThriftClient: The engine[local-1714879506640] alive probe fails
org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)
...
03:25:16.126 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:BD 23 DE B6 16 56 4E 2B 97 3C 09 74 89 F5 C9 26, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11886 ERROR KyuubiSyncThriftClient: Mark the engine[local-1714879506640] not alive with no recent alive probe success: 2001 ms exceeds timeout 1000 ms
```
Success log
```
16:57:46.859 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 WARN KyuubiSyncThriftClient: The engine[local-1715101059872] alive probe fails
...
16:57:46.860 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 ERROR KyuubiSyncThriftClient: Mark the engine[local-1715101059872] not alive with no recent alive probe success: 1001 ms exceeds timeout 1000 ms
16:57:47.860 engine-alive-probe-TSessionHandle(sessionId:THandleIdentifier(guid:B4 9D 71 83 6D 15 4D 8D BE DA 65 75 27 5D 4E D8, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-11609 WARN KyuubiSyncThriftClient: Removing Clients for TSessionHandle(sessionId:THandleIdentifier(guid:9D AA D5 C2 9B E4 43 D7 BE 81 D0 99 EA 5B 9E 37, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38))
```
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6375 from beryllw/kyuubi_6172.
Closes#6172
991798b86 [wangjunbo] [KYUUBI #6172][TASK][EASY] Support to interrupt the thrift request immediately after marking the engine not alive
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6373
## Describe Your Solution 🔧
`ThriftUtils#EMPTY_ROW_SET` is not suitable for OperationLog , so I add a new `LOG_EMPTY_ROW_SET`
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6378 from wForget/KYUUBI-6373.
Closes#6373
f25aa593f [wforget] revert
ef87b3f4c [wforget] revert and fix npe
7219d35de [wforget] add test
06135fe0e [wforget] fix test
c0bd29078 [wforget] [KYUUBI #6373] Add empty RowSet for OpertationLog
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Return `sessionName` and `totalOperations` within `SessionData`.
Return `appStartTime` with list batches.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6359 from turboFei/total_op.
Closes#6359
5fb3fb8e4 [Wang, Fei] fix compile
fc3970595 [Wang, Fei] comments
0e7ff009e [Wang, Fei] return app time
bb4b64902 [Wang, Fei] dto
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
Unify the words and provide the tag for the waiting submission case.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Manually review.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6366 from pan3793/improve-log.
Closes#6366
30e3fdee0 [Cheng Pan] fix style
a3b36ea33 [Cheng Pan] Improve log message for YARN getApplicationInfoByTag
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6244
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6351 from beryllw/kyuubi-6244.
Closes#6244
603431087 [wangjunbo] [KYUUBI #6244] Fix test logs upload error
460b25985 [wangjunbo] [KYUUBI #6244] Fix test logs upload error
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6338
## Describe Your Solution 🔧
Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6340 from wForget/KYUUBI-6338.
Closes#6338
acd73a16b [wforget] fix
5e64ed3a5 [wforget] comments
48b8ec6cd [wforget] fix style and address comments
ca653492c [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
…t to users
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6329 from yanghua/KYUUBI-6322.
Closes#6322
d081df75c [yanghua] [KYUUBI #6322] Expose more config items of server internal rest client to users
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: yanghua <yanghua1127@gmail.com>
# 🔍 Description
Support to filter sessions/operations with session type.
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6049 from turboFei/batch_interactive.
Closes#6049
68390c774 [Wang, Fei] add ut
bfc2cb8c2 [Fei Wang] save
c1979c7ea [Fei Wang] saev
Lead-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
Now for kubernetes application, there is no runtime AppURL return in the application report.
In fact, we can get the driver svc and get the spark-ui port, then build the Spark UI URL same with that of spark.
074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L67)074ddc2825/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala (L96-L102)
Service Selector example:
```
selector:
kyuubi-unique-tag: bf5fa281-0b2f-4096-aa0a-13a463c147a6
spark-app-selector: spark-bf074093a7994954a89101eb2831bd1e
```
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6318 from turboFei/k8s_spark_ui.
Closes#6321
f77999ef0 [Wang, Fei] with label selector does not work
4e8d5767b [Wang, Fei] typo
3f1eb545c [Wang, Fei] comments
6f0d01d94 [Wang, Fei] ut
9edfde034 [Wang, Fei] ut
65d5e8175 [Wang, Fei] using pattern
9b21d30df [Wang, Fei] Support to get kubernetes app url
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
We found that some pods were not deleted if kyuubi server restarted
The root cause is that:
com.google.common.cache::cleanup will not trigger removalListener action, we shall use `invalidateAll` instead.
Testing:
Nothing print for below code:
```
import com.google.common.cache.{Cache, CacheBuilder, RemovalNotification}
var removed = 0
val cache: Cache[String, String] = CacheBuilder.newBuilder()
.removalListener((notification: RemovalNotification[String, String]) => {
removed += 1
println("removed: " + removed)
}).build()
(0 until 1000).foreach { i =>
cache.put(i.toString, i.toString)
}
cache.cleanup()
```
Replacing `cache.cleanup()` with `cache.invalidateAll()`, the output is expected.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6319 from turboFei/app_k8s_leak.
Closes#6320
ff2adfc60 [Wang, Fei] invalidate all
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6294
## Describe Your Solution 🔧
The netty jars come from `grpc-netty`, `arrow-memory-netty`, use `netty-bom` and `grpc-bom` to simplify the dependency management.
Run `./build/dependency.sh --replace`
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6310 from PorterZhang2021/issue-6294.
Closes#6294
327030170 [Cheng Pan] simplify netty dependency management
cdb435d01 [PorterZhang2021] [# 6294] Prune unused Netty libraries
e0676ed5a [PorterZhang2021] Finished Prune unused Netty libraries [#6294]
Lead-authored-by: PorterZhang2021 <PorterZhang2021@outlook.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6299
## Describe Your Solution 🔧
when disabling web ui, return 404 page
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6311 from lsm1/branch-support-disable-webui.
Closes#6299
aa96c2737 [senmiaoliu] remove enable.html
998504710 [senmiaoliu] fix style
a2622cbbc [senmiaoliu] disable web ui
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
* [KYUUBI #6278] Update DB state when the query fallback to yarn and the batch app is terminal
* update all engine info
---------
Co-authored-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6253
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
<img width="1251" alt="image" src="https://github.com/apache/kyuubi/assets/18713676/b654a300-8c79-4461-9fba-4ad1c913accc">
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6275 from lsm1/branch-jdbc-engine-on-yarn.
Closes#6253
5ed4af041 [senmiaoliu] fix style
86e032688 [senmiaoliu] fix style
b3e114445 [senmiaoliu] fix style
bf33de300 [senmiaoliu] fix style
c38404918 [senmiaoliu] jdbc engine on yarn
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
# 🔍 Description
as title
## Issue References 🔗
This pull request fixes#6291
## Describe Your Solution 🔧
Add a rule to check if it's null
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6292 from zhaohehuhu/dev-0411.
Closes#6291
a58eed609 [hezhao2] refactor
40ad5e0fa [hezhao2] Avoid NPE in MySQLErrPacket
e02be17b4 [dupeng] [KYUUBI #6288] [KYUUBI #6283] Improve the message of distribution built without enabling web ui
Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
Improve message of distribution built without enabling web ui.
## Issue References 🔗
This pull request fixes#6283
## Describe Your Solution 🔧
Here is my implement:
<img width="1512" alt="image" src="https://github.com/apache/kyuubi/assets/34719039/fc5e859e-1c92-44a1-8e27-316986c5d9c3">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6288 from dupen01/issue-6283.
Closes#6288
a5886e271 [dupeng] revise the message of page
bad14a748 [dupeng] improve the message of distribution built without enabling web ui
Authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
when some text is selected, only send the selected text.
## Issue References 🔗
This pull request fixes#6248
## Describe Your Solution 🔧
The Monaco Editor API is called to get the selected SQL statement, and without any other dependency.
<img width="581" alt="image" src="https://github.com/apache/kyuubi/assets/34719039/ce6ac5ae-b1fc-4778-a57e-dfda9ad51275">
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6272 from dupen01/issue-6248.
Closes#6248
67d47cd4e [dupeng] Improved nonempty judgment
e0767bb64 [dupeng] Finished task "Web SQL Editor supports run SQL snippets #6248"
Authored-by: dupeng <dunett@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6216
## Describe Your Solution 🔧
Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
SessionLimiterSuite.test("test session limiter with ip deny list")
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6217 from wForget/KYUUBI-6216.
Closes#6216
e0a058c1d [wforget] update ctl
f66426eda [wforget] typo
5c5308ad8 [wforget] typo
959242e9b [wforget] Support to deny some client ips to make connection
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6249.
## Describe Your Solution 🔧
According to the plan, we will no longer support Flink 1.16, and this PR will try to remove Flink 1.16.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GHA
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6259 from slfan1989/drop_flink1.16_support.
Closes#6249
808acc2fb [Cheng Pan] Update docs/deployment/migration-guide.md
fc5ecf6fd [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
8d8f9ded4 [Shilun Fan] [KYUUBI #6249] Fix CheckStyle.
7a1d974b3 [Shilun Fan] [KYUUBI #6249] Drop support for Flink 1.16.
Lead-authored-by: Shilun Fan <slfan1989@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 4.5.2 to 4.5.3.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/vitejs/vite/blob/v4.5.3/packages/vite/CHANGELOG.md">vite's changelog</a>.</em></p>
<blockquote>
<h2>4.5.3 (2024-03-24)</h2>
<ul>
<li>fix: <code>fs.deny</code> with globs with directories (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16250">#16250</a>) (<a href="https://github.com/vitejs/vite/commit/96a7f3a">96a7f3a</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16250">#16250</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="aac695e9f8"><code>aac695e</code></a> release: v4.5.3</li>
<li><a href="96a7f3a41e"><code>96a7f3a</code></a> fix: <code>fs.deny</code> with globs with directories (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16250">#16250</a>)</li>
<li>See full diff in <a href="https://github.com/vitejs/vite/commits/v4.5.3/packages/vite">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6257 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/vite-4.5.3.
Closes#6257
2e4d3be38 [Cheng Pan] Update kyuubi-server/web-ui/pnpm-lock.yaml
4cbc7f170 [Cheng Pan] Update kyuubi-server/web-ui/pnpm-lock.yaml
15ad7ec63 [Cheng Pan] Update kyuubi-server/web-ui/pnpm-lock.yaml
81f63e412 [Cheng Pan] Update kyuubi-server/web-ui/pnpm-lock.yaml
c7b51991f [Cheng Pan] Update kyuubi-server/web-ui/pnpm-lock.yaml
c237e67dc [dependabot[bot]] ⬆️ Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
Convert the spark batch conf key if not start with `spark.`, it is impacted by #5767
```
protected def convertConfigKey(key: String): String = {
if (key.startsWith("spark.")) {
key
} else if (key.startsWith("hadoop.")) {
"spark.hadoop." + key
} else {
"spark." + key
}
}
```
## Describe Your Solution 🔧
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6256 from turboFei/convert_key.
Closes#5767
d76655e05 [Wang, Fei] Fix break
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6223
Even the user specify `spark.jars`, but they can not access the classes in jars with Scala code.
## Describe Your Solution 🔧
Add the jars into repl classpath.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
UT.
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6235 from turboFei/scala_repl_urls.
Closes#6223
344502660 [Wang, Fei] scala 2.13
cc6e28989 [Wang, Fei] todo
a8b373167 [Wang, Fei] refine
65b438ccf [Wang, Fei] remove scala reflect check
eb257c7a8 [Wang, Fei] using -classpath
e1c6f0e11 [Wang, Fei] revert 2.13
15d37662d [Wang, Fei] repl
41ebe1011 [Wang, Fei] fix ut
ed5d344f8 [Wang, Fei] info
1cdd82ab4 [Wang, Fei] comment
aa4292dac [Wang, Fei] fix
a10cfa5e0 [Wang, Fei] ut
63fdb8877 [Wang, Fei] Use global.classPath.asURLs instead of class loader urls
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
## Issue References 🔗
This pull request fixes https://github.com/apache/kyuubi/issues/6228
issue Description:
Using the branch kyuubi 1.9.0
1. The kyuubi sql editor frontend dropdown menu cannot select the submission type.
Solution: Set disabled to false.
Issue 2: After making the change in step one, the sql editor can select different types of sql type, but regardless of the selected type, the backend always initiates a spark engine. Upon investigation, it was found that the frontend parameter settings were inappropriate. The key in the JSON and the constructor parameter names in the backend request object did not match, leading to parsing failure, and the parameters were always null.
Solution: Modify the parameter name.
## Types of changes 🔖
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
The kyuubi sql editor frontend dropdown menu cannot select the submission type.
regardless of the selected type, the backend always initiates a spark engine
#### Behavior With This Pull Request 🎉
Manual testing. After modification, the backend request can obtain the parameters normally and initiate the corresponding type of engine.
#### Related Unit Tests
---
# Checklist 📝
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6227 from JiaLiangC/fix_sqleditor_opensession_parameters.
Closes#6228
e24b2d975 [jialiang] Fix Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor
Authored-by: jialiang <jialiangcaimd@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request implement a feature - Run HiveSQLEngine on kerberized YARN
## Describe Your Solution 🔧
Introduced two configs:
- kyuubi.engine.principal
- kyuubi.engine.keytab
When do submit to a kerberized YARN, submitter uploads `kyuubi.engine.keytab` to application's staging dir.
YARN NodeManager downloads keytab to AM's working directory. AM logins to Kerberos using the principal and keytab
**Note**
I've tried to run HiveSQLEngine with only DelegationTokens but failed.
Take SQL `SELECT * FROM a` as an example:
Hive handles this simple TableScan SQL by reading directly from table's hdfs file.
When Hive invokes `FileInputFormat.getSplits` during reading, `java.io.IOException: Delegation Token can be issued only with kerberos or web authentication` will be thrown.
The simplified stacktrace from IDEA is as below:
```
getDelegationToken:734, DFSClient (org.apache.hadoop.hdfs)
getDelegationToken:2072, DistributedFileSystem (org.apache.hadoop.hdfs)
collectDelegationTokens:108, DelegationTokenIssuer (org.apache.hadoop.security.token)
addDelegationTokens:83, DelegationTokenIssuer (org.apache.hadoop.security.token)
obtainTokensForNamenodesInternal:143, TokenCache (org.apache.hadoop.mapreduce.security)
obtainTokensForNamenodesInternal:102, TokenCache (org.apache.hadoop.mapreduce.security)
obtainTokensForNamenodes:81, TokenCache (org.apache.hadoop.mapreduce.security)
listStatus:221, FileInputFormat (org.apache.hadoop.mapred)
getSplits:332, FileInputFormat (org.apache.hadoop.mapred)
getNextSplits:372, FetchOperator (org.apache.hadoop.hive.ql.exec)
getRecordReader:304, FetchOperator (org.apache.hadoop.hive.ql.exec)
getNextRow:459, FetchOperator (org.apache.hadoop.hive.ql.exec)
pushRow:428, FetchOperator (org.apache.hadoop.hive.ql.exec)
fetch:147, FetchTask (org.apache.hadoop.hive.ql.exec)
getResults:2208, Driver (org.apache.hadoop.hive.ql)
getNextRowSet:494, SQLOperation (org.apache.hive.service.cli.operation)
getNextRowSetInternal:105, HiveOperation (org.apache.kyuubi.engine.hive.operation)
```
Theoretically, it can be solved by add AM DelegationTokens into
`org.apache.hadoop.hive.ql.exec.FetchOperator.job.credentials`.
But actually, it is impossible without modifying Hive's source code.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
HiveSQLEngine can not run on a kerberized YARN
#### Behavior With This Pull Request 🎉
HiveSQLEngine can run on a kerberized YARN
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6199 from zhouyifan279/kerberized-hive-engine-on-yarn.
Closes#6199
383d1cdcb [zhouyifan279] Fix tests
458493a91 [zhouyifan279] Warn if run Hive on YARN without principal and keytab
118afe280 [zhouyifan279] Warn if run Hive on YARN without principal and keytab
41fed0c44 [zhouyifan279] Ignore Principal&Keytab when hadoop security is no enabled.
9e2d86237 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/hive/HiveProcessBuilder.scala
5ae0a3eac [zhouyifan279] Remove redundant checks
5d3013aaf [zhouyifan279] Use principal & keytab in Local mode
5733dfdcb [zhouyifan279] Use principal & keytab in Local mode
85ce9bb7a [zhouyifan279] Use principal & keytab in Local mode
061223dbe [zhouyifan279] Resolve comments
e706936e7 [zhouyifan279] Resolve comments
f84c7bccc [zhouyifan279] Support run HiveSQLEngine on kerberized YARN
4d262c847 [zhouyifan279] Support run HiveSQLEngine on kerberized YARN
Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
We found that, a user has no active sessions, but kyuubi said that the user reach the max limit sessions per user.
Now, we increase the session limiter for user when opening session and decrease it when closing session.
But if the user open session failed, it will not decrease the session limiter.
This pr fix session limiter leak issue when failed to open session.
Before open session, add session handle into sessionHandleMap, and invoke SessionManager::closeSession when failed to open session.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6204 from turboFei/limiter_leak.
Closes#6204
c0f2969fc [Wang, Fei] refine
98fda9438 [Wang, Fei] fix leak
Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
# 🔍 Description
## Issue References 🔗
We meet some issue, I want to check the operation sessionHandle and the operation timeline.
I found that, it is difficult for me to check it from kyuubi log.
So, In this pr, I log the operation change log into separate file.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6185 from turboFei/audit_operation.
Closes#6185
38b01921c [Wang, Fei] op audit
Lead-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.4 to 1.15.6.
<details>
<summary>Commits</summary>
<ul>
<li><a href="35a517c586"><code>35a517c</code></a> Release version 1.15.6 of the npm package.</li>
<li><a href="c4f847f851"><code>c4f847f</code></a> Drop Proxy-Authorization across hosts.</li>
<li><a href="8526b4a1b2"><code>8526b4a</code></a> Use GitHub for disclosure.</li>
<li><a href="b1677ce001"><code>b1677ce</code></a> Release version 1.15.5 of the npm package.</li>
<li><a href="d8914f7982"><code>d8914f7</code></a> Preserve fragment in responseUrl.</li>
<li>See full diff in <a href="https://github.com/follow-redirects/follow-redirects/compare/v1.15.4...v1.15.6">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/kyuubi/network/alerts).
</details>
Closes#6188 from dependabot[bot]/dependabot/npm_and_yarn/kyuubi-server/web-ui/follow-redirects-1.15.6.
Closes#6188
4f36009cb [Cheng Pan] fix
b4de22f7c [dependabot[bot]] ⬆️ Bump follow-redirects in /kyuubi-server/web-ui
Lead-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes#6165
## Describe Your Solution 🔧
Add HttpClient options for engine ui proxy servlet.
- idleTimeout
- maxConnections
- maxThreads
- requestBufferSize
- responseBufferSize
- timeout
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
Engine UI proxy requests with large headers( > 4k) are failed with 500 http status.
#### Behavior With This Pull Request 🎉
Increase the value of `kyuubi.frontend.rest.proxy.jetty.client.requestBufferSize` properly(ex: 5k), request with large header has succeeded with 200 status.
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6192 from minyk/kyuubi-6165.
Closes#6165
6ef26aa43 [minyk] fix kyuubi version of configs
1812a96eb [minyk] apply spotless
3527cada9 [minyk] set initial parameters to the httpclient of proxy servletholder.
Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
They have stopped patching the JDK 1.5 jars that Hadoop uses (see [HADOOP-18540](https://issues.apache.org/jira/browse/HADOOP-18540)).
The new artifacts have similar names - but the names are like bcprov-jdk18on as opposed to bcprov-jdk15on.
CVE-2023-33201 is an example of a security issue that seems only to be fixed in the JDK 1.8 artifacts (ie no JDK 1.5 jar has the fix).
https://www.bouncycastle.org/releasenotes.html#r1rv77 latest current release but the CVE was fixed in 1.74.
To be clear, Kyuubi only uses BouncyCastle for testing, the CVE does not affect Kyuubi distribution.
## Describe Your Solution 🔧
Bump BouncyCastle from 1.67 to 1.77, and change the artifactId from `*-jdk15on` to `*jdk18on`.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
Pass GA.
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6177 from pan3793/bouncycastle.
Closes#6177
8595b98c1 [Cheng Pan] Bump BouncyCastle from 1.67 to 1.77
b9e7123f6 [Cheng Pan] Bump bouncycastle from 1.67 to 1.77
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
