Commit Graph

691 Commits

Author SHA1 Message Date
Cheng Pan
6eb24bc47a
[KYUUBI #7164] Bump Byte Buddy 1.17.6
### Why are the changes needed?

Support JDK 25

Full Release Notes: https://github.com/raphw/byte-buddy/releases

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7164 from pan3793/bytebuddy-1.17.6.

Closes #7164

9d4f45a8c [Cheng Pan] Revert "Bump maven shade plugin 3.6.0"
d3dc66862 [Cheng Pan] upgrade asm for maven-shade-plugin
9afe01915 [Cheng Pan] Bump maven shade plugin 3.6.0
1b8a99d71 [Cheng Pan] Bump Byte Buddy 1.17.6

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-08-08 01:22:29 +08:00
Cheng Pan
dc9c75b8e6
[KYUUBI #7162] Bump kafka-clients 3.9.1
### Why are the changes needed?

https://kafka.apache.org/cve-list.html

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7162 from pan3793/kafka-3.9.1.

Closes #7162

108e5690b [Cheng Pan] Bump kafka-clients 3.9.1

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-08-05 16:58:59 +08:00
Cheng Pan
d2cfc871be
[KYUUBI #7144] Remove direct usage of sun.misc.Signal
### Why are the changes needed?

This PR replaces the `sun.misc.Signal` with the Kyuubi wrapped one, see https://github.com/apache/kyuubi-shaded/pull/64, which allows Kyuubi to use any of Java 8+ to compile with `-release:8` while still ensuring compatibility with Java 8.

### How was this patch tested?

Pass GHA.

Local tested by building against JDK 21, running on JDK 8, everything works as expected.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7144 from pan3793/signal.

Closes #7144

64804060a [Cheng Pan] enforcer.maxJdkVersion
2eb263573 [Cheng Pan] cleanup
b699c345a [Cheng Pan] remove staging repo
12eeae3ab [Cheng Pan] fix
08eba7695 [Cheng Pan] RC
53e4e6cce [Cheng Pan] Use Kyuubi Signal

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-07-28 15:29:27 +08:00
Cheng Pan
31ce37f526
[KYUUBI #7147] Enforce to use Kyuubi's PlainSASLServer for PLAIN mechanism
### Why are the changes needed?

Close #7147

### How was this patch tested?

UT is updated, I also tested it with log4j2 kafka appended (kafka uses SASL/PLAIN authN).

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7145 from pan3793/sasl-plain.

Closes #7147

dd2f7b0a8 [Cheng Pan] rm staging repo
c5c622918 [Cheng Pan] SASL PLAIN
a5331a624 [Cheng Pan] RC

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-07-24 15:18:43 +08:00
Cheng Pan
8bea66a412
[KYUUBI #7151] Bump commons-lang3 3.18.0
### Why are the changes needed?

This version contains CVE-2025-48924 fix.

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7151 from pan3793/lang3-3.18.

Closes #7151

fbbedce33 [Cheng Pan] Bump commons-lang3 3.18.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-07-24 14:46:51 +08:00
Cheng Pan
364f062852
[KYUUBI #7103] Bump Delta 4.0.0 and enable Delta tests for Spark 4.0
### Why are the changes needed?

https://github.com/delta-io/delta/releases/tag/v4.0.0

### How was this patch tested?

GHA.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #7103 from pan3793/delta-4.0.

Closes #7103

febaa11ab [Cheng Pan] Bump Delta 4.0.0 and enable Delta tests for Spark 4.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-06-19 13:32:17 +08:00
Cheng Pan
7313042c0f
[KYUUBI #7104] Bump Maven 3.9.10
### Why are the changes needed?

Upgrade Maven to the latest version to speed up `build/mvn` downloading, as the previous versions are not available at https://dlcdn.apache.org/maven/maven-3/

### How was this patch tested?

Pass GHA,

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7104 from pan3793/maven-3.9.10.

Closes #7104

48aa9a232 [Cheng Pan] Bump Maven 3.9.10

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-06-19 12:54:28 +08:00
Cheng Pan
526110dfe7
[KYUUBI #6928] Bump Spark 4.0.0
### Why are the changes needed?

Test Spark 4.0.0 RC1
https://lists.apache.org/thread/3sx86qhnmot1p519lloyprxv9h7nt2xh

### How was this patch tested?

GHA.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6928 from pan3793/spark-4.0.0.

Closes #6928

a910169bd [Cheng Pan] Bump Spark 4.0.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-05-26 12:23:22 +08:00
Cheng Pan
873503e662
[KYUUBI #7073] Retry 3 times on deploying to nexus
### Why are the changes needed?

Retry on deploying failure to overcome the transient issues.

### How was this patch tested?

Review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7073 from pan3793/deploy-retry.

Closes #7073

f42bd663b [Cheng Pan] Retry 3 times on deploying to nexus

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-05-23 01:55:40 +08:00
Cheng Pan
fc654cfbd3
[KYUUBI #7063] Bump Kyuubi Shaded 0.5.0
### Why are the changes needed?

https://kyuubi.apache.org/shaded-release/0.5.0.html

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7063 from pan3793/kyuubi-shaded-0.5.0.

Closes #7063

b202a7c83 [Cheng Pan] Update pom.xml
417914529 [Cheng Pan] Bump Kyuubi Shaded 0.5.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-05-22 17:19:07 +08:00
Cheng Pan
4df682bf83
[KYUUBI #7062] Bump Delta Lake 3.3.1
### Why are the changes needed?

https://github.com/delta-io/delta/releases/tag/v3.3.1

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7062 from pan3793/delta-3.3.1.

Closes #7062

0fc1df8f9 [Cheng Pan] Bump DeltaLake 3.3.1

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-05-15 22:29:31 +08:00
Wang, Fei
70c03ef998
[KYUUBI #7046] Bump dropwizard metrics version to 4.2.30
### Why are the changes needed?

Bump to use the latest 4.2.x, seems mainly for dependency upgrading.

- https://github.com/dropwizard/metrics/releases/tag/v4.2.27
- https://github.com/dropwizard/metrics/releases/tag/v4.2.28
- https://github.com/dropwizard/metrics/releases/tag/v4.2.29
- https://github.com/dropwizard/metrics/releases/tag/v4.2.30

### How was this patch tested?

GA.
### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7046 from turboFei/bump_codahale.

Closes #7046

51d2a5522 [Wang, Fei] Bump codahale metrics version to 4.2.30

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-04-27 15:56:27 +08:00
Wang, Fei
4cbff4d192 [KYUUBI #7045] Expose jetty metrics
### Why are the changes needed?

Expose the jetty metrics to help detect issue.
Refer: https://metrics.dropwizard.io/4.2.0/manual/jetty.html

### How was this patch tested?

<img width="1425" alt="image" src="https://github.com/user-attachments/assets/ac8c9a48-eaa1-48ee-afec-6f33980d4270" />

<img width="1283" alt="image" src="https://github.com/user-attachments/assets/c2fa444b-6337-4662-832b-3d02f206bd13" />

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7045 from turboFei/metrics_jetty.

Closes #7045

122b93f3d [Wang, Fei] metrics
45a73e7cd [Wang, Fei] metrics

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
2025-04-25 00:02:56 -07:00
Wang, Fei
3e638b61c3 [KYUUBI #7044] Bump jetty version to 9.4.57.v20241219
### Why are the changes needed?

Bump to latest jetty 9.x, see https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.57.v20241219

<img width="1222" alt="image" src="https://github.com/user-attachments/assets/8d2734ee-52f2-4ba9-b22a-660e3b202b7f" />

### How was this patch tested?

GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7044 from turboFei/jetty_version.

Closes #7044

740560dd0 [Wang, Fei] Upgrade jetty version

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
2025-04-25 00:01:34 -07:00
Wang, Fei
fa99183354 [KYUUBI #7023] Upgrade kubernetes client version to 6.13.5
### Why are the changes needed?

Upgrade the kubernetes client, https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.5

### How was this patch tested?

GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7023 from turboFei/k8s_client.

Closes #7023

3e3ac634f [Wang, Fei] 6.16.5
df5aa011f [Wang, Fei] upgrade

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
2025-04-15 11:47:13 -07:00
Cheng Pan
22ce315870
[KYUUBI #6965] Bump Scala 2.13.16
### Why are the changes needed?

Scala 2.13.16 release notes: https://github.com/scala/scala/releases/tag/v2.13.16

Also, bump the silencer plugin to 1.7.19 to support Scala 2.13.16

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6965 from pan3793/scala-2.13.6.

Closes #6965

141e75864 [Cheng Pan] Bump silencer 1.7.19
9f09127e7 [Cheng Pan] Bump Scala 2.13.16

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-03-07 23:32:33 +08:00
Wang, Fei
a305bdc035
[KYUUBI #6960] Bump log4j version to 2.24.3 to fix ConcurrentModificationException in LoggerContext
### Why are the changes needed?

Bump the log4j version to fix below issue:
```
2025-03-04 22:27:58.291 WARN [main-SendThread(xxxx:2181)] org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
: java.lang.ExceptionInInitializerError
        at org.apache.log4j.Logger.getLogger(Logger.java:35)
        at org.apache.kyuubi.shaded.zookeeper.Login.<init>(Login.java:44)
        at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:228)
        at org.apache.kyuubi.shaded.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131)
        at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:990)
        at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1042)
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1657)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
        at java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1216)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
        at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
        at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
        at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:379)
        at org.apache.logging.log4j.core.config.Configurator.setLevel(Configurator.java:344)
        at org.apache.log4j.legacy.core.CategoryUtil.setLevel(CategoryUtil.java:131)
        at org.apache.log4j.Category.setLevel(Category.java:643)
        at org.apache.log4j.Category.setLevel(Category.java:638)
        at org.apache.log4j.spi.RootLogger.setLevel(RootLogger.java:60)
        at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:70)
        ... 6 more
```

It is fixed in https://github.com/apache/logging-log4j2/releases/tag/rel%2F2.24.3

https://github.com/apache/logging-log4j2/issues/3234
### How was this patch tested?

Existing GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6960 from turboFei/log4j_version.

Closes #6960

48b076c3e [Wang, Fei] Upgrade log4j version to 2.24.3

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-03-05 15:03:18 +08:00
Cheng Pan
d5b01fa3e2
[KYUUBI #6939] Bump Spark 3.5.5
### Why are the changes needed?

Test Spark 3.5.5 Release Notes

https://spark.apache.org/releases/spark-release-3-5-5.html

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6939 from pan3793/spark-3.5.5.

Closes #6939

8c0288ae5 [Cheng Pan] ga
78b0e72db [Cheng Pan] nit
686a7b0a9 [Cheng Pan] fix
d40cc5bba [Cheng Pan] Bump Spark 3.5.5

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-03-03 13:42:09 +08:00
Bowen Liang
0f4906d1b0 [KYUUBI #6674] Bump Scalafmt to 3.9.x
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Without this PR:
- use and conform Scalafmt 3.7.x
- forcing all the imports from the same parent package, which is even violation the max length of the line

With this PR:
- use and conform Scalafmt 3.9.x
- Scalafmt 3.8.2 changes the binpack style: https://github.com/scalameta/scalafmt/releases/tag/v3.8.2
- change to `binPack.importSelectors=singleLine`, minimizing the impacts to existed code

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6674 from bowenliang123/scalafmt38.

Closes #6674

d9f4f9ddc [Bowen Liang] 3.9.0
438a07d61 [Bowen Liang] reformat
2df50669c [liangbowen] bump scalafmt to 3.8

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2025-02-20 11:00:14 +08:00
Cheng Pan
a5301e52b9
[KYUUBI #6914] Bump Iceberg 1.8.0
### Why are the changes needed?

https://iceberg.apache.org/releases/#180-release

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6914 from pan3793/iceberg-1.8.0.

Closes #6914

f7e9b7e51 [Cheng Pan] fix
f44dee4e6 [Cheng Pan] Use Iceberg 1.7 for Spark 3.3
cd0bdda6a [Cheng Pan] rm staging repo
881a01581 [Cheng Pan] Bump Iceberg 1.8.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-02-14 17:07:09 +08:00
Cheng Pan
eda385e9e3
[KYUUBI #6916] Bump Delta 3.3.0 for Spark 3.5
### Why are the changes needed?

https://github.com/delta-io/delta/releases/tag/v3.3.0

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6916 from pan3793/delta-3.3.0.

Closes #6916

51f6c1cdf [Cheng Pan] Bump Delta 3.3.0 for Spark 3.5

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-02-14 10:26:59 +08:00
Cheng Pan
81742586e8
[KYUUBI #6917] Bump Hudi 1.0.1
### Why are the changes needed?

https://hudi.apache.org/releases/release-1.0.1

### How was this patch tested?

Pass GHA

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6917 from pan3793/hudi-1.0.1.

Closes #6917

b25414bd3 [Cheng Pan] Bump Hudi 1.0.1

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-02-14 10:25:19 +08:00
Cheng Pan
d49c6314d0
[KYUUBI #6915] Fix ClickHouse integration tests
### Why are the changes needed?

I observed ClickHouse integration test failure in GHA, after some investigation, the root cause is https://github.com/testcontainers/testcontainers-java/pull/9942

```
/entrypoint.sh: neither CLICKHOUSE_USER nor CLICKHOUSE_PASSWORD is set, disabling network access for user 'default'
```

In short, the recent ClickHouse docker image does not allow the `default` user to connect without a password, unfortunately, `testcontainers-scala-clickhosue` does not expose API to set CLICKHOSUE_USER and CLICKHOUSE_PASSWORD, as a workaround, I pin `clickhouse-server:24.3.15`(the latest version has no such restriction) until a fixed version of Testcontainers available.

This PR also switches the `clickhouse-jdbc`'s classifier from `http` to `shaded`, the reason is, `http` does not ship ApacheHttpClient5, previously, it happened to work because `iceberg-runtime-spark3.5_2.12` packaged un-relocated ApacheHttpClient5 classes, but it gets fixed in Iceberg 1.8.0, then `clickhouse-jdbc:http` stop working.
```
java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpRequest
```

Additionally, this PR bumps `clickhouse-jdbc` from 0.6.0 to 0.6.5.

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6915 from pan3793/fix-ch-test.

Closes #6915

996f095e0 [Cheng Pan] Pin clickhouse-server:24.3.15
d633df07c [Cheng Pan] Bump clickhouse-jdbc 0.6.5
214c8a227 [Cheng Pan] Fix ClickHouse integration tests

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-02-14 10:23:20 +08:00
Wang, Fei
8742334073 [KYUUBI #6871] Revert "[KYUUBI #6865] [TEST] Set kyuubi.zookeeper.embedded.client.port.address to localhost for testing"
This reverts commit e8cbff32d4.

### Why are the changes needed?

### How was this patch tested?

### Was this patch authored or co-authored using generative AI tooling?

Closes #6871 from turboFei/revert_zk.

Closes #6871

Closes #6865

1c61f9d85 [Wang, Fei] Revert "[KYUUBI #6865] [TEST] Set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing"

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
2024-12-26 10:08:26 -08:00
Wang, Fei
e8cbff32d4 [KYUUBI #6865] [TEST] Set kyuubi.zookeeper.embedded.client.port.address to localhost for testing
### Why are the changes needed?

Similar with `kyuubi.frontend.bind.host`, it is better to set `kyuubi.zookeeper.embedded.client.port.address` to `localhost` for testing as well.

### How was this patch tested?

GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6865 from turboFei/zk_bind_host.

Closes #6865

9044e284b [Wang, Fei] [TEST] Set kyuubi.zookeeper.embedded.client.port.address for testing

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
2024-12-25 22:07:44 -08:00
Cheng Pan
14e12e9aa4
[KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
### Why are the changes needed?

It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.

To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.

PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.

### How was this patch tested?

Manually test.

Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
  <Appenders>
    <Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
      <JsonTemplateLayout>
        <EventTemplateAdditionalField key="app" value="kyuubi"/>
        <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
        <EventTemplateAdditionalField key="host" value="${hostName}"/>
      </JsonTemplateLayout>
      <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
      <Property name="compression.type" value="gzip"/>
    </Kafka>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="kafka"/>
    </Root>
  </Loggers>
</Configuration>
```

Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
![Xnip2024-12-25_03-18-52](https://github.com/user-attachments/assets/e1b5853a-3800-4363-8ce4-7e78d0928c6a)

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6861 from pan3793/structured-logging.

Closes #6861

9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-25 17:22:53 +08:00
Cheng Pan
1d1e8a0a3b
[KYUUBI #6842] Bump Spark 3.5.4
### Why are the changes needed?

Spark 3.5.4 is released https://spark.apache.org/releases/spark-release-3-5-4.html

### How was this patch tested?

Pas GHA

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6842 from pan3793/spark-3.5.4.

Closes #6842

0fb7ad8a0 [Cheng Pan] ga
8eacc9c97 [Cheng Pan] Spark 3.5.4 RC2
0721fa401 [Cheng Pan] fix
49e98a201 [Cheng Pan] maven repo
951db0c82 [Cheng Pan] Spark 3.5.4

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-23 11:21:45 +08:00
Cheng Pan
f2cacd3f18
[KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
### Why are the changes needed?

I'd like to include `kafka-clients` in the Kyuubi binary distribution tarball to enable the out-of-box support for sinking Kyuubi events to Kafka.

- Kafka is an important component in modern data platforms, and is a defacto message queue implementation, especially in the big data domain
- `kafka-clients` is released under Apache License V2, has no legal issue
- `kafka-clients` is quite a light lib, has no third-party deps except for `slf4j-api` and a few optional compression libs
- `kafka-clients` uses "none" compression as default, in practice, "gzip"(delegate to JDK gzip algorithm, no additional libs are required) already performs well for non-extreme cases

Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking logging to Kafka, which also requires `kafka-clients` in the classpath, I have some initial ideas to forward both Kyuubi server's and engine bootstrap processes log to Kafka in a structured format, will send another PR to add docs to guide users in configuring that.

### How was this patch tested?

Review

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6836 from pan3793/kafka-lib.

Closes #6836

b069eb199 [Cheng Pan] Ship kafka-clients in binary distribution tarball without compression libs

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-05 13:59:29 +08:00
Cheng Pan
9c8b2c3b32
[KYUUBI #6838] [INFRA] Update archive mailing list address
### Why are the changes needed?

Replace the obsolete address with a new one.

### How was this patch tested?

Review

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6838 from pan3793/mail.

Closes #6838

858e6cc72 [Cheng Pan] [INFRA] Update archive mailing list address

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-04 23:35:01 +08:00
naive-zhang
eb1b5996c9
[KYUUBI #6815] JDBC Engine supports Oracle
# Description

Currently, Kyuubi supports JDBC engines with limited dialects, and I extend the dialects to support Oracle.
* Introduce Oracle support in JDBC Engine
* Adding dialects and tests for Oracle

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Add tests of `OperationWithOracleEngineSuite`, `OracleOperationSuite`, `OracleSessionSuite` and `OracleStatementSuite`.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6815 from naive-zhang/jdbc-oracle.

Closes #6815

0ffad5b6b [native-zhang] add some brief comments on the caller side for the implementation of Oracle JDBC engine
6f469a135 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ae70710e6 [Cheng Pan] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
171d06b9e [native-zhang] use another implementation of transform decimal into int, in engine instead of KyuubiBaseResultSet
7cb74d28e [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
ccd7cae8b [naive-zhang] remove redundant override methods in OracleSQLDialect.scala
a7da4a646 [naive-zhang] remove redundant impl of getTableTypesOperation in OracleSQLDialect.scala
70b49fcba [naive-zhang] Use the single line string if SQL fits in one line, otherwise  write it in a pretty style
e58348460 [naive-zhang] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala
b33e97a08 [naive-zhang] remove redundant testcontainers-scala-oracle-xe dependency in pom.xml
4c967b98e [naive-zhang] use gvenzl/oracle-free:23.5-slim with docker-compose for test case
0215e6d49 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
d688b4706 [naive-zhang] change oracle image into gvenzl/oracle-free:23.5-slim
abf983727 [naive-zhang] fix code style checking error in KyuubiConf.scala
d1e82edb1 [naive-zhang] fix code style checking error in settings.md
aa2e2e9ba [naive-zhang] adjust wired space in OracleSQLDialect
b43cea421 [naive-zhang] add oracle configuration for kyuubi.engine.jdbc.connection.provider
397c1cfec [naive-zhang] Merge branch 'apache:master' into jdbc-oracle
2f1b5ed0b [naive-zhang] add jdbc support for Oracle

Lead-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: native-zhang <xinsen.zhang.0571@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-02 23:41:57 +08:00
chengpeiming
bfa9f2ac4f
[KYUUBI #6826] Bump log4j from 2.24.1 to 2.24.2
### Why are the changes needed?
As  pan3793  pointed out that version 2.24.1 had critical bug in https://github.com/apache/kyuubi/pull/6774#issuecomment-2499523805
Bump bump log4j from 2.24.1 to 2.24.2
- Release:https://logging.apache.org/log4j/2.x/release-notes.html#release-notes-2-24-2

### How was this patch tested?

GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6826 from pionCham/bump-log4j-version.

Closes #6826

150713f02 [chengpeiming] Bump log4j to 2.24.2

Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: chengpeiming <chengpeiming@chengpeimingdeMacBook-Pro.local>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-11-29 00:29:49 +08:00
Wang, Fei
cbbb0622bd
[KYUUBI #6818] Upgrade mysql jdbc version to 8.4.0
# 🔍 Description
## Issue References 🔗

Fix CVE: https://github.com/advisories/GHSA-m6vm-37g8-gqvh
## Describe Your Solution 🔧

Upgrade to 8.4.0, the latest 8.x version.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

GA.
---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6818 from turboFei/mysql_version.

Closes #6818

954d3c91e [Wang, Fei] Upgrade mysql jdbc version to 8.4.0

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2024-11-22 17:39:33 +08:00
Bowen Liang
c391d169fe
[KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- fix the compilation error on Zulu JDK8 (Zulu 8.82.0.21-cA-macos-aarch64 with M4 Max chip on MacOS 15.1), when running `build/mvn clean install -pl :kyuubi-util -DskipTests -am` command to build a pure Java module,  by replacing version code sytle from 1.8 to 8.

![image](https://github.com/user-attachments/assets/d4d473b5-8db7-49cf-b7d5-6cc23541496c)

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6813 from bowenliang123/java8.

Closes #6813

1a1e3f3a7 [Bowen Liang] set java.version to 8

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-11-19 17:22:59 +08:00
Bowen Liang
903e333ec0
[KYUUBI #6807] [BUILD] Bump maven download plugin from 1.8.1 to 1.11.3
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6807 from pionCham/bump-maven-plugin-version.

Closes #6807

6582a194c [Bowen Liang] bump to 1.11.3
1505449a6 [chengpeiming] Bump maven plugin download version to 1.11.2

Lead-authored-by: Bowen Liang <bowenliang@apache.org>
Co-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
2024-11-19 15:00:39 +08:00
Bowen Liang
27e7254930 [KYUUBI #6773] Bump commons-codec from 1.15 to 1.17.1
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump commons-codec from 1.15 (Seq 2020) to 1.17.1 (Jul 2024): https://commons.apache.org/proper/commons-codec/changes-report.html

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6773 from bowenliang123/commons-codec-1.17.

Closes #6773

347ef0fe9 [Bowen Liang] update

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-11-15 11:20:04 +08:00
chengpeiming
a4eaacd850
[KYUUBI #6804] Bump Iceberg from 1.6.1 to 1.7.0
# 🔍 Description
## Issue References 🔗

Apache Iceberg 1.7.0 release https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.7.0

## Describe Your Solution 🔧
- Bump Apache Iceberg to 1.7.0
- As Apache Iceberg 1.7.0 drops support for Java 8 and building with Java 11, keep it in 1.6.x for Java 8

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6804 from pionCham/bump-iceberg-version.

Closes #6804

0896ac768 [Bowen Liang] keep iceberg 1.6.1 in playground
eba16ae6c [chengpeiming] Specify the iceberg version in java-8 profile
3b160ddd6 [chengpeiming] Bump iceberg version

Lead-authored-by: chengpeiming <chengpeiming@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2024-11-14 18:25:09 +08:00
Bowen Liang
d22ad7fd22
[KYUUBI #6774] Bump log4j from 2.20.0 to 2.24.1
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump log4j from 2.20.0(Feb 2023) to 2.24.1 (Sep 2024) : https://logging.apache.org/log4j/2.x/release-notes.html#release-notes-2-24-1

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6774 from bowenliang123/log4j-2.24.

Closes #6774

98a89ef8a [Bowen Liang] bump log4j from 2.20.0 to 2.24.1

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
2024-11-14 18:18:17 +08:00
chengpeiming
ad53eefe4b [KYUUBI #6808] [BUILD]Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧
Fix the warning of `maven-surefire-plugin` that show `Parameter 'systemProperties' is deprecated. Use systemPropertyVariables instead.` in the build.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6808 from pionCham/use-systemPropertyVariables.

Closes #6808

668bfe252 [chengpeiming] Replace  systemProperties with systemPropertyVariables in maven-surefire-plugin
0e74b4725 [chengpeiming] Revert "Replace systemProperties with systemPropertyVariables"
797384a5b [chengpeiming] Replace systemProperties with systemPropertyVariables

Authored-by: chengpeiming <chengpeiming@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-11-13 22:41:23 +08:00
Bowen Liang
dddb03722d [KYUUBI #6778] [BUILD] allow frontend plugin not to inherits Maven's http proxy config
# 🔍 Description
## Issue References 🔗

## Describe Your Solution 🔧

- previously, `pnpm install` and `npm install` are forced to inherits HTTP proxy configs from Maven by the frontend maven plugin (https://github.com/eirslett/frontend-maven-plugin?tab=readme-ov-file#proxy-settings)
- adding a parameter `maven.plugin.frontend.inheritsProxyConfigFromMave` to control whether it inherits the proxy configs with false as default value

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6778 from bowenliang123/frontend-build-proxy.

Closes #6778

c8eb2e4f1 [Bowen Liang] inheritsProxyConfigFromMaven

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-11-12 15:20:45 +08:00
Bowen Liang
d3520ddbce [KYUUBI #6769] [RELEASE] Bump 1.11.0-SNAPSHOT
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Preparing v1.11.0-SNAPSHOT after branch-1.10 cut

```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6769 from bowenliang123/bump-1.11.

Closes #6769

6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-23 17:10:56 +08:00
wforget
1e9d68b000 [KYUUBI #6368] Flink engine supports user impersonation
# 🔍 Description
## Issue References 🔗

This pull request fixes #6368

## Describe Your Solution 🔧

Support impersonation mode for flink sql engine.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

Test in hadoop-testing env.

Connection:

```
beeline -u "jdbc:hive2://hadoop-master1.orb.local:10009/default;hive.server2.proxy.user=spark;principal=kyuubi/_HOSTTEST.ORG?kyuubi.engine.type=FLINK_SQL;flink.execution.target=yarn-application;kyuubi.engine.share.level=CONNECTION;kyuubi.engine.flink.doAs.enabled=true;"
```

sql:

```
select 1;
```

result:

![image](https://github.com/apache/kyuubi/assets/17894939/4bde3e4e-0dac-4e09-ac7c-a2c3a3607a13)

launch engine command:

```
2024-06-12 03:22:10.242 INFO KyuubiSessionManager-exec-pool: Thread-62 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/flink-1.18.1/bin/flink run-application \
	-t yarn-application \
	-Dyarn.ship-files=/opt/flink/opt/flink-sql-client-1.18.1.jar;/opt/flink/opt/flink-sql-gateway-1.18.1.jar;/etc/hive/conf/hive-site.xml \
	-Dyarn.application.name=kyuubi_CONNECTION_FLINK_SQL_spark_6170b9aa-c690-4b50-938f-d59cca9aa2d6 \
	-Dyarn.tags=KYUUBI,6170b9aa-c690-4b50-938f-d59cca9aa2d6 \
	-Dcontainerized.master.env.FLINK_CONF_DIR=. \
	-Dcontainerized.master.env.HIVE_CONF_DIR=. \
	-Dyarn.security.appmaster.delegation.token.services=kyuubi \
	-Dsecurity.delegation.token.provider.HiveServer2.enabled=false \
	-Dsecurity.delegation.token.provider.hbase.enabled=false \
	-Dexecution.target=yarn-application \
	-Dsecurity.module.factory.classes=org.apache.flink.runtime.security.modules.JaasModuleFactory;org.apache.flink.runtime.security.modules.ZookeeperModuleFa
ctory \
	-Dsecurity.delegation.token.provider.hadoopfs.enabled=false \
	-c org.apache.kyuubi.engine.flink.FlinkSQLEngine /opt/apache-kyuubi-1.10.0-SNAPSHOT-bin/externals/engines/flink/kyuubi-flink-sql-engine_2.12-1.10.0-SNAPS
HOT.jar \
	--conf kyuubi.session.user=spark \
	--conf kyuubi.client.ipAddress=172.20.0.5 \
	--conf kyuubi.engine.credentials=SERUUwACJnRocmlmdDovL2hhZG9vcC1tYXN0ZXIxLm9yYi5sb2NhbDo5MDgzRQAFc3BhcmsEaGl2ZShreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2Fs
QFRFU1QuT1JHigGQCneevIoBkC6EIrwWDxSg03pnAB8dA295wh+Dim7Fx4FNxhVISVZFX0RFTEVHQVRJT05fVE9LRU4ADzE3Mi4yMC4wLjU6ODAyMEEABXNwYXJrAChreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiL
mxvY2FsQFRFU1QuT1JHigGQCneekIoBkC6EIpBHHBSket0SQnlXT5EIMN0U2fUKFRIVvBVIREZTX0RFTEVHQVRJT05fVE9LRU4PMTcyLjIwLjAuNTo4MDIwAA== \
	--conf kyuubi.engine.flink.doAs.enabled=true \
	--conf kyuubi.engine.hive.extra.classpath=/opt/hadoop/share/hadoop/client/*:/opt/hadoop/share/hadoop/mapreduce/* \
	--conf kyuubi.engine.share.level=CONNECTION \
	--conf kyuubi.engine.submit.time=1718162530017 \
	--conf kyuubi.engine.type=FLINK_SQL \
	--conf kyuubi.frontend.protocols=THRIFT_BINARY,REST \
	--conf kyuubi.ha.addresses=hadoop-master1.orb.local:2181 \
	--conf kyuubi.ha.engine.ref.id=6170b9aa-c690-4b50-938f-d59cca9aa2d6 \
	--conf kyuubi.ha.namespace=/kyuubi_1.10.0-SNAPSHOT_CONNECTION_FLINK_SQL/spark/6170b9aa-c690-4b50-938f-d59cca9aa2d6 \
	--conf kyuubi.server.ipAddress=172.20.0.5 \
	--conf kyuubi.session.connection.url=hadoop-master1.orb.local:10009 \
	--conf kyuubi.session.engine.startup.waitCompletion=false \
	--conf kyuubi.session.real.user=spark
```

launch engine log:

![image](https://github.com/apache/kyuubi/assets/17894939/590463a8-2858-47a2-8897-0ddfbe3ffdf6)

jobmanager job:

```
2024-06-12 03:22:26,400 INFO  org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Loading delegation token providers
2024-06-12 03:22:26,992 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenProvider [] - Renew delegation token with engine credentials: SERUUwACJnRocmlmdDovL2hhZG9vcC1tYXN0ZXIxLm9yYi5sb2NhbDo5MDgzRQAFc3BhcmsEaGl2ZShreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2FsQFRFU1QuT1JHigGQCneevIoBkC6EIrwWDxSg03pnAB8dA295wh+Dim7Fx4FNxhVISVZFX0RFTEVHQVRJT05fVE9LRU4ADzE3Mi4yMC4wLjU6ODAyMEEABXNwYXJrAChreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2FsQFRFU1QuT1JHigGQCneekIoBkC6EIpBHHBSket0SQnlXT5EIMN0U2fUKFRIVvBVIREZTX0RFTEVHQVRJT05fVE9LRU4PMTcyLjIwLjAuNTo4MDIwAA==
2024-06-12 03:22:27,100 INFO  org.apache.kyuubi.engine.flink.FlinkEngineUtils              [] - Add new unknown token Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 05 73 70 61 72 6b 04 68 69 76 65 28 6b 79 75 75 62 69 2f 68 61 64 6f 6f 70 2d 6d 61 73 74 65 72 31 2e 6f 72 62 2e 6c 6f 63 61 6c 40 54 45 53 54 2e 4f 52 47 8a 01 90 0a 77 9e bc 8a 01 90 2e 84 22 bc 16 0f
2024-06-12 03:22:27,104 WARN  org.apache.kyuubi.engine.flink.FlinkEngineUtils              [] - Ignore token with earlier issue date: Kind: HDFS_DELEGATION_TOKEN, Service: 172.20.0.5:8020, Ident: (token for spark: HDFS_DELEGATION_TOKEN owner=spark, renewer=, realUser=kyuubi/hadoop-master1.orb.localTEST.ORG, issueDate=1718162529936, maxDate=1718767329936, sequenceNumber=71, masterKeyId=28)
2024-06-12 03:22:27,104 INFO  org.apache.kyuubi.engine.flink.FlinkEngineUtils              [] - Update delegation tokens. The number of tokens sent by the server is 2. The actual number of updated tokens is 1.
......
4-06-12 03:22:29,414 INFO  org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Starting tokens update task
2024-06-12 03:22:29,415 INFO  org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - New delegation tokens arrived, sending them to receivers
2024-06-12 03:22:29,422 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updating delegation tokens for current user
2024-06-12 03:22:29,422 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[10, 13, 10, 9, 8, 10, 16, -78, -36, -49, -17, -5, 49, 16, 1, 16, -100, -112, -60, -127, -8, -1, -1, -1, -1, 1]
2024-06-12 03:22:29,422 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[0, 5, 115, 112, 97, 114, 107, 4, 104, 105, 118, 101, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -68, -118, 1, -112, 46, -124, 34, -68, 22, 15]
2024-06-12 03:22:29,422 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service:172.20.0.5:8020 Identifier:[0, 5, 115, 112, 97, 114, 107, 0, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -112, -118, 1, -112, 46, -124, 34, -112, 71, 28]
2024-06-12 03:22:29,422 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updated delegation tokens for current user successfully

```

taskmanager log:

```
2024-06-12 03:45:06,622 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Receive initial delegation tokens from resource manager
2024-06-12 03:45:06,627 INFO  org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - New delegation tokens arrived, sending them to receivers
2024-06-12 03:45:06,628 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updating delegation tokens for current user
2024-06-12 03:45:06,629 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[10, 13, 10, 9, 8, 10, 16, -78, -36, -49, -17, -5, 49, 16, 1, 16, -100, -112, -60, -127, -8, -1, -1, -1, -1, 1]
2024-06-12 03:45:06,630 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[0, 5, 115, 112, 97, 114, 107, 4, 104, 105, 118, 101, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -68, -118, 1, -112, 46, -124, 34, -68, 22, 15]
2024-06-12 03:45:06,630 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service:172.20.0.5:8020 Identifier:[0, 5, 115, 112, 97, 114, 107, 0, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -112, -118, 1, -112, 46, -124, 34, -112, 71, 28]
2024-06-12 03:45:06,636 INFO  org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updated delegation tokens for current user successfully
2024-06-12 03:45:06,636 INFO  org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation tokens sent to receivers
```

#### Related Unit Tests

---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6383 from wForget/KYUUBI-6368.

Closes #6368

47df43ef0 [wforget] remove doAsEnabled
984b96c74 [wforget] update settings.md
c7f8d474e [wforget] make generateTokenFile conf to internal
8632176b1 [wforget] address comments
2ec270e8a [wforget] licenses
ed0e22f4e [wforget] separate kyuubi-flink-token-provider module
b66b855b6 [wforget] address comment
d4fc2bd1d [wforget] fix
1a3dc4643 [wforget] fix style
825e2a7a0 [wforget] address comments
a679ba1c2 [wforget] revert remove renewer
cdd499b95 [wforget] fix and comment
19caec6c0 [wforget] pass token to submit process
b2991d419 [wforget] fix
7c3bdde1b [wforget] remove security.delegation.tokens.enabled check
8987c9176 [wforget] fix
5bd8cfe7c [wforget] fix
08992642d [wforget] Implement KyuubiDelegationToken Provider/Receiver
fa16d7def [wforget] enable delegation token manager
e50db7497 [wforget] [KYUUBI #6368] Support impersonation mode for flink sql engine

Authored-by: wforget <643348094@qq.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-21 17:32:39 +08:00
wforget
e99a737e63
[KYUUBI #6755] Upgrade default flink version to 1.20.0
# 🔍 Description

Upgrade default flink version to 1.20.0

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6755 from wForget/minor.

Closes #6755

e968502a6 [wforget] comment
c0094c760 [wforget] comment
b74a0bb75 [wforget] comment
1aea2216c [wforget] Upgrade default flink version to 1.20.0

Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-10-18 21:26:01 +08:00
Bowen Liang
85dd5a52ef [KYUUBI #6742] Bump Jetty from 9.4.54 to 9.4.56
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

 - to fix CVE-2024-8184 reported in https://github.com/apache/kyuubi/security/dependabot/75

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6742 from bowenliang123/jetty-9.4.56.

Closes #6742

1e8fee555 [Bowen Liang] update
96db527d6 [Bowen Liang] bumpu jetty to 9.4.56

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 21:33:03 +08:00
Bowen Liang
ad07ae9398 [KYUUBI #6743] Bump protobuf from 3.25.4 to 3.25.5
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- to fix CVE-2024-8184 reported in https://github.com/apache/kyuubi/security/dependabot/72

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6743 from bowenliang123/protobuf-3.25.5.

Closes #6743

a9a574041 [Bowen Liang] bump protobuf from 3.25.4 to 3.25.5

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 21:32:25 +08:00
Bowen Liang
b9beec2630 [KYUUBI #6745] Bump testcontainers-scala from 0.41.0 to 0.41.4
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Bump testcontainers-scala from 0.41.0 (released in Aug 2023) to 0.41.4 (released in Jun 2024)

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6745 from bowenliang123/testcontainers-scala-0.41.4.

Closes #6745

06551cb8d [Bowen Liang] bump testcontainers-scala from 0.41.0 to 0.41.4

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 21:30:10 +08:00
Bowen Liang
45eebda431 [KYUUBI #6746] Bump commons-lang3 from 3.13.0 to 3.17.0
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- Bump commons-lang3 from 3.13.0 ( 2023-07-23 ) to 3.17.0 ( 2024-08-24 ), https://commons.apache.org/proper/commons-lang/changes-report.html#a3.17.0

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6746 from bowenliang123/commonslang3-3.17.

Closes #6746

d72467bfb [Bowen Liang] bump commons-lang3 from 3.13.0 to 3.17.0

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 21:29:05 +08:00
Bowen Liang
a1a5a9d27f [KYUUBI #6747] Bump Guava from 32.1.3 to 33.3.1
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- Bump guava from 32.1.3 (Oct 11, 2023) to 33.3.1 (Sep 23, 2024 ) : https://github.com/google/guava/releases/tag/v33.3.1

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6747 from bowenliang123/guava-33.3.1.

Closes #6747

39abf1b23 [Bowen Liang] bump guava from 32.1.3 to 33.3.1

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 21:28:01 +08:00
madlnu
04468a60c9 [KYUUBI #6737] Bump vertx-grpc and vertx-core from 4.5.1 to 4.5.3
# 🔍 Description

## Describe Your Solution 🔧

Bump vertx-core from 4.5.1 to 4.5.3 to fix CVE-2024-1300 and CVE-2024-1023

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪
Build and ran locally

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6737 from Madhukar525722/vertx.

Closes #6737

7c758e082 [madlnu] Bump vertx-grpc from 4.5.1 to 4.5.3

Authored-by: madlnu <madlnu@visa.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-17 15:14:53 +08:00
Bowen Liang
daeff6bfd8 [KYUUBI #6733] Bump SQLlite from 3.42 to 3.46
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump SQLlite from 3.42.0 (May 2023) to 3.46 (May 2024)
- SQLlite changelog: https://sqlite.org/changes.html

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6733 from bowenliang123/sqllite-346.

Closes #6733

a86bea232 [Bowen Liang] update

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-16 11:00:39 +08:00
Bowen Liang
9ef38b9582 [KYUUBI #6712] [UI] Bump pnpm to v9 and nodejs to v20 LTS
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump `pnpm` to v9.11, release note: https://github.com/pnpm/pnpm/releases/tag/v9.11.0
  - updated pnpm lockfile by running `pnpm install --fix-lockfile`
- bump `nodejs` version from v18 to v20 (LTS version, released on 18 Apr 2023)
- update related github actions, aligning the pnpm and nodejs versions

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6712 from bowenliang123/pnpm9.

Closes #6712

ca6e425f4 [liangbowen] nit
a9ea92216 [liangbowen] bump pnpm to v9

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2024-09-26 16:26:20 +08:00