From f2cacd3f1875c97d9d14532e780aee62ce57ab7d Mon Sep 17 00:00:00 2001 From: Cheng Pan Date: Thu, 5 Dec 2024 13:59:29 +0800 Subject: [PATCH] [KYUUBI #6836] Ship `kafka-clients` in binary distribution tarball without compression libs ### Why are the changes needed? I'd like to include `kafka-clients` in the Kyuubi binary distribution tarball to enable the out-of-box support for sinking Kyuubi events to Kafka. - Kafka is an important component in modern data platforms, and is a defacto message queue implementation, especially in the big data domain - `kafka-clients` is released under Apache License V2, has no legal issue - `kafka-clients` is quite a light lib, has no third-party deps except for `slf4j-api` and a few optional compression libs - `kafka-clients` uses "none" compression as default, in practice, "gzip"(delegate to JDK gzip algorithm, no additional libs are required) already performs well for non-extreme cases Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking logging to Kafka, which also requires `kafka-clients` in the classpath, I have some initial ideas to forward both Kyuubi server's and engine bootstrap processes log to Kafka in a structured format, will send another PR to add docs to guide users in configuring that. ### How was this patch tested? Review ### Was this patch authored or co-authored using generative AI tooling? No. Closes #6836 from pan3793/kafka-lib. Closes #6836 b069eb199 [Cheng Pan] Ship kafka-clients in binary distribution tarball without compression libs Authored-by: Cheng Pan Signed-off-by: Cheng Pan --- LICENSE-binary | 3 --- NOTICE-binary | 16 ---------------- dev/dependencyList | 3 --- pom.xml | 15 ++++++++++++++- 4 files changed, 14 insertions(+), 23 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index e78047f54..6b4c15fa7 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -306,8 +306,6 @@ io.vertx:vertx-grpc com.squareup.retrofit2:retrofit com.squareup.okhttp3:okhttp org.apache.kafka:kafka-clients -org.lz4:lz4-java -org.xerial.snappy:snappy-java org.xerial:sqlite-jdbc BSD @@ -319,7 +317,6 @@ jline:jline com.thoughtworks.paranamer:paranamer com.google.protobuf:protobuf-java-util com.google.protobuf:protobuf-java -com.github.luben:zstd-jni org.postgresql:postgresql Eclipse Distribution License - v 1.0 diff --git a/NOTICE-binary b/NOTICE-binary index b9bcdb214..708624e3f 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -1063,19 +1063,3 @@ which can be obtained at: * license/LICENSE.kafka.txt (Apache License 2.0) * HOMEPAGE: * https://github.com/apache/kafka - -This product optionally depends on 'snappy-java', Snappy compression and -decompression for Java, which can be obtained at: - - * LICENSE: - * license/LICENSE.snappy-java.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/xerial/snappy-java - -This product optionally depends on 'lz4-java', Lz4 compression and -decompression for Java, which can be obtained at: - - * LICENSE: - * license/LICENSE.lz4-java.txt (Apache License 2.0) - * HOMEPAGE: - * https://github.com/lz4/lz4-java diff --git a/dev/dependencyList b/dev/dependencyList index 2d4e4fd34..ab6a0499b 100644 --- a/dev/dependencyList +++ b/dev/dependencyList @@ -128,7 +128,6 @@ log4j-api/2.24.2//log4j-api-2.24.2.jar log4j-core/2.24.2//log4j-core-2.24.2.jar log4j-slf4j-impl/2.24.2//log4j-slf4j-impl-2.24.2.jar logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar -lz4-java/1.8.0//lz4-java-1.8.0.jar metrics-core/4.2.26//metrics-core-4.2.26.jar metrics-jmx/4.2.26//metrics-jmx-4.2.26.jar metrics-json/4.2.26//metrics-json-4.2.26.jar @@ -173,7 +172,6 @@ simpleclient_tracer_otel_agent/0.16.0//simpleclient_tracer_otel_agent-0.16.0.jar slf4j-api/1.7.36//slf4j-api-1.7.36.jar snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar snakeyaml/2.2//snakeyaml-2.2.jar -snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar sqlite-jdbc/3.46.1.3//sqlite-jdbc-3.46.1.3.jar swagger-annotations/2.2.1//swagger-annotations-2.2.1.jar swagger-core/2.2.1//swagger-core-2.2.1.jar @@ -185,4 +183,3 @@ units/1.7//units-1.7.jar vertx-core/4.5.3//vertx-core-4.5.3.jar vertx-grpc/4.5.3//vertx-grpc-4.5.3.jar zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar -zstd-jni/1.5.5-1//zstd-jni-1.5.5-1.jar diff --git a/pom.xml b/pom.xml index c6055b1b1..731463632 100644 --- a/pom.xml +++ b/pom.xml @@ -1053,7 +1053,20 @@ org.apache.kafka kafka-clients ${kafka.version} - true + + + com.github.luben + zstd-jni + + + org.lz4 + lz4-java + + + org.xerial.snappy + snappy-java + +