586f6008bd
# 🔍 Description Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118, which breaks the binary compatibility of `SparkUITab` and `WebUIPage` that Kyuubi used, thus breaking the previous assumption of Kyuubi Spark SQL engine: single jar built with default Spark version, compatible with all supported versions of Spark runtime. ## Describe Your Solution 🔧 This PR uses bytebuddy to dynamically generate classes and Java reflection find and dispatch method invocation in runtime, to recover the existing compatibility of Kyuubi Spark SQL engine. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Build with Spark 3.5 ``` build/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided -Pspark-3.5 ``` It produces both Scala 2.12 and 2.13 Spark SQL engine jars - `kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar` - `kyuubi-spark-sql-engine_2.13-1.10.0-SNAPSHOT.jar` Run with Spark 3.4 Scala 2.12 <img width="1639" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/caeef30d-7467-4942-a56a-88a7c93ef7cc"> Run with Spark 3.5 Scala 2.13 <img width="1639" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/c339c1e9-c07f-4952-9a57-098b832c889f"> Run with Spark 4.0.0-preview1 Scala 2.13 <img width="1639" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/a3fb6e77-b27e-4634-8acf-245a26b39d2b"> --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6399 from pan3793/ui-4.0. Closes #6399 e0104f6df [Cheng Pan] nit a2f9df4fa [Cheng Pan] nit c369ab2e3 [Cheng Pan] nit ec1c45f66 [Cheng Pan] nit 3e05744d6 [Cheng Pan] fix a7e38cc1e [Cheng Pan] nit fa14a0d98 [Cheng Pan] refactor 9d0ce6111 [Cheng Pan] A work version fc78b58e4 [Cheng Pan] fix startup d74c1c0fe [Cheng Pan] fix 50066f563 [Cheng Pan] nit f5ad4c760 [Cheng Pan] Kyuubi UI supports Spark 4.0 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|---|---|---|
| .. | ||
| kyuubi-chat-engine | ||
| kyuubi-download | ||
| kyuubi-flink-sql-engine | ||
| kyuubi-hive-sql-engine | ||
| kyuubi-jdbc-engine | ||
| kyuubi-spark-sql-engine | ||
| kyuubi-trino-engine | ||