From bc394a933bfa2f3c5b631476c0e7ee0d9a7f1d83 Mon Sep 17 00:00:00 2001 From: PorterZhang2021 Date: Wed, 15 May 2024 12:43:09 +0800 Subject: [PATCH] [KYUUBI #6305][FOLLOWUP] Improve package Spark SQL engine both Scala 2.12 and 2.13 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # :mag: Description ## Issue References πŸ”— This pull request fixes #6305 ## Describe Your Solution πŸ”§ ### Solution 1 use `` - Inappropriate I found a way to use , roughly as follows: ```xml scala-2.12 2.12 ``` After specifying, I attempted to use 1. `build/mvn install -DskipTests -Dmaven.javadoc.skip=true -Dmaven.scaladoc.skip=true - Dmaven.source.skip -Pscala-2.12 -Pscala-2.13 -pl: kyuubi-spark-sql-engine -am` 2. `build/mvn install -DskipTests -Dmaven.javadoc.skip=true -Dmaven.scaladoc.skip=true - Dmaven.source.skip -Pscala-2.13 -Pscala-2.12 -pl: kyuubi-spark-sql-engine -am` #### Problem But in the end, it was found that if both '-Pscala-2.12' and '-Pscala-2.13' are used at the same time, '-Pscala2.13' will be selected by default, which may not be a good solution to this problem. ### Solution2 Later, I thought about whether it was possible to filter the parameter '$' internally. It is effective. ## Types of changes :bookmark: - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan πŸ§ͺ 1. scala2.12 ![scala_2 12η‰ˆζœ¬spark-sql-engine](https://github.com/apache/kyuubi/assets/96274454/a9f82433-8f73-4bb0-b0f7-7d2725435a0e) ![scala_2 12_select ζ΅‹θ―•](https://github.com/apache/kyuubi/assets/96274454/fe1cb601-8d19-4dc0-a248-c045176b87a6) 2. scala2.13 ![image](https://github.com/apache/kyuubi/assets/96274454/d79ae1cc-64fe-4079-b706-37a21b00b627) ![scala_2 13η‰ˆζœ¬ζ΅‹θ―•](https://github.com/apache/kyuubi/assets/96274454/a4084499-12ea-4150-982a-67aacd16c184) --- # Checklist πŸ“ - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6330 from PorterZhang2021/followup-6297. Closes #6305 5524b13f5 [Cheng Pan] nit cccb94d98 [PorterZhang2021] [# 6297] imporve Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up fb03d60c2 [Porter Zhang] Merge branch 'apache:master' into followup-6297 393435ea9 [PorterZhang2021] [# 6297] imporve Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up 0b49f60e6 [Porter Zhang] Merge branch 'apache:master' into followup-6297 f7c7a65c8 [PorterZhang2021] [# 6297] Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up 3d2926afc [Porter Zhang] Merge branch 'apache:master' into followup-6297 eb6406148 [PorterZhang2021] [followup-issue6297] improve issue6297 956ac4955 [PorterZhang2021] [# 6297] Package Spark SQL engine both Scala 2.12 and 2.13 Follow Up Lead-authored-by: PorterZhang2021 Co-authored-by: Porter Zhang <96274454+PorterZhang2021@users.noreply.github.com> Co-authored-by: Cheng Pan Co-authored-by: PorterZhang2021 Co-authored-by: Porter Zhang Signed-off-by: Cheng Pan --- build/dist | 17 ++++++++++++++--- pom.xml | 7 +++++++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/build/dist b/build/dist index ea58f7a29..b2b4b4abe 100755 --- a/build/dist +++ b/build/dist @@ -247,18 +247,28 @@ BUILD_COMMAND=("$MVN" clean install $MVN_DIST_OPT $@) echo -e "\nBuilding with..." # shellcheck disable=SC2145 echo -e "\$ ${BUILD_COMMAND[@]}\n" +"${BUILD_COMMAND[@]}" + +FILTERED_ARGS=() +# shellcheck disable=SC2045 +for arg in "$@"; do + if [[ $arg == *scala-2.12* ]]; then + FILTERED_ARGS+=("${arg//scala-2.12/scala-2.13}") + elif [[ $arg == *scala-2.13* ]]; then + FILTERED_ARGS+=("${arg//scala-2.13/scala-2.12}") + fi +done # shellcheck disable=SC2050 if [ "$SCALA_VERSION" = "2.12" ]; then - EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT $@ -Pscala-2.13 -pl :kyuubi-spark-sql-engine_2.13 -am) + EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT ${FILTERED_ARGS[@]} -Pscala-2.13 -pl :kyuubi-spark-sql-engine_2.13 -am) else - EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT $@ -Pscala-2.12 -pl :kyuubi-spark-sql-engine_2.12 -am) + EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT ${FILTERED_ARGS[@]} -pl :kyuubi-spark-sql-engine_2.12 -am) fi # shellcheck disable=SC2145 echo -e "\$ ${EXTRA_SPARK_ENGINE_BUILD_COMMAND[@]}\n" -"${BUILD_COMMAND[@]}" "${EXTRA_SPARK_ENGINE_BUILD_COMMAND[@]}" # Make directories @@ -307,6 +317,7 @@ done cp "$KYUUBI_HOME/externals/kyuubi-flink-sql-engine/target/kyuubi-flink-sql-engine_${SCALA_VERSION}-${VERSION}.jar" "$DISTDIR/externals/engines/flink/" # Copy spark engines +# shellcheck disable=SC2045 for scala_version in 2.12 2.13; do cp "$KYUUBI_HOME/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_${scala_version}-${VERSION}.jar" "$DISTDIR/externals/engines/spark/" done diff --git a/pom.xml b/pom.xml index bd29b5427..8773caf0f 100644 --- a/pom.xml +++ b/pom.xml @@ -1950,6 +1950,13 @@ + + scala-2.12 + + 2.12 + + + scala-2.13