apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile

Browse Source 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767

In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`.

After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1257 from pan3793/pom.

Closes #1257

1d871b6c [Cheng Pan] nit
874f30c1 [Cheng Pan] Address comments
00a32b25 [Cheng Pan] nit
445666ee [Cheng Pan] typo
bd96097e [Cheng Pan] nit
e85116cd [Cheng Pan] nit
98347c8c [Cheng Pan] Update doc
6615d7cc [Cheng Pan] Exclude HudiTest in cross test
087d3fde [Cheng Pan] Add spark 3.2 binary test
ccff63c1 [Cheng Pan] update GA
23b81dcd [Cheng Pan] Address comments
3055c2a8 [Cheng Pan] nit
49a41e86 [Cheng Pan] nit
8d400798 [Cheng Pan] Reduce unnecessary maven profile
ced8d987 [Cheng Pan] Reduce unnecessary maven profile

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
This commit is contained in:
Cheng Pan 2021-10-22 14:51:42 +08:00 committed by ulysses-you
parent f5b792a0c7
commit 589b23f578
No known key found for this signature in database
GPG Key ID: 4C500BC62D576766
15 changed files with 63 additions and 79 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.github/workflows/master.yml vendored
View File

@ -39,16 +39,18 @@ jobs:
- java: 8
spark: '3.1'
spark-hadoop: '2.7'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.2.0 -Dspark.archive.name=spark-3.2.0-bin-hadoop3.2.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-1'
codecov: 'true'
- java: 8
spark: '3.2'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-2'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'

2
.github/workflows/nightly.yml vendored
View File

@ -11,7 +11,7 @@ jobs:
strategy:
matrix:
profiles:
- '-Pspark-master -pl :kyuubi-spark-sql-engine_2.12,:kyuubi-common_2.12,:kyuubi-ha_2.12,:kyuubi-zookeeper_2.12,:kyuubi-spark-monitor_2.12'
- '-Pspark-master -pl externals/kyuubi-spark-sql-engine -am'
env:
SPARK_LOCAL_IP: localhost
steps:

4
.github/workflows/publish-snapshot.yml vendored
View File

@ -36,4 +36,6 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -DskipTests -Pkyuubi-extension-spark-3-1,spark-provided -s ./build/release/asf-settings.xml
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am

2
build/release/create-package.sh
View File

@ -75,7 +75,7 @@ package_binary() {
echo "Creating binary release tarball ${BIN_TGZ_FILE}"
${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1
${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pspark-3.1
cp "${BIN_TGZ_FILE}" "${RELEASE_DIR}"

6
build/release/release.sh
View File

@ -96,6 +96,12 @@ upload_svn_staging() {
upload_nexus_staging() {
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml"
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.1 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-1 -am
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.2 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-2 -am
}
finalize_svn() {

8
dev/kyuubi-codecov/pom.xml
View File

@ -111,6 +111,14 @@
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>

9
dev/kyuubi-extension-spark-3-1/pom.xml
View File

@ -155,13 +155,4 @@
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>kyuubi-extension-spark-3-1</id>
<properties>
<spark.version>3.1.2</spark.version>
</properties>
</profile>
</profiles>
</project>

10
dev/kyuubi-extension-spark-3-2/pom.xml
View File

@ -161,14 +161,4 @@
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>kyuubi-extension-spark-3-2</id>
<properties>
<spark.version>3.2.0</spark.version>
</properties>
</profile>
</profiles>
</project>

34
dev/kyuubi-extension-spark-common/pom.xml
View File

@ -135,13 +135,11 @@
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
@ -159,24 +157,12 @@
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>kyuubi-extension-spark-3-1</id>
<properties>
<spark.version>3.1.2</spark.version>
</properties>
</profile>
<profile>
<id>kyuubi-extension-spark-3-2</id>
<properties>
<spark.version>3.2.0</spark.version>
</properties>
</profile>
</profiles>
</project>

2
docs/community/release.md
View File

@ -159,7 +159,7 @@ export RELEASE_RC_NO=<RC number, e.g. 0>
```shell
build/mvn versions:set -DgenerateBackupPoms=false \
-DnewVersion="${RELEASE_VERSION}" \
-Pkubernetes,kyuubi-extension-spark-3-1,spark-block-cleaner,tpcds
-Pspark-3.1,spark-block-cleaner
git commit -am "[RELEASE] Bump ${RELEASE_VERSION}"
```

18
docs/sql/rules.md
View File

@ -15,8 +15,6 @@
- limitations under the License.
-->
<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY [org.apache.kyuubi.engine.spark.udf.KyuubiUDFRegistrySuite] -->
<div align=center>
![](../imgs/kyuubi_logo.png)
@ -53,10 +51,18 @@ And don't worry, Kyuubi will support the new Apache Spark version in future. Tha
## How to use Kyuubi SQL extension
1. you need to choose Apache Spark branch-3.1 or higher version with Kyuubi binary tgz.
2. if you want to compile Kyuubi by yourself, the maven opt should add `-Pkyuubi-extension-spark-3-1`
3. move the jar(`kyuubi-extension-spark-*.jar`) which is in `$KYUUBI_HOME/extension` into `$SPARK_HOME/jars`
4. add a config into `spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
| Kyuubi Spark SQL extension | Supported Spark version(s) | Available since | EOL | Bundled in Binary release tarball | Maven profile
| -------------------------- | -------------------------- | ---------------- | ---------------- | --------------------------------- | -------------
| kyuubi-extension-spark-3-1 | 3.1.x | 1.3.0-incubating | N/A | 1.3.0-incubating | spark-3.1
| kyuubi-extension-spark-3-2 | 3.2.x | 1.4.0-incubating | N/A | 1.4.0-incubating | spark-3.2
1. Check the matrix that if you are using the supported Spark version, and find the corresponding Kyuubi Spark SQL Extension jar
2. Get the Kyuubi Spark SQL Extension jar
1. Each Kyuubi binary release tarball only contains one default version of Kyuubi Spark SQL Extension jar, if you are looking for such version, you can find it under `$KYUUBI_HOME/extension`
2. All supported versions of Kyuubi Spark SQL Extension jar will be deployed to [Maven Central](https://search.maven.org/search?q=kyuubi-extension-spark)
3. If you like, you can compile Kyuubi Spark SQL Extension jar by yourself, please activate the corresponding Maven's profile on you compile command, i.e. you can get Kyuubi Spark SQL Extension jar for Spark 3.1 under `dev/kyuubi-extension-spark-3-1/target` when compile with `-Pspark-3.1`
3. Put the Kyuubi Spark SQL extension jar `kyuubi-extension-spark-*.jar` into `$SPARK_HOME/jars`
4. Enable `KyuubiSparkSQLExtension`, i.e. add a config into `$SPARK_HOME/conf/spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
Now, you can enjoy the Kyuubi SQL Extension, and also Kyuubi provides some configs to make these feature easy to use.

2
docs/sql/z-order-benchmark.md
View File

@ -15,8 +15,6 @@
- limitations under the License.
-->
<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY [org.apache.kyuubi.engine.spark.udf.KyuubiUDFRegistrySuite] -->
<div align=center>
![](../imgs/kyuubi_logo.png)

8
externals/kyuubi-download/pom.xml vendored
View File

@ -57,6 +57,14 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>

24
pom.xml
View File

@ -1775,6 +1775,10 @@
<delta.version>1.0.0</delta.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-1</module>
</modules>
</profile>
<profile>
@ -1783,6 +1787,10 @@
<spark.version>3.2.0</spark.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-2</module>
</modules>
</profile>
<profile>
@ -1826,22 +1834,6 @@
</properties>
</profile>
<profile>
<id>kyuubi-extension-spark-3-1</id>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-1</module>
</modules>
</profile>
<profile>
<id>kyuubi-extension-spark-3-2</id>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-2</module>
</modules>
</profile>
<profile>
<id>spark-block-cleaner</id>
<modules>

5
tools/spark-block-cleaner/pom.xml
View File

@ -55,10 +55,5 @@
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<resources>
<resource>
<directory>${project.basedir}/src/main/resources</directory>
</resource>
</resources>
</build>
</project>