apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-1413][FOLLOWUP] Rename celeborn-client-spark-3-4 back to celeborn-client-spark-3

Browse Source 
### What changes were proposed in this pull request?

This PR partially reverts the change of https://github.com/apache/celeborn/pull/2813, namely, restores the renaming of `celeborn-client-spark-3`

### Why are the changes needed?

The renaming is not necessary, and might cause some confusion, for example, I wrongly interpreted the `spark-3-4` as Spark 3.4, it also increases the backport efforts for branch-0.5

### Does this PR introduce _any_ user-facing change?

No, it's dev only, before/after this change, the end users always use the shaded client

```
celeborn-client-spark-2-shaded_2.11-0.6.0-SNAPSHOT.jar
celeborn-client-spark-3-shaded_2.12-0.6.0-SNAPSHOT.jar
celeborn-client-spark-4-shaded_2.13-0.6.0-SNAPSHOT.jar
```

### How was this patch tested?

Pass GA.

Closes #3133 from pan3793/CELEBORN-1413-followup.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
This commit is contained in:
Cheng Pan 2025-03-04 22:25:10 +08:00 committed by mingji
parent 15e34eca6e
commit e85207e2c7
32 changed files with 35 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.github/workflows/maven.yml vendored
View File

@ -133,12 +133,8 @@ jobs:
run: |
SPARK_BINARY_VERSION=${{ matrix.spark }}
SPARK_MAJOR_VERSION=${SPARK_BINARY_VERSION%%.*}
SPARK_MODULE_NAME=$SPARK_MAJOR_VERSION
if [[ $SPARK_MAJOR_VERSION == "3" ]]; then
SPARK_MODULE_NAME="3-4"
fi
PROFILES="-Pgoogle-mirror,spark-${{ matrix.spark }}"
TEST_MODULES="client-spark/common,client-spark/spark-${SPARK_MODULE_NAME},client-spark/spark-${SPARK_MAJOR_VERSION}-columnar-common,client-spark/spark-${SPARK_MAJOR_VERSION}-shaded,tests/spark-it"
TEST_MODULES="client-spark/common,client-spark/spark-${SPARK_MAJOR_VERSION},client-spark/spark-${SPARK_MAJOR_VERSION}-columnar-common,client-spark/spark-${SPARK_MAJOR_VERSION}-shaded,tests/spark-it"
build/mvn $PROFILES -pl $TEST_MODULES -am clean install -DskipTests
build/mvn $PROFILES -pl $TEST_MODULES -Dspark.shuffle.sort.io.plugin.class=${{ matrix.shuffle-plugin-class }} test
- name: Upload test log

2
client-spark/spark-3-columnar-common/pom.xml
View File

@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>

2
client-spark/spark-3-columnar-shuffle/pom.xml
View File

@ -49,7 +49,7 @@
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>

2
client-spark/spark-3-shaded/pom.xml
View File

@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

2
client-spark/spark-3-4/pom.xml → client-spark/spark-3/pom.xml
View File

@ -24,7 +24,7 @@
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Client for Spark 3 and 4</name>

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/CelebornPartitionUtil.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/CelebornPartitionUtil.java
View File

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/CelebornShuffleDataIO.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/CelebornShuffleDataIO.java
View File

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriter.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriter.java
View File

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriter.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriter.java
View File

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
View File

0
client-spark/spark-3-4/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java → client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/SparkVersionUtil.scala → client-spark/spark-3/src/main/scala/org/apache/spark/SparkVersionUtil.scala
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/celeborn/ExceptionMakerHelper.scala → client-spark/spark-3/src/main/scala/org/apache/spark/celeborn/ExceptionMakerHelper.scala
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala → client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleHandle.scala → client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleHandle.scala
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala → client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReader.scala
View File

0
client-spark/spark-3-4/src/main/scala/org/apache/spark/shuffle/celeborn/ShuffleFetchFailureReportTaskCleanListener.scala → client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/ShuffleFetchFailureReportTaskCleanListener.scala
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/CelebornPartitionUtilSuiteJ.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/CelebornPartitionUtilSuiteJ.java
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/CelebornShuffleWriterSuiteBase.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/CelebornShuffleWriterSuiteBase.java
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriterSuiteJ.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/HashBasedShuffleWriterSuiteJ.java
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/ShuffleManagerHook.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/ShuffleManagerHook.java
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriterSuiteJ.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/SortBasedShuffleWriterSuiteJ.java
View File

0
client-spark/spark-3-4/src/test/java/org/apache/spark/shuffle/celeborn/TestCelebornShuffleManager.java → client-spark/spark-3/src/test/java/org/apache/spark/shuffle/celeborn/TestCelebornShuffleManager.java
View File

0
client-spark/spark-3-4/src/test/resources/log4j.properties → client-spark/spark-3/src/test/resources/log4j.properties
View File

0
client-spark/spark-3-4/src/test/resources/log4j2-test.xml → client-spark/spark-3/src/test/resources/log4j2-test.xml
View File

0
client-spark/spark-3-4/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleManagerSuite.scala → client-spark/spark-3/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleManagerSuite.scala
View File

0
client-spark/spark-3-4/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReaderSuite.scala → client-spark/spark-3/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleReaderSuite.scala
View File

2
client-spark/spark-4-shaded/pom.xml
View File

@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

2
dev/dependencies.sh
View File

@ -169,7 +169,7 @@ case "$MODULE" in
SBT_PROJECT="celeborn-client-spark-2"
;;
"spark-3"*) # Match all versions starting with "spark-3"
MVN_MODULES="client-spark/spark-3-4"
MVN_MODULES="client-spark/spark-3"
SBT_PROJECT="celeborn-client-spark-3"
;;
"flink-1.16")

14
pom.xml
View File

@ -1372,7 +1372,7 @@
<id>spark-3.0</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1392,7 +1392,7 @@
<id>spark-3.1</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1412,7 +1412,7 @@
<id>spark-3.2</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1431,7 +1431,7 @@
<id>spark-3.3</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1450,7 +1450,7 @@
<id>spark-3.4</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1469,7 +1469,7 @@
<id>spark-3.5</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-3.5-columnar-shuffle</module>
<module>client-spark/spark-3-shaded</module>
@ -1488,7 +1488,7 @@
<id>spark-4.0</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3-4</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-4-columnar-shuffle</module>
<module>client-spark/spark-4-shaded</module>

14
project/CelebornBuild.scala
View File

@ -783,7 +783,7 @@ object Spark24 extends SparkClientProjects {
object Spark30 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -797,7 +797,7 @@ object Spark30 extends SparkClientProjects {
object Spark31 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -811,7 +811,7 @@ object Spark31 extends SparkClientProjects {
object Spark32 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -825,7 +825,7 @@ object Spark32 extends SparkClientProjects {
object Spark33 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -842,7 +842,7 @@ object Spark33 extends SparkClientProjects {
object Spark34 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -856,7 +856,7 @@ object Spark34 extends SparkClientProjects {
object Spark35 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-3"
val sparkClientShadedProjectPath = "client-spark/spark-3-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-3-shaded"
@ -872,7 +872,7 @@ object Spark35 extends SparkClientProjects {
object Spark40 extends SparkClientProjects {
val sparkClientProjectPath = "client-spark/spark-3-4"
val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-4"
val sparkClientShadedProjectPath = "client-spark/spark-4-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-4-shaded"

28
tests/spark-it/pom.xml
View File

@ -115,13 +115,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -133,13 +133,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -151,13 +151,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -169,13 +169,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -187,13 +187,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -205,13 +205,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
@ -223,13 +223,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3-4_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>