apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
main
celeborn/client-spark/spark-4-shaded
History
sychen c0512c67fd [CELEBORN-2022] Spark4 Client should package commons-io 
### What changes were proposed in this pull request?

### Why are the changes needed?
```java
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/celeborn/shaded/org/apache/commons/io/output/ByteArrayOutputStream
	at org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO.<init>(CelebornShuffleDataIO.java:41)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:75)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:53)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2834)
	at scala.collection.immutable.List.flatMap(List.scala:294)
	at scala.collection.immutable.List.flatMap(List.scala:79)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2826)
	at org.apache.spark.shuffle.ShuffleDataIOUtils$.loadShuffleDataIO(ShuffleDataIOUtils.scala:35)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:633)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:3055)
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
local test

Closes #3304 from cxzl25/CELEBORN-2022.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
2025-06-04 17:56:45 +08:00
..
src/main/resources/META-INF [CELEBORN-1413] Support Spark 4.0 2024-12-24 18:12:27 +08:00
pom.xml [CELEBORN-2022] Spark4 Client should package commons-io 2025-06-04 17:56:45 +08:00