20a629a432
### What changes were proposed in this pull request? Fix NPE if `celeborn.client.spark.fetch.cleanFailedShuffle` is true. This PR also refine the code for `FailedShuffleCleaner`. ### Why are the changes needed? `failedShuffleCleaner` is null in executor end. ``` 25/07/29 17:58:40 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[CoarseGrainedExecutorBackend-stop-executor,5,main] java.lang.NullPointerException: Cannot invoke "org.apache.celeborn.spark.FailedShuffleCleaner.reset()" because "this.failedShuffleCleaner" is null at org.apache.spark.shuffle.celeborn.SparkShuffleManager.stop(SparkShuffleManager.java:272) ~[celeborn-client-spark-3-shaded_2.12-0.6.0-rc3.jar:?] ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. Closes #3401 from turboFei/fix_npe_cleaner. Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|---|---|---|
| .. | ||
| src | ||
| pom.xml | ||