6a097944cf
### What changes were proposed in this pull request? Fixes the FetchFailure handling logic in shouldReportShuffleFetchFailure method to properly handle cases where TaskSetManager cannot be found for a given task ID. ### Why are the changes needed? The current implementation incorrectly reports FetchFailure when TaskSetManager is not found, which leads to false positive failures in normal fault tolerance scenarios. This happens because: 1. Executor Lost scenarios: When executors are lost due to resource preemption or failures, the associated TaskSetManager gets cleaned up, making it unavailable for lookup 2. Stage cancellation: Cancelled or completed stages may have their TaskSetManager removed These are all normal scenarios in Spark's fault tolerance mechanism and should not be treated as shuffle failures. The current behavior can cause unnecessary job failures and confusion in debugging actual shuffle issues. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT, Long-running Production Validation Closes #3339 from gaoyajun02/CELEBORN-2042. Authored-by: gaoyajun02 <gaoyajun02@meituan.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|---|---|---|
| .. | ||
| flink-it | ||
| kubernetes-it | ||
| mr-it | ||
| spark-it | ||
| tez-it | ||