celeborn/assets/spark-patch
wangshengjie d659e06d45 [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files
### What changes were proposed in this pull request?
Add logic to support avoid sorting shuffle files for Reduce mode when optimize skew partitions

### Why are the changes needed?
Current logic need sorting shuffle files when read Reduce mode skew partition shuffle files, we found some shuffle sorting timeout and performance issue

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Cluster test and uts

Closes #2373 from wangshengjie123/optimize-skew-partition.

Lead-authored-by: wangshengjie <wangshengjie3@xiaomi.com>
Co-authored-by: wangshengjie3 <wangshengjie3@xiaomi.com>
Co-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Shuang <lvshuang.tb@gmail.com>
Co-authored-by: wangshengjie3 <soldier.sj.wang@gmail.com>
Co-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
2025-02-19 16:57:44 +08:00
..
Celeborn_Columnar_Shuffle_spark3.patch
Celeborn_Dynamic_Allocation_spark2.patch
Celeborn_Dynamic_Allocation_spark3_0.patch
Celeborn_Dynamic_Allocation_spark3_1.patch
Celeborn_Dynamic_Allocation_spark3_2.patch
Celeborn_Dynamic_Allocation_spark3_3.patch
Celeborn_Dynamic_Allocation_spark3_4.patch
Celeborn-Optimize-Skew-Partitions-spark3_2.patch [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files 2025-02-19 16:57:44 +08:00
Celeborn-Optimize-Skew-Partitions-spark3_3.patch [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files 2025-02-19 16:57:44 +08:00
Celeborn-Optimize-Skew-Partitions-spark3_4.patch [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files 2025-02-19 16:57:44 +08:00
Celeborn-Optimize-Skew-Partitions-spark3_5.patch [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files 2025-02-19 16:57:44 +08:00