d659e06d45
### What changes were proposed in this pull request? Add logic to support avoid sorting shuffle files for Reduce mode when optimize skew partitions ### Why are the changes needed? Current logic need sorting shuffle files when read Reduce mode skew partition shuffle files, we found some shuffle sorting timeout and performance issue ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Cluster test and uts Closes #2373 from wangshengjie123/optimize-skew-partition. Lead-authored-by: wangshengjie <wangshengjie3@xiaomi.com> Co-authored-by: wangshengjie3 <wangshengjie3@xiaomi.com> Co-authored-by: Fu Chen <cfmcgrady@gmail.com> Co-authored-by: Shuang <lvshuang.tb@gmail.com> Co-authored-by: wangshengjie3 <soldier.sj.wang@gmail.com> Co-authored-by: Fei Wang <fwang12@ebay.com> Co-authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| diagram | ||
| grafana | ||
| spark-patch | ||