7571e10ad5
### What changes were proposed in this pull request? Whenever a `WorkerPartitionReader` is recreated (due celeborn worker restarts / any other chunk fetch failure), the entire shuffle partition file is re-read from beginning, discarding already read chunks in `CelebornInputStream` based on the batchIdSet metadata maintained. This can be improved (only for cases where shuffle data is unreplicated) by skipping already read chunk id since they'd be discarded anyway. This improves overall shuffle read performance (reducer's total time, network usage etc). ### Why are the changes needed? Allow skipping already read shuffle chunks ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UTs added Closes #3132 from saurabhd336/skipReadChunks. Authored-by: Saurabh Dubey <saurabhd336@uber.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| client.md | ||
| columnar-shuffle.md | ||
| ha.md | ||
| index.md | ||
| master.md | ||
| metrics.md | ||
| network-module.md | ||
| network.md | ||
| quota.md | ||
| worker.md | ||