Sanskar Modi f527b22b4d [CELEBORN-1410] Combine multiple ShuffleBlockInfo into a single ShuffleBlockInfo ... ### What changes were proposed in this pull request? Merging smaller `ShuffleBlockInfo` corresponding into same mapID, such that size of each block does not exceeds `celeborn.shuffle.chunk.size` ### Why are the changes needed? As sorted ShuffleBlocks are contiguous, we can compact multiple `ShuffleBlockInfo` into one as long as the size of compacted one does not exceeds half of `celeborn.shuffle.chunk.size`. This way we can decrease the number of ShuffleBlock objects. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing UTs Closes #2524 from s0nskar/CELEBORN-1410. Lead-authored-by: Sanskar Modi <sanskarmodi97@gmail.com> Co-authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>		2024-05-23 21:22:45 +08:00
..
main	[CELEBORN-1410] Combine multiple ShuffleBlockInfo into a single ShuffleBlockInfo	2024-05-23 21:22:45 +08:00
test	[CELEBORN-914] Support memory file storage	2024-05-23 21:05:52 +08:00