caojiaqing d64e0091f1 [CELEBORN-785] Add worker side partition hard split threshold ... ### What changes were proposed in this pull request? Add a configuration `celeborn.worker.shuffle.partitionSplit.max` to ensure that, in soft mode, individual partition files are limited to a size smaller than the configured value ### Why are the changes needed? In soft mode, there may be situations where individual partition files are exceptionally large, which can result in excessively long sort times in skewed scenarios. ### Does this PR introduce _any_ user-facing change? `celeborn.worker.shuffle.partitionSplit.max` defalut value 2g ### How was this patch tested? none Closes #1701 from JQ-Cao/785. Authored-by: caojiaqing <caojiaqing@bilibili.com> Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>		2023-07-11 14:14:41 +08:00
..
src	[CELEBORN-785] Add worker side partition hard split threshold	2023-07-11 14:14:41 +08:00
pom.xml	[CELEBORN-650][TEST] Upgrade scalatest and unify mockito version	2023-06-09 10:04:14 +08:00