celeborn/worker
zky.zhoukeyong 334f0e5ba4 [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
### What changes were proposed in this pull request?
Make max components configurable for FileWriter#flushBuffer.

### Why are the changes needed?
When max components of ```CompositeByteBuf``` is too big (hard coded 256 before this PR), netty's offheap memory
usage will be several times bigger than true usage:
```
 Direct memory usage: 1044.0 MiB/4.0 GiB, disk buffer size: 255.9 MiB
```
When set to 1, netty's memory usage will be very close to disk buffer:
```
Direct memory usage: 376.0 MiB/4.0 GiB, disk buffer size: 353.0 MiB
```
but when it set too low, for example 1, performance might decrease, especially for sort pusher:
```
max components:   1 vs. 16
shuffle write time:   34s vs. 30s
```
For hash pusher, difference is not so big:
```
max components:   1 vs. 8
shuffle write time:   25s vs. 23s
```
This PR makes the parameter configurable.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Passes GA, and manual test.

Closes #1697 from waitinfuture/782.

Lead-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
Co-authored-by: Keyong Zhou <waitinfuture@gmail.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-07-10 18:00:07 +08:00
..
src [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer 2023-07-10 18:00:07 +08:00
pom.xml [CELEBORN-650][TEST] Upgrade scalatest and unify mockito version 2023-06-09 10:04:14 +08:00