celeborn/worker
mingji 7be05b430b [CELEBORN-1133] Refactor fileinfo
### What changes were proposed in this pull request?
Rename FileWriter to PartitionLocationDataWriter, add storageManager, delete fileinfo, and flusher in the constructor.

FileInfo(userIdentifier,partitionSplitEnabled,fileMeta)
– NonMemoryFileInfo(streams,filePath,storageType,bytesFlushed)
– MemoryFileInfo(length,buffer)

FileMeta
– reduceFileMeta(chunkOffsets,sorted)
– mapFileMeta(bufferSize,numSubPartitions)

### Why are the changes needed?
1. To make concepts more clear.
2. To support memory storage and HDFS slot management.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
GA and cluster test with worker kill.

Closes #2130 from FMX/b1133.

Lead-authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Co-authored-by: Keyong Zhou <zky.zhoukeyong@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2024-01-02 21:26:10 +08:00
..
src [CELEBORN-1133] Refactor fileinfo 2024-01-02 21:26:10 +08:00
pom.xml [CELEBORN-977] Support RocksDB as recover DB backend 2023-09-19 09:20:33 +08:00