apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
f226424b9a
celeborn/client-spark
History
sychen bc3bd460e8 [CELEBORN-1544][FOLLOWUP] ShuffleWriter needs to catch exception and call abort to avoid memory leaks 
### What changes were proposed in this pull request?
This PR aims to fix a possible memory leak in ShuffleWriter.

Introduce a private abort method, which can be called to release memory when an exception occurs.

### Why are the changes needed?
https://github.com/apache/celeborn/pull/2661 Call the close method in the finally block, but the close method has `shuffleClient.mapperEnd`, which is dangerous for incomplete tasks, and the data may be inaccurate.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
GA

Closes #2663 from cxzl25/CELEBORN-1544-followup.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2024-08-21 14:45:57 +08:00
..
common [CELEBORN-1544][FOLLOWUP] ShuffleWriter needs to catch exception and call abort to avoid memory leaks 2024-08-21 14:45:57 +08:00
spark-2 [CELEBORN-1544][FOLLOWUP] ShuffleWriter needs to catch exception and call abort to avoid memory leaks 2024-08-21 14:45:57 +08:00
spark-2-shaded [INFRA] Remove incubator/incubating for graduation 2024-03-27 13:54:47 +08:00
spark-3 [CELEBORN-1544][FOLLOWUP] ShuffleWriter needs to catch exception and call abort to avoid memory leaks 2024-08-21 14:45:57 +08:00
spark-3-columnar-shuffle [CELEBORN-1495] CelebornColumnDictionary supports dictionary of float and double column type 2024-07-11 10:53:00 +08:00
spark-3-shaded [INFRA] Remove incubator/incubating for graduation 2024-03-27 13:54:47 +08:00