SteNicholas 4e8e8c2310 [CELEBORN-1094] Optimize mechanism of ChunkManager expired shuffle key cleanup to avoid memory leak ... ### What changes were proposed in this pull request? The `cleaner` of `Worker` executes the `StorageManager#cleanupExpiredShuffleKey` to clean expired shuffle keys with daemon cached thread pool. The optimization speeds up cleaning including expired shuffle keys of ChunkManager to avoid memory leak. ### Why are the changes needed? `ChunkManager#streams` could lead memory leak when the speed of cleanup is slower than expiration for expired shuffle of worker. The behavior that `ChunkStreamManager` cleanup expired shuffle key should be optimized to avoid memory leak, which causes that the VM thread of worker is 100%. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `WorkerSuite#clean up`. Closes #2053 from SteNicholas/CELEBORN-1094. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>		2023-11-02 15:46:07 +08:00
..
src	[CELEBORN-1094] Optimize mechanism of ChunkManager expired shuffle key cleanup to avoid memory leak	2023-11-02 15:46:07 +08:00
pom.xml	[CELEBORN-977] Support RocksDB as recover DB backend	2023-09-19 09:20:33 +08:00