apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-1844][FOLLOWUP] alway try to use memory storage if available

Browse Source 
### What changes were proposed in this pull request?
Try to use memory storage first if it is available.
To increase performance, if a cluster is set to use MEMORY and HDFS.

### Why are the changes needed?
To keep the old behavior as in release 0.5, always try to use memory storage first because the slots allocator won't allocate slots on memory storage.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
GA.

Closes #3352 from FMX/b1844-2.

Authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
This commit is contained in:
mingji 2025-07-10 15:55:29 +08:00
parent 0fa600ade1
commit 532cedbfd2
1 changed files with 9 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StoragePolicy.scala
View File

@ -185,8 +185,15 @@ class StoragePolicy(conf: CelebornConf, storageManager: StorageManager, source:
if (evict) {
0
} else {
order.get.indexOf(
partitionDataWriterContext.getPartitionLocation.getStorageInfo.getType.name())
// keep the old behavior, always try to use memory if worker
// has configured to use memory storage, because slots allocator
// will not allocate slots on memory storage
if (order.contains(StorageInfo.Type.MEMORY.name())) {
order.get.indexOf(StorageInfo.Type.MEMORY.name())
} else {
order.get.indexOf(
partitionDataWriterContext.getPartitionLocation.getStorageInfo.getType.name())
}
}
val maxSize = order.get.length