apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-2006] LifecycleManager should avoid parsing shufflePartitionType every time

Browse Source 
### What changes were proposed in this pull request?

### Why are the changes needed?
`org.apache.celeborn.client.LifecycleManager.getPartitionType` may be called frequently, but in Spark scenario it requires each parsing configuration, which is not necessary.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
GA

Closes #3273 from cxzl25/CELEBORN-2006.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Wang, Fei <fwang12@ebay.com>
This commit is contained in:
sychen 2025-05-22 17:23:07 -07:00 committed by Wang, Fei
parent 634343e200
commit a554261f32
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala
View File

@ -329,8 +329,10 @@ class LifecycleManager(val appUniqueId: String, val conf: CelebornConf) extends
rpcEnv.address.port
}
private val partitionType = conf.shufflePartitionType
def getPartitionType(shuffleId: Int): PartitionType = {
shufflePartitionType.getOrDefault(shuffleId, conf.shufflePartitionType)
shufflePartitionType.getOrDefault(shuffleId, partitionType)
}
override def receive: PartialFunction[Any, Unit] = {