apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
4cb4701ede
celeborn/client
History
Shuang 28a99ded8d [CELEBORN-687] Fix shuffleResourceExists, reduce unexpected slot release request 
### What changes were proposed in this pull request?
Check ShufflePartitionLocationInfo whether empty or not for every worker

### Why are the changes needed?
Actually shuffleResources would only remove related partitionLocations after stageEnd , then workers with empty partitionLocations will left(for speculative task), so shuffleResourceExists need check ShufflePartitionLocationInfo for every worker otherwise it would print wrong message and send release slot requests twice.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Manual test

### Before this pr

<img width="1252" alt="image" src="https://github.com/apache/incubator-celeborn/assets/28799061/06b71162-e78b-4163-8f52-24b50bc6c540">

![image](https://github.com/apache/incubator-celeborn/assets/28799061/fec263e0-9641-4d17-a837-ab03c36c5e6d)

### After this pr

![image](https://github.com/apache/incubator-celeborn/assets/28799061/8f2f9653-ff58-4a6e-ae06-1023922ca5bf)

Closes #1599 from RexXiong/CELEBORN-687.

Authored-by: Shuang <lvshuang.tb@gmail.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-06-16 18:37:48 +08:00
..
src [CELEBORN-687] Fix shuffleResourceExists, reduce unexpected slot release request 2023-06-16 18:37:48 +08:00
pom.xml [CELEBORN-147][IT]Extraction of common integration test cases (#1092) 2022-12-29 12:03:09 +08:00