apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
0e5008db19
celeborn/client-spark
History
TongWei1105 0583cdb5a8 [CELEBORN-1048] Align fetchWaitTime metrics to spark implementation 
### What changes were proposed in this pull request?
Align fetchWaitTime metrics to spark implementation

### Why are the changes needed?
In our production environment, there are variations in the fetchWaitTime metric for the same stage of the same job.

ON YARN ESS:
![image](https://github.com/apache/incubator-celeborn/assets/68682646/601a8315-1317-48dc-b9a6-7ea651d5122d)
ON CELEBORN
![image](https://github.com/apache/incubator-celeborn/assets/68682646/e00ed60f-3789-4330-a7ed-fdd5754acf1d)
Then, based on the implementation of Spark ShuffleBlockFetcherIterator, I made adjustments to the fetchWaitTime metrics code

Now, looks like more reasonable, 
![image](https://github.com/apache/incubator-celeborn/assets/68682646/ce5e46e4-8ed2-422e-b54b-cd594aad73dd)
### Does this PR introduce _any_ user-facing change?
no

### How was this patch tested?
yes, tested in our production environment

Closes #2000 from TongWei1105/CELEBORN-1048.

Lead-authored-by: TongWei1105 <vvtwow@gmail.com>
Co-authored-by: Keyong Zhou <zhouky@apache.org>
Co-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-11-02 15:27:30 +08:00
..
common [CELEBORN-1032][FOLLOWUP] Use scheduleWithFixedDelay instead of scheduleAtFixedRate in threads pool of master and worker 2023-10-27 11:20:28 +08:00
spark-2 [CELEBORN-1048] Align fetchWaitTime metrics to spark implementation 2023-11-02 15:27:30 +08:00
spark-2-shaded [CELEBORN-1003] Correct the LICENSE and NOTICE for shaded client jars 2023-09-28 19:23:54 +08:00
spark-3 [CELEBORN-1048] Align fetchWaitTime metrics to spark implementation 2023-11-02 15:27:30 +08:00
spark-3-columnar-shuffle [CELEBORN-1022][TEST] Update log level from FATAL to ERROR for console output in unit tests 2023-10-09 15:56:05 +08:00
spark-3-shaded [CELEBORN-1003] Correct the LICENSE and NOTICE for shaded client jars 2023-09-28 19:23:54 +08:00