276ab979a4
### What changes were proposed in this pull request? Unify the size and file count of active shuffle metrics for `MasterSource` and `WorkerSource`. Follow up #2171. ### Why are the changes needed? `MasterSource` uses `PartitionWritten` and `PartitionFileCount` metrics as the size and file count of active shuffle for all workers. Meanwhile, `WorkerSource` uses `ActiveShuffleSize` and `ActiveShuffleFileCount` metrics as the size and file count of active shuffle for a worker including master replica and slave replica. It's recommended to unify the size and file count of active shuffle metrics between `MasterSource` and `WorkerSource`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Internal tests. Closes #2186 from SteNicholas/CELEBORN-1187. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| diagram | ||
| grafana | ||
| img | ||
| spark-patch | ||