169b6f6973
### What changes were proposed in this pull request? 1. `ShuffleFallbackPolicy` supports `ShuffleFallbackCount` metric to provide the shuffle fallback count of each fallback policy. 2. Introduce `ShuffleTotalCount` metric to record the total count of shuffle. 3. Fix Spark 2 does not increment shuffle count via `LifecycleManager`. ### Why are the changes needed? The implementations of `ShuffleFallbackPolicy` does not support `ShuffleFallbackCount` metric at present. Meanwhile, Bilibili production practice needs `ShuffleFallbackCount` of different `ShuffleFallbackPolicy`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Cluster test. Closes #2891 from SteNicholas/CELEBORN-1685. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| diagram | ||
| grafana | ||
| spark-patch | ||