celeborn/service
Wang, Fei 6c4f6c7b6c
[CELEBORN-1758] Remove the empty user resource consumption from worker heartbeat
### What changes were proposed in this pull request?

1. report the  `resourceConsumptionSnapshot` from `storageManager` directly in the worker heartbeat, which does not contain the empty user resource consumption
2. For RESTful API, do not return the empty user resource consumption as well.

### Why are the changes needed?
878a83cfa7/common/src/main/scala/org/apache/celeborn/common/meta/WorkerInfo.scala (L239-L248)

Currently, we never remove the user resource consumption even the sub resource consumptions is empty, and create a `ResourceConsumption(0, 0, 0, 0)` instead.

I am afraid that, the worker will report more and more empty user resource consumption to master, once one of their slots assigned to this worker.

Likes:
<img width="813" alt="image" src="https://github.com/user-attachments/assets/64932552-dc29-4a43-aed4-557419628b23">

So, I think we just need to report the `resourceConsumptionSnapshot` from `storageManager` directly, which does not contain the empty user resource consumption.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?
GA.

Closes #2967 from turboFei/reduce_report.

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: SteNicholas <programgeek@163.com>
2024-12-06 08:20:54 +08:00
..
src [CELEBORN-1758] Remove the empty user resource consumption from worker heartbeat 2024-12-06 08:20:54 +08:00
pom.xml [CELEBORN-1477][FOLLOWUP] Remove scala binary version from openapi-client artifactId 2024-11-01 15:08:00 +08:00