celeborn/common
SteNicholas 893085c9c0 [CELEBORN-1473] TransportClientFactory should register netty memory metric with source for shared pooled ByteBuf allocator
### What changes were proposed in this pull request?

`TransportClientFactory` registers netty memory metric with source of `TransportContext` for shared pooled ByteBuf allocator.

Address https://github.com/apache/celeborn/pull/2585#issuecomment-2183720137.

### Why are the changes needed?

The default value of `celeborn.network.memory.allocator.share` is true, which means that enables shared memory allocator at default. Meanwhile, `RpcEnv#create` does not create `TransportClientFactory` with source, which cause lack of `NettyMemoryMetric` for `TransportClientFactory`.

Therefore, `TransportClientFactory` should also register `NettyMemoryMetric` with `WorkerSource` of `TransportContext` for netty memory metrics of shared pooled memory.

![Untitled-2024-06-23-0500](https://github.com/user-attachments/assets/e527be5f-d155-4129-ad49-31a351616b58)

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

- `celeborn.network.memory.allocator.allowCache=false`
```
$ curl http://celeborn-worker:9096/metrics|grep metrics_shared_pool_used
metrics_shared_pool_1_usedHeapMemory_Value{hostName="celeborn-worker",role="Worker"} 29360128 1722505389446
metrics_shared_pool_1_usedDirectMemory_Value{hostName="celeborn-worker",role="Worker"} 188743680 1722505389446
```

- `celeborn.network.memory.allocator.allowCache=true`
```
$ curl http://celeborn-worker:9096/metrics|grep metrics_shared_pool_used
metrics_shared_pool_1_usedHeapMemory_Value{hostName="celeborn-worker",role="Worker"} 8388608 1722505646795
metrics_shared_pool_1_usedDirectMemory_Value{hostName="celeborn-worker",role="Worker"} 8388608 1722505646795
metrics_shared_pool_0_usedHeapMemory_Value{hostName="celeborn-worker",role="Worker"} 281018368 1722505646795
metrics_shared_pool_0_usedDirectMemory_Value{hostName="celeborn-worker",role="Worker"} 1438646272 1722505646796
```

Closes #2652 from SteNicholas/CELEBORN-1473.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
2024-08-01 23:11:44 +08:00
..
benchmarks [CELEBORN-744] Add Benchmark framework and ComputeIfAbsentBenchmark 2023-06-29 20:19:30 +08:00
src [CELEBORN-1473] TransportClientFactory should register netty memory metric with source for shared pooled ByteBuf allocator 2024-08-01 23:11:44 +08:00
pom.xml [CELEBORN-1521] Introduce celeborn-spi module for authentication extensions 2024-07-25 00:52:00 -07:00