celeborn/common
SteNicholas 464a7c71a9 [CELEBORN-1651] Support ratio threshold of unhealthy disks for excluding worker
### What changes were proposed in this pull request?

Support ratio threshold of unhealthy disks for excluding worker with `celeborn.master.excludeWorker.unhealthyDiskRatioThreshold`.

### Why are the changes needed?

We often encounter issues such as disk input/output errors in production practice. When a bad disk occurs, the worker will be maintained to decommission for repairing the machine disk. The reason is that generally the fault will be repaired in time after it is discovered. It is possible that the machine will not trigger all disk failures if it is out of warranty. It can be replaced directly when it is under warranty. If the disk fails after it is out of warranty, you need to purchase the disk yourself for replacement. At the same time, submitting the disk for repair at one time will affect the failure rate judgment of the system group and scenario. In addition, the occurrence of bad disks will bring about some management problems, such as continuous alarms, and the handling of disk failures is relatively customized.

Therefore, it's recommended to configure ratio threshold of unhealthy disks for excluding worker, which does not need to wait for all unhealthy disks to exclude corresponding worker.

### Does this PR introduce _any_ user-facing change?

Introduce `celeborn.master.excludeWorker.unhealthyDiskRatioThreshold` to configure max ratio of unhealthy disks for excluding worker.

### How was this patch tested?

Cluster test.

Closes #2812 from SteNicholas/CELEBORN-1651.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
2024-10-24 11:43:22 +08:00
..
benchmarks [CELEBORN-744] Add Benchmark framework and ComputeIfAbsentBenchmark 2023-06-29 20:19:30 +08:00
src [CELEBORN-1651] Support ratio threshold of unhealthy disks for excluding worker 2024-10-24 11:43:22 +08:00
pom.xml [CELEBORN-1521] Introduce celeborn-spi module for authentication extensions 2024-07-25 00:52:00 -07:00