celeborn/worker
sychen d7e900fa9a [CELEBORN-959] Use Java API to obtain disk capacity information instead of df command
### What changes were proposed in this pull request?
Use Java API to obtain disk capacity information.

bf605c8acc/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java (L84-L104)

599bb77c45/jdk/src/solaris/native/java/io/UnixFileSystem_md.c (L439-L467)

### Why are the changes needed?

Some OS does not support the `df -B1` command, and the worker will throw an `ArrayIndexOutOfBoundsException` exception.

We can replace the df command with the Java API, which is more general.

```java
23/09/08 22:03:25,522 ERROR [worker-disk-checker] LocalDeviceMonitor: Device check failed.
java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: -4
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at org.apache.celeborn.common.util.Utils$.tryWithTimeoutAndCallback(Utils.scala:858)
	at org.apache.celeborn.service.deploy.worker.storage.DeviceMonitor$.highDiskUsage(DeviceMonitor.scala:258)
	at org.apache.celeborn.service.deploy.worker.storage.LocalDeviceMonitor$$anon$1.$anonfun$run$9(DeviceMonitor.scala:136)
	at org.apache.celeborn.service.deploy.worker.storage.LocalDeviceMonitor$$anon$1.$anonfun$run$9$adapted(DeviceMonitor.scala:135)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at org.apache.celeborn.service.deploy.worker.storage.LocalDeviceMonitor$$anon$1.$anonfun$run$2(DeviceMonitor.scala:135)
	at org.apache.celeborn.service.deploy.worker.storage.LocalDeviceMonitor$$anon$1.$anonfun$run$2$adapted(DeviceMonitor.scala:110)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at org.apache.celeborn.service.deploy.worker.storage.LocalDeviceMonitor$$anon$1.run(DeviceMonitor.scala:110)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -4
	at org.apache.celeborn.service.deploy.worker.storage.DeviceMonitor$.$anonfun$highDiskUsage$1(DeviceMonitor.scala:240)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at org.apache.celeborn.common.util.Utils$$anon$3.call(Utils.scala:851)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
```

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes #1892 from cxzl25/CELEBORN-959.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-09-11 17:42:29 +08:00
..
src [CELEBORN-959] Use Java API to obtain disk capacity information instead of df command 2023-09-11 17:42:29 +08:00
pom.xml [CELEBORN-888][WORKER] Tweak the logic and add unit tests for the MemoryManager#currentServingState method 2023-08-23 17:20:31 +08:00