celeborn/dev/deps
sychen beed2a85b0
[CELEBORN-977] Support RocksDB as recover DB backend
### What changes were proposed in this pull request?

### Why are the changes needed?

LevelDB does not support mac arm version.

```java
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8: dlopen(/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8, 0x0001): tried: '/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (fat file, but missing compatible architecture (have 'x86_64,i386', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (no such file), '/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (fat file, but missing compatible architecture (have 'x86_64,i386', need 'arm64'))]
  	at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
  	at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
  	at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
  	at org.apache.celeborn.service.deploy.worker.shuffledb.LevelDBProvider.initLevelDB(LevelDBProvider.java:49)
  	at org.apache.celeborn.service.deploy.worker.shuffledb.DBProvider.initDB(DBProvider.java:30)
  	at org.apache.celeborn.service.deploy.worker.storage.StorageManager.<init>(StorageManager.scala:197)
  	at org.apache.celeborn.service.deploy.worker.Worker.<init>(Worker.scala:109)
  	at org.apache.celeborn.service.deploy.worker.Worker$.main(Worker.scala:734)
  	at org.apache.celeborn.service.deploy.worker.Worker.main(Worker.scala)
```

The released `leveldbjni-all` for `org.fusesource.leveldbjni` does not support AArch64 Linux, we need to use `org.openlabtesting.leveldbjni`.

See https://issues.apache.org/jira/browse/HADOOP-16614

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
local test

Closes #1913 from cxzl25/CELEBORN-977.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-19 09:20:33 +08:00
..
dependencies-client-flink-1.14 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-flink-1.15 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-flink-1.17 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-spark-2.4 [CELEBORN-929][INFRA] Add dependencies check CI 2023-09-07 14:02:07 +08:00
dependencies-client-spark-3.0 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-spark-3.1 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-spark-3.2 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-spark-3.3 [CELEBORN-879] Add dev/dependencies.sh for audit dependencies 2023-08-26 15:59:20 +08:00
dependencies-client-spark-3.4 [CELEBORN-929][INFRA] Add dependencies check CI 2023-09-07 14:02:07 +08:00
dependencies-client-spark-3.5 [CELEBORN-929][INFRA] Add dependencies check CI 2023-09-07 14:02:07 +08:00
dependencies-server [CELEBORN-977] Support RocksDB as recover DB backend 2023-09-19 09:20:33 +08:00