apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-1280] Change default value of celeborn.worker.graceful.shutdown.recoverDbBackend to ROCKSDB

Browse Source 
### What changes were proposed in this pull request?

Change the default value of `celeborn.worker.graceful.shutdown.recoverDbBackend` from `LEVELDB` to `ROCKSDB`.

### Why are the changes needed?

Because the LevelDB support will be removed, the default value of `celeborn.worker.graceful.shutdown.recoverDbBackend` could be changed to ROCKSDB instead of LEVELDB for preparation of LevelDB deprecation.

Backport:
 [[SPARK-45351][CORE] Change spark.shuffle.service.db.backend default value to ROCKSDB](https://github.com/apache/spark/pull/43142)
 [[SPARK-45413][CORE] Add warning for prepare drop LevelDB support](https://github.com/apache/spark/pull/43217)

### Does this PR introduce _any_ user-facing change?

The default value of `celeborn.worker.graceful.shutdown.recoverDbBackend` is changed from `LEVELDB` to `ROCKSDB`.

### How was this patch tested?

No.

Closes #2320 from SteNicholas/CELEBORN-1280.

Lead-authored-by: SteNicholas <programgeek@163.com>
Co-authored-by: Nicholas Jiang <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
This commit is contained in:
SteNicholas 2024-02-23 14:53:24 +08:00 committed by mingji
parent 621a719d8d
commit b9bdea3c72
No known key found for this signature in database
GPG Key ID: 6392F71F37356FA0
4 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
View File

@ -3002,12 +3002,12 @@ object CelebornConf extends Logging {
val WORKER_GRACEFUL_SHUTDOWN_RECOVER_DB_BACKEND: ConfigEntry[String] =
buildConf("celeborn.worker.graceful.shutdown.recoverDbBackend")
.categories("worker")
.doc("Specifies a disk-based store used in local db. LEVELDB or ROCKSDB.")
.doc("Specifies a disk-based store used in local db. ROCKSDB or LEVELDB (deprecated).")
.version("0.4.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValues(Set("LEVELDB", "ROCKSDB"))
.createWithDefault("LEVELDB")
.createWithDefault("ROCKSDB")
val WORKER_PARTITION_SORTER_SHUTDOWN_TIMEOUT: ConfigEntry[Long] =
buildConf("celeborn.worker.graceful.shutdown.partitionSorter.shutdownTimeout")

2
docs/configuration/worker.md
View File

@ -75,7 +75,7 @@ license: |
| celeborn.worker.graceful.shutdown.checkSlotsFinished.timeout | 480s | false | The wait time of waiting for the released slots to be committed or destroyed during worker graceful shutdown. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.enabled | false | false | When true, during worker shutdown, the worker will wait for all released slots to be committed or destroyed. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.partitionSorter.shutdownTimeout | 120s | false | The wait time of waiting for sorting partition files during worker graceful shutdown. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.recoverDbBackend | LEVELDB | false | Specifies a disk-based store used in local db. LEVELDB or ROCKSDB. | 0.4.0 | |
| celeborn.worker.graceful.shutdown.recoverDbBackend | ROCKSDB | false | Specifies a disk-based store used in local db. ROCKSDB or LEVELDB (deprecated). | 0.4.0 | |
| celeborn.worker.graceful.shutdown.recoverPath | &lt;tmp&gt;/recover | false | The path to store DB. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.saveCommittedFileInfo.interval | 5s | false | Interval for a Celeborn worker to flush committed file infos into Level DB. | 0.3.1 | |
| celeborn.worker.graceful.shutdown.saveCommittedFileInfo.sync | false | false | Whether to call sync method to save committed file infos into Level DB to handle OS crash. | 0.3.1 | |

3
docs/migration.md
View File

@ -29,6 +29,9 @@ license: |
- Since 0.5.0, Celeborn configurations support new tag `isDynamic` to represent whether the configuration is dynamic config.
- Since 0.5.0, Celeborn changed the default value of `celeborn.worker.graceful.shutdown.recoverDbBackend` from `LEVELDB` to `ROCKSDB`, which means Celeborn will use RocksDB store for recovery backend.
To restore the behavior before Celeborn 0.5, you can set `celeborn.worker.graceful.shutdown.recoverDbBackend` to `LEVELDB`.
## Upgrading from 0.3 to 0.4
- Since 0.4.0, Celeborn won't be compatible with Celeborn client that versions below 0.3.0.

6
worker/src/main/java/org/apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java
View File

@ -20,14 +20,20 @@ package org.apache.celeborn.service.deploy.worker.shuffledb;
import java.io.File;
import java.io.IOException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Note: code copied from Apache Spark. */
public class DBProvider {
private static final Logger logger = LoggerFactory.getLogger(DBProvider.class);
public static DB initDB(DBBackend dbBackend, File dbFile, StoreVersion version)
throws IOException {
if (dbFile != null) {
switch (dbBackend) {
case LEVELDB:
org.iq80.leveldb.DB levelDB = LevelDBProvider.initLevelDB(dbFile, version);
logger.warn("The LEVELDB is deprecated. Please use ROCKSDB instead.");
return levelDB != null ? new LevelDB(levelDB) : null;
case ROCKSDB:
org.rocksdb.RocksDB rocksDB = RocksDBProvider.initRockDB(dbFile, version);