[CELEBORN-1010] Update docs about spark.shuffle.service.enabled

### What changes were proposed in this pull request?
To clarify a spark config to work with Celeborn.

### Why are the changes needed?
After some tests, I found that Spark 3.1 and newer can work with Celeborn with `spark.shuffle.service.enabled=true`.

ExternalShuffleBlockResolver won't check the shuffle manager's type since Spark 3.1 and newer.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
I tested two scenarios about this PR.
1. Check whether Spark can release the executors in time.
2. Check data correctness by running TPC-DS.
All checks are good.

Closes #1955 from FMX/CELEBORN-1010.

Authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
This commit is contained in:
mingji 2023-10-08 09:15:42 +08:00 committed by zky.zhoukeyong
parent 785a035eca
commit 95c9ccfc3e
3 changed files with 3 additions and 0 deletions

View File

@ -234,6 +234,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
# celeborn master
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
spark.shuffle.service.enabled false
# options: hash, sort

View File

@ -167,6 +167,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
# celeborn master
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
spark.shuffle.service.enabled false
# options: hash, sort

View File

@ -47,6 +47,7 @@ spark.celeborn.master.endpoints clb-master:9097
spark.celeborn.client.push.replicate.enabled true
spark.celeborn.client.spark.shuffle.writer hash
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
spark.shuffle.service.enabled false
spark.sql.adaptive.localShuffleReader.enabled false