[CELEBORN-1010] Update docs about spark.shuffle.service.enabled
### What changes were proposed in this pull request? To clarify a spark config to work with Celeborn. ### Why are the changes needed? After some tests, I found that Spark 3.1 and newer can work with Celeborn with `spark.shuffle.service.enabled=true`. ExternalShuffleBlockResolver won't check the shuffle manager's type since Spark 3.1 and newer. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? I tested two scenarios about this PR. 1. Check whether Spark can release the executors in time. 2. Check data correctness by running TPC-DS. All checks are good. Closes #1955 from FMX/CELEBORN-1010. Authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com> Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
This commit is contained in:
parent
785a035eca
commit
95c9ccfc3e
@ -234,6 +234,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
|
||||
|
||||
# celeborn master
|
||||
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
|
||||
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
|
||||
spark.shuffle.service.enabled false
|
||||
|
||||
# options: hash, sort
|
||||
|
||||
@ -167,6 +167,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
|
||||
|
||||
# celeborn master
|
||||
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
|
||||
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
|
||||
spark.shuffle.service.enabled false
|
||||
|
||||
# options: hash, sort
|
||||
|
||||
@ -47,6 +47,7 @@ spark.celeborn.master.endpoints clb-master:9097
|
||||
spark.celeborn.client.push.replicate.enabled true
|
||||
|
||||
spark.celeborn.client.spark.shuffle.writer hash
|
||||
# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
|
||||
spark.shuffle.service.enabled false
|
||||
spark.sql.adaptive.localShuffleReader.enabled false
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user