From 95c9ccfc3e13c68f7bcdb1d3c06859b915501e2e Mon Sep 17 00:00:00 2001 From: mingji Date: Sun, 8 Oct 2023 09:15:42 +0800 Subject: [PATCH] [CELEBORN-1010] Update docs about `spark.shuffle.service.enabled` ### What changes were proposed in this pull request? To clarify a spark config to work with Celeborn. ### Why are the changes needed? After some tests, I found that Spark 3.1 and newer can work with Celeborn with `spark.shuffle.service.enabled=true`. ExternalShuffleBlockResolver won't check the shuffle manager's type since Spark 3.1 and newer. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? I tested two scenarios about this PR. 1. Check whether Spark can release the executors in time. 2. Check data correctness by running TPC-DS. All checks are good. Closes #1955 from FMX/CELEBORN-1010. Authored-by: mingji Signed-off-by: zky.zhoukeyong --- README.md | 1 + docs/deploy.md | 1 + docs/developers/glutensupport.md | 1 + 3 files changed, 3 insertions(+) diff --git a/README.md b/README.md index 5876f2b4f..1f86a9203 100644 --- a/README.md +++ b/README.md @@ -234,6 +234,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer # celeborn master spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097 +# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer spark.shuffle.service.enabled false # options: hash, sort diff --git a/docs/deploy.md b/docs/deploy.md index b618d8e5c..b91f74ced 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -167,6 +167,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer # celeborn master spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097 +# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer spark.shuffle.service.enabled false # options: hash, sort diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md index d3b7b01df..3879e353f 100644 --- a/docs/developers/glutensupport.md +++ b/docs/developers/glutensupport.md @@ -47,6 +47,7 @@ spark.celeborn.master.endpoints clb-master:9097 spark.celeborn.client.push.replicate.enabled true spark.celeborn.client.spark.shuffle.writer hash +# This is not necessary if your Spark external shuffle service is Spark 3.1 or newer spark.shuffle.service.enabled false spark.sql.adaptive.localShuffleReader.enabled false