From f2e9043028a57fc7d4df3d11bad5b7f9227bc098 Mon Sep 17 00:00:00 2001 From: Weijie Guo Date: Wed, 6 Nov 2024 10:52:34 +0800 Subject: [PATCH] [CELEBORN-1687] Highlight flink session cluster issue in doc ### What changes were proposed in this pull request? If we use celeborn shuffle service, we can't submit both batch and streaming to the same flink session cluster. This should be highlight in doc. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need. Closes #2879 from reswqa/session-doc. Authored-by: Weijie Guo Signed-off-by: mingji --- README.md | 2 +- docs/deploy.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 95c2c2a38..17c625c06 100644 --- a/README.md +++ b/README.md @@ -312,7 +312,7 @@ spark.executor.userClassPathFirst false ### Deploy Flink client -**Important: Only Flink batch jobs are supported for now.** +**Important: Only Flink batch jobs are supported for now. Due to the Shuffle Service in Flink is cluster-granularity, if you want to use Celeborn in a session cluster, it will not be able to submit both streaming and batch job to the same cluster. We plan to get rid of this restriction for Hybrid Shuffle mode in a future release.** Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`. diff --git a/docs/deploy.md b/docs/deploy.md index 30909603f..d151ef494 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -210,7 +210,7 @@ spark.executor.userClassPathFirst false ## Deploy Flink client -**Important: Only Flink batch jobs are supported for now.** +**Important: Only Flink batch jobs are supported for now. Due to the Shuffle Service in Flink is cluster-granularity, if you want to use Celeborn in a session cluster, it will not be able to submit both streaming and batch job to the same cluster. We plan to get rid of this restriction for Hybrid Shuffle mode in a future release.** Celeborn release binary contains clients for Flink 1.14.x, Flink 1.15.x, Flink 1.17.x, Flink 1.18.x, Flink 1.19.x and Flink 1.20.x, copy the corresponding client jar into Flink's `lib/` directory: