apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
a7bbbd05c4
celeborn/docs
History
zky.zhoukeyong a7bbbd05c4 [CELEBORN-797] Decrease writeTime metric sampling frequency to improve perf 
### What changes were proposed in this pull request?
1. Decrease writeTime metric sampling frequency to improve perf
2. Set default value of ```celeborn.<module>.push.timeoutCheck.threads``` and ```celeborn.<module>.fetch.timeoutCheck.threads``` to 4

### Why are the changes needed?
Following are test cases
case 1: ```spark.sparkContext.parallelize(1 to 8000, 8000).flatMap( _ => (1 to 15000000).iterator.map(num => num)).repartition(8000).count``` // shuffle 1.1T data
case 2: ```spark.sparkContext.parallelize(1 to 8000, 8000).flatMap( _ => (1 to 30000000).iterator.map(num => num)).repartition(8000).count``` // shuffle 2.2T data
Following are e2e time of shuffle write stage
||Sort pusher before|Sort pusher after|Hash pusher before|Hash pusher after|
|----|----|----|----|-----|
|case1|4.4min|4.1min|4.4min|3.9min|
|case2|9.1min|8.4min|9.7min|8.5min|

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Passes GA and manual test.

Closes #1718 from waitinfuture/797.

Authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-07-14 20:51:50 +08:00
..
assets/logo [CELEBORN-499] Move version specific resource to main repo (#1429) 2023-04-14 16:20:51 +08:00
configuration [CELEBORN-797] Decrease writeTime metric sampling frequency to improve perf 2023-07-14 20:51:50 +08:00
celeborn_ratis_shell.md [CELEBORN-638] Migrate configurations celeborn.ha.master.* to celeborn.master.ha.* 2023-06-16 18:18:26 +08:00
cluster_planning.md [CELEBORN-570] Update docs about monitor and deployment. (#1478) 2023-05-08 17:07:42 +08:00
deploy_on_k8s.md [CELEBORN-713] Local network binding support IP or FQDN 2023-06-27 09:42:11 +08:00
deploy.md [CELEBORN-751] Rename remain rss related class name and filenames etc 2023-07-04 10:20:08 +08:00
migration.md [CELEBORN-751] Rename remain rss related class name and filenames etc 2023-07-04 10:20:08 +08:00
monitoring.md [CELEBORN-726] Update data replication terminology from master/slave to primary/replica for configurations and metrics 2023-06-29 09:47:02 +08:00
README.md [CELEBORN-499] Move version specific resource to main repo (#1429) 2023-04-14 16:20:51 +08:00
upgrade.md [CELEBORN-680][DOC] Refresh celeborn configurations in doc 2023-06-15 13:59:38 +08:00

README.md

hide license
navigation
toc
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Apache Celeborn (Incubating)