apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
375e855d42
celeborn/docs/configuration
History
mingji e0c00ecd38 [CELEBORN-839][MR] Support Hadoop MapReduce 
### What changes were proposed in this pull request?
1. Map side merge and push.
2. Support hadoop2 & 3.
3. Reduce in-memory merge.
4. Integrate LifecycleManager to RmApplicationMaster.

### Why are the changes needed?
Ditto.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
Cluster.

I tested this PR on a cluster with a 4x 16 CPU 64G Mem 4ESSD cluster.
Hadoop 2.8.5

1TB Terasort, 8400 mappers, 1000 reducers
Celeborn 81min vs MR shuffle 89min
![mr1](https://github.com/apache/incubator-celeborn/assets/4150993/a3cf6493-b6ff-4c03-9936-4558cf22761d)
![mr2](https://github.com/apache/incubator-celeborn/assets/4150993/9119ffb4-6996-4b77-bcdf-cbd6db5c096f)

1GB wordcount, 8 mappers, 8 reducers
Celeborn 35s VS MR shuffle 38s
![mr3](https://github.com/apache/incubator-celeborn/assets/4150993/907dce24-16b7-4788-ab5d-5b784fd07d47)
![mr4](https://github.com/apache/incubator-celeborn/assets/4150993/8e8065b9-6c46-4c8d-9e71-45eed8e63877)

Closes #1830 from FMX/CELEBORN-839.

Lead-authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Co-authored-by: Ethan Feng <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-09-14 14:12:53 +08:00
..
client.md [CELEBORN-839][MR] Support Hadoop MapReduce 2023-09-14 14:12:53 +08:00
columnar-shuffle.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
ha.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
index.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
master.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
metrics.md [CELEBORN-882][WORKER][METRICS] Add Pause Push Data Time Count Metrics & Dashboard Panel 2023-09-12 17:45:26 +08:00
network.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
quota.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
worker.md [CELEBORN-920] Worker sends its load to Master through heartbeat 2023-08-26 13:58:37 +08:00