apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
e0c00ecd38
celeborn/dev
History
mingji e0c00ecd38 [CELEBORN-839][MR] Support Hadoop MapReduce 
### What changes were proposed in this pull request?
1. Map side merge and push.
2. Support hadoop2 & 3.
3. Reduce in-memory merge.
4. Integrate LifecycleManager to RmApplicationMaster.

### Why are the changes needed?
Ditto.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
Cluster.

I tested this PR on a cluster with a 4x 16 CPU 64G Mem 4ESSD cluster.
Hadoop 2.8.5

1TB Terasort, 8400 mappers, 1000 reducers
Celeborn 81min vs MR shuffle 89min
![mr1](https://github.com/apache/incubator-celeborn/assets/4150993/a3cf6493-b6ff-4c03-9936-4558cf22761d)
![mr2](https://github.com/apache/incubator-celeborn/assets/4150993/9119ffb4-6996-4b77-bcdf-cbd6db5c096f)

1GB wordcount, 8 mappers, 8 reducers
Celeborn 35s VS MR shuffle 38s
![mr3](https://github.com/apache/incubator-celeborn/assets/4150993/907dce24-16b7-4788-ab5d-5b784fd07d47)
![mr4](https://github.com/apache/incubator-celeborn/assets/4150993/8e8065b9-6c46-4c8d-9e71-45eed8e63877)

Closes #1830 from FMX/CELEBORN-839.

Lead-authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Co-authored-by: Ethan Feng <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-09-14 14:12:53 +08:00
..
deps [CELEBORN-929][INFRA] Add dependencies check CI 2023-09-07 14:02:07 +08:00
checkout_pr.sh [INFRA] Inroduce checkout_pr.sh shell script (#968) 2022-11-14 22:28:43 +08:00
dependencies.sh [CELEBORN-929][INFRA] Add dependencies check CI 2023-09-07 14:02:07 +08:00
merge_pr.py [CELEBORN-937][INFRA] Improve branch suggestion for backporting 2023-09-01 00:20:42 +08:00
reformat [CELEBORN-839][MR] Support Hadoop MapReduce 2023-09-14 14:12:53 +08:00