mingji
|
e0c00ecd38
|
[CELEBORN-839][MR] Support Hadoop MapReduce
### What changes were proposed in this pull request?
1. Map side merge and push.
2. Support hadoop2 & 3.
3. Reduce in-memory merge.
4. Integrate LifecycleManager to RmApplicationMaster.
### Why are the changes needed?
Ditto.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Cluster.
I tested this PR on a cluster with a 4x 16 CPU 64G Mem 4ESSD cluster.
Hadoop 2.8.5
1TB Terasort, 8400 mappers, 1000 reducers
Celeborn 81min vs MR shuffle 89min
1GB wordcount, 8 mappers, 8 reducers
Celeborn 35s VS MR shuffle 38s
Closes #1830 from FMX/CELEBORN-839.
Lead-authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Co-authored-by: Ethan Feng <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
|
2023-09-14 14:12:53 +08:00 |
|
Ethan Feng
|
114b1b4d62
|
[CELEBORN-548][FLINK] Support flink 1.17. (#1472)
|
2023-05-05 23:00:49 +08:00 |
|
Ethan Feng
|
93d2f106e0
|
[CELEBORN-548][FLINK] Support flink 1.15. (#1463)
|
2023-05-04 15:23:59 +08:00 |
|
Cheng Pan
|
a16ba0e807
|
[CELEBORN-180][BUILD] Script for creating binary release artifact (#1129)
|
2023-01-03 12:58:42 +08:00 |
|
Cheng Pan
|
7105f98829
|
[CELEBORN-160][BUILD] Spilt CI workflow (#1107)
|
2022-12-21 23:47:01 +08:00 |
|
Shuang
|
f3f104870c
|
[CELEBORN-75] Initialize flink plugin module (#1027)
|
2022-12-07 15:53:00 +08:00 |
|
Cheng Pan
|
c88ce306be
|
Use Spotless to auto check and reformat Java/Scala code (#497)
|
2022-09-01 21:19:56 +08:00 |
|