apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
ec2959fccb
celeborn/client-spark
History
Fu Chen b9c9c00697 [CELEBORN-683][SPARK][PERF] Avoid calling CelebornConf.get multi-time when columnar shuffle wri… 
…te is enabled.

### What changes were proposed in this pull request?

as title.

### Why are the changes needed?

frame graph and stage duration before:

![截屏2023-06-15 下午4 49 04](https://github.com/apache/incubator-celeborn/assets/8537877/6fe7f7f6-fd36-42ec-a6a1-9a4943022dc8)

![截屏2023-06-15 下午4 57 53](https://github.com/apache/incubator-celeborn/assets/8537877/077f6c22-4dc9-497a-affe-ddba9200fe28)

frame graph and stage duration after:

![截屏2023-06-15 下午4 37 45](https://github.com/apache/incubator-celeborn/assets/8537877/d6ae7aa6-95c7-490e-a0ae-c110e6a83e5a)

![截屏2023-06-15 下午4 58 12](https://github.com/apache/incubator-celeborn/assets/8537877/e8dd5c3b-94d9-47d7-a644-4897acef43ad)

### Does this PR introduce _any_ user-facing change?

No, only perf improvement.

### How was this patch tested?

tested locally.

Closes #1595 from cfmcgrady/columnar-conf.

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-06-15 17:52:23 +08:00
..
common [CELEBORN-648][SPARK] Improve perf of SendBufferPool and logs about memory 2023-06-09 09:45:27 +08:00
spark-2 [CELEBORN-673][SPARK][PERF] Improve the perf of sort-based shuffle write 2023-06-13 19:07:04 +08:00
spark-2-shaded [Feature] Shade netty native libraries (#908) 2022-11-02 19:02:03 +08:00
spark-3 [CELEBORN-683][SPARK][PERF] Avoid calling CelebornConf.get multi-time when columnar shuffle wri… 2023-06-15 17:52:23 +08:00
spark-3-shaded Exclude unused files from Spark shaded client (#942) 2022-11-09 11:20:33 +08:00