zky.zhoukeyong 8b6bc35997 [CELEBORN-1300] Optimize CelebornInputStreamImpl's memory usage ... ### What changes were proposed in this pull request? To avoid too much memory usage when CelebornShuffleReader creates input streams. This PR does the following: 1. Constructor of `CelebornInputStream` does not fetch chunk 2. `compressedBuf` and `rawDataBuf` are created first time `fillBuffer` is called 3. When `fillBuffer` returns false, which means the inputstream is exhausted, `close` is called and resource released 4. `CelebornFetchFailureSuite` is only run for Spark 3.0 and newer ### Why are the changes needed? ditto ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? GA and e2e test. Closes #2348 from waitinfuture/1300. Lead-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com> Co-authored-by: Keyong Zhou <waitinfuture@gmail.com> Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>		2024-03-05 14:03:11 +08:00
..
flink-it	[CELEBORN-1235] Start test nodes in random ports to allow multiple builds run in the same ci server	2024-01-27 12:51:53 +08:00
kubernetes-it	[MINOR] Fix config type of imagePullSecrets in values.yaml for helm	2024-01-19 14:02:00 +08:00
mr-it	[CELEBORN-1255][FOLLOWUP] Fix MR UT	2024-02-19 14:11:49 +08:00
spark-it	[CELEBORN-1300] Optimize CelebornInputStreamImpl's memory usage	2024-03-05 14:03:11 +08:00