zky.zhoukeyong 8b6bc35997 [CELEBORN-1300] Optimize CelebornInputStreamImpl's memory usage ... ### What changes were proposed in this pull request? To avoid too much memory usage when CelebornShuffleReader creates input streams. This PR does the following: 1. Constructor of `CelebornInputStream` does not fetch chunk 2. `compressedBuf` and `rawDataBuf` are created first time `fillBuffer` is called 3. When `fillBuffer` returns false, which means the inputstream is exhausted, `close` is called and resource released 4. `CelebornFetchFailureSuite` is only run for Spark 3.0 and newer ### Why are the changes needed? ditto ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? GA and e2e test. Closes #2348 from waitinfuture/1300. Lead-authored-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com> Co-authored-by: Keyong Zhou <waitinfuture@gmail.com> Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>		2024-03-05 14:03:11 +08:00
..
src	[CELEBORN-1300] Optimize CelebornInputStreamImpl's memory usage	2024-03-05 14:03:11 +08:00
pom.xml	[CELEBORN-713] Local network binding support IP or FQDN	2023-06-27 09:42:11 +08:00