183e18f1bc
 [](https://github.com/yaooqinn/kyuubi/pull/404)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #404 from yaooqinn/dra. b8b755c [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi 9bb47b1 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi e786787 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
1.7 KiB
1.7 KiB
## Shuffle Behavior
# spark.shuffle.compress true
# spark.shuffle.detectCorrupt true
# spark.shuffle.detectCorrupt.useExtraMemory true
# spark.shuffle.file.buffer 64k
# spark.shuffle.unsafe.file.output.buffer 64k
# spark.shuffle.spill.diskWriteBufferSize 8k
# spark.shuffle.spill.compress true
# spark.shuffle.mapOutput.dispatcher.numThreads 12
# spark.shuffle.mapOutput.parallelAggregationThreshold 5000
# spark.shuffle.readHostLocalDisk true
# spark.shuffle.io.maxRetries 10
# spark.shuffle.io.retryWait 6s
# spark.shuffle.io.preferDirectBufs false
# spark.shuffle.io.serverThreads 8
# spark.shuffle.io.clientThreads 8
# spark.shuffle.io.connectionTimeout 240s
# spark.shuffle.registration.timeout 6000
# spark.shuffle.registration.maxAttempts 10
# spark.shuffle.sync false
# spark.shuffle.useOldFetchProtocol true
# spark.shuffle.unsafe.fastMergeEnabled true
# spark.shuffle.minNumPartitionsToHighlyCompress 100
# spark.network.maxRemoteBlockSizeFetchToMem 128m
# spark.reducer.maxSizeInFlight 48m
# spark.reducer.maxReqsInFlight 256
# spark.reducer.maxBlocksInFlightPerAddress 256