a76c344042
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #951 from pan3793/license. Closes #951 4629eecd [Cheng Pan] Fix c45a0784 [Cheng Pan] nit b9a46b42 [Cheng Pan] pin license header at first line 80d1a71b [Cheng Pan] nit b2a46e4c [Cheng Pan] Update f6acaaf8 [Cheng Pan] minor ef99183f [Cheng Pan] Add license header on all docs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
47 lines
2.5 KiB
Markdown
47 lines
2.5 KiB
Markdown
<!--
|
|
- Licensed to the Apache Software Foundation (ASF) under one or more
|
|
- contributor license agreements. See the NOTICE file distributed with
|
|
- this work for additional information regarding copyright ownership.
|
|
- The ASF licenses this file to You under the Apache License, Version 2.0
|
|
- (the "License"); you may not use this file except in compliance with
|
|
- the License. You may obtain a copy of the License at
|
|
-
|
|
- http://www.apache.org/licenses/LICENSE-2.0
|
|
-
|
|
- Unless required by applicable law or agreed to in writing, software
|
|
- distributed under the License is distributed on an "AS IS" BASIS,
|
|
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
- See the License for the specific language governing permissions and
|
|
- limitations under the License.
|
|
-->
|
|
|
|
```properties
|
|
## Shuffle Behavior
|
|
# spark.shuffle.compress true
|
|
# spark.shuffle.detectCorrupt true
|
|
# spark.shuffle.detectCorrupt.useExtraMemory true
|
|
# spark.shuffle.file.buffer 64k
|
|
# spark.shuffle.unsafe.file.output.buffer 64k
|
|
# spark.shuffle.spill.diskWriteBufferSize 8k
|
|
# spark.shuffle.spill.compress true
|
|
# spark.shuffle.mapOutput.dispatcher.numThreads 12
|
|
# spark.shuffle.mapOutput.parallelAggregationThreshold 5000
|
|
# spark.shuffle.readHostLocalDisk true
|
|
# spark.shuffle.io.maxRetries 10
|
|
# spark.shuffle.io.retryWait 6s
|
|
# spark.shuffle.io.preferDirectBufs false
|
|
# spark.shuffle.io.serverThreads 8
|
|
# spark.shuffle.io.clientThreads 8
|
|
# spark.shuffle.io.connectionTimeout 240s
|
|
# spark.shuffle.registration.timeout 6000
|
|
# spark.shuffle.registration.maxAttempts 10
|
|
# spark.shuffle.sync false
|
|
# spark.shuffle.useOldFetchProtocol true
|
|
# spark.shuffle.unsafe.fastMergeEnabled true
|
|
# spark.shuffle.minNumPartitionsToHighlyCompress 100
|
|
# spark.network.maxRemoteBlockSizeFetchToMem 128m
|
|
# spark.reducer.maxSizeInFlight 48m
|
|
# spark.reducer.maxReqsInFlight 256
|
|
# spark.reducer.maxBlocksInFlightPerAddress 256
|
|
```
|