---
license: |
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      https://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
---

<!--begin-include-->
| Key | Default | isDynamic | Description | Since | Deprecated |
| --- | ------- | --------- | ----------- | ----- | ---------- |
| celeborn.&lt;module&gt;.fetch.timeoutCheck.interval | 5s | false | Interval for checking fetch data timeout. It only support setting <module> to `data` since it works for shuffle client fetch data. | 0.3.0 |  | 
| celeborn.&lt;module&gt;.fetch.timeoutCheck.threads | 4 | false | Threads num for checking fetch data timeout. It only support setting <module> to `data` since it works for shuffle client fetch data. | 0.3.0 |  | 
| celeborn.&lt;module&gt;.heartbeat.interval | 60s | false | The heartbeat interval between worker and client. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `fetch`, it works for worker fetch server. If you are using the "celeborn.client.heartbeat.interval", please use the new configs for each module according to your needs or replace it with "celeborn.push.heartbeat.interval" and "celeborn.fetch.heartbeat.interval".  | 0.3.0 | celeborn.client.heartbeat.interval | 
| celeborn.&lt;module&gt;.io.backLog | 0 | false | Requested maximum length of the queue of incoming connections. Default 0 for no backlog. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. |  |  | 
| celeborn.&lt;module&gt;.io.clientThreads | 0 | false | Number of threads used in the client thread pool. Default to 0, which is 2x#cores. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data, of which default value is determined by celeborn.<module>.io.threads . If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. |  |  | 
| celeborn.&lt;module&gt;.io.conflictAvoidChooser.enable | false | false | Whether to use conflict avoid event executor chooser in the client thread pool. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. | 0.5.4 |  | 
| celeborn.&lt;module&gt;.io.connectTimeout | &lt;value of celeborn.network.connect.timeout&gt; | false | Socket connect timeout. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `replicate`, it works for the replicate client of worker replicating data to peer worker. |  |  | 
| celeborn.&lt;module&gt;.io.connectionTimeout | &lt;value of celeborn.network.timeout&gt; | false | Connection active timeout. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server or client of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. |  |  | 
| celeborn.&lt;module&gt;.io.lazyFD | true | false | Whether to initialize FileDescriptor lazily or not. If true, file descriptors are created only when data is going to be transferred. This can reduce the number of open files. If setting <module> to `fetch`, it works for worker fetch server. |  |  | 
| celeborn.&lt;module&gt;.io.maxRetries | 3 | false | Max number of times we will try IO exceptions (such as connection timeouts) per request. If set to 0, we will not do any retries. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. If setting <module> to `push`, it works for Flink shuffle client push data. |  |  | 
| celeborn.&lt;module&gt;.io.mode | &lt;undefined&gt; | false | Netty EventLoopGroup backend, available options: NIO, EPOLL. If epoll mode is available, the default IO mode is EPOLL; otherwise, the default is NIO. |  |  | 
| celeborn.&lt;module&gt;.io.numConnectionsPerPeer | 1 | false | Number of concurrent connections between two nodes. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. |  |  | 
| celeborn.&lt;module&gt;.io.preferDirectBufs | true | false | If true, we will prefer allocating off-heap byte buffers within Netty. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server or client of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. |  |  | 
| celeborn.&lt;module&gt;.io.receiveBuffer | 0b | false | Receive buffer size (SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth = 10Gbps buffer size should be ~ 1.25MB. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server or client of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. | 0.2.0 |  | 
| celeborn.&lt;module&gt;.io.retryWait | 5s | false | Time that we will wait in order to perform a retry after an IOException. Only relevant if maxIORetries > 0. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. If setting <module> to `push`, it works for Flink shuffle client push data. | 0.2.0 |  | 
| celeborn.&lt;module&gt;.io.saslTimeout | 30s | false | Timeout for a single round trip of auth message exchange, in milliseconds. | 0.5.0 |  | 
| celeborn.&lt;module&gt;.io.sendBuffer | 0b | false | Send buffer size (SO_SNDBUF). If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `data`, it works for shuffle client push and fetch data. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server or client of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. | 0.2.0 |  | 
| celeborn.&lt;module&gt;.io.serverThreads | 0 | false | Number of threads used in the server thread pool. Default to 0, which is 2x#cores. If setting <module> to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`, works for master or worker. If setting <module> to `push`, it works for worker receiving push data. If setting <module> to `replicate`, it works for replicate server of worker replicating data to peer worker. If setting <module> to `fetch`, it works for worker fetch server. |  |  | 
| celeborn.&lt;module&gt;.io.threads | 8 | false | Default number of threads used in the server and client thread pool. This specifies thread configuration based on JVM's allocation of cores. If setting <module> to `data`, it works for shuffle client push and fetch data. |  |  | 
| celeborn.&lt;module&gt;.push.timeoutCheck.interval | 5s | false | Interval for checking push data timeout. If setting <module> to `data`, it works for shuffle client push data. If setting <module> to `push`, it works for Flink shuffle client push data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. | 0.3.0 |  | 
| celeborn.&lt;module&gt;.push.timeoutCheck.threads | 4 | false | Threads num for checking push data timeout. If setting <module> to `data`, it works for shuffle client push data. If setting <module> to `push`, it works for Flink shuffle client push data. If setting <module> to `replicate`, it works for replicate client of worker replicating data to peer worker. | 0.3.0 |  | 
| celeborn.&lt;role&gt;.rpc.dispatcher.threads | &lt;value of celeborn.rpc.dispatcher.threads&gt; | false | Threads number of message dispatcher event loop for roles |  |  | 
| celeborn.io.maxDefaultNettyThreads | 64 | false | Max default netty threads | 0.3.2 |  | 
| celeborn.network.advertise.preferIpAddress | &lt;value of celeborn.network.bind.preferIpAddress&gt; | false | When `true`, prefer to use IP address, otherwise FQDN for advertise address. | 0.6.0 |  | 
| celeborn.network.bind.preferIpAddress | true | false | When `true`, prefer to use IP address, otherwise FQDN. This configuration only takes effects when the bind hostname is not set explicitly, in such case, Celeborn will find the first non-loopback address to bind. | 0.3.0 |  | 
| celeborn.network.bind.wildcardAddress | false | false | When `true`, the bind address will be set to a wildcard address, while the advertise address will remain as whatever is set by `celeborn.network.advertise.preferIpAddress`. The wildcard address is a special local IP address, and usually refers to 'any' and can only be used for bind operations. In the case of IPv4, this is 0.0.0.0 and in the case of IPv6 this is ::0. This is helpful in dual-stack environments, where the service must listen to both IPv4 and IPv6 clients. | 0.6.0 |  | 
| celeborn.network.connect.timeout | 10s | false | Default socket connect timeout. | 0.2.0 |  | 
| celeborn.network.memory.allocator.numArenas | &lt;undefined&gt; | false | Number of arenas for pooled memory allocator. Default value is Runtime.getRuntime.availableProcessors, min value is 2. | 0.3.0 |  | 
| celeborn.network.memory.allocator.verbose.metric | false | false | Whether to enable verbose metric for pooled allocator. | 0.3.0 |  | 
| celeborn.network.timeout | 240s | false | Default timeout for network operations. | 0.2.0 |  | 
| celeborn.port.maxRetries | 1 | false | When port is occupied, we will retry for max retry times. | 0.2.0 |  | 
| celeborn.rpc.askTimeout | 60s | false | Timeout for RPC ask operations. It's recommended to set at least `240s` when `HDFS` is enabled in `celeborn.storage.availableTypes` | 0.2.0 |  | 
| celeborn.rpc.connect.threads | 64 | false |  | 0.2.0 |  | 
| celeborn.rpc.dispatcher.threads | 0 | false | Threads number of message dispatcher event loop. Default to 0, which is availableCore. | 0.3.0 | celeborn.rpc.dispatcher.numThreads | 
| celeborn.rpc.dump.interval | 60s | false | min interval (ms) for RPC framework to dump performance summary | 0.6.0 |  | 
| celeborn.rpc.inbox.capacity | 0 | false | Specifies size of the in memory bounded capacity. | 0.5.0 |  | 
| celeborn.rpc.io.threads | &lt;undefined&gt; | false | Netty IO thread number of NettyRpcEnv to handle RPC request. The default threads number is the number of runtime available processors. | 0.2.0 |  | 
| celeborn.rpc.lookupTimeout | 30s | false | Timeout for RPC lookup operations. | 0.2.0 |  | 
| celeborn.rpc.retryWait | 1s | false | Time to wait before next retry on RpcTimeoutException. | 0.5.4 |  | 
| celeborn.rpc.slow.interval | &lt;undefined&gt; | false | min interval (ms) for RPC framework to log slow RPC | 0.6.0 |  | 
| celeborn.rpc.slow.threshold | 1s | false | threshold for RPC framework to log slow RPC | 0.6.0 |  | 
| celeborn.shuffle.io.maxChunksBeingTransferred | &lt;undefined&gt; | false | The max number of chunks allowed to be transferred at the same time on shuffle service. Note that new incoming connections will be closed when the max number is hit. The client will retry according to the shuffle retry configs (see `celeborn.<module>.io.maxRetries` and `celeborn.<module>.io.retryWait`), if those limits are reached the task will fail with fetch failure. | 0.2.0 |  | 
| celeborn.ssl.&lt;module&gt;.enabled | false | false | Enables SSL for securing wire traffic. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.enabledAlgorithms | &lt;undefined&gt; | false | A comma-separated list of ciphers. The specified ciphers must be supported by JVM.<br/>The reference list of protocols can be found in the "JSSE Cipher Suite Names" section of the Java security guide. The list for Java 11, for example, can be found at [this page](https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#jsse-cipher-suite-names)<br/>Note: If not set, the default cipher suite for the JRE will be used | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.keyStore | &lt;undefined&gt; | false | Path to the key store file.<br/> The path can be absolute or relative to the directory in which the process is started. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.keyStorePassword | &lt;undefined&gt; | false | Password to the key store. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.protocol | TLSv1.2 | false | TLS protocol to use.<br/> The protocol must be supported by JVM.<br/> The reference list of protocols can be found in the "Additional JSSE Standard Names" section of the Java security guide. For Java 11, for example, the list can be found [here](https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#additional-jsse-standard-names) | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.sslHandshakeTimeoutMs | 10s | false | The timeout for the SSL handshake (in milliseconds). The default value is set to the current Netty default. This is applicable for `rpc_app` and `rpc_service` modules | 0.5.4 |  | 
| celeborn.ssl.&lt;module&gt;.trustStore | &lt;undefined&gt; | false | Path to the trust store file.<br/> The path can be absolute or relative to the directory in which the process is started. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.trustStorePassword | &lt;undefined&gt; | false | Password for the trust store. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.trustStoreReloadIntervalMs | 10s | false | The interval at which the trust store should be reloaded (in milliseconds), when enabled. This setting is mostly only useful for server components, not applications. | 0.5.0 |  | 
| celeborn.ssl.&lt;module&gt;.trustStoreReloadingEnabled | false | false | Whether the trust store should be reloaded periodically.<br/> This setting is mostly only useful for Celeborn services (masters, workers), and not applications. | 0.5.0 |  | 
<!--end-include-->
