celeborn/docs/developers/slotsallocation.md

license

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Slots allocation

This article describes the detailed design of Celeborn workers' slots allocation. Slots allocation is the core components about how Celeborn distribute workload amount workers. We have achieved two approaches of slots allocation.

Principle

Allocate slots to local disks unless explicit assigned to HDFS.

LoadAware

Related configs

celeborn.master.slot.assign.policy LOADAWARE
celeborn.master.slot.assign.loadAware.numDiskGroups 5
celeborn.master.slot.assign.loadAware.diskGroupGradient 0.1
celeborn.master.slot.assign.loadAware.flushTimeWeight 0
celeborn.master.slot.assign.loadAware.fetchTimeWeight 0
[spark.client.]celeborn.storage.availableTypes HDD,SSD

Detail

Load-aware slots allocation will take following elements into consideration.

• disk's fetch time
• disk's flush time
• disk's usable space
• disk's used slot

Slots allocator will find out all worker involved in this allocation and sort their disks by disk's average flushtime * flush time weight + disk's average fetch time * fetch time weight. After getting the sorted disks list, Celeborn will split the disks into celeborn.master.slot.assign.loadAware.numDiskGroups groups. The slots number to be placed into a disk group is controlled by the celeborn.master.slot.assign.loadAware.diskGroupGradient which means that a group's allocated slots number will be (1+celeborn.master.slot.assign.loadAware.diskGroupGradient) times to the group's slower than it. For example, there is 5 groups, G1 , G2, G3, G4 and G5. If the G5 is allocated 100 slots. Other groups will be G4:110, G3:121, G2:133, G1:146.

After Celeborn has decided the slots number of a disk group, slots will be distributed in disks of a disk group. Each disk has a usableSlots which is calculated by (disk's usable space)/(average partition size)-usedSlots. The slots number to allocate in a disk is calculated by slots of this disk group * ( current disk's usableSlots / the sum of all disks' usableSlots in this group). For example, G5 need to allocate 100 slots and have 3 disks D1 with usable slots 100, D2 with usable slots 50, D3 with usable slots 20. The distribution will be D1:59, D2: 29, D3: 12.

If all slots can be place in disk groups, the slots allocation process is done.

requested slots are more than all usable slots, slots can not be placed into disks. Worker will need to allocate these slots to workers with local disks one by one.

RoundRobin

Detail

Roundrobin slots allocation will distribute all slots into all registered workers with disks. Celeborn will treat all workers as an array and place 1 slots in a worker until all slots are allocated. If a worker has multiple disks, the chosen disk index is (monotone increasing disk index +1) % disk count.

Celeborn Worker's Behavior

1. When reserve slots Celeborn worker will decide a slot be placed in local disks or HDFS when reserve slots.
2. If a partition is evicted from memory, the partition might be placed in HDFS.
3. If a slot is explicitly assigned to HDFS, worker will put the slot in HDFS.