apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-972][HELM] Enhance workingdirDiskCapacity unit parsing and fix ConfigMap is not effected for workerStatefuleSet

Browse Source 
### What changes were proposed in this pull request?
1. fix the issue with the configmap not being mounted for worker
2. fix compatability with different workingdir's capacity byte unit types, e.g.  Gi, Ti.

### Why are the changes needed?

1. in previous pr the configmap is removed from value.yaml, but worker-statefulset.yaml still use this config, so worker pod can't mount the /opt/celeborn/conf to configmap volume.

2. in previous pr capacity is appended for workingdir, but unit type Gi is not suppored by byteStringTransformer
``` java
  java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.Invalid suffix: "gi"

celeborn.worker.storage.dirs=/mnt/disk1:disktype=SSD:capacity=100Gi,/mnt/disk2:disktype=SSD:capacity=100Gi,/mnt/disk3:disktype=SSD:capacity=100Gi,/mnt/disk4:disktype=SSD:capacity=100Gi
```

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?
manual on k8s
1. the issue of configMap not being mounted
- befor this pr
```shell
kubectl get pod celeborn-worker-0 -o yaml |grep conf
  - configMap:
      name: celeborn-conf
      - configMap:
```
- after this pr
``` shell
kubectl get pod celeborn-worker-0 -o yaml |grep conf
    - mountPath: /opt/celeborn/conf
  - configMap:
      name: celeborn-conf
      - configMap:
```
2. compatilbiy
the NumberFormatException is not thrown after this pr.

Closes #1906 from zhongqiangczq/helm-fix.

Authored-by: zhongqiang.czq <zhongqiang.czq@alibaba-inc.com>
Signed-off-by: zhongqiang.czq <zhongqiang.czq@alibaba-inc.com>
This commit is contained in:
zhongqiang.czq 2023-09-14 11:58:26 +08:00
parent 3c8d89ca99
commit 8b4fe73d4f
2 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
charts/celeborn/templates/worker-statefulset.yaml
View File

@ -114,11 +114,9 @@ spec:
name: metrics
protocol: TCP
volumeMounts:
{{- if .Values.configmap }}
- mountPath: /opt/celeborn/conf
name: {{ include "celeborn.fullname" . }}-volume
readOnly: true
{{- end }}
{{- range $index, $volume := .Values.volumes.worker }}
- name: {{ $.Release.Name }}-worker-vol-{{ $index }}
mountPath: {{ .mountPath }}

10
common/src/main/java/org/apache/celeborn/common/util/JavaUtils.java
View File

@ -226,14 +226,24 @@ public class JavaUtils {
.put("b", ByteUnit.BYTE)
.put("k", ByteUnit.KiB)
.put("kb", ByteUnit.KiB)
.put("kib", ByteUnit.KiB)
.put("ki", ByteUnit.KiB)
.put("m", ByteUnit.MiB)
.put("mb", ByteUnit.MiB)
.put("mib", ByteUnit.MiB)
.put("mi", ByteUnit.MiB)
.put("g", ByteUnit.GiB)
.put("gb", ByteUnit.GiB)
.put("gib", ByteUnit.GiB)
.put("gi", ByteUnit.GiB)
.put("t", ByteUnit.TiB)
.put("tb", ByteUnit.TiB)
.put("tib", ByteUnit.TiB)
.put("ti", ByteUnit.TiB)
.put("p", ByteUnit.PiB)
.put("pb", ByteUnit.PiB)
.put("pib", ByteUnit.PiB)
.put("pi", ByteUnit.PiB)
.build();
/**