apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #6521] [K8S][HELM] Implement new configuration approach

Browse Source 
# 🔍 Description
## Issue References 🔗

This pull request changes Helm chart configuration approach as discussed in https://github.com/apache/kyuubi/issues/6123

## Describe Your Solution 🔧

Suggested implementation makes chart configuration more general and more flexible. It allows to configure Kyuubi (and its engines) by setting configuration file contents to chart properties or by providing configuration files through existing ConfigMaps and Secrets. Also users are not limited by predefined number of files and can put any files to configuration directories.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

### 1. Test suite "Kyuubi configuration"

Property file `values-kyuubi-files.yaml`
```yaml
kyuubiConf:
  dir: /opt/kyuubi/conf
  files:
    'kyuubi-env.sh': |
      #!/usr/bin/env bash
      export KYUUBI_TEST=true
    'kyuubi-custom.properties': |
      kyuubi.custom=true
  filesFrom:
    - configMap:
        name: kyuubi-configs
```

ConfigMap `kyuubi-configs` from `configmap-kyuubi-configs.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kyuubi-configs
data:
  'kyuubi-test.properties': |
    kyuubi.config.test=true
```

#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-kyuubi-files.yaml -s templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml
```

#### Configuration files are in place
```shell
$ kubectl create -f configmap-kyuubi-configs.yaml
$ helm install kyuubi charts/kyuubi -f values-kyuubi-files.yaml
$ kubectl exec kyuubi-0 -- ls conf
kyuubi-custom.properties
kyuubi-env.sh
kyuubi-test.properties
```

### 2. Test suite "Spark configuration"

Property file `values-spark-files.yaml`
```yaml
sparkConf:
  dir: /opt/spark/conf
  files:
    'spark-env.sh': |
      #!/usr/bin/env bash
      export SPARK_TEST=true
    'spark-custom.properties': |
      spark.custom=true
  filesFrom:
    - configMap:
        name: spark-configs
```

ConfigMap `spark-configs` from `configmap-spark-configs.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-configs
data:
  'spark-test.properties': |
    spark.config.test=true
```

#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-spark-files.yaml -s templates/kyuubi-spark-configmap.yaml -s templates/kyuubi-statefulset.yaml
```

#### Configuration files are in place
```shell
$ kubectl create -f configmap-spark-configs.yaml
$ helm install kyuubi charts/kyuubi -f values-spark-files.yaml
$ kubectl exec kyuubi-0 -- ls ../spark/conf
spark-custom.properties
spark-env.sh
spark-test.properties
```

3. Test suite "Custom kyuubi-defaults.conf from existing ConfigMap overwrites kyuubi-defaults.conf from values"

Property file `values-kyuubi-defaults.yaml`
```yaml
kyuubiConf:
  dir: /opt/kyuubi/conf
  files:
    'kyuubi-defaults.conf': |
      custom.from.values=true
  filesFrom:
    - configMap:
        name: kyuubi-defaults-config
```

ConfigMap `kyuubi-defaults-config` from `configmap-kyuubi-defaults.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kyuubi-defaults-config
data:
  'kyuubi-defaults.conf': |
    custom.from.configmap=true
```

#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-kyuubi-defaults.yaml -s templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml
```

#### Content of `kyuubi-defaults.conf` comes from ConfigMap
```shell
$ kubectl create -f configmap-kyuubi-defaults.yaml
$ helm install kyuubi charts/kyuubi -f values-kyuubi-defaults.yaml
$ kubectl exec kyuubi-0 -- ls conf
kyuubi-defaults.conf
$ kubectl exec kyuubi-0 -- cat conf/kyuubi-defaults.conf
custom.from.configmap=true
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6521 from dnskr/implement-new-helm-chart-configuration-approach.

Closes #6521

452dca32f [dnskr] Fix empty value type
14829f342 [dnskr] Revert "[REVERT BEFORE MERGE] Use 'master-snapshot' image tag"
8d90f424c [dnskr] Move default properties from 'kyuubi-defaults.conf' to --conf args
6b3c77f94 [dnskr] [REVERT BEFORE MERGE] Use 'master-snapshot' image tag
fe7c17a68 [dnskr] [K8S][HELM] Implement new configuration approach

Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
dnskr 2024-11-22 17:47:07 +08:00 committed by Kent Yao
parent 160bf58042
commit b0036fdf92
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
4 changed files with 78 additions and 97 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
charts/kyuubi/templates/kyuubi-configmap.yaml
View File

@ -22,30 +22,6 @@ metadata:
labels:
{{- include "kyuubi.labels" . | nindent 4 }}
data:
{{- with .Values.kyuubiConf.kyuubiEnv }}
kyuubi-env.sh: |
{{- tpl . $ | nindent 4 }}
{{- end }}
kyuubi-defaults.conf: |
## Helm chart provided Kyuubi configurations
kyuubi.kubernetes.namespace={{ .Release.Namespace }}
kyuubi.frontend.connection.url.use.hostname=false
kyuubi.frontend.thrift.binary.bind.port={{ .Values.server.thriftBinary.port }}
kyuubi.frontend.thrift.http.bind.port={{ .Values.server.thriftHttp.port }}
kyuubi.frontend.rest.bind.port={{ .Values.server.rest.port }}
kyuubi.frontend.mysql.bind.port={{ .Values.server.mysql.port }}
kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
# Kyuubi Metrics
kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
## User provided Kyuubi configurations
{{- with .Values.kyuubiConf.kyuubiDefaults }}
{{- tpl . $ | nindent 4 }}
{{- end }}
{{- with .Values.kyuubiConf.log4j2 }}
log4j2.xml: |
{{- tpl . $ | nindent 4 }}
{{- with .Values.kyuubiConf.files }}
{{- tpl (toYaml .) $ | nindent 2 }}
{{- end }}

17
charts/kyuubi/templates/kyuubi-spark-configmap.yaml
View File

@ -22,19 +22,6 @@ metadata:
labels:
{{- include "kyuubi.labels" . | nindent 4 }}
data:
{{- with .Values.sparkConf.sparkEnv }}
spark-env.sh: |
{{- tpl . $ | nindent 4 }}
{{- end }}
{{- with .Values.sparkConf.sparkDefaults }}
spark-defaults.conf: |
{{- tpl . $ | nindent 4 }}
{{- end }}
{{- with .Values.sparkConf.log4j2 }}
log4j2.properties: |
{{- tpl . $ | nindent 4 }}
{{- end }}
{{- with .Values.sparkConf.metrics }}
metrics.properties: |
{{- tpl . $ | nindent 4 }}
{{- with .Values.sparkConf.files }}
{{- tpl (toYaml .) $ | nindent 2 }}
{{- end }}

42
charts/kyuubi/templates/kyuubi-statefulset.yaml
View File

@ -59,14 +59,28 @@ spec:
{{- with .Values.command }}
command: {{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
{{- with .Values.args }}
{{- if .Values.args }}
args: {{- tpl (toYaml .) $ | nindent 12 }}
{{- else }}
args:
- ./bin/kyuubi
- run
- --conf kyuubi.kubernetes.namespace={{ .Release.Namespace }}
- --conf kyuubi.frontend.connection.url.use.hostname=false
- --conf kyuubi.frontend.thrift.binary.bind.port={{ .Values.server.thriftBinary.port }}
- --conf kyuubi.frontend.thrift.http.bind.port={{ .Values.server.thriftHttp.port }}
- --conf kyuubi.frontend.rest.bind.port={{ .Values.server.rest.port }}
- --conf kyuubi.frontend.mysql.bind.port={{ .Values.server.mysql.port }}
- --conf kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
- --conf kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
- --conf kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
- --conf kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
{{- end }}
env:
- name: KYUUBI_CONF_DIR
value: {{ .Values.kyuubiConfDir }}
value: {{ .Values.kyuubiConf.dir }}
- name: SPARK_CONF_DIR
value: {{ .Values.sparkConfDir }}
value: {{ .Values.sparkConf.dir }}
{{- with .Values.env }}
{{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
@ -109,9 +123,9 @@ spec:
{{- end }}
volumeMounts:
- name: conf
mountPath: {{ .Values.kyuubiConfDir }}
mountPath: {{ .Values.kyuubiConf.dir }}
- name: conf-spark
mountPath: {{ .Values.sparkConfDir }}
mountPath: {{ .Values.sparkConf.dir }}
{{- with .Values.volumeMounts }}
{{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
@ -120,11 +134,21 @@ spec:
{{- end }}
volumes:
- name: conf
configMap:
name: {{ .Release.Name }}
projected:
sources:
- configMap:
name: {{ .Release.Name }}
{{- with .Values.kyuubiConf.filesFrom }}
{{- tpl (toYaml .) $ | nindent 14 }}
{{- end }}
- name: conf-spark
configMap:
name: {{ .Release.Name }}-spark
projected:
sources:
- configMap:
name: {{ .Release.Name }}-spark
{{- with .Values.sparkConf.filesFrom }}
{{- tpl (toYaml .) $ | nindent 14 }}
{{- end }}
{{- with .Values.volumes }}
{{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}

88
charts/kyuubi/values.yaml
View File

@ -145,53 +145,40 @@ server:
# clientIP:
# timeoutSeconds: 10800
# $KYUUBI_CONF_DIR directory
kyuubiConfDir: /opt/kyuubi/conf
# Kyuubi configuration files
kyuubiConf:
# The value (templated string) is used for kyuubi-env.sh file
# See example at conf/kyuubi-env.sh.template and https://kyuubi.readthedocs.io/en/master/configuration/settings.html#environments for more details
kyuubiEnv: ~
# kyuubiEnv: |
# #!/usr/bin/env bash
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
# export SPARK_HOME=/opt/spark
# export FLINK_HOME=/opt/flink
# export HIVE_HOME=/opt/hive
# The value (templated string) is used for kyuubi-defaults.conf file
# See https://kyuubi.readthedocs.io/en/master/configuration/settings.html#kyuubi-configurations for more details
kyuubiDefaults: ~
# kyuubiDefaults: |
# $KYUUBI_CONF_DIR directory
dir: /opt/kyuubi/conf
# Configuration files from the specified keys (file name) and values (file content)
files: ~
#files:
# 'kyuubi-defaults.conf': |
# kyuubi.authentication=NONE
# kyuubi.frontend.bind.host=10.0.0.1
# kyuubi.engine.type=SPARK_SQL
# kyuubi.engine.share.level=USER
# kyuubi.session.engine.initialize.timeout=PT3M
# kyuubi.ha.addresses=zk1:2181,zk2:2181,zk3:2181
# kyuubi.ha.namespace=kyuubi
# The value (templated string) is used for log4j2.xml file
# See example at conf/log4j2.xml.template https://kyuubi.readthedocs.io/en/master/configuration/settings.html#logging for more details
log4j2: ~
# Configuration files from the list of existing ConfigMaps and Secrets
filesFrom: []
#filesFrom:
#- configMap:
# name: kyuubi-configs
#- secret:
# name: kyuubi-secrets
#- secret:
# name: ssl-secrets
# items:
# - key: key-store
# path: certs/keystore.jks
# - key: trust-store
# path: certs/truststore.jks
# $SPARK_CONF_DIR directory
sparkConfDir: /opt/spark/conf
# Spark configuration files
# Spark configuration, see https://github.com/apache/spark/tree/master/conf and Spark documentation for more details
sparkConf:
# The value (templated string) is used for spark-env.sh file
# See example at https://github.com/apache/spark/blob/master/conf/spark-env.sh.template and Spark documentation for more details
sparkEnv: ~
# sparkEnv: |
# #!/usr/bin/env bash
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
# export SPARK_LOG_DIR=/opt/spark/logs
# export SPARK_LOG_MAX_FILES=5
# The value (templated string) is used for spark-defaults.conf file
# See example at https://github.com/apache/spark/blob/master/conf/spark-defaults.conf.template and Spark documentation for more details
sparkDefaults: ~
# sparkDefaults: |
# $SPARK_CONF_DIR directory
dir: /opt/spark/conf
# Configuration files from the specified keys (file name) and values (file content)
files: ~
#files:
# 'spark-defaults.conf': |
# spark.submit.deployMode=cluster
# spark.kubernetes.container.image=apache/spark:3.5.0
# spark.kubernetes.authenticate.driver.serviceAccountName=spark
@ -207,13 +194,20 @@ sparkConf:
# spark.hadoop.fs.s3a.path.style.access=true
# spark.hadoop.fs.s3a.fast.upload=true
# The value (templated string) is used for log4j2.properties file
# See example at https://github.com/apache/spark/blob/master/conf/log4j2.properties.template and Spark documentation for more details
log4j2: ~
# The value (templated string) is used for metrics.properties file
# See example at https://github.com/apache/spark/blob/master/conf/metrics.properties.template and Spark documentation for more details
metrics: ~
# Configuration files from the list of existing ConfigMaps and Secrets
filesFrom: []
#filesFrom:
#- configMap:
# name: spark-configs
#- secret:
# name: spark-secrets
#- secret:
# name: ssl-secrets
# items:
# - key: key-store
# path: certs/keystore.jks
# - key: trust-store
# path: certs/truststore.jks
# Command to launch Kyuubi server (templated)
command: ~