apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #6580] [K8S][HELM] Improve metrics configuration

Browse Source 
# 🔍 Description
## Issue References 🔗

This pull request fixes https://github.com/apache/kyuubi/issues/6565

## Describe Your Solution 🔧

- Fixed misleading `kyuubi.metrics` properties in `charts/kyuubi/templates/kyuubi-configmap.yaml`.
- Fixed condition to create `PodMonitor`, `ServiceMonitor` and `PrometheusRule`.
- Move metrics related properties to `metrics` property tree.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Disabled metrics - prometheus port is not renderred
```shell
helm template kyuubi charts/kyuubi --set metrics.enabled=false \
                                   -s templates/kyuubi-statefulset.yaml \
                                   -s templates/kyuubi-headless-service.yaml
```

#### JMX reporter only - prometheus port is not renderred
```shell
helm template kyuubi charts/kyuubi --set metrics.reporters="JMX" \
                                   -s templates/kyuubi-statefulset.yaml \
                                   -s templates/kyuubi-headless-service.yaml
```

#### Default properties - prometheus port is renderred in StatefulSet and Headless Service
```shell
helm template kyuubi charts/kyuubi -s templates/kyuubi-statefulset.yaml \
                                   -s templates/kyuubi-headless-service.yaml
```

#### Default properties - PodMonitor is not renderred
```shell
helm template kyuubi charts/kyuubi -s templates/kyuubi-podmonitor.yaml
```

#### Default properties - ServiceMonitor is not renderred
```shell
helm template kyuubi charts/kyuubi -s templates/kyuubi-servicemonitor.yaml
```

#### Default properties - PrometheusRule is not renderred
```shell
helm template kyuubi charts/kyuubi -s templates/kyuubi-alert.yaml
```

#### Enabled metrics, PodMonitor, ServiceMonitor, PrometheusRule, PROMETHEUS reporter and port set to 9999
```shell
helm template kyuubi charts/kyuubi --set metrics.enabled=true \
                                   --set metrics.reporters="PROMETHEUS\, JMX" \
                                   --set metrics.prometheusPort=9999 \
                                   --set metrics.podMonitor.enabled=true \
                                   --set metrics.serviceMonitor.enabled=true \
                                   --set metrics.prometheusRule.enabled=true \
                                   -s templates/kyuubi-statefulset.yaml \
                                   -s templates/kyuubi-headless-service.yaml \
                                   -s templates/kyuubi-configmap.yaml \
                                   -s templates/kyuubi-podmonitor.yaml \
                                   -s templates/kyuubi-servicemonitor.yaml \
                                   -s templates/kyuubi-alert.yaml
```

#### Install the chart and test Prometheus endpoint
```shell
helm install kyuubi charts/kyuubi --set metrics.enabled=true \
                                  --set metrics.reporters="PROMETHEUS\, JMX" \
                                  --set metrics.prometheusPort=9999
...
kyuubikyuubi-0:/opt/kyuubi$ curl 127.0.0.1:9999/metrics
# HELP kyuubi_buffer_pool_mapped_count Generated from Dropwizard metric import (metric=kyuubi.buffer_pool.mapped.count, type=com.codahale.metrics.jvm.JmxAttributeGauge)
# TYPE kyuubi_buffer_pool_mapped_count gauge
kyuubi_buffer_pool_mapped_count 0.0
# HELP kyuubi_gc_MarkSweepCompact_time Generated from Dropwizard metric import (metric=kyuubi.gc.MarkSweepCompact.time, type=com.codahale.metrics.jvm.GarbageCollectorMetricSet$$Lambda$227/1493158871)
# TYPE kyuubi_gc_MarkSweepCompact_time gauge
kyuubi_gc_MarkSweepCompact_time 91.0
...
```
---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6580 from dnskr/helm-improve-monitoring-configuration.

Closes #6580

1f20cabef [dnskr] [K8S][HELM] Improve metrics configuration

Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: dnskr <dnskrv88@gmail.com>
This commit is contained in:
dnskr 2024-08-06 14:54:09 +02:00
parent 35941ed375
commit 44fcf28720
7 changed files with 44 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
charts/kyuubi/templates/kyuubi-alert.yaml
View File

@ -15,7 +15,7 @@
limitations under the License. limitations under the License.
*/}} */}}
{{- if and .Values.monitoring.prometheus.enabled (eq .Values.metricsReporters "PROMETHEUS") .Values.prometheusRule.enabled }} {{- if and .Values.metrics.enabled (.Values.metrics.reporters | nospace | splitList "," | has "PROMETHEUS") .Values.metrics.prometheusRule.enabled }}
apiVersion: monitoring.coreos.com/v1 apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule kind: PrometheusRule
metadata: metadata:
@ -24,5 +24,5 @@ metadata:
{{- include "kyuubi.labels" . | nindent 4 }} {{- include "kyuubi.labels" . | nindent 4 }}
spec: spec:
groups: groups:
{{- toYaml .Values.prometheusRule.groups | nindent 4 }} {{- toYaml .Values.metrics.prometheusRule.groups | nindent 4 }}
{{- end }} {{- end }}

5
charts/kyuubi/templates/kyuubi-configmap.yaml
View File

@ -37,8 +37,9 @@ data:
kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }} kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
# Kyuubi Metrics # Kyuubi Metrics
kyuubi.metrics.enabled={{ .Values.monitoring.prometheus.enabled }} kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
kyuubi.metrics.reporters={{ .Values.metricsReporters }} kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
## User provided Kyuubi configurations ## User provided Kyuubi configurations
{{- with .Values.kyuubiConf.kyuubiDefaults }} {{- with .Values.kyuubiConf.kyuubiDefaults }}

7
charts/kyuubi/templates/kyuubi-headless-service.yaml
View File

@ -30,11 +30,10 @@ spec:
port: {{ tpl $frontend.service.port $ }} port: {{ tpl $frontend.service.port $ }}
targetPort: {{ $frontend.port }} targetPort: {{ $frontend.port }}
{{- end }} {{- end }}
{{- if .Values.monitoring.prometheus.enabled }} {{- if and .Values.metrics.enabled (.Values.metrics.reporters | nospace | splitList "," | has "PROMETHEUS") }}
- name: prometheus - name: prometheus
port: {{ .Values.monitoring.prometheus.port }} port: {{ .Values.metrics.prometheusPort }}
targetPort: {{ .Values.monitoring.prometheus.port }} targetPort: prometheus
{{- end }} {{- end }}
selector: selector:
{{- include "kyuubi.selectorLabels" $ | nindent 4 }} {{- include "kyuubi.selectorLabels" $ | nindent 4 }}

4
charts/kyuubi/templates/kyuubi-podmonitor.yaml
View File

@ -15,7 +15,7 @@
limitations under the License. limitations under the License.
*/}} */}}
{{- if and .Values.monitoring.prometheus.enabled (eq .Values.metricsReporters "PROMETHEUS") .Values.podMonitor.enabled }} {{- if and .Values.metrics.enabled (.Values.metrics.reporters | nospace | splitList "," | has "PROMETHEUS") .Values.metrics.podMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1 apiVersion: monitoring.coreos.com/v1
kind: PodMonitor kind: PodMonitor
metadata: metadata:
@ -27,5 +27,5 @@ spec:
matchLabels: matchLabels:
app: {{ .Release.Name }} app: {{ .Release.Name }}
podMetricsEndpoints: podMetricsEndpoints:
{{- toYaml .Values.podMonitor.podMetricsEndpoint | nindent 4 }} {{- toYaml .Values.metrics.podMonitor.podMetricsEndpoints | nindent 4 }}
{{- end }} {{- end }}

8
charts/kyuubi/templates/kyuubi-servicemonitor.yaml
View File

@ -15,20 +15,20 @@
limitations under the License. limitations under the License.
*/}} */}}
{{- if and .Values.monitoring.prometheus.enabled (eq .Values.metricsReporters "PROMETHEUS") .Values.serviceMonitor.enabled }} {{- if and .Values.metrics.enabled (.Values.metrics.reporters | nospace | splitList "," | has "PROMETHEUS") .Values.metrics.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1 apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor kind: ServiceMonitor
metadata: metadata:
name: {{ .Release.Name }} name: {{ .Release.Name }}
labels: labels:
{{- include "kyuubi.labels" . | nindent 4 }} {{- include "kyuubi.labels" . | nindent 4 }}
{{- if .Values.serviceMonitor.labels }} {{- if .Values.metrics.serviceMonitor.labels }}
{{- toYaml .Values.serviceMonitor.labels | nindent 4 }} {{- toYaml .Values.metrics.serviceMonitor.labels | nindent 4 }}
{{- end }} {{- end }}
spec: spec:
selector: selector:
matchLabels: matchLabels:
{{- include "kyuubi.selectorLabels" . | nindent 6 }} {{- include "kyuubi.selectorLabels" . | nindent 6 }}
endpoints: endpoints:
{{- toYaml .Values.serviceMonitor.endpoints | nindent 4 }} {{- toYaml .Values.metrics.serviceMonitor.endpoints | nindent 4 }}
{{- end }} {{- end }}

4
charts/kyuubi/templates/kyuubi-statefulset.yaml
View File

@ -80,9 +80,9 @@ spec:
containerPort: {{ $frontend.port }} containerPort: {{ $frontend.port }}
{{- end }} {{- end }}
{{- end }} {{- end }}
{{- if .Values.monitoring.prometheus.enabled }} {{- if and .Values.metrics.enabled (.Values.metrics.reporters | nospace | splitList "," | has "PROMETHEUS") }}
- name: prometheus - name: prometheus
containerPort: {{ .Values.monitoring.prometheus.port }} containerPort: {{ .Values.metrics.prometheusPort }}
{{- end }} {{- end }}
{{- if .Values.livenessProbe.enabled }} {{- if .Values.livenessProbe.enabled }}
livenessProbe: livenessProbe:

47
charts/kyuubi/values.yaml
View File

@ -145,12 +145,6 @@ server:
# clientIP: # clientIP:
# timeoutSeconds: 10800 # timeoutSeconds: 10800
monitoring:
# Exposes metrics in Prometheus format
prometheus:
enabled: true
port: 10019
# $KYUUBI_CONF_DIR directory # $KYUUBI_CONF_DIR directory
kyuubiConfDir: /opt/kyuubi/conf kyuubiConfDir: /opt/kyuubi/conf
# Kyuubi configuration files # Kyuubi configuration files
@ -279,31 +273,34 @@ affinity: {}
# Kyuubi pods security context # Kyuubi pods security context
securityContext: {} securityContext: {}
# Monitoring Kyuubi - Server Metrics # Metrics configuration
# PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format metrics:
metricsReporters: ~ # Enable metrics system, used for 'kyuubi.metrics.enabled' property
enabled: true
# A comma-separated list of metrics reporters, used for 'kyuubi.metrics.reporters' property
reporters: PROMETHEUS
# Prometheus port, used for 'kyuubi.metrics.prometheus.port' property
prometheusPort: 10019
# Prometheus pod monitor # PodMonitor by Prometheus Operator
podMonitor: podMonitor:
# If enabled, podMonitor for operator's pod will be created # Enable PodMonitor creation
enabled: false enabled: false
# The podMetricsEndpoint contains metrics information such as port, interval, scheme, and possibly other relevant details. # List of pod endpoints serving metrics to be scraped by Prometheus, see Prometheus Operator docs for more details
# This information is used to configure the endpoint from which Prometheus can scrape and collect metrics for a specific Pod in Kubernetes. podMetricsEndpoints: []
podMetricsEndpoint: []
# Prometheus service monitor # ServiceMonitor by Prometheus Operator
serviceMonitor: serviceMonitor:
# If enabled, ServiceMonitor resources for Prometheus Operator are created # Enable ServiceMonitor creation
enabled: false enabled: false
# The endpoints section in a ServiceMonitor specifies the metrics information for each target endpoint. # List of service endpoints serving metrics to be scraped by Prometheus, see Prometheus Operator docs for more details
# This allows you to collect metrics from multiple Services across your Kubernetes cluster in a standardized and automated way.
endpoints: [] endpoints: []
# Additional labels that can be used so ServiceMonitor will be discovered by Prometheus # Additional labels to be used to make ServiceMonitor discovered by Prometheus
labels: {} labels: {}
# Rules for the Prometheus Operator # PrometheusRule by Prometheus Operator
prometheusRule: prometheusRule:
# If enabled, a PrometheusRule resource for Prometheus Operator is created # Enable PrometheusRule creation
enabled: false enabled: false
# Contents of Prometheus rules file # Content of Prometheus rule file
groups: [] groups: []