---
title: Server-Side Apply
authors:
  - "@joshvanl"
reviewers:
  - @jetstack/team-cert-manager
approvers:
  - @jetstack/team-cert-manager
editor: "@joshvanl"
creation-date: 2022-01-18
last-updated: 2022-01-18
status: implementable
---

# Server-Side Apply

## Table of Contents

<!-- toc -->
- [Summary](#summary)
  * [Conflicts](#conflicts)
  * [Deleting Fields](#deleting-fields)
- [Field Manager and User Agent](#field-manager-and-user-agent)
    + [Known Field Managers](#known-field-managers)
  * [Scheme](#scheme)
  * [Field Manager](#field-manager)
  * [User Agent](#user-agent)
- [Implementation Consideration](#implementation-consideration)
  * [API Call Code Changes](#api-call-code-changes)
  * [Force parameter](#force-parameter)
  * [client-go Testing](#client-go-testing)
  * [API Changes](#api-changes)
- [Feature Gate](#feature-gate)
- [Migration](#migration)
<!-- /toc -->

## Summary

Server-Side Apply is a [Kubernetes
feature](https://kubernetes.io/docs/reference/using-api/server-side-apply/)
whereby clients writing to a resource that is managed by more than one client
can

- declare what fields that client manages, and
- make decisions on what to do if there is a conflict with another client on
  what a field value should be.

Server-Side Apply works by the client sending a PATCH API request with a
`Content-Type` header with the value `application/apply-patch+yaml`. The
`fieldManager=<my-field-manager>` URL query can be optionally sent which
instructs which field manager to use (or alternatively, will be derived from the
client's user agent). This API call tells the API server that the client wishes
to manage and set the fields and values of the resource (or alternatively remove
managed fields which are omitted) as they appear in the body. A client's managed
fields are defined as the fields that are labelled with the client's manager
name `<my-field-manager>` (or client's user agent equivalent).

Server-Side Apply is useful for cert-manager for 2 reasons:

### Conflicts

cert-manager controllers, namely the certificates controllers, have multiple
controllers that are writing to the same resource type. This leads to cases
where UPDATE operations result in resource version conflicts. This in turn,
results in error logs that regularly confuse users and miss-attribute the
problem they are having, as well as needless re-queuing churn through the
controller, slowing cert-manager down.

### Deleting Fields

Without Server-Side Apply, it is difficult for cert-manager to reason about some
fields that should be _deleted_ on an UPDATE operation during reconciliation.
One such example are Certificate SecretTemplate's Annotations and Labels. If a
key was removed from either the Annotation or Label field on the SecretTemplate,
the next reconciliation would not know which keys should be preserved or removed
(as they may have been annotated or labelled from a third party). This problem
occurs in several places where copying or transforming state from one resource
to another happens.

Managed fields define ownership of applied fields. cert-manager is therefore
able to observe a discrepancy occurring from, for example, a previous
SecretTemplate state and performing another apply (omitting those keys),
resulting in those fields being removed on the Secret. Without managed fields,
cert-manager would have to create a third state store in either annotations or
status fields on that resource.

## Field Manager and User Agent

The Field Manager is a string denoting the manager who owns a field or set of
fields on a resource. This string is set by the client at API call time and is
derived by either the client setting the `fieldManager` URL query explicitly, or
calculated from the client's user agent ([characters preceding the first "/",
quote unprintable characters and then trim what's beyond the 128
limit](https://github.com/kubernetes/kubernetes/blob/9a75e7b0fd1b567f774a3373be640e19b33e7ef1/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go#L252)).

cert-manager will use a consistent naming scheme for both the user agent prefix
and field manager across all components. Each component will have a human
readable name that describes that component, and is used for both its user agent
and field manager. Using the same name enables better auditing trails that helps
debugging and improves telemetry.

#### Known Field Managers

This is the list of known field managers. This might not be an exhaustive list
as they will be discovered during the changes.

```text
cert-manager # base field manager of cert-manager controller, used when no distinct component
cert-manager-leader-election
cert-manager-acme-[orders,challenges]
cert-manager-certificates-[issuing,trigger,keymanager,readiness]
cert-manager-certificaterequests-[acme,approver,ca,selfsigned,vault,venafi]
cert-manager-clusterissuers-[acme,ca,selfsigned,vault,venafi]
cert-manager-issuers-[acme,ca,selfsigned,vault,venafi]
cert-manager-cainjector # base field manager of cert-manager cainjector
cert-manager-webhook # base field manager of cert-manager webhook
cert-manager-cmctl
```


### Scheme

### Field Manager

The field manager is derived from the component in cert-manager, prefixed by the
string `cert-manager`:

```text
cert-manager-<cert-manager component>
```

e.g.

```text
cert-manager-certificates-issuing
```

### User Agent

The user agent is inspired by the [client-go default user
agent](https://github.com/kubernetes/client-go/blob/664b1a6c8ce9c92ce65bef3f9833b402449c98d2/rest/config.go#L499),
and is defined with the following:

```text
<component field manager>/<cert-manager version> (<Operating System>/<Architecture>) cert-manager/<git commit>
```

e.g:

```text
cert-manager-certificates-issuing/v1.7.0 (Linux/amd64) cert-manager/0b686b8f38c8c7442744c9224d18e780ee7f244a
```

## Implementation Consideration

### API Call Code Changes

All CREATE and UPDATE operations in cert-manager controllers are to be replaced
with Apply. For this change, controllers will need to be modified so that for
these calls, _only_ the fields in which they manage should be included in the
PATCH API call. This means each controller will need to define exactly which
fields they are concerned with. A package will be created which all API calls
will use. This package will implement the logic for checking the feature gate
for consistency across the project whilst minimising disruption to existing code
paths.

### Force parameter

Some fields, such as the Certificate Issuing Condition are managed by more than
one controller (issuing and trigger Certificate controllers, and cmctl), and as
such, will need to make use of the `force` parameter in their API calls. This
option tells the API server to revoke management of that field from the previous
owner, overwrite the field, and change owner ship to the new client. Since some
fields, such as the Issuing Condition, may have an undefined number of potential
managers (both internal and external to the cert-manager controller), using the
same manager for things is not a possibility. You can read more about the
`force` paramerter on the Kubernetes documentation on
[Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/),
and in particular the
[Conflicts](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts)
section.

Use of the `force` parameter will not overwrite other controllers progress
since, 1 controllers only update the fields that they manage, 2 controllers who
manage the same fields should never be writing to the same field at the same
time (as they should be gated by those or other fields presence and values, such
as the Issuing condition for the issuing and trigger controllers).

There is likely no Apply call that cert-manager does not set the `force`
parameter to true since it, [never wants to give up ownership claim, and always
wants to overwrite
values](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts).
See
[here](https://kubernetes.io/docs/reference/using-api/server-side-apply/#using-server-side-apply-in-a-controller).

### client-go Testing

The [fake client-go client](https://github.com/kubernetes/client-go/issues/970)
does not support the Apply PATCH call for mocking API calls and events. This
means that significant controller unit-testing will either need to moved to
testing against a real API server as integration tests, the controller
[test framework must add custom support for Apply](https://github.com/jetstack/cert-manager/blob/master/pkg/controller/test/context_builder.go),
or a new testing framework should be developed. We can also PR this upstream but
will take time to be released so a stop gap would always be needed.

### API Changes

Some API fields will need to have some metadata updated to function better under
Server-Side Apply. One such example is adding `x-kubernetes-list-type=map` and
`x-kubernetes-list-map-keys=Type` to the [Certificates Status Condition
slice](https://github.com/jetstack/cert-manager/blob/0ca1ce9a6a1d7c311afd4b3e786975759249132a/pkg/apis/certmanager/v1/types_certificate.go#L385),
so that controllers are able to apply distinct condition types, without
conflicting with other controller conditions (i.e. the Ready and Issuing
conditions). Integration tests will be able to ensure cert-manager have set
these API type tags correctly.

## Feature Gate

Placing the Apply functionality behind a feature gate should be required.
Placing this functionality behind a feature gate would allow the cert-manager
authors gain confidence about its correctness, and ensure there are no
regressions in the stability of controller reconciliation.

To reach graduation, the cert-manager authors should consider the Server-Side
Apply implementation for cert-manager to be safe for end users. One strategy for
gaining confidence that it is a safe change is to solicit feedback from end
users who have the feature enabled. Graduation should ideally be done over no
more than one or two releases, however the project shouldn't proceed with
removing the gate until the authors are confident.

## Migration

For an exiting cluster with cert-manager and resources installed, a user should
be able to safely enable the feature or upgrade cert-manager with the feature
enabled by default without any down time or loss of data.

Although testing is needed, it _should_ be the case that an upgrade/feature
enable that all controllers on the next reconcile that applies, will take
ownership of that component's fields and set the correct values. Fields will not
be deleted that should be however (i.e. Annotation or Label from a previous
SecretTemplate) as the field manager has changed (though this is the same
behaviour as today).
