Merge pull request #4758 from JoshVanL/design-server-side-apply
Design Server Side Apply
This commit is contained in:
jetstack-bot
GitHub
commit
0c454ea72e
242
design/20220118.server-side-apply.md
Normal file
242
design/20220118.server-side-apply.md
Normal file
@ -0,0 +1,242 @@
|
||||
---
|
||||
title: Server-Side Apply
|
||||
authors:
|
||||
- "@joshvanl"
|
||||
reviewers:
|
||||
- @jetstack/team-cert-manager
|
||||
approvers:
|
||||
- @jetstack/team-cert-manager
|
||||
editor: "@joshvanl"
|
||||
creation-date: 2022-01-18
|
||||
last-updated: 2022-01-18
|
||||
status: implementable
|
||||
---
|
||||
|
||||
# Server-Side Apply
|
||||
|
||||
## Table of Contents
|
||||
|
||||
<!-- toc -->
|
||||
- [Summary](#summary)
|
||||
* [Conflicts](#conflicts)
|
||||
* [Deleting Fields](#deleting-fields)
|
||||
- [Field Manager and User Agent](#field-manager-and-user-agent)
|
||||
+ [Known Field Managers](#known-field-managers)
|
||||
* [Scheme](#scheme)
|
||||
* [Field Manager](#field-manager)
|
||||
* [User Agent](#user-agent)
|
||||
- [Implementation Consideration](#implementation-consideration)
|
||||
* [API Call Code Changes](#api-call-code-changes)
|
||||
* [Force parameter](#force-parameter)
|
||||
* [client-go Testing](#client-go-testing)
|
||||
* [API Changes](#api-changes)
|
||||
- [Feature Gate](#feature-gate)
|
||||
- [Migration](#migration)
|
||||
<!-- /toc -->
|
||||
|
||||
## Summary
|
||||
|
||||
Server-Side Apply is a [Kubernetes
|
||||
feature](https://kubernetes.io/docs/reference/using-api/server-side-apply/)
|
||||
whereby clients writing to a resource that is managed by more than one client
|
||||
can
|
||||
|
||||
- declare what fields that client manages, and
|
||||
- make decisions on what to do if there is a conflict with another client on
|
||||
what a field value should be.
|
||||
|
||||
Server-Side Apply works by the client sending a PATCH API request with a
|
||||
`Content-Type` header with the value `application/apply-patch+yaml`. The
|
||||
`fieldManager=<my-field-manager>` URL query can be optionally sent which
|
||||
instructs which field manager to use (or alternatively, will be derived from the
|
||||
client's user agent). This API call tells the API server that the client wishes
|
||||
to manage and set the fields and values of the resource (or alternatively remove
|
||||
managed fields which are omitted) as they appear in the body. A client's managed
|
||||
fields are defined as the fields that are labelled with the client's manager
|
||||
name `<my-field-manager>` (or client's user agent equivalent).
|
||||
|
||||
Server-Side Apply is is useful for cert-manager for 2 reasons:
|
||||
|
||||
### Conflicts
|
||||
|
||||
cert-manager controllers, namely the certificates controllers, have multiple
|
||||
controllers that are writing to the same resource type. This leads to cases
|
||||
where UPDATE operations result in resource version conflicts. This in turn,
|
||||
results in error logs that regularly confuse users and miss-attribute the
|
||||
problem they are having, as well as needless re-queuing churn through the
|
||||
controller, slowing cert-manager down.
|
||||
|
||||
### Deleting Fields
|
||||
|
||||
Without Server-Side Apply, it is difficult for cert-manager to reason about some
|
||||
fields that should be _deleted_ on an UPDATE operation during reconciliation.
|
||||
One such example are Certificate SecretTemplate's Annotations and Labels. If a
|
||||
key was removed from either the Annotation or Label field on the SecretTemplate,
|
||||
the next reconciliation would not know which keys should be preserved or removed
|
||||
(as they may have been annotated or labelled from a third party). This problem
|
||||
occurs in several places where copying or transforming state from one resource
|
||||
to another happens.
|
||||
|
||||
Managed fields define ownership of applied fields. cert-manager is therefore
|
||||
able to observe a discrepancy occurring from, for example, a previous
|
||||
SecretTemplate state and performing another apply (omitting those keys),
|
||||
resulting in those fields being removed on the Secret. Without managed fields,
|
||||
cert-manager would have to create a third state store in either annotations or
|
||||
status fields on that resource.
|
||||
|
||||
## Field Manager and User Agent
|
||||
|
||||
The Field Manager is a string denoting the manager who owns a field or set of
|
||||
fields on a resource. This string is set by the client at API call time and is
|
||||
derived by either the client setting the `fieldManager` URL query explicitly, or
|
||||
calculated from the client's user agent ([characters preceding the first "/",
|
||||
quote unprintable characters and then trim what's beyond the 128
|
||||
limit](https://github.com/kubernetes/kubernetes/blob/9a75e7b0fd1b567f774a3373be640e19b33e7ef1/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go#L252)).
|
||||
|
||||
cert-manager will use a consistent naming scheme for both the user agent prefix
|
||||
and field manager across all components. Each component will have a human
|
||||
readable name that describes that component, and is used for both its user agent
|
||||
and field manager. Using the same name enables better auditing trails that helps
|
||||
debugging and improves telemetry.
|
||||
|
||||
#### Known Field Managers
|
||||
|
||||
This is the list of known field managers. This might not be an exhaustive list
|
||||
as they will be discovered during the changes.
|
||||
|
||||
```text
|
||||
cert-manager # base field manager of cert-manager controller, used when no distinct component
|
||||
cert-manager-leader-election
|
||||
cert-manager-acme-[orders,challenges]
|
||||
cert-manager-certificates-[issuing,trigger,keymanager,readiness]
|
||||
cert-manager-certificaterequests-[acme,approver,ca,selfsigned,vault,venafi]
|
||||
cert-manager-clusterissuers-[acme,ca,selfsigned,vault,venafi]
|
||||
cert-manager-issuers-[acme,ca,selfsigned,vault,venafi]
|
||||
cert-manager-cainjector # base field manager of cert-manager ca-injector
|
||||
cert-manager-webhook # base field manager of cert-manager webhook
|
||||
cert-manager-cmctl
|
||||
```
|
||||
|
||||
|
||||
### Scheme
|
||||
|
||||
### Field Manager
|
||||
|
||||
The field manager is derived from the component in cert-manager, prefixed by the
|
||||
string `cert-manager`:
|
||||
|
||||
```text
|
||||
cert-manager-<cert-manager component>
|
||||
```
|
||||
|
||||
e.g.
|
||||
|
||||
```text
|
||||
cert-manager-certificates-issuing
|
||||
```
|
||||
|
||||
### User Agent
|
||||
|
||||
The user agent is inspired by the [client-go default user
|
||||
agent](https://github.com/kubernetes/client-go/blob/664b1a6c8ce9c92ce65bef3f9833b402449c98d2/rest/config.go#L499),
|
||||
and is defined with the following:
|
||||
|
||||
```text
|
||||
<component field manager>/<cert-manager version> (<Operating System>/<Architecture>) cert-manager/<git commit>
|
||||
```
|
||||
|
||||
e.g:
|
||||
|
||||
```text
|
||||
cert-manager-certificates-issuing/v1.7.0 (Linux/amd64) cert-manager/0b686b8f38c8c7442744c9224d18e780ee7f244a
|
||||
```
|
||||
|
||||
## Implementation Consideration
|
||||
|
||||
### API Call Code Changes
|
||||
|
||||
All CREATE and UPDATE operations in cert-manager controllers are to be replaced
|
||||
with Apply. For this change, controllers will need to be modified so that for
|
||||
these calls, _only_ the fields in which they manage should be included in the
|
||||
PATCH API call. This means each controller will need to define exactly which
|
||||
fields they are concerned with. A package will be created which all API calls
|
||||
will use. This package will implement the logic for checking the feature gate
|
||||
for consistency across the project whilst minimising disruption to existing code
|
||||
paths.
|
||||
|
||||
### Force parameter
|
||||
|
||||
Some fields, such as the Certificate Issuing Condition are managed by more than
|
||||
one controller (issuing and trigger Certificate controllers, and cmctl), and as
|
||||
such, will need to make use of the `force` parameter in their API calls. This
|
||||
option tells the API server to revoke management of that field from the previous
|
||||
owner, overwrite the field, and change owner ship to the new client. Since some
|
||||
fields, such as the Issuing Condition, may have an undefined number of potential
|
||||
managers (both internal and external to the cert-manager controller), using the
|
||||
same manager for things is not a possibility. You can read more about the
|
||||
`force` paramerter on the Kubernetes documentation on
|
||||
[Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/),
|
||||
and in particular the
|
||||
[Conflicts](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts)
|
||||
section.
|
||||
|
||||
Use of the `force` parameter will not overwrite other controllers progress
|
||||
since, 1 controllers only update the fields that they manage, 2 controllers who
|
||||
manage the same fields should never be writing to the same field at the same
|
||||
time (as they should be gated by those or other fields presence and values, such
|
||||
as the Issuing condition for the issuing and trigger controllers).
|
||||
|
||||
There is likely no Apply call that cert-manager does not set the `force`
|
||||
parameter to true since it, [never wants to give up ownership claim, and always
|
||||
wants to overwrite
|
||||
values](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts).
|
||||
See
|
||||
[here](https://kubernetes.io/docs/reference/using-api/server-side-apply/#using-server-side-apply-in-a-controller).
|
||||
|
||||
### client-go Testing
|
||||
|
||||
The [fake client-go client](https://github.com/kubernetes/client-go/issues/970)
|
||||
does not support the Apply PATCH call for mocking API calls and events. This
|
||||
means that significant controller unit-testing will either need to moved to
|
||||
testing against a real API server as integration tests, the controller
|
||||
[test framework must add custom support for Apply](https://github.com/jetstack/cert-manager/blob/master/pkg/controller/test/context_builder.go),
|
||||
or a new testing framework should be developed. We can also PR this upstream but
|
||||
will take time to be released so a stop gap would always be needed.
|
||||
|
||||
### API Changes
|
||||
|
||||
Some API fields will need to have some metadata updated to function better under
|
||||
Server-Side Apply. One such example is adding `x-kubernetes-list-type=map` and
|
||||
`x-kubernetes-list-map-keys=Type` to the [Certificates Status Condition
|
||||
slice](https://github.com/jetstack/cert-manager/blob/0ca1ce9a6a1d7c311afd4b3e786975759249132a/pkg/apis/certmanager/v1/types_certificate.go#L385),
|
||||
so that controllers are able to apply distinct condition types, without
|
||||
conflicting with other controller conditions (i.e. the Ready and Issuing
|
||||
conditions). Integration tests will be able to ensure cert-manager have set
|
||||
these API type tags correctly.
|
||||
|
||||
## Feature Gate
|
||||
|
||||
Placing the Apply functionality behind a feature gate should be required.
|
||||
Placing this functionality behind a feature gate would allow the cert-manager
|
||||
authors gain confidence about its correctness, and ensure there are no
|
||||
regressions in the stability of controller reconciliation.
|
||||
|
||||
To reach graduation, the cert-manager authors should consider the Server-Side
|
||||
Apply implementation for cert-manager to be safe for end users. One strategy for
|
||||
gaining confidence that it is a safe change is to solicit feedback from end
|
||||
users who have the feature enabled. Graduation should ideally be done over no
|
||||
more than one or two releases, however the project shouldn't proceed with
|
||||
removing the gate until the authors are confident.
|
||||
|
||||
## Migration
|
||||
|
||||
For an exiting cluster with cert-manager and resources installed, a user should
|
||||
be able to safely enable the feature or upgrade cert-manager with the feature
|
||||
enabled by default without any down time or loss of data.
|
||||
|
||||
Although testing is needed, it _should_ be the case that an upgrade/feature
|
||||
enable that all controllers on the next reconcile that applies, will take
|
||||
ownership of that component's fields and set the correct values. Fields will not
|
||||
be deleted that should be however (i.e. Annotation or Label from a previous
|
||||
SecretTemplate) as the field manager has changed (though this is the same
|
||||
behaviour as today).
|
||||
Loading…
Reference in New Issue
Block a user