Every CNCF project written in Go (Argo CD, cert-manager, Karpenter, KEDA, Cilium's operator, Flux) is a set of controllers built from the same pieces: client-go, informers, work queues and a reconcile function. LFX Mentorship and GSoC tasks on these projects assume you can read that code. In platform engineering interviews, "write an operator" or "what happens when two controllers update the same object" is the question that separates people who have run Kubernetes from people who can extend it. This lesson gives you the vocabulary and one complete, working operator.

The API server: groups, versions, kinds and resources

The API server is an HTTP server with a database behind it (etcd). Everything it stores is addressed by four names. A group is a family of APIs (apps, batch, networking.k8s.io, or your own webapp.example.com); the original core group has no name and lives under /api/v1. A version (v1alpha1, v1beta1, v1) says how stable the schema is. A kind is the Go type of one object (Deployment). A resource is the plural, lower-case name in the URL (deployments). The pair of group, version and resource is called a GVR, and group, version and kind is a GVK; kubectl api-resources lists them all.

Two more endpoints matter. Discovery (/apis and /apis/apps/v1) tells clients which groups, versions and resources exist; kubectl reads it to turn kubectl get deploy into the right URL. OpenAPI (/openapi/v3) publishes the schema of every type, used by kubectl explain and client-side validation. The aggregation layer mounts a separate API server (metrics-server is the common one) under /apis/metrics.k8s.io, so one endpoint serves both.

An object on the wire

Every object has the same shape. apiVersion and kind identify the type. metadata carries the name, namespace, labels, annotations and the fields the system owns. spec is what you want; status is what the controller observed. The fields in metadata that a controller author must know:

• resourceVersion: an opaque string that changes on every write. Clients must pass it back unmodified; it is the key to concurrency control and to watches.
• generation: an integer the server increments only when spec changes. A controller copies it into status.observedGeneration so a reader can tell whether the status is about the current spec.
• uid: unique for the life of the object; a new object with the same name gets a new uid.
• ownerReferences: the list of objects that own this one, with controller: true on the one managing it. When the owner is deleted, the garbage collector deletes the children.
• finalizers: a list of strings. While it is non-empty, a delete only sets deletionTimestamp; the object stays until every controller removes its own entry.

Optimistic concurrency, watches and admission

The API server has no locks. An update carries the resourceVersion the client last read; if the stored object has moved on, the server answers 409 Conflict and the client must read again, re-apply its change and retry. retry.RetryOnConflict(retry.DefaultRetry, fn) in client-go does exactly that, with 5 tries 10 ms apart. Two controllers can therefore write the same object safely as long as both retry.

A watch is a long HTTP request (?watch=1&resourceVersion=10245) that streams ADDED, MODIFIED and DELETED events after that version. etcd keeps about 5 minutes of history, so a client that reconnects too late gets 410 Gone and must list again and watch from the new version. With allowWatchBookmarks=true the server sends occasional BOOKMARK events carrying only a resourceVersion, so the client can restart from a fresh point. This list-then-watch cycle is what an informer does for you.

Before a write reaches etcd it passes the admission chain: mutating admission (built-in plugins, then mutating webhooks, which may change the object), schema validation, then validating admission (built-in plugins, ValidatingAdmissionPolicy written in CEL, and validating webhooks). An admission webhook is an HTTPS server you run; the API server calls it with an AdmissionReview and waits for allow or deny.

The controller pattern, precisely

A controller is a loop that moves the actual state toward the desired state. The rules below are the ones the Kubernetes contributor guide states for every controller in the tree.

• Level-based, not edge-based. The reconcile function never looks at the event ("replicas went from 2 to 3"). It reads the current state ("the WebApp says 3, the Deployment has 2") and fixes the difference. If the controller was down for an hour, it catches up on restart because the level is still there; an edge would have been missed.
• One key at a time. Events are turned into keys (namespace/name) on a work queue. The queue dedupes, so 50 events for one object become one reconcile, and it guarantees no two workers handle the same key at once.
• Idempotent. Running reconcile twice must produce the same result as running it once. Create only what is missing; update only what differs.
• Requeue with backoff. Return an error and the key goes back on the queue through AddRateLimited: the default controller rate limiter starts at 5 ms per item, doubles on each failure up to 1000 s, and also caps the whole controller at 10 retries per second with a burst of 100. Return success and the failure count is forgotten.
• Report through status. Set status.conditions (type, status, reason, message, lastTransitionTime, observedGeneration) rather than logging. Ready=True with observedGeneration equal to metadata.generation is how a user or Argo CD knows the object is done.
• Own what you create. Set an owner reference on every child. Garbage collection then deletes the Deployment when the WebApp goes, and the informer on Deployments can map a child event back to the owner key.
• Finalizers for external cleanup. If the controller created something outside the cluster (an S3 bucket, a DNS record), add a finalizer, delete the external thing when deletionTimestamp is set, then remove the finalizer.
• Leader election. Run two or three replicas but let only one reconcile, by holding a Lease object in the cluster. controller-runtime defaults are a 15 s lease, a 10 s renew deadline and a 2 s retry period.

The informer also resyncs: periodically it replays every cached object through the update handler (controller-runtime defaults to 10 hours), so a reconcile that silently failed to requeue still gets another chance.

The controller loop: informer and cache on the left feeding a work queue and Reconcile, the API server and a WebApp custom resource owning a Deployment and a Service on the right, with resourceVersion, requeue and 409 Conflict marked

CustomResourceDefinitions

A CRD adds a new resource to the API server without any code. The server stores, validates and serves it exactly like a built-in type. The schema is OpenAPI v3; x-kubernetes-validations adds CEL rules such as self.replicas <= 10. A CRD may serve several versions; exactly one has storage: true, and conversion between versions is either strategy: None (same fields, only apiVersion changes) or strategy: Webhook (your code converts). Two subresources matter: status, which makes /status a separate endpoint so a controller can update status without touching spec and users cannot change status through a normal update, and scale, which lets kubectl scale and the HorizontalPodAutoscaler work on your type by pointing at .spec.replicas and .status.replicas.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.webapp.example.com
spec:
  group: webapp.example.com
  names: {kind: WebApp, plural: webapps, singular: webapp, shortNames: [wa]}
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
        scale: {specReplicasPath: .spec.replicas, statusReplicasPath: .status.replicas}
      additionalPrinterColumns:     # what kubectl get webapps shows
        - {name: Replicas, type: integer, jsonPath: .spec.replicas}
        - {name: Ready, type: integer, jsonPath: .status.readyReplicas}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [image]
              properties:
                image: {type: string}
                replicas: {type: integer, minimum: 1, maximum: 50, default: 1}
                port: {type: integer, default: 8080}
            status:
              type: object
              properties:
                readyReplicas: {type: integer}
                observedGeneration: {type: integer}

With kubebuilder you never write this file; it is generated from Go struct tags and markers.

client-go: clientset, informers and listers

k8s.io/client-go is the Go library every controller uses. The clientset is the typed client: clientset.AppsV1().Deployments("shop").Get(ctx, "api", metav1.GetOptions{}) returns a *appsv1.Deployment. The dynamic client works on unstructured.Unstructured maps for types it does not know at compile time, which is what tools like Argo CD and kubectl use. Polling the API server from every controller would not scale, so client-go provides the informer: a reflector that lists then watches one resource, a local cache (the indexer), and event handlers. A lister is a read-only view of that cache, so Get and List in a reconcile never hit the network. The shared informer factory gives every controller in a process one informer per resource instead of one watch each. Cache reads can be slightly stale; that is fine, because a write from stale data fails with a 409 and the retry reads fresh data.

controller-runtime and kubebuilder

sigs.k8s.io/controller-runtime wraps client-go into three ideas. The Manager holds the shared cache, the client, leader election, metrics and health endpoints, and starts every controller. A Reconciler implements Reconcile(ctx, req) (Result, error). The builder wires watches: For(&WebApp{}) makes the controller reconcile WebApps; Owns(&Deployment{}) watches Deployments and enqueues the owner on any change; Watches(...) with a map function handles anything else. The client is one client.Client for every type, reading from the cache and writing to the server; r.Status().Update writes the status subresource.

kubebuilder scaffolds all of this and generates the boring parts with controller-gen from markers (comments that start with +kubebuilder:):

kubebuilder init --domain example.com --repo example.com/webapp-operator
kubebuilder create api --group webapp --version v1 --kind WebApp   # answer y to resource and controller
make manifests   # CRD YAML, RBAC ClusterRole and webhook config from markers -> config/
make generate    # zz_generated.deepcopy.go: DeepCopy methods every runtime.Object needs
make install     # kubectl apply the CRDs into the current cluster
make run         # run the manager on your laptop against the cluster in ~/.kube/config
make test        # envtest: a real kube-apiserver and etcd, no kubelet or controllers
make docker-build docker-push IMG=registry.example.com/webapp-operator:0.1.0
make deploy IMG=registry.example.com/webapp-operator:0.1.0

The generated tree is api/v1/webapp_types.go (the structs), internal/controller/webapp_controller.go (the reconciler), cmd/main.go (the manager), config/ (kustomize bases for CRD, RBAC, manager, samples). The Operator SDK is a framework from the Operator Framework project that builds on the same controller-runtime library and adds Helm and Ansible operator types plus packaging for OLM; a Go operator from either tool looks the same.

The WebApp operator, end to end

The types file. Markers drive the CRD above: validation, printer columns, the status subresource.

// needs: sigs.k8s.io/controller-runtime
package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type WebAppSpec struct {
	Image string `json:"image"`
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=50
	// +kubebuilder:default=1
	Replicas int32 `json:"replicas,omitempty"`
	// +kubebuilder:default=8080
	Port int32 `json:"port,omitempty"`
}

type WebAppStatus struct {
	ReadyReplicas      int32              `json:"readyReplicas,omitempty"`
	ObservedGeneration int64              `json:"observedGeneration,omitempty"`
	Conditions         []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.readyReplicas
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
type WebApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              WebAppSpec   `json:"spec,omitempty"`
	Status            WebAppStatus `json:"status,omitempty"`
}

The reconciler. Read the four numbered steps; everything else is plumbing.

// needs: sigs.k8s.io/controller-runtime
package controller

// +kubebuilder:rbac:groups=webapp.example.com,resources=webapps,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=webapp.example.com,resources=webapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 1. Read the object from the cache. Gone means deleted: nothing to do, GC removes the children.
	var app webappv1.WebApp
	if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Deployment: create it if missing, otherwise bring replicas and image in line.
	dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: app.Name, Namespace: app.Namespace}}
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
		labels := map[string]string{"app": app.Name}
		dep.Spec.Replicas = &app.Spec.Replicas
		dep.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
		dep.Spec.Template.ObjectMeta.Labels = labels
		dep.Spec.Template.Spec.Containers = []corev1.Container{{
			Name: "web", Image: app.Spec.Image,
			Ports: []corev1.ContainerPort{{ContainerPort: app.Spec.Port}},
		}}
		return controllerutil.SetControllerReference(&app, dep, r.Scheme) // ownerReference, controller=true
	})
	if err != nil {
		return ctrl.Result{}, err // a 409 Conflict lands here and is requeued with backoff
	}

	// 3. Service: same pattern. Owned, so deleting the WebApp deletes it too.
	svc := &corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: app.Name, Namespace: app.Namespace}}
	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, svc, func() error {
		svc.Spec.Selector = map[string]string{"app": app.Name}
		svc.Spec.Ports = []corev1.ServicePort{{Port: 80, TargetPort: intstr.FromInt32(app.Spec.Port)}}
		return controllerutil.SetControllerReference(&app, svc, r.Scheme)
	}); err != nil {
		return ctrl.Result{}, err
	}

	// 4. Status: report what the Deployment observed, through the status subresource.
	app.Status.ReadyReplicas = dep.Status.ReadyReplicas
	app.Status.ObservedGeneration = app.Generation
	ready := dep.Status.ReadyReplicas == app.Spec.Replicas
	meta.SetStatusCondition(&app.Status.Conditions, metav1.Condition{
		Type: "Ready", Status: boolToCondition(ready), Reason: "Reconciled",
		Message: fmt.Sprintf("%d/%d replicas ready", dep.Status.ReadyReplicas, app.Spec.Replicas),
		ObservedGeneration: app.Generation,
	})
	return ctrl.Result{}, r.Status().Update(ctx, &app)
}

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.WebApp{}).
		Owns(&appsv1.Deployment{}). // Deployment events enqueue the owning WebApp
		Owns(&corev1.Service{}).
		Complete(r)
}

There is no finalizer here because every child is in the cluster and owner references are enough. Add one only for resources outside Kubernetes.

Testing, packaging and admission webhooks

envtest starts a real kube-apiserver and etcd binary on your laptop (fetched by setup-envtest, pointed to by KUBEBUILDER_ASSETS), with no kubelet, scheduler or built-in controllers. Your test creates a WebApp, runs the manager, and asserts that a Deployment with the right replica count appears. Because no controllers run, dep.Status.ReadyReplicas stays 0 and garbage collection never happens; tests must set status by hand or not depend on it. For an end-to-end check, kind create cluster, make install, make run and kubectl apply -f config/samples/.

Packaging is an image (the scaffold's Dockerfile builds a distroless, non-root manager), the generated RBAC (a ClusterRole with exactly the verbs in the markers, a ServiceAccount, a binding), the CRDs, and a Deployment with --leader-elect. Teams ship this as a Helm chart with the CRDs in the chart's crds/ directory, where Helm installs them once and never upgrades them, which is why CRD changes need their own release step.

Admission webhooks in Go: kubebuilder create webhook --group webapp --version v1 --kind WebApp --defaulting --programmatic-validation scaffolds a CustomDefaulter (Default) and a CustomValidator (ValidateCreate, ValidateUpdate, ValidateDelete) registered through ctrl.NewWebhookManagedBy(mgr, &WebApp{}).WithDefaulter(...).WithValidator(...).Complete(). Two gotchas cost people days. First, the API server only speaks HTTPS to webhooks and verifies the certificate against caBundle in the webhook configuration; the scaffold uses cert-manager to issue the serving certificate and inject the CA with the cert-manager.io/inject-ca-from annotation, so a missing or expired certificate makes every create fail. Second, failurePolicy defaults to Fail: if your webhook pod is down or slower than timeoutSeconds (default 10 s, maximum 30 s), every matching request in the cluster is rejected, including the one that would restart your webhook. Scope the webhook with namespaceSelector and objectSelector, keep the timeout short, and use Ignore for anything that is not a security control.

Where the design came from, and how to read the source

The design is described in the paper "Borg, Omega, and Kubernetes": Borg had a central master with per-job state machines; Omega replaced it with a shared, transactional store that many independent schedulers watched optimistically; Kubernetes kept Omega's shared store idea but put an API server in front of it, so that every component is a client that watches and writes through one audited door. The 409 conflict you handle in a controller is the direct descendant of Omega's optimistic transactions.

The kubernetes/kubernetes repository: cmd/ holds the binaries (cmd/kube-apiserver, cmd/kube-controller-manager, cmd/kubelet); pkg/ holds their implementation; staging/src/k8s.io/ holds the libraries that are published as separate modules (client-go, apimachinery, api) and are developed here first. The Deployment controller is pkg/controller/deployment/deployment_controller.go; syncDeployment is its reconcile function and sync.go holds the scaling logic. To run one test:

make test WHAT=./pkg/controller/deployment                     # one package
go test ./pkg/controller/deployment -run ^TestSyncDeploymentCreatesReplicaSet$ -v

Start from main, follow the flags into the options struct, then into Run; the tests next to each file are the best documentation of what a function promises.

One reconcile, step by step: the user scales the WebApp shop from 2 to 3.

Before: WebApp shop has spec.replicas: 2, resourceVersion: 5, generation: 1, status.readyReplicas: 2. Its Deployment shop has replicas: 2, resourceVersion: 4, 2 ready pods.

1. kubectl patch webapp shop --type merge -p '{"spec":{"replicas":3}}'. The API server runs admission, the schema allows 3 (maximum 50), writes the object with resourceVersion: 6 and bumps generation to 2 because spec changed.
2. The WebApp informer in the operator receives MODIFIED with rv 6, replaces the cached copy, and the event handler adds the key shop/shop to the work queue. If the user had patched three times in a second there would still be one item in the queue.
3. A worker takes the key and calls Reconcile. It reads the WebApp (rv 6, replicas 3) and the Deployment (rv 4, replicas 2) from the cache: no network calls.
4. CreateOrUpdate sees the Deployment differs and sends PUT .../deployments/shop carrying rv 4. Meanwhile the Deployment controller in kube-controller-manager has just written status.observedGeneration on the same Deployment, so the stored rv is 7. The API server answers 409 Conflict.
5. Reconcile returns the error. The queue calls AddRateLimited: the first failure waits 5 ms (next ones would be 10, 20, 40 ms and so on), then the key is back in the queue.
6. The informer has by now received the Deployment's MODIFIED at rv 7 (the Owns watch), so the second reconcile reads a fresh copy and sends the update with rv 7. The server stores rv 8. The reconcile sets status.readyReplicas to 2 (still what the Deployment reports), observedGeneration: 2, Ready=False with message 2/3 replicas ready, and returns nil; the queue forgets the failure count.
7. The ReplicaSet controller creates a third pod. When it passes readiness, the Deployment's status.readyReplicas becomes 3, which is a Deployment MODIFIED event, which enqueues the owner shop/shop again.
8. That reconcile finds nothing to change in the Deployment, writes status.readyReplicas: 3 and Ready=True. Count the status writes: two, and the Deployment writes: one successful plus one rejected. Count the reconciles: three for one user action, each cheap because reads come from the cache.

The two numbers an interviewer listens for: the controller made zero API reads (all cache), and the only retry was driven by a real conflict, not by a timer.