Almost every real cluster is installed with Helm: the ingress controller, cert-manager, Prometheus, External Secrets, the AWS Load Balancer Controller, and the company's own services. Interviewers ask "how do you deploy the same app to dev and prod without copying YAML", "what does helm upgrade do if the new version crashes", "where does Helm keep its state" and "why did my ConfigMap change not restart the pods". The questions look small, but each one tests whether you have shipped with Helm or only read about it. A take-home task of "write a chart for this service" is common for platform roles.

Why templating

One service needs the same Deployment, Service, Ingress and HPA in three environments. Only a few things differ: the image tag, the replica count, the hostname, the resource sizes. Copying four YAML files three times means twelve files that drift apart. Templating keeps one copy of the structure and injects the differences from a small per-environment file.

In practice the three live together: Helm for anything you install from outside, Helm or Kustomize for your own services, and Argo CD rendering either from Git. Both Helm and Kustomize are open source and Helm is a CNCF graduated project.

Chart structure

helm create api writes a starter chart. The folder, trimmed to what matters:

api/
  Chart.yaml            # name, chart version, appVersion, dependencies
  values.yaml           # default values; every key the templates read
  values.schema.json    # optional JSON Schema that rejects bad values
  charts/               # dependency charts, downloaded by helm dependency update
  templates/
    _helpers.tpl        # named templates (labels, fullname); files starting with _ render nothing
    deployment.yaml
    service.yaml
    ingress.yaml
    hpa.yaml
    serviceaccount.yaml
    NOTES.txt           # printed after install; a template too
    tests/
      test-connection.yaml   # a Pod with the helm.sh/hook: test annotation

Chart.yaml has two versions and people confuse them. version is the chart's own version and must change every time any file in the chart changes. appVersion is the version of the software inside and is only informational; it is the default image tag in the standard template. apiVersion: v2 marks a chart for Helm 3 and later.

The template language

Helm templates are Go text/template with extra functions (most from the Sprig library, a few from Helm itself). The things you must be able to read and write:

• Values and built-in objects. .Values.replicaCount comes from values.yaml. .Release.Name, .Release.Namespace, .Chart.Name, .Chart.Version and .Chart.AppVersion come from the release and Chart.yaml. .Capabilities.KubeVersion tells you the cluster version so a chart can pick an API version.
• Pipelines and functions. {{ .Values.image.tag | default .Chart.AppVersion | quote }} reads left to right: take the tag, fall back to appVersion if empty, wrap in quotes. toYaml turns a map into YAML text; nindent 12 adds a newline and indents every line by 12 spaces; include "api.labels" . renders a named template to a string so you can pipe it; tpl renders a string from values as a template; required "image.repository is required" .Values.image.repository fails the render with your message.
• Control flow. {{- if .Values.ingress.enabled }} ... {{- end }}, {{- range .Values.env }}, {{- with .Values.nodeSelector }} (changes . to that value inside the block and skips it when empty).
• Named templates. Defined in _helpers.tpl with {{- define "api.labels" -}} ... {{- end }} and used with include. Use include, not template, because template cannot be piped into nindent.
• Whitespace. {{- eats the whitespace and newline before the action, -}} eats it after. Without the dash, every if line leaves a blank line in the output. Most indentation bugs are a missing dash or indent used where nindent was needed.

A complete small chart, file by file

The chart deploys an HTTP API on EKS. The ServiceAccount takes an annotation that is used by IRSA; with EKS Pod Identity the association is made with aws eks create-pod-identity-association and the ServiceAccount needs no annotation, so the annotation block is optional.

# Chart.yaml
apiVersion: v2
name: api
description: The shop API
type: application
version: 0.3.0        # bump on every change to the chart
appVersion: "1.7.2"   # the image tag by default; quoted because it is a string
kubeVersion: ">=1.29.0-0"

# values.yaml: every key with a working default for a laptop
replicaCount: 1
image:
  repository: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/api
  tag: ""             # empty means use Chart.appVersion
  pullPolicy: IfNotPresent
serviceAccount:
  create: true
  annotations: {}     # IRSA: eks.amazonaws.com/role-arn: arn:aws:iam::...:role/api
service:
  port: 80
  targetPort: 8080
ingress:
  enabled: false
  className: alb      # the AWS Load Balancer Controller on EKS; ingress-nginx is retired
  host: api.example.com
  tls: false
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi
env:
  LOG_LEVEL: info

# templates/_helpers.tpl
{{- define "api.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end }}

{{- define "api.labels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{- define "api.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "api.fullname" . }}
  labels:
    {{- include "api.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "api.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "api.selectorLabels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      serviceAccountName: {{ include "api.fullname" . }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ required "image.repository is required" .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.targetPort }}
          envFrom:
            - configMapRef:
                name: {{ include "api.fullname" . }}
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 20
            periodSeconds: 10
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "api.fullname" . }}
data:
  {{- range $key, $value := .Values.env }}
  {{ $key }}: {{ $value | quote }}
  {{- end }}

# templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "api.fullname" . }}
  labels:
    {{- include "api.labels" . | nindent 4 }}
spec:
  type: ClusterIP
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
      name: http
  selector:
    {{- include "api.selectorLabels" . | nindent 4 }}

# templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "api.fullname" . }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  {{- if .Values.ingress.tls }}
  tls:
    - hosts: [{{ .Values.ingress.host | quote }}]
      secretName: {{ include "api.fullname" . }}-tls
  {{- end }}
  rules:
    - host: {{ .Values.ingress.host | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "api.fullname" . }}
                port:
                  name: http
{{- end }}

# templates/hpa.yaml
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "api.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "api.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}

# templates/serviceaccount.yaml
{{- if .Values.serviceAccount.create }}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "api.fullname" . }}
  {{- with .Values.serviceAccount.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
{{- end }}

# templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "api.fullname" . }}-test-connection
  annotations:
    "helm.sh/hook": test
spec:
  restartPolicy: Never
  containers:
    - name: wget
      image: busybox:1.36
      command: ["wget", "-qO-", "{{ include "api.fullname" . }}:{{ .Values.service.port }}/healthz"]

Two details carry weight. The Deployment omits replicas when the HPA is on, otherwise every helm upgrade would reset the count the HPA chose. The checksum/config annotation hashes the rendered ConfigMap into the pod template, so a config change produces a new pod template and a rollout; without it, pods keep the old values until they happen to restart.

Values design

• Defaults that work. helm install api ./api with no flags must produce a running app on a laptop. Prod is the override, not the default.
• One values file per environment, small and only the differences: values-dev.yaml, values-prod.yaml. Files given with -f are merged left to right, and --set key=value wins over all files. Maps merge key by key; lists are replaced whole.
• Schema validation. values.schema.json is a JSON Schema that Helm checks on install, upgrade, lint and template against the final merged values. It turns "ingress.host is a number" into an error before anything reaches the cluster.

{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["image"],
  "properties": {
    "replicaCount": {"type": "integer", "minimum": 1},
    "image": {
      "type": "object",
      "required": ["repository"],
      "properties": {"repository": {"type": "string", "minLength": 1}, "tag": {"type": "string"}}
    },
    "ingress": {
      "type": "object",
      "properties": {"enabled": {"type": "boolean"}, "host": {"type": "string", "format": "hostname"}}
    }
  }
}

• No secrets in values. A values file lives in Git and is copied into the release secret. Put a database password in a kind: ExternalSecret template that pulls from AWS Secrets Manager through the External Secrets Operator, and let the chart only know the secret's name.

The release lifecycle

Values files merged into one values object, rendered through the chart templates into manifests, applied to the API server and recorded as a release secret, with a revision timeline for install, upgrade, failed upgrade and rollback

A release is one installed instance of a chart in one namespace. Helm stores every revision of every release as a Secret in the release's namespace, named sh.helm.release.v1.<release>.v<N> with type helm.sh/release.v1. The secret holds the chart, the merged values and the rendered manifests, gzip-compressed and base64-encoded. There is no server component; Helm 3 removed Tiller, and helm is a client that talks to the API server with your kubeconfig and your RBAC.

helm install api ./api -n shop --create-namespace -f values-prod.yaml      # revision 1
helm upgrade api ./api -n shop -f values-prod.yaml --set image.tag=1.8.0   # revision 2
helm upgrade --install api ./api -n shop -f values-prod.yaml \
  --wait --timeout 5m --rollback-on-failure                                 # CI-friendly: install or upgrade, wait, undo on failure
helm history api -n shop            # every revision with status and description
helm rollback api 1 -n shop         # creates revision 3 with the content of revision 1
helm status api -n shop             # NOTES.txt and the last state
helm get values api -n shop         # the values used, without the defaults
helm get manifest api -n shop       # what was sent to the cluster
helm uninstall api -n shop          # deletes the objects and the release secrets
helm uninstall api -n shop --keep-history   # keeps the secrets so rollback is still possible

The flags that matter in CI: --wait makes Helm block until Deployments have their ready replicas, Services have endpoints and Jobs finish, up to --timeout (default 5m0s). --rollback-on-failure (Helm 4; the Helm 3 name was --atomic) rolls back to the last good revision if the upgrade fails. --history-max limits how many revision secrets are kept (default 10). On upgrade Helm computes a three-way merge between the last rendered manifest, the live object and the new manifest, so a field another controller added (an injected sidecar, an HPA-owned replica count) survives the upgrade.

Hooks are templates with the helm.sh/hook annotation that run at a point in the lifecycle: pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete, pre-rollback, post-rollback and test. The typical hook is a database migration Job:

# templates/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "api.fullname" . }}-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"                      # lower runs first; default 0
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          command: ["./migrate", "up"]

Helm waits for a hook Job to finish before applying the rest; a failed Job fails the release. Hook resources are not part of the release, so helm uninstall does not delete them: the delete policy (before-hook-creation, hook-succeeded, hook-failed) does.

Dependencies and umbrella charts

A chart can depend on others, listed in Chart.yaml and fetched into charts/ by helm dependency update:

# Chart.yaml (dependencies section)
dependencies:
  - name: redis
    version: "20.x.x"
    repository: oci://registry-1.docker.io/bitnamicharts
    condition: redis.enabled      # a values key that turns the subchart on or off
    tags: [cache]

Values for a subchart live under its name: redis.auth.enabled: false. condition points at a boolean values path; if it is set, it wins over tags, which enable a group of subcharts at once. An umbrella chart is a chart with no templates of its own that lists a whole system (api, worker, redis) as dependencies, so one helm install brings up the stack. It is convenient for demos and risky in production, because one upgrade touches everything.

Publishing: OCI registries and ECR

Chart repositories used to be an index.yaml on an HTTP server. The current way is an OCI registry, the same kind that holds container images, and ECR supports it:

helm package ./api                                          # api-0.3.0.tgz
aws ecr create-repository --repository-name api --region ap-south-1   # repo name must equal the chart name
aws ecr get-login-password --region ap-south-1 | \
  helm registry login --username AWS --password-stdin 123456789012.dkr.ecr.ap-south-1.amazonaws.com
helm push api-0.3.0.tgz oci://123456789012.dkr.ecr.ap-south-1.amazonaws.com/
helm install api oci://123456789012.dkr.ecr.ap-south-1.amazonaws.com/api --version 0.3.0 -n shop

The chart version becomes the OCI tag, which is why it must change on every edit. Provenance: helm package --sign produces a .prov file signed with a GPG key, and helm install --verify refuses a chart whose signature does not match. Teams on a supply-chain track sign the OCI artifact with cosign instead.

Testing a chart

Helm with Argo CD

Argo CD can take a chart as an Application source. It does not run helm install. It runs helm template with the values from the Application (valueFiles, values, valuesObject, parameters, in that order of increasing precedence) and applies the rendered manifests itself. There is no release secret, so helm ls shows nothing, and helm rollback does not exist: rollback is a Git revert. Helm hooks are mapped onto Argo CD hooks (pre-install and pre-upgrade both become PreSync, helm.sh/hook-weight becomes a sync wave); test hooks are skipped. The release name defaults to the Application name, and Argo CD 3.5 and later render with the Helm 4 binary. In this model values files live in the GitOps repository and every environment change is a reviewed commit, which is the subject of the next lesson.

An upgrade that fails and the rollback timeline. The api release is at revision 4, chart 0.3.0, image tag 1.7.2, 3 replicas, with maxUnavailable: 0. At 10:00 CI runs:

helm upgrade --install api oci://123456789012.dkr.ecr.ap-south-1.amazonaws.com/api \
  --version 0.3.1 -n shop -f values-prod.yaml --set image.tag=1.8.0 \
  --wait --timeout 5m --rollback-on-failure

Step 1 (10:00:00). Helm merges values: values.yaml from chart 0.3.1, then values-prod.yaml, then the --set. Schema passes. It renders 7 manifests and finds one pre-upgrade hook, the migrate Job with weight -5.

Step 2 (10:00:02). Helm creates the Job and waits. The migration adds an index and finishes in 40 seconds with exit code 0. Policy hook-succeeded deletes the Job.

Step 3 (10:00:42). Helm applies the manifests. Only the Deployment changed (new image, new chart label), so the API server starts a rolling update: new ReplicaSet, 1 surge pod.

Step 4 (10:00:42 to 10:05:42). The new pod starts, but 1.8.0 reads a config key that does not exist in prod and exits. CrashLoopBackOff. Because maxUnavailable is 0, the 3 old pods stay ready and users see nothing. --wait keeps polling the Deployment: it wants 3 ready replicas on the new ReplicaSet and never gets them.

Step 5 (10:05:42). The 5 minute timeout fires. Helm marks revision 5 as failed and, because of --rollback-on-failure, performs a rollback: it writes revision 6 with the manifests of revision 4 (image 1.7.2) and applies them. The Deployment's pod template is now identical to the old ReplicaSet, so Kubernetes scales it back to 3 and deletes the crashing ReplicaSet. No pre-rollback hooks exist, so the migration is not reversed; adding an index is backwards compatible, which is the rule for migrations that run before a rollout.

Step 6 (10:05:50). helm history api -n shop shows: 4 deployed-then-superseded, 5 failed, 6 deployed with description "Rollback to 4". Total user-facing impact: zero errors, about 6 minutes of CI time.

Count the cost of the alternative. Without --wait, Helm returns "deployed" at 10:00:43, CI goes green, and the broken ReplicaSet sits in the cluster until a human reads kubectl get pods. Without maxUnavailable: 0, the controller could have removed one old pod first and run on 2 of 3 for 5 minutes, which is a 33 percent capacity cut during the window. The flags and the Deployment strategy together are what made the failure invisible.