"How does code get from a merged pull request to production" is the question every DevOps interview asks in some form, and in 2026 the expected answer is a GitOps pipeline: CI builds an image, a commit changes the desired state, Argo CD applies it. Platform teams are hired to run exactly this for dozens of product teams, so the follow-ups go deep: multi-tenancy, drift, rollbacks, secrets, and how a canary gets judged. The 100xDevs cohort covers Argo CD in week 37.2; this lesson takes that material to the level a platform interview expects.

What GitOps is

The OpenGitOps project (a CNCF working group) defines GitOps with four principles (version 1.0.0):

1. Declarative. The desired state is expressed declaratively: Kubernetes manifests, Helm values, Kustomize overlays, not scripts of commands.
2. Versioned and immutable. The desired state is stored in a way that enforces immutability and versioning and keeps a complete history. Git is the usual store, hence the name, but an OCI registry also qualifies.
3. Pulled automatically. Software agents pull the desired state from the source. Nobody pushes to the cluster.
4. Continuously reconciled. The agents keep observing the actual state and applying the desired state, not only when a commit lands.

The last two principles separate GitOps from "CI runs kubectl apply": only GitOps keeps correcting the cluster after the merge.

Push versus pull

In an interview, give the credential argument first: a compromised CI runner in a push model has admin on production; in a pull model it can at most push a commit, which is reviewed and visible.

Argo CD architecture

GitOps flow from a developer pull request through CI, ECR and a GitOps repo pull request to Argo CD pulling and syncing inside the EKS cluster, with rollback shown as a git revert

Argo CD is a CNCF graduated project that runs in the argocd namespace as a few Deployments:

The controller re-checks Git every 3 minutes by default (timeout.reconciliation in the argocd-cm ConfigMap), at once on a webhook from GitHub or GitLab, and sees a cluster change within seconds through the watch API.

The Application resource

An Application is the unit of GitOps: one source, one destination, one sync policy.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-api-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # deleting the Application deletes its objects
spec:
  project: shop-team
  source:
    repoURL: https://github.com/example/gitops
    targetRevision: main
    path: charts/api                 # a Helm chart in the repo
    helm:
      valueFiles:
        - ../../envs/prod/api-values.yaml
  destination:
    server: https://kubernetes.default.svc   # the cluster Argo CD runs in
    namespace: shop
  syncPolicy:
    automated:
      prune: true       # delete objects removed from Git (default false)
      selfHeal: true    # revert changes made outside Git (default false)
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 3
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 5m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas       # the HPA owns this field

Each Application has two statuses. Sync status compares Git with the cluster: Synced or OutOfSync. Health status reads the objects themselves: Healthy, Progressing, Degraded, Missing, Suspended. A Deployment with a crash-looping new ReplicaSet is Synced and Degraded at once.

Without automated, Argo CD only reports OutOfSync and a human clicks Sync. Without prune, an object deleted from Git stays in the cluster. Without selfHeal, a kubectl edit is reported, not reverted. Production usually runs all three.

AppProject: limits for teams

An AppProject is the multi-tenancy boundary: which repositories an Application may use, which clusters and namespaces it may deploy to, and which kinds it may create.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop-team
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/gitops
  destinations:
    - server: https://kubernetes.default.svc
      namespace: shop
    - server: https://kubernetes.default.svc
      namespace: shop-*
  clusterResourceWhitelist: []          # no cluster-scoped objects at all
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota               # only the platform team sets quotas
  syncWindows:
    - kind: deny
      schedule: "0 18 * * 5"            # cron: Fridays 18:00
      duration: 62h                     # until Monday 08:00
      timeZone: Asia/Kolkata
      applications: ["*-prod"]
      manualSync: true                  # a human can still sync in an emergency
  roles:
    - name: deployer
      policies:
        - p, proj:shop-team:deployer, applications, sync, shop-team/*, allow
      groups:
        - shop-team-engineers

An Application whose destination is outside its project's list is refused before any sync, so a team cannot write namespace: kube-system into its own Application.

Many apps: app-of-apps and ApplicationSets

With 50 teams nobody writes Applications by hand. Two patterns:

• App-of-apps. One root Application points at a folder of Application manifests. Syncing the root creates the children; the repo becomes the list of what is installed. Simple, and still common for bootstrapping a cluster (ingress controller, cert-manager, External Secrets, monitoring).
• ApplicationSet. A template plus a generator that produces Applications. Generators: list (a fixed list of environments), clusters (every cluster Argo CD knows, filtered by labels), git (one Application per directory or per file in a repo), pullRequest (one preview environment per open pull request, torn down on merge), and matrix and merge to combine them. The Argo CD docs recommend ApplicationSets over app-of-apps.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: shop-services
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/example/gitops
        revision: main
        directories:
          - path: envs/prod/*          # one Application per service folder
  template:
    metadata:
      name: "{{path.basename}}-prod"
    spec:
      project: shop-team
      source:
        repoURL: https://github.com/example/gitops
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: shop
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Adding a service is adding a folder; removing the folder removes the Application and, with prune, its objects.

Sync waves, hooks, health and diffs

• Sync waves. The annotation argocd.argoproj.io/sync-wave: "-1" orders objects: lower waves first, with a 2 second gap and a health check between waves. Namespaces and CRDs in wave -1, a database in wave 0, the app in 1.
• Hooks. argocd.argoproj.io/hook: PreSync runs a Job before the apply (migrations), PostSync after it is healthy (smoke tests), SyncFail for cleanup. argocd.argoproj.io/hook-delete-policy: HookSucceeded removes the finished Job. Helm pre-install and pre-upgrade hooks map to PreSync automatically.
• Health checks. Argo CD judges built-in kinds and many CRDs (Rollout, Certificate); for your own CRD you add a Lua health script to argocd-cm.
• ignoreDifferences. Some fields are rightly changed by controllers after apply: replicas by an HPA, caBundle in a webhook by cert-manager, injected sidecars. Listing them with jsonPointers or jqPathExpressions keeps the app Synced while the controller does its job. Add RespectIgnoreDifferences=true to syncOptions so a sync does not overwrite them.
• Sync options. CreateNamespace=true, ServerSideApply=true for large objects and shared ownership, PruneLast=true to delete old objects only after new ones are healthy, Replace=true for a rare immutable-field change (delete and recreate, with downtime).

RBAC, SSO and notifications

Users log in through SSO (an OIDC provider, or Dex in front of GitHub or LDAP). Permissions are rules in the argocd-rbac-cm ConfigMap: p, role:shop-deployer, applications, sync, shop-team/*, allow grants, and g, shop-team-engineers, role:shop-deployer binds an SSO group. policy.default: role:readonly lets everyone see and nobody change. The notifications controller sends events to Slack or a webhook on triggers such as on-sync-failed and on-health-degraded, so a failed production sync pings the owning team.

From CI to the cluster

The split of responsibilities is the core of the design:

1. A developer merges a pull request to the application repo.
2. CI (GitHub Actions) builds the image, scans it, pushes it to ECR and reads back the digest sha256:.... Tags are mutable, digests are not, so the GitOps repo pins the digest.
3. CI checks out the GitOps repo, changes one line in envs/staging/api-values.yaml (image.digest: sha256:...) and opens a pull request. For staging the PR can auto-merge; for prod a human approves.
4. Argo CD sees the commit (webhook or next poll), renders the chart with the new values, finds the Deployment OutOfSync, applies it, and reports Healthy once the rollout finishes.

CI never holds a kubeconfig. Argo CD Image Updater is an alternative to step 3: it watches ECR for new tags matching a rule (semver, digest, newest build) and commits the change itself, which skips the review, so most teams use it only for dev.

Repository layout and promotion

What goes where: application code and its Dockerfile in the app repo; the chart or Kustomize base and per-environment values in the GitOps repo; add-ons and AppProjects in a platform folder only the platform team can merge to. Promotion is a pull request copying the digest from envs/staging to envs/prod, which CI can open after staging has been healthy for an hour. Git history then answers "what is in prod and since when".

Rollback the GitOps way

kubectl rollout undo and helm rollback are wrong answers in a GitOps interview, because self-heal would put the bad version straight back. The right answer: git revert <merge commit> on the GitOps repo, merge it, and Argo CD syncs the previous digest the same way it synced the bad one. For a faster emergency revert, Argo CD's UI and argocd app rollback re-apply an earlier synced revision, but only when auto-sync is off for that app, and the revert commit must still follow. Neither path reverts a database migration, which is why migrations must be backwards compatible with the previous version.

Secrets in a GitOps repo

Git must never hold a plaintext secret, and Argo CD does not solve that by itself. Three options:

ESO is the default on EKS: one source of truth with IAM audit, rotation as a Secrets Manager feature, and a GitOps repo that is safe to make public. ESO authenticates with IRSA or Pod Identity, so no AWS key exists in the cluster.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: api-db
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: api-db                       # the Kubernetes Secret ESO creates
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: prod/shop/api             # the Secrets Manager entry
        property: url

Flux: the other CNCF option

Flux is the other CNCF graduated GitOps tool: controllers driven by custom resources, with no UI.

Progressive delivery with Argo Rollouts

A rolling update only checks readiness probes. Progressive delivery sends a small share of real traffic to the new version, measures it, and advances or aborts on the numbers. Argo Rollouts does this with a Rollout resource that replaces Deployment (same pod template, a different strategy).

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
  namespace: shop
spec:
  replicas: 10
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/api@sha256:2222222222222222222222222222222222222222222222222222222222222222
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
  strategy:
    canary:
      canaryService: api-canary        # two Services you create; the controller patches their selectors
      stableService: api-stable
      trafficRouting:
        alb:
          ingress: api                  # an Ingress whose backend is api-stable with port name use-annotation
          servicePort: 80               # the controller writes a weighted actions annotation on it
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service
            value: api-canary
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - setWeight: 25
        - pause: {duration: 5m}
        - setWeight: 50
        - pause: {duration: 10m}
        - pause: {}                      # wait for a human: kubectl argo rollouts promote api

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: shop
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 1m
      failureLimit: 3                  # three failed measurements are tolerated; the fourth aborts the rollout
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: https://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service}}", code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service}}"}[2m]))

Without trafficRouting, the controller approximates the percentage with pod counts (1 canary pod of 10 is 10 percent). With trafficRouting through the AWS Load Balancer Controller (the Rollout controller writes an alb.ingress.kubernetes.io/actions.api-stable annotation and the ALB splits traffic between the api-stable and api-canary target groups), Istio or Gateway API, the stable set stays at full size and the proxy splits requests exactly. failureLimit is the number of failed measurements tolerated, so failureLimit: 3 aborts on the fourth failure. A failed analysis aborts the rollout, sets the canary weight to 0 and marks it Degraded; a fixed commit starts a new rollout.

Blue-green is the other strategy: activeService and previewService, the new ReplicaSet comes up at full size behind preview, prePromotionAnalysis runs on it, and promotion (manual with autoPromotionEnabled: false, or after autoPromotionSeconds) switches the active Service's selector to the new hash. The old set stays for scaleDownDelaySeconds (default 30) so an instant switch back is possible. Flagger, from the Flux family, does the same job with a Canary resource wrapping a normal Deployment.

Argo CD syncs the Rollout from Git and reports Progressing while the canary runs, Degraded if the analysis fails.

When Argo CD is down: break-glass

Argo CD being down does not take workloads down; it only stops changes. For a change that cannot wait:

1. Declare it in the incident channel and name the person applying.
2. Make the change in Git first, on a branch, so the intended state exists.
3. Apply it by hand from that branch with kubectl apply -k envs/prod/api or helm template ... | kubectl apply -f -, using a break-glass IAM role whose use is logged to CloudTrail and alerts the platform team.
4. Merge the branch. When Argo CD returns it finds the cluster already matches Git and reports Synced; if the manual change differed, self-heal corrects it to Git, which is what you want.
5. Write the postmortem item: why Argo CD was down and why the change could not wait.

Git first, then cluster, so self-heal cannot undo the fix.

A canary that fails analysis, with the blast radius in numbers. The shop API serves 2,000 requests per second with 10 replicas. The Rollout above runs steps 10 percent for 5 minutes, 25 percent for 5 minutes, 50 percent for 10 minutes, then a manual promote. Analysis starts at step 1 and measures success rate every minute; failureLimit: 3 tolerates three failed measurements, so the fourth aborts.

A new image has a bug that returns HTTP 500 for 3 percent of requests.

Step 1 (minute 0 to 5): weight 10. The canary receives 2,000 x 0.10 = 200 requests per second. Over 5 minutes that would be 200 x 300 = 60,000 requests. The success rate measured each minute is 0.97, below the 0.99 condition, so measurements at minutes 1, 2, 3 and 4 fail. The fourth failure passes failureLimit: 3 at minute 4.

That is faster than the step: the rollout aborts during step 1, not after it. The canary weight goes back to 0 at minute 4, the ALB action annotation sends 100 percent to api-stable, and the Rollout is Degraded. Total exposure: 200 x 240 = 48,000 requests to the canary, of which 3 percent, 1,440, failed.

Compare with a plain Deployment rolling update at maxSurge: 25%: within about 2 minutes all 10 pods are new and every request hits the bug. In 4 minutes that is 2,000 x 240 = 480,000 requests and 14,400 errors, ten times more, and nothing stops it until a human notices a dashboard.

Put it against the error budget. At 99.9 percent monthly availability and 2,000 rps, the monthly request count is 2,000 x 30 x 86,400 = 5,184,000,000 and the budget is 0.1 percent of that, 5,184,000 failed requests. The canary spent 1,440 of them, 0.03 percent of the budget. The plain rollout spent 14,400, and it would keep spending until rolled back.

What happens next in GitOps terms: Argo CD shows the Application Synced (Git and cluster agree on the Rollout spec) but Degraded (the Rollout reports the abort). The notifications controller fires on-health-degraded to the team's Slack channel. The developer opens a fix, CI pushes a new digest, the GitOps PR is merged, and the next Rollout starts at weight 10 again. No revert is needed because the canary never became stable; if the bug had been found after promotion, the fix would be git revert of the promotion commit, and Argo CD would roll the stable set back through the same canary steps unless the Rollout's steps are skipped with kubectl argo rollouts promote --full.