Security questions are in every DevOps interview now, and they are concrete: "how does your pipeline get AWS credentials", "where do your secrets live", "what stops someone deploying a root container", "what do you do when a key leaks". A candidate who answers "we use IAM" loses to one who says "GitHub Actions assumes a role through OIDC, the trust policy pins the repo and branch, no key exists to leak". The same controls are what a platform team is hired to build. Audits (an access review, a SOC 2 or ISO question, a customer security questionnaire) ask for the table at the end of this lesson, layer by layer.

Shift left: security as a pipeline property

"Shift left" means the check happens where the change is made, as a failing build, not as a finding six weeks later. The order of controls follows the artifact: secret scanning on every push, dependency and image scanning on every build, policy checks on every manifest and every Terraform plan, signature verification at admission, detection at runtime. Each control is cheap on its own; together they make the common attacks (a leaked key, a vulnerable base image, a root container with the node's disk mounted) impossible to deploy by accident.

The software supply chain from commit through CI, dependencies, build and scan, SBOM and signing, registry, admission and runtime, with the security control and the verification command at each step

Identity and access: short-lived credentials everywhere

Least privilege means every identity (human, pipeline, pod) can do exactly what its job needs and nothing more. The second rule matters as much: no long-lived keys. A static access key in a CI secret is a credential that never expires, is copied into logs and laptops, and is the first thing an attacker looks for. The replacement is a token that proves identity to the cloud and is exchanged for credentials that expire in an hour.

The CI side in GitHub Actions: the workflow requests a token with permissions: id-token: write, and the AWS role's trust policy pins the subject claim so only one repository and branch can assume it.

# .github/workflows/deploy.yaml (the job that touches AWS)
permissions:
  id-token: write    # required to request the OIDC JWT
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@<40-character-commit-sha>   # pin every action by commit SHA, not by a movable tag
      - uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-shop-deploy
          aws-region: ap-south-1

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"},
      "StringLike": {"token.actions.githubusercontent.com:sub": "repo:shop-org/shop-api:ref:refs/heads/main"}
    }
  }]
}

Without the sub condition, any GitHub repository in the world could assume the role. That single line is the most common OIDC mistake.

Secrets management

A secret is anything that grants access: passwords, API keys, tokens, private keys, certificates, database connection strings with a password inside, webhook signing secrets, and the kubeconfig itself. Rules: a secret lives in one place (AWS Secrets Manager or HashiCorp Vault), is written to Git never, reaches the workload at runtime, and rotates on a schedule.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-provider
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payments-provider        # the Kubernetes Secret ESO creates
    creationPolicy: Owner
  data:
    - secretKey: PAYMENT_PROVIDER_KEY
      remoteRef:
        key: prod/payments/provider  # the name in Secrets Manager
        property: api_key

Scan every repository for leaked secrets, in CI and on the history: gitleaks detect --source . --redact fails the build on a match, and GitHub secret scanning with push protection blocks the push itself. When a key leaks: rotate first, then investigate. The order matters because the investigation takes hours and the attacker needs minutes. Revoke the key, issue a new one, deploy it, then read CloudTrail for every call the old key made (aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...), then remove it from Git history and write the incident note. Deleting the commit is not rotation; the key is already in a dozen clones.

Supply chain security

The supply chain is every step between a developer's commit and a running container. Each step can be attacked: a typosquatted package, a compromised CI action, a registry tag that someone re-pushed. The controls, in pipeline order:

• SBOM (software bill of materials): a list of every package in the image, generated at build time with trivy image --format cyclonedx --output sbom.json IMAGE and attached to the image as an attestation. When the next log4j happens, you query SBOMs instead of rebuilding everything to find out.
• Image scanning: the gate policy is fail on CRITICAL, warn on HIGH. trivy image --severity CRITICAL --ignore-unfixed --exit-code 1 IMAGE in CI fails the build on a fixable critical vulnerability, and a second run with --severity HIGH --exit-code 0 prints HIGH findings as warnings; ECR scan-on-push catches images that arrive another way. Scan the base image and the Terraform and Kubernetes manifests too (trivy config .).
• Pinning by digest: FROM python:3.12-slim@sha256:... and uses: actions/checkout@<40-char SHA>. A tag can be moved; a digest cannot. Renovate or Dependabot updates the pins with a pull request you can review.
• Signing and verification: cosign sign IMAGE@DIGEST in CI uses keyless signing, where the signature is tied to the CI job's OIDC identity and recorded in the public Rekor transparency log. At admission, Kyverno verifies: cosign verify IMAGE --certificate-identity-regexp '^https://github.com/shop-org/shop-api/' --certificate-oidc-issuer https://token.actions.githubusercontent.com.
• SLSA levels (Supply-chain Levels for Software Artifacts, build track, from the v1.2 specification): L0 no guarantees; L1 provenance exists, generated by the build platform, showing how the package was built (prevents mistakes, trivial to forge); L2 builds run on a hosted platform that generates and signs the provenance, and consumers validate it (forging requires an explicit attack); L3 a hardened hosted platform that prevents builds from influencing one another and keeps the signing secrets away from user-defined build steps. GitHub Actions with the official SLSA generator workflow reaches L3 for provenance.

Dependency confusion and typosquatting. Dependency confusion is when a package manager configured with both a private and a public registry fetches a public package with the same name as your internal one, because the public version number is higher. Typosquatting is a public package named reqeusts or colourama that someone installs by mistake. Both are stopped by a lockfile with hashes, a private registry that proxies and pins the public one, scoping internal names (@shop/ in npm), and go mod verify or pip install --require-hashes.

Container and Kubernetes hardening

The Pod Security Standards define three profiles: privileged, baseline, restricted. The restricted profile is what every application namespace should enforce, through the namespace label pod-security.kubernetes.io/enforce: restricted. A pod that passes it:

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  securityContext:
    runAsNonRoot: true                 # the image must declare a non-root USER
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault             # the runtime's default syscall filter
  containers:
    - name: api
      image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/api@sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
      securityContext:
        allowPrivilegeEscalation: false  # no setuid binaries gain root
        readOnlyRootFilesystem: true     # writes go to an emptyDir, not the image layer
        capabilities:
          drop: ["ALL"]                  # add back only NET_BIND_SERVICE if you must bind 80
      resources:
        requests: {cpu: 250m, memory: 256Mi}
        limits: {memory: 512Mi}
      volumeMounts:
        - {name: tmp, mountPath: /tmp}
  volumes:
    - {name: tmp, emptyDir: {}}

Beyond the pod: no privileged pods (privileged: true is root on the node), no hostPath (mounting / or the container socket is the standard container escape), no hostNetwork, hostPID, hostIPC, NetworkPolicy default deny per namespace with explicit allow rules for DNS, the services the app calls, and the ingress controller. Admission policies with Kyverno enforce all of this and more (a required registry, no latest tag, required labels, signature verification):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-digest-from-ecr
spec:
  rules:
    - name: image-from-ecr-by-digest
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        failureAction: Enforce         # Audit only writes a PolicyReport
        message: "images must come from our ECR and be pinned by digest"
        pattern:
          spec:
            containers:
              - image: "123456789012.dkr.ecr.ap-south-1.amazonaws.com/*@sha256:*"

The OWASP Kubernetes Top Ten (2025 edition) as a checklist, with the control for each: K01 insecure workload configurations (restricted profile), K02 overly permissive authorization (RBAC with named verbs, no cluster-admin for workloads), K03 secrets management failures (ESO, etcd encryption, no secrets in env dumps), K04 lack of cluster-level policy enforcement (Kyverno or Gatekeeper), K05 missing network segmentation (NetworkPolicy default deny), K06 overly exposed components (private API endpoint, no public kubelet or dashboard), K07 misconfigured and vulnerable components (upgrade cadence, add-on versions), K08 cluster-to-cloud lateral movement (block IMDS from pods, Pod Identity with least privilege), K09 broken authentication (SSO, no static tokens), K10 inadequate logging and monitoring (audit logs on, Falco, CloudTrail).

Cloud hardening

• Private clusters and VPC endpoints: the EKS API endpoint is private (or public with a CIDR allow-list), nodes have no public IP, and traffic to ECR, S3, STS and Secrets Manager goes through VPC endpoints so it never crosses the internet and the NAT bill drops.
• Security groups are stateful allow-lists per ENI: the database group allows 5432 from the node or pod security group only, not from a CIDR.
• Encryption at rest with KMS: EBS volumes, S3 buckets, RDS and Aurora, and EKS Secrets envelope encryption, each with a customer-managed key so you control rotation and the key policy.
• CloudTrail records every API call in every region to an S3 bucket with object lock; GuardDuty reads CloudTrail, VPC flow logs and DNS logs and raises findings such as a credential used from an unusual IP or an EKS pod talking to a known bad host. Both are on in every account, by Organizations policy.
• IMDSv2 and the hop limit: a pod that can reach 169.254.169.254 can read the node's IAM credentials. Set the node launch template to http_tokens = "required" and http_put_response_hop_limit = 1, so the metadata response does not survive the extra hop into the pod network. Workloads then get credentials only through Pod Identity.

Application-side controls DevOps owns

TLS everywhere: cert-manager issues certificates from Let's Encrypt or a private CA, the ingress or Gateway terminates TLS 1.2 and 1.3 only, and inside the cluster a mesh or application TLS covers service-to-service traffic. A WAF (AWS WAF or Cloudflare) in front of the load balancer blocks known injection patterns and bots. Rate limits at the edge and the gateway, per IP and per API token, with quotas per customer, are the first abuse control; the full design is in the scaling lesson. SSRF protection (server-side request forgery, where the app is tricked into making a request on the attacker's behalf) is partly application code and partly network: block pod egress to the metadata endpoint and to the VPC CIDR except the services the app needs, through NetworkPolicy and the IMDS hop limit. A typical application security audit covers exactly these: cross-tenant access (a request with tenant A's token reading tenant B's rows, found by replaying requests with swapped IDs), SSRF through a URL parameter, and reconnect storms where clients hammered a recovering service; the fixes were authorization checks at the data layer, egress policy, and jittered reconnect backoff with a rate limit.

Policy as code, runtime detection, compliance

Compliance basics the auditor asks for: audit logs enabled (EKS control plane audit logs to CloudWatch, CloudTrail, application access logs) with a stated retention (90 days hot, one year cold is a common choice; label it as yours); quarterly access reviews where every IAM role, GitHub team and RBAC binding has an owner who confirms it is still needed; and evidence, which means the policy report, the scan output and the review ticket, not a promise.

Layer, control, how to verify

A leaked AWS key, from alert to closed ticket. At 11:20 GitHub secret scanning emails you: an AKIA... access key was found in a commit to shop-org/data-tools, pushed at 11:04. The key belongs to the IAM user etl-batch, created two years ago, with AmazonS3FullAccess.

Step 1, rotate first (11:22). aws iam create-access-key --user-name etl-batch makes a new key, you put it into Secrets Manager, the ETL job reads it at its next start, and at 11:31 aws iam update-access-key --user-name etl-batch --access-key-id AKIA...OLD --status Inactive, then delete it. Elapsed exposure: 11:04 to 11:31, 27 minutes.

Step 2, scope the damage (11:35). aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...OLD --start-time 2026-11-03T11:00:00Z lists every call the key made in the window. You see the usual 14 PutObject calls from the ETL host's IP, and 3 ListBuckets and 1 GetObject from an IP in another country at 11:19. The repository was public for 15 minutes. One object was read: exports/customers-2026-10.csv, 48,000 rows. That is a data breach and needs the privacy process, not just an engineering fix.

Step 3, shrink the blast radius (same day). The user had full S3 access to 31 buckets; the job needs s3:PutObject on one prefix. Replace the IAM user with a role assumed by the ETL runner through Pod Identity, with a policy for arn:aws:s3:::shop-exports/etl/* only. Now there is no key to leak, and a repeat exposes one prefix instead of 31 buckets.

Step 4, stop the next one. Enable push protection on every repository in the organization so the push is blocked, not reported after. Add gitleaks to the CI template. Route GuardDuty's credential exfiltration and anomalous-location findings to the on-call channel, which would have paged you at 11:19 instead of 11:20 by email.

Step 5, the numbers for the note. Exposure window 27 minutes; 4 unauthorized calls; 1 object, 48,000 customer rows; root cause: a long-lived key with 31 buckets of access in a repository with no push protection; fixes: rotation in 9 minutes, key removed, role with one prefix, push protection organization-wide.