Senior Kubernetes interviews are about production behaviour, not objects: "why is this pod OOMKilled when the node has free memory", "why did the HPA scale to max and stay there", "how would you upgrade a cluster with 300 services and no maintenance window", "can this developer delete that Secret". Each has a precise answer. On the job, these settings are the difference between a cluster that absorbs a traffic spike and one that pages you. Platform roles in Indian product companies and GCCs expect you to own them, and the eks-cluster-platform and autoscaling-web-app design problems are built on them.

Requests, limits and the kernel

A request is what the scheduler reserves for a container and what the HPA measures utilisation against. A limit is what the kernel enforces. They are enforced differently.

CPU is compressible. A CPU request becomes a cgroup weight, so under contention a pod gets at least its share and can use idle CPU above it. A CPU limit becomes a CFS quota: the container may run for limit worth of time in each 100 ms period and is then paused until the next period, even if the node is idle. That pause is throttling, and it adds latency that you only see in container_cpu_cfs_throttled_periods_total. A request needing 30 ms of CPU under a 250m limit (25 ms per period) always crosses a period boundary. The case against CPU limits: requests already protect neighbours through weights, so limits add latency without adding safety. Many teams set CPU requests and no CPU limits, keeping limits for untrusted or batch work.

Memory is not compressible. A memory request is used for scheduling and eviction decisions; a memory limit is a hard cgroup ceiling and exceeding it gets the process killed by the kernel with OOMKilled in the pod's last state, exit code 137. The node can have free memory; the limit is per container. Set memory requests from measured usage and make the limit equal to the request for anything that matters.

resources:
  requests:
    cpu: 400m        # measured p99 of per-pod usage, see the worked example
    memory: 512Mi
  limits:
    memory: 512Mi    # equal to the request; no CPU limit

QoS classes, eviction, LimitRange and ResourceQuota

The kubelet assigns a QoS class from the resources alone. Guaranteed: every container has CPU and memory requests equal to limits. Burstable: at least one request or limit is set but the pod is not Guaranteed. BestEffort: nothing set. When a node runs short of memory or disk (the defaults are memory.available<100Mi, nodefs.available<10%, imagefs.available<15%), the kubelet evicts pods: first those whose usage exceeds their requests, ordered by priority and by how far over they are; Guaranteed pods and Burstable pods under their requests go last. A BestEffort pod has a request of zero, so any usage is over it, which is why it goes first.

A LimitRange sets per-container defaults and bounds in a namespace, so a Deployment with no resources still gets a request. A ResourceQuota caps the namespace's total requests, limits, storage and object counts; once a quota exists, every pod must set the quoted resources or it is rejected.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: payments
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi
      max:
        memory: 4Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 120Gi
    persistentvolumeclaims: "20"

Probes and the three ways they cause outages

A startup probe runs first and disables the other two until it passes; give it failureThreshold x periodSeconds of at least the worst-case start time (30 x 10 = 300 s in the docs example). Readiness controls membership in Service endpoints and never restarts anything. Liveness restarts the container.

The three outage patterns. First, liveness stricter than the app: a 1 s timeout on a JVM under GC pauses, so healthy pods are restarted in a loop while traffic lands on fewer and fewer pods. Second, a readiness probe that checks a dependency (the database) during a dependency incident: every pod goes unready at once, the Service has no endpoints and a partial outage becomes total. Third, no readiness probe, or one that returns 200 before the app has loaded its config: a rollout with maxUnavailable: 0 replaces working pods with pods that cannot serve, and the rollout succeeds. Rules: liveness checks only the process, readiness may check what the pod needs to serve, and a failing shared dependency should degrade the pod, not unready it. The bad-deploy-readiness-probes problem is pattern three.

Scheduling controls

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2          # or maxUnavailable: 25%; percentages round up
  selector:
    matchLabels:
      app: api

kubectl drain NODE cordons the node (marks it unschedulable) and evicts every pod through the Eviction API, which honours PDBs: an eviction that would break a budget is refused and drain retries. Deleting a pod directly does not check PDBs. A PDB with minAvailable equal to the replica count can never be satisfied during a drain and blocks every node upgrade: the most common PDB mistake.

Autoscaling at three levels

HPA, VPA and Karpenter side by side: what each reads, what each writes, and a timeline of a 3x traffic spike

The HorizontalPodAutoscaler needs metrics-server for CPU and memory. Every 15 s it computes desiredReplicas = ceil(currentReplicas x currentMetric / targetMetric), ignores changes within a 10 percent tolerance, and scales up at once (at most the larger of 4 pods or 100 percent per 15 s) but scales down using the highest recommendation of the last 300 s (behavior.scaleDown.stabilizationWindowSeconds). Utilisation is a percentage of the request, so a pod without requests cannot be autoscaled on CPU. Custom metrics (requests per second, queue depth) arrive through the Prometheus adapter as Pods or External metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60     # of the CPU request
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60          # remove at most 20 percent per minute

The VerticalPodAutoscaler watches real usage over days and recommends requests. updateMode: "Off" only writes recommendations into the VPA status, which is the safe way to size a service; Recreate restarts pods to apply them; InPlaceOrRecreate resizes in place where the cluster supports it. Never let VPA and HPA both act on CPU for the same pods: one changes the request and the other scales on utilisation of that request.

KEDA is for event-driven scaling. A ScaledObject names the Deployment and a trigger (SQS queue length, Kafka consumer lag, a Prometheus query), and KEDA creates and feeds an HPA with that metric. It also scales to zero when the trigger is idle and back to one when work arrives, which an HPA cannot do.

Cluster autoscaling adds nodes when pods are Pending for lack of resources. The Cluster Autoscaler works on node groups: it picks a group whose instance type fits the pods and raises the Auto Scaling group's desired count, then removes nodes that have been under 50 percent requested for 10 minutes (scale-down-unneeded-time), unless a pod there has a strict PDB, local storage or no controller. Karpenter has no node groups. A NodePool lists the allowed instance families, architectures and capacity types; for each batch of Pending pods it computes the cheapest instance that fits them and launches it directly, usually ready within a couple of minutes. Consolidation (consolidationPolicy: WhenEmptyOrUnderutilized) replaces several underused nodes with fewer or cheaper ones, and disruption.budgets limits how many nodes it may disrupt at once, including a schedule for "none during business hours". For spot, Karpenter handles the interruption notice by draining the node before reclaim; listing spot and on-demand together lets it fall back when spot is unavailable.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h                      # replace nodes after 30 days (AMI refresh)
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: 10%
      - schedule: "0 9 * * mon-fri"
        duration: 9h
        nodes: "0"                           # no consolidation in business hours
  limits:
    cpu: "500"

RBAC, Pod Security Standards and admission control

RBAC has four objects. A Role is a list of rules (API groups, resources, verbs) inside a namespace; a ClusterRole is the same cluster-wide or for non-namespaced resources (nodes, PVs). A RoleBinding gives a Role or ClusterRole to subjects (users, groups, ServiceAccounts) inside one namespace; a ClusterRoleBinding gives a ClusterRole everywhere. Rules are additive only: there is no deny, and a user's permissions are the union of every binding that names them or their groups. Pods act as their ServiceAccount; on EKS, users come from IAM access entries. ClusterRoles can aggregate: a ClusterRole with aggregationRule collects the rules of every ClusterRole carrying a label, which is how the built-in edit and view roles pick up rules for CRDs.

kubectl auth can-i delete secrets -n payments --as dev-priya          # yes or no
kubectl auth can-i --list -n payments --as-group=team-payments        # everything a group may do
kubectl auth can-i get pods --as=system:serviceaccount:ci:deployer    # a ServiceAccount

Pod Security Standards are three levels: privileged (anything), baseline (no host namespaces, no privileged containers, no hostPath), restricted (also non-root, dropped capabilities, seccomp, no privilege escalation). They are applied per namespace with labels, each in a mode: enforce rejects, warn returns a warning, audit records it.

kubectl label ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

Admission webhooks run after authorisation and before the write. A mutating webhook changes the object (inject a sidecar, add labels); a validating webhook accepts or rejects it. Kyverno writes policies as YAML; Gatekeeper uses Rego. Every cluster ends up with three: require CPU and memory requests, forbid the latest tag, require runAsNonRoot. Start each in Audit; switch to Enforce once the report is empty.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-no-latest
spec:
  rules:
    - name: requests-set
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "every container needs CPU and memory requests"
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    cpu: "?*"
                    memory: "?*"
    - name: no-latest-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "use a pinned image tag, not latest"
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Upgrades, node operations, multi-tenancy and backups

Kubernetes releases three minor versions a year and the version skew policy fixes the order: the API server first; controller-manager and scheduler may be one minor older than it; kubelet may be up to three minors older but never newer; kubectl within one minor either way. So an upgrade is control plane, then add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI, each with a compatible version per minor), then nodes. EKS gives each minor 14 months of standard support and then 12 months of extended support at a higher price; at the end of extended support AWS upgrades the control plane for you, without notice, and your nodes and add-ons are then your problem. Check the EKS version page each planning cycle; in November 2026 the standard-support versions are 1.34 to 1.37.

A managed node group upgrade is a surge: it launches new nodes first (the larger of twice the AZ count or maxUnavailable, default 1), then cordons old nodes maxUnavailable at a time, labels them to leave load balancers, drains them, waits for pods to move, and terminates them. Pods that cannot be evicted within 15 minutes fail the upgrade with PodEvictionFailure, which is nearly always a PDB that can never be satisfied or a pod that tolerates every taint. Test the upgrade in staging with the same PDBs, and read the deprecated API list for the target minor (the apiserver_requested_deprecated_apis metric shows who still uses them).

Node problems have signatures. NotReady means the kubelet has not reported for the grace period: read kubectl describe node conditions, then journalctl -u kubelet on the node. MemoryPressure and DiskPressure mean the kubelet is evicting; disk pressure is usually images or container logs, and the kubelet garbage-collects unused images once disk use passes 85 percent (imageGCHighThresholdPercent) down to 80. Pods on a node that stays NotReady are evicted after the default tolerationSeconds: 300 for the unreachable taint.

Multi-tenancy is a choice of boundary. A namespace per team on a shared cluster is cheap and gives RBAC, quota, NetworkPolicy and PSS boundaries, but shares the control plane, the kernel and every add-on. A cluster per environment (dev, staging, prod) is the common split, so an upgrade or a broken add-on in dev cannot touch prod. Split further when a tenant needs a different Kubernetes version or security level, or when one team's API load slows the control plane for all.

Backups are two different things. On EKS, etcd is backed up by AWS and you cannot restore it yourself, so the real backup of cluster state is Git (every object applied by GitOps, lesson 19). Application state is Velero: it stores object manifests in S3 and snapshots PVCs through CSI VolumeSnapshots or copies file data with its node agent, on a schedule (velero schedule create daily --schedule="0 1 * * *"), and restores with velero restore create --from-backup. A backup never restored is not a backup; restore into a scratch namespace monthly.

The production checklist

Sizing a service and its autoscaling for a 3x spike. The api service serves 1,000 requests per second at normal load. A load test of one pod at 200 requests per second shows CPU p99 of 380m, mean 240m, and memory under 420 MiB.

Step 1, requests and limits. CPU request from the p99, rounded: 400m. Memory request and limit from the p99 plus about 20 percent headroom: 420 x 1.2 = 504, rounded to 512Mi. No CPU limit. The pod is Burstable on CPU (no limit) but memory behaves as Guaranteed.

Step 2, replicas at normal load. 1,000 / 200 = 5 pods carry the load at the measured mean of 240m each, which is 240 / 400 = 60 percent of the request.

Step 3, the HPA target. Pick a target that scales out before pods saturate. At 60 percent the HPA is at target and does nothing; with the 10 percent tolerance a scale-out starts when the ratio passes 1.1, that is 66 percent, 264m per pod, 220 requests per second per pod, well before p99 usage reaches the request. Set averageUtilization: 60, minReplicas: 5.

Step 4, the spike. Traffic goes to 3,000 requests per second. With 5 pods that is 600 requests per second per pod, which needs 600 / 200 x 240 = 720m per pod: 180 percent of the request. The HPA computes ceil(5 x 180 / 60) = 15 replicas. The default scale-up policy allows doubling per 15 s, so it goes 5 to 10 at the first tick and 10 to 15 at the second.

Step 5, nodes. Nodes are m6i.xlarge (4 vCPU, 16 GiB) with about 3,500m allocatable after system reserves and DaemonSets. By CPU, floor(3500 / 400) = 8 pods fit per node; by memory 8 x 512Mi = 4 GiB, no constraint. The existing nodes have room for 3 more pods, so 3 of the 10 new pods schedule at once and 7 go Pending. 7 x 400m = 2,800m fits on one node, so Karpenter launches one m6i.xlarge.

Step 6, time. Karpenter sees the Pending pods within seconds and launches the instance; it joins and is Ready typically a minute or two later, so plan for about 120 s. Add 20 s for image pull and readiness: the 7 Pending pods serve at about t = 140 s. Until then 8 pods carry 3,000 requests per second, 375 each, needing 450m each, 112 percent of the request; with no CPU limit they burst into idle CPU, but the nodes do not have 3,600m spare, so p99 latency rises. If that is unacceptable, buy headroom: minReplicas: 8, or an over-provisioning Deployment of low-priority pause pods that real pods preempt instantly.

Step 7, scale-down. When traffic returns to 1,000, 15 pods sit at 20 percent and the recommendation is 5. The 600 s window in the HPA above holds replicas until every recommendation in the last 10 minutes agrees, then the 20 percent per minute policy walks replicas down from 15 to 5 over several minutes instead of in one jump. Karpenter consolidates the empty node after consolidateAfter: 1m.