Most Kubernetes incidents are networking or storage incidents wearing a different name: a 502 from the ingress, a pod that cannot reach a Service, DNS lookups that fail one time in fifty, a PVC that stays Pending because the only free node is in the wrong zone. Interviewers test this layer because it separates people who can deploy to Kubernetes from people who can run it. On EKS the questions get specific: why nodes run out of pod IPs, why an EBS volume pins a pod to one availability zone, and when you should not run the database in the cluster at all.

The networking model and the CNI plugin

Kubernetes makes three promises about the network. Every pod gets its own IP address. Every pod can talk to every other pod, on any node, using that IP, with no NAT in between. Agents on a node can talk to every pod on it. A CNI plugin (Container Network Interface) delivers them: when kubelet creates a pod, it calls the plugin, which creates the pod's network interface, assigns the IP and programs whatever routes or tunnels make the three promises true.

The VPC CNI pod limit is the number interviewers ask for: (number of ENIs x (IPs per ENI - 1)) + 2. An m5.large has 3 ENIs and 10 IPs per ENI, so 3 x 9 + 2 = 29 pods. Prefix delegation (ENABLE_PREFIX_DELEGATION=true) assigns a /28 prefix (16 addresses) to each ENI slot instead of one IP, which raises the limit far above 29; EKS caps managed node groups at 110 pods for instances with fewer than 30 vCPUs. Each /28 must be a contiguous free block in the subnet, so a fragmented subnet gives InsufficientCidrBlocks errors. When the VPC runs out of IPv4, the options are custom networking with a secondary CIDR (pods in 100.64.0.0/10) or an IPv6 cluster, which AWS recommends.

Services under the hood

A ClusterIP is a virtual IP: no interface anywhere has it. The API server allocates it from the Service CIDR.

The EndpointSlice controller watches pods matching the Service selector and writes their IPs into EndpointSlice objects (up to 100 endpoints per slice by default), labelled kubernetes.io/service-name. Each endpoint carries conditions: serving (the pod passes readiness), terminating (it has a deletion timestamp) and ready (serving and not terminating). A pod that fails readiness drops out of the slice within one sync; the Service object never changes.

kube-proxy on every node watches EndpointSlices and programs the kernel so that packets to 10.100.21.7:80 are rewritten to one of the ready pod IPs. In iptables mode (the Linux default today, with nftables planned as the future default) this is a chain of rules with random selection; with tens of thousands of endpoints the rule updates get slow. ipvs mode uses a kernel hash table and offers schedulers such as rr, lc and mh but is marked deprecated. Cilium replaces kube-proxy entirely: its eBPF socket load balancer rewrites the destination at connect() time, so the packet leaves the pod already addressed to the backend.

An HTTP request travelling from a client through Route 53, an ALB, and node-level Service rules to pods with VPC IPs on two nodes, with the EndpointSlice feeding both the ALB targets and the node rules

• sessionAffinity: ClientIP pins a client IP to one pod for sessionAffinityConfig.clientIP.timeoutSeconds (default 10800, three hours). Cookies need an ingress proxy.
• externalTrafficPolicy: Local on a NodePort or LoadBalancer Service sends traffic only to pods on the receiving node and preserves the client source IP; nodes without a pod fail the health check.
• trafficDistribution: PreferSameZone asks kube-proxy to prefer endpoints in the caller's zone, which cuts cross-AZ data transfer charges.
• A headless Service (clusterIP: None) gets no virtual IP; DNS returns every pod IP, and kube-proxy ignores it. StatefulSets need one.

DNS: how a name becomes an address

CoreDNS runs as a Deployment in kube-system behind the Service kube-dns. Every Service gets name.namespace.svc.cluster.local (an A or AAAA record for ClusterIP Services, the pod IPs for headless ones), and every port gets an SRV record. A pod in namespace payments gets this /etc/resolv.conf with the default dnsPolicy: ClusterFirst:

nameserver 10.100.0.10
search payments.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The ndots:5 line causes a famous class of incidents. The resolver counts the dots in the name. If there are fewer than 5, it tries every search suffix first and the bare name last. api becomes api.payments.svc.cluster.local on the first try: one query. db.example.com has 2 dots, so the resolver tries db.example.com.payments.svc.cluster.local, then .svc.cluster.local, then .cluster.local, gets NXDOMAIN three times, and only then asks for db.example.com: four queries, each doubled into A and AAAA. On EKS the node's VPC suffix (ec2.internal or <region>.compute.internal) is a fourth search entry, so one external lookup is five names, ten packets when the resolver also asks AAAA for each. A trailing dot (db.example.com.) marks the name absolute and skips the search list. The fixes: fully qualified names with a trailing dot for external hosts, or a lower ndots for that pod:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

The CoreDNS config is a ConfigMap called coredns holding a Corefile. The default server block is .:53 { errors; health; ready; kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure; fallthrough in-addr.arpa ip6.arpa; ttl 30 }; prometheus :9153; forward . /etc/resolv.conf; cache 30; loop; reload; loadbalance }. The kubernetes plugin answers cluster names, forward sends everything else to the node's resolvers (on EKS, the VPC resolver), cache 30 caches for 30 seconds. A second block such as corp.internal:53 { forward . 10.20.0.2 } sends one domain elsewhere.

NodeLocal DNSCache runs a caching agent as a DaemonSet on every node. Pods keep asking the same kube-dns IP, but the node agent answers from its cache and talks to CoreDNS over TCP. This removes the cross-node hop and a UDP conntrack race that causes the "one lookup in a thousand times out after 5 seconds" symptom (the intermittent-dns-failures problem).

Exposing HTTP: Ingress, Gateway API and the AWS Load Balancer Controller

An Ingress says "for host X and path Y, send to Service Z". It does nothing by itself; an ingress controller, chosen with ingressClassName, watches Ingress objects and configures a proxy.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: shop
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing   # default is internal
    alb.ingress.kubernetes.io/target-type: ip           # pods as targets; default is instance
    alb.ingress.kubernetes.io/group.name: shop-public   # Ingresses with this name share one ALB
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
  tls:
    - hosts: [shop.example.com]  # the ALB controller finds a matching ACM cert from these hosts
      secretName: shop-tls       # used by in-cluster proxies only: tls.crt and tls.key, filled by cert-manager
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix     # Exact, Prefix or ImplementationSpecific
            backend:
              service:
                name: web
                port:
                  number: 80

Two facts changed the answer in 2026. The Kubernetes docs now state that the Ingress API is frozen and recommend Gateway API instead. And ingress-nginx, the controller most tutorials use, was retired by the Kubernetes project: best-effort maintenance ended in March 2026 with no security fixes after that. Existing installs keep working, but do not start a new platform on it. On EKS the common choices are the AWS Load Balancer Controller (an ALB per Ingress, or one ALB shared by every Ingress with the same group.name; target-type: ip registers pod IPs directly, instance goes through a NodePort and needs a NodePort Service) or an in-cluster proxy behind an NLB (Envoy Gateway, Cilium, Traefik, NGINX Gateway Fabric). TLS certificates come from ACM when the ALB terminates TLS (certificate-arn annotation) or from cert-manager when the proxy does: a ClusterIssuer for Let's Encrypt plus the annotation cert-manager.io/cluster-issuer: letsencrypt, and the Secret is created and renewed for you.

Gateway API is the replacement, and its core resources (GatewayClass, Gateway, HTTPRoute) have been GA (v1) in the Standard channel since release v1.0. It splits the job by role: the infrastructure provider supplies a GatewayClass (which controller), the cluster operator creates a Gateway (listeners, ports, TLS, which namespaces may attach through allowedRoutes), and application teams create HTTPRoute objects in their own namespaces. Things that needed vendor annotations on Ingress are typed fields: header matches, weighted backendRefs for canaries, RequestHeaderModifier, RequestRedirect and URLRewrite filters. The AWS Load Balancer Controller implements it too (HTTPRoute on ALB, TCP and UDP routes on NLB), and ingress2gateway converts existing Ingress objects.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: public-gw          # a Gateway owned by the platform team
      namespace: infra
  hostnames: [shop.example.com]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: api
          port: 80
          weight: 90
        - name: api-canary
          port: 80
          weight: 10

NetworkPolicy: who may talk to whom

By default every pod accepts traffic from every pod in the cluster and can reach anything. A NetworkPolicy selects pods and lists what may reach them (ingress) and what they may reach (egress). It only works if the CNI enforces it (VPC CNI with network policy enabled, Cilium, Calico); without that the object is accepted and ignored. Rules in one from list item with both namespaceSelector and podSelector are ANDed; separate items are ORed. The EKS best practices guide says to start every namespace with deny-all and add rules one at a time.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}                 # every pod in the namespace
  policyTypes: [Ingress, Egress]  # isolate both directions
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: checkout   # callers in one namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: infra
          podSelector:
            matchLabels:
              app.kubernetes.io/name: envoy-gateway   # the ingress proxy, both labels required
      ports:
        - protocol: TCP
          port: 8080

An egress rule has the same shape with egress and to; for a database outside the cluster use ipBlock: {cidr: 10.50.0.0/16} with port 5432.

People forget to pair deny-all with allow-dns: without it nothing resolves and every call times out instead of being refused. With the ALB in ip mode the "ingress controller" is the ALB itself, so the allow rule is an ipBlock for the ALB subnets.

A service mesh (Istio, Linkerd, Cilium's mesh mode) adds mutual TLS, retries, timeouts and per-request metrics between services through a proxy next to each pod or in the node's eBPF datapath. Lesson 29 covers when it is worth the complexity.

Storage: from emptyDir to EBS

A PersistentVolume (PV) is a piece of storage in the cluster; a PersistentVolumeClaim (PVC) is a request for storage by size and access mode that a pod mounts; a StorageClass says how to make a PV when a PVC asks for one (dynamic provisioning), through a CSI driver. On EKS the EBS CSI driver (ebs.csi.aws.com) is the usual one.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3                     # the driver default; iops and throughput are tunable
  encrypted: "true"
  csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Delete           # Retain keeps the EBS volume after the PVC is deleted
allowVolumeExpansion: true      # edit spec.resources.requests.storage on the PVC to grow it
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data
  namespace: payments
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3-encrypted
  resources:
    requests:
      storage: 100Gi

Access modes. ReadWriteOnce (RWO) means one node at a time, which is what a block device like EBS can do; many pods on that node can share it, and ReadWriteOncePod restricts it to one pod. ReadWriteMany (RWX) means many nodes at once, which needs a shared file system: on AWS that is EFS through the EFS CSI driver, slower and priced per GB stored, but the only answer when three replicas must share uploads. A PVC asking for RWX from an EBS StorageClass stays Pending forever.

Zones. An EBS volume exists in one availability zone and can only attach to instances in that zone. With volumeBindingMode: Immediate the volume would be created when the PVC is created, in whatever zone the provisioner picks, and the pod would then be forced into that zone, possibly one with no free node. WaitForFirstConsumer delays provisioning until the scheduler has placed the pod, then creates the volume in that zone. From then on the pod is pinned: if the zone's nodes are full, the pod is Pending with "volume node affinity conflict". Snapshots (VolumeSnapshot in snapshot.storage.k8s.io/v1, from a PVC; restore by creating a PVC with dataSource pointing at the snapshot) are how you move data across zones. Expansion is online for EBS; shrinking is not supported. reclaimPolicy decides what happens to the volume when the PVC is deleted: Delete (the default for dynamic PVs) removes the disk, Retain leaves it as a Released PV for a human to deal with.

StatefulSets and the database question

A StatefulSet is for pods that need identity: stable names (pg-0, pg-1), a stable DNS entry each (pg-0.pg.payments.svc.cluster.local through the headless Service named in serviceName), their own PVC each from volumeClaimTemplates, and ordered operations: pod N is created only after pod N-1 is Running and Ready, rollouts go from the highest ordinal down, and scale-down removes the highest first. PVCs are kept when pods are deleted or the set is scaled down, so pg-1 gets its old disk back (persistentVolumeClaimRetentionPolicy can change that). podManagementPolicy: Parallel turns off the ordering.

Run a database in Kubernetes when an operator handles failover, backups and upgrades (CloudNativePG for PostgreSQL, Strimzi for Kafka), when the team will own those procedures, and when the workload is many small databases or dev environments. Use RDS or Aurora when the database is the business-critical one or the team is small. The cost difference is real but smaller than the operational difference; the stateful-backup-restore problem puts numbers on it.

Symptom to layer to command

Sizing pod IPs for an EKS node group and seeing why prefix delegation matters. A team plans a node group of m5.large instances for 300 pods of a web application plus the usual DaemonSets (VPC CNI, kube-proxy, a log agent, a metrics agent: 4 per node).

Step 1, the limit without prefix delegation. An m5.large supports 3 ENIs with 10 IPv4 addresses each. The EKS formula is (3 x (10 - 1)) + 2 = 29 pods per node. Two of those are the CNI and kube-proxy, which use the node's own IP, so 27 pods get pod IPs, and 2 of those 27 are the log and metrics agents. That leaves 25 application pods per node, and 300 / 25 = 12 nodes.

Step 2, what those nodes actually carry. An m5.large has 2 vCPU and 8 GiB. If each web pod requests 100m CPU and 128 MiB, 25 pods need 2.5 vCPU and 3.1 GiB. The CPU requests already exceed what the node has, so IPs are not the real limit here; CPU is. Pick 20 pods per node for CPU, so 15 nodes, and the IP cap of 29 never binds.

Step 3, change the pod to a small worker. Now each pod requests 50m CPU and 64 MiB. 25 pods need 1.25 vCPU and 1.6 GiB, so the node is less than half used and the IP cap of 29 is what stops us. 12 nodes run at about 60 percent CPU and 20 percent memory, paid for in full. This is the case for prefix delegation.

Step 4, with prefix delegation. Each ENI slot now holds a /28 (16 addresses), so one ENI slot gives what used to need 16 slots. The node is no longer IP-limited and EKS allows up to 110 pods on this instance size. With 110 pods per node, 300 pods need ceil(300 / 110) = 3 nodes for IPs, but CPU would be 110 x 50m = 5.5 vCPU on a 2 vCPU node, so pick about 36 pods per node (36 x 50m = 1.8 vCPU), which is ceil(300 / 36) = 9 nodes instead of 12. The saving is 3 of 12 nodes, 25 percent of the node group, at the cost of needing contiguous /28 blocks in each subnet.

Step 5, the subnet. 9 nodes, each eventually holding up to 3 ENIs, each ENI with a handful of /28 prefixes. If the CNI keeps one spare prefix per ENI (WARM_PREFIX_TARGET, set to 1 in the add-on and Helm defaults), a node with 36 pods holds 3 prefixes in use plus 1 spare, 64 addresses, and 9 nodes consume 576 addresses plus 9 node IPs. A /24 subnet (251 usable addresses after the 5 AWS reserves) per zone is too small; a /22 (1019 usable) per zone leaves room for the node group to double. Subnet size is the decision people get wrong first and fix last, because changing it means new subnets and new nodes.