"Write a Dockerfile for this service" and "what is the difference between a container and a VM" are in almost every DevOps screening round, including the Infosys, Wipro and TCS basics. On the job, image size, build time and image security are the first things a platform team is asked to fix, and "works on my machine" is the bug containers exist to kill. Kubernetes, ECS and every CI system assume you can build a small, non-root, reproducible image; the next six lessons stand on this one.

What a container really is

A container is not a small virtual machine. It is an ordinary process that the kernel shows a different view of the system. Three kernel features do the work:

• Namespaces change what the process can see: its own process tree, network stack, mount table, hostname, users and IPC. Inside the container ps shows your app as PID 1; on the host it is PID 48213.
• cgroups (control groups) limit what it can use: CPU, memory, PIDs, I/O. --memory=512m is a cgroup limit that the kernel's OOM killer enforces.
• A layered filesystem (overlayfs) gives it a root directory assembled from read-only image layers plus one writable layer on top.

Because the container shares the host kernel, it starts in milliseconds and costs almost nothing beyond the process. That is also the security caveat: a kernel bug is a shared bug.

Docker Desktop on a Mac or Windows laptop runs a small Linux VM with the containers inside it, which is why localhost behaves differently there.

Images, layers and the build cache

An image is an ordered list of read-only layers plus a JSON config (entrypoint, env, exposed ports). Each instruction that changes the filesystem (FROM, RUN, COPY, ADD) produces a layer; ENV, WORKDIR, USER, EXPOSE, CMD only change the config. At run time the union filesystem stacks the layers: the process sees one tree, the top layer wins for any path, and a file deleted in a higher layer is hidden by a whiteout while the bytes below still exist. That explains two classics: RUN apt-get install followed by a separate RUN rm -rf /var/lib/apt/lists does not shrink the image (do both in one RUN), and a secret copied in one layer and deleted in the next is still in the image.

The build cache works per layer. BuildKit reuses a layer when the instruction and its inputs (the files a COPY reads, the command string of a RUN) are identical and every layer before it was reused. One miss invalidates everything after it, so order instructions from least to most often changed: base image, system packages, dependency manifest, dependency install, source, build.

Two stacks of image layers showing which layers rebuild after a source edit in a bad and a good Dockerfile order, next to a running container with its writable layer on top of read-only image layers

A production multi-stage Dockerfile

One stage compiles with all the tools; a second, smaller stage copies only the result, so the compiler, dev dependencies and source never reach the final image.

# syntax=docker/dockerfile:1
# ---- build stage: has npm, dev dependencies and the TypeScript compiler ----
# Pinned major and OS. Node 24 is an LTS line (checked October 2026).
FROM node:24-bookworm-slim AS build
WORKDIR /app
# Manifests only, so the install layer below stays cached until a dependency changes.
COPY package.json package-lock.json ./
# Exact versions from the lock file; the cache mount keeps npm's download cache between builds.
RUN --mount=type=cache,target=/root/.npm npm ci
# Source last: edits here never re-run npm ci.
COPY . .
# Compile, then drop dev dependencies from node_modules.
RUN npm run build && npm prune --omit=dev

# ---- runtime stage: only node, the built app and production dependencies ----
FROM node:24-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
COPY --from=build --chown=node:node /app/dist ./dist
COPY --from=build --chown=node:node /app/node_modules ./node_modules
COPY --from=build --chown=node:node /app/package.json ./
# The official image ships this unprivileged user, uid 1000.
USER node
# Documentation only; -p or the orchestrator publishes the port.
EXPOSE 3000
# healthcheck.js is a 10-line script that GETs /healthz on localhost and exits 0 or 1.
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 CMD ["node", "dist/healthcheck.js"]
# Exec form: node is PID 1 and receives SIGTERM directly.
CMD ["node", "dist/server.js"]

The .dockerignore next to it; without it COPY . . drags node_modules, .git and .env into the build context (slow upload, cache misses, leaked secrets):

node_modules
.git
.env
*.log
dist
Dockerfile

In Go the output is one static binary: build in golang:1.27-bookworm with CGO_ENABLED=0 go build -o /server ./cmd/server, then FROM gcr.io/distroless/static-debian13:nonroot and COPY --from=build /server /server: a few megabytes. HEALTHCHECK defaults are interval 30s, timeout 30s, start-period 0s, retries 3; exit code 0 means healthy, 1 unhealthy.

Base images and size

The musl gotcha: Alpine uses musl instead of glibc. Prebuilt native binaries (Node addons like sharp, Python wheels built for glibc, Go programs with CGO) either fail to load or must be compiled against musl. If a build suddenly needs apk add build-base python3 and takes five minutes, slim Debian is the better trade: 50 MB more, no surprises. Sizes above are rough classes for the Node family, not measurements; docker history <image> shows which layer costs what.

Tags, digests and registries

A tag (api:1.4.2) is a mutable pointer; whoever pushes next moves it. A digest (api@sha256:9f86d0...) is the content hash and cannot change. Humans use tags; automation that must be reproducible pins digests: production Kubernetes manifests, base images in regulated builds. The rule that prevents most "it changed under us" incidents: never deploy latest, tag every build with the git SHA, and let the deploy tool resolve the tag to a digest.

A registry stores images. Docker Hub is the public default. Private options: Amazon ECR (IAM auth, scanning, lifecycle rules that delete old tags), GitHub Container Registry (GHCR, authenticated by the Actions token, so CI needs no secret) and GitLab's registry. Pushing to ECR:

REPO=123456789012.dkr.ecr.ap-south-1.amazonaws.com/api
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin "${REPO%/*}"      # token valid 12 hours
docker build -t "$REPO:$(git rev-parse --short HEAD)" .
docker push "$REPO:$(git rev-parse --short HEAD)"
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/example/api:1.4.2 --push .   # multi-arch for Graviton and x86

An ImagePullBackOff in Kubernetes is almost always a tag typo, no pull secret or node role for the registry, or an amd64-only image on an arm64 node.

docker run flags that matter

docker run -d --name api \
  -p 127.0.0.1:3000:3000 \          # host:container; bind to localhost so only Nginx reaches it
  --env-file /etc/app/env \         # secrets from a host file, never baked into the image
  -e NODE_ENV=production \
  -v api-data:/app/data \           # named volume: survives the container
  -v /srv/app/config:/app/config:ro \   # bind mount: a host path, read-only
  --memory=512m --cpus=1.0 \        # cgroup limits; over memory means SIGKILL, exit code 137
  --restart=unless-stopped \        # restart on crash and daemon start, not after docker stop
  --network app-net \               # user-defined bridge: containers resolve each other by name
  --read-only --tmpfs /tmp \        # immutable root filesystem; scratch space in memory
  --user 1000:1000 \                # override the image's user if it forgot USER
  --cap-drop ALL \                  # no Linux capabilities; add back only what the app needs
  api:1.4.2

Restart policies are no (default), on-failure[:max-retries], always and unless-stopped. --init adds a tiny PID 1 that forwards signals and reaps zombies, useful when the entrypoint is a shell script. docker run itself exits 125 when the daemon fails to start the container, 126 when the command cannot be invoked and 127 when it is not found; anything else is the app's exit code.

Networking modes

Each container on a bridge gets its own IP on a private subnet (172.17.0.0/16 by default), and -p 8080:80 is an iptables rule forwarding host port 8080 into it. From a container, the host is host.docker.internal on Docker Desktop; on Linux add --add-host host.docker.internal:host-gateway.

Volumes, bind mounts and logs

The writable layer dies with the container, so anything that must survive goes in a volume. A named volume (-v pgdata:/var/lib/postgresql/data) lives under /var/lib/docker/volumes/ and is right for database files: Docker manages it, it works the same on every OS, and the container's user owns it. A bind mount (-v $(pwd)/src:/app/src) maps a host path and is right for development (edit on the host, the container sees it) and config files. The permission trap: the container process runs as uid 1000 (node) but the host directory is owned by uid 501 on a Mac or root on a server, so writes fail with EACCES. Fix it by matching uids (--user $(id -u):$(id -g)), by chown on the host, or with a named volume. A tmpfs mount (--tmpfs /tmp) keeps scratch data in memory.

Logs: a container writes to stdout and stderr, nothing else. The default json-file driver stores them under /var/lib/docker/containers/, growing without limit unless you set --log-opt max-size=10m --log-opt max-file=3 or the same in /etc/docker/daemon.json. Full disks from container logs are a common VM outage. Looking inside:

docker logs -f --since 10m api                 # stdout and stderr of the main process
docker exec -it api sh                         # a shell inside (not on distroless; use the :debug image)
docker inspect api --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.RestartCount}}'
docker stats --no-stream                       # live CPU, memory and I/O per container
docker system df                               # disk used by images, containers, volumes, build cache

Docker Compose for a local multi-service stack

Compose describes several containers, their network and volumes in one file and starts them with docker compose up -d. Services share a user-defined network, so the app reaches the database at the hostname postgres.

# compose.yaml
services:
  api:
    build:
      context: .
      target: build            # the build stage, with dev dependencies, for local work
    command: npm run dev
    ports:
      - "127.0.0.1:3000:3000"
    env_file: .env
    environment:
      DATABASE_URL: postgres://app:app@postgres:5432/app
      REDIS_URL: redis://redis:6379
    volumes:
      - ./src:/app/src         # bind mount: edits on the host reload in the container
    depends_on:
      postgres:
        condition: service_healthy     # wait for the health check, not only for the process
      redis:
        condition: service_started
    restart: unless-stopped

  postgres:
    image: postgres:18-bookworm
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app           # local only; production uses a secrets manager
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data    # named volume: the database survives docker compose down
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 10s

  redis:
    image: redis:8-bookworm
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    volumes:
      - redisdata:/data

volumes:
  pgdata:
  redisdata:

depends_on with condition: service_healthy fixes the classic first-run failure where the app starts before Postgres accepts connections. docker compose down -v also deletes the named volumes: a clean database. docker compose logs -f api and docker compose exec api sh mirror the single-container commands.

Deploying a monorepo to VMs with Docker, and its limits

The 100xDevs week 27 pattern for a monorepo (web, api, worker in one repo): one Dockerfile per service with the repo root as build context; CI builds all three images on every merge to main, tags them with the git SHA and pushes to ECR or GHCR; each VM has a compose.yaml referencing the images by tag; the deploy step is ssh vm "export TAG=abc123 && docker compose pull && docker compose up -d". Nginx on the VM or an ALB routes traffic. For a small team on two or three VMs it is the right amount of machinery.

Its limits are what Kubernetes is for. Scaling means editing a Compose file on every VM by hand. A VM failure takes its containers with it and nothing reschedules them. Rolling updates with health gating are a shell script you now own. Secrets are files per VM. Service discovery across VMs does not exist, so every cross-VM address is hard-coded. When two of these hurt, you have rebuilt a worse scheduler, and the Kubernetes lessons start here.

Container security basics

• Non-root: USER node in the image, or --user at run time. A root process that escapes is root on the host, and it can write to every bind mount.
• Read-only root filesystem: --read-only --tmpfs /tmp; an attacker with code execution cannot drop a binary.
• No secrets in the image: not in ENV or ARG (visible in docker history), not in a copied .env. Pass them at run time with --env-file, Compose secrets or a secrets manager; at build time use RUN --mount=type=secret.
• Drop capabilities (--cap-drop ALL) and never --privileged for an app.
• Scan: trivy image --severity HIGH,CRITICAL --exit-code 1 api:1.4.2 in CI fails the build on known vulnerabilities; --ignore-unfixed hides those without a fix yet. Small bases mean fewer findings, which is the real argument for distroless.
• Pin and sign: pin base images and production digests, sign with cosign, verify at admission. The supply-chain problem in the related list goes further.

Troubleshooting: the patterns

containerd, the OCI standards and Kubernetes

Docker is three things: a CLI, a daemon (dockerd) with the build and run API, and underneath it containerd, the runtime that pulls images, manages layers and starts containers through runc, which does the namespace and cgroup setup. The Open Container Initiative (OCI) standardised the image format, the runtime spec and the distribution (registry) API, so these parts can be swapped: an image built with Docker, Buildah or ko is an OCI image and any OCI runtime can run it.

Kubernetes talks to a runtime through the Container Runtime Interface (CRI). containerd implements CRI directly, so nodes run containerd and no Docker daemon; the Docker shim was removed from the kubelet in 1.24. On an EKS node you use crictl ps and crictl logs, not docker ps. The split: Docker (or Podman) is the developer tool for building and running locally, containerd is the production runtime, and the image is the contract between them.

Build time for a Node monorepo service, bad order versus good order. The api service has a 180 MB node_modules from npm ci, which takes 90 s on the CI runner; npm run build takes 40 s; copying the source into the image takes 2 s. The team merges 22 pull requests a day and CI builds the image on every merge. Most merges touch only source; about one in ten changes a dependency.

Step 1, the bad Dockerfile: COPY . . then RUN npm ci then RUN npm run build. Any source change invalidates COPY . ., and every layer after it is rebuilt. Cost per source-only merge: 2 + 90 + 40 = 132 s. Dependency merges cost the same 132 s. Daily: 22 x 132 = 2,904 s, about 48 minutes of build time.

Step 2, the good Dockerfile: COPY package.json package-lock.json ./, RUN npm ci, COPY . ., RUN npm run build. A source-only merge leaves the manifest layer and the npm ci layer cached; it rebuilds COPY . . (1 s, fewer files than the manifest-plus-source copy of the bad order) and the build (40 s): 1 + 2 + 40 = 43 s, counting the 2 s for the cached checks and copy. A dependency merge still rebuilds from the manifest layer: about 133 s. Daily: 20 source merges x 43 s + 2 dependency merges x 133 s = 860 + 266 = 1,126 s, about 19 minutes. Saving: 29 minutes of runner time a day, and each developer waits 43 s instead of 132 s for the build step.

Step 3, the cache must survive between builds. On a laptop the local BuildKit cache does this by default. On a fresh CI runner every build starts cold, so the good order alone gains nothing; you also need a remote cache: docker buildx build --cache-from type=registry,ref=ghcr.io/example/api:buildcache --cache-to type=registry,ref=ghcr.io/example/api:buildcache,mode=max. With it, the runner downloads the cached npm ci layer (180 MB, roughly 10 s on a fast link) instead of running the install (90 s).

Step 4, size. Both orders produce the same final image, because order changes cache hits, not content. Size comes from the multi-stage split: the build stage holds node_modules with dev dependencies (say 420 MB) and the TypeScript source; the runtime stage copies only dist/ and the pruned production node_modules (180 MB) onto node:24-bookworm-slim (about 200 MB). Final image about 380 MB instead of about 1.2 GB for a single-stage build on the full node:24-bookworm base, and the pull on each new node is three times faster. The numbers here are assumptions for the arithmetic; measure yours with docker history.