A small team runs an API on three EC2 virtual machines with no Kubernetes. Each release is a tarball in S3 with a sha256 file next to it. Today the deploy is a person typing scp and mv by hand, and last month two engineers deployed at the same time and left a box half-updated. Write deploy.sh <version> <env> in bash that downloads and verifies the release, switches an atomic current symlink, health-checks the service with retries, rolls back on failure, refuses to run concurrently, cleans up on any exit, logs with timestamps and keeps only the last 5 releases.

Usage: ./deploy.sh <version> <prod|staging> [--dry-run]
       ./deploy.sh                 # self-test in a temp directory, prints check lines

Environment overrides (defaults in brackets):
  APP_ROOT       [/srv/myapp]            releases/, current, shared/ live here
  ARTIFACT_DIR   [/var/artifacts/myapp]  stands in for "aws s3 cp"
  KEEP_RELEASES  [5]   HEALTH_RETRIES [3]   HEALTH_SLEEP [1]

Functions (one responsibility each):
  acquire_lock          mkdir-based lock, breaks a lock whose pid is dead
  fetch_artifact VER    copy tarball + .sha256 to a temp dir, sha256sum -c
  install_release VER   untar into releases/VER.partial, then mv to releases/VER
  switch_current VER    ln -sfn into current.new, mv -Tf current.new current
  health_check          3 attempts, reads current/health ("ok" = HTTP 200)
  prune_releases N      keep the newest N by version, never the current one
  cleanup               trap EXIT: rm temp dir and lock, keep the exit code

Exit codes: 0 ok or no-op, 1 unexpected error, 2 usage, 3 lock held,
            4 health check failed and rolled back, 5 artifact missing or bad checksum

Artifact store (S3 in real life, a directory here):
  myapp-1.1.0.tar.gz            contents: app.sh, health, VERSION
  myapp-1.1.0.tar.gz.sha256     "<64 hex chars>  myapp-1.1.0.tar.gz" (sha256sum format)

Layout on the VM, APP_ROOT=/srv/myapp:
  .deploy.lock/pid              exists only while a deploy runs; holds the pid
  releases/1.0.0/               never modified after extraction
  releases/1.1.0/ENVIRONMENT    written at install: "prod" or "staging"
  current -> releases/1.1.0     relative symlink; systemd unit runs /srv/myapp/current/app.sh
  shared/                       logs and config that survive releases

Health contract: current/health contains "ok" when the service is healthy.
In production this is replaced by: curl -fsS --max-time 2 localhost:8080/healthz

Log line: 2026-11-03T10:41:07Z [INFO] current -> releases/1.1.0

1. Parse arguments into VERSION, ENV_NAME, DRY_RUN. Check the version against ^[0-9]+\.[0-9]+\.[0-9]+$ so that nothing like ../.. can become part of a path later.
2. Install the traps before anything is created: trap cleanup EXIT and trap 'on_error $LINENO $?' ERR. From here on, every exit path (normal return, exit N, a set -e failure, SIGINT, SIGTERM) runs cleanup.
3. Take the lock with mkdir "$APP_ROOT/.deploy.lock". mkdir is a single atomic system call that either creates the directory or fails because it exists, so two processes cannot both succeed. Write the pid inside. If the directory exists and its pid is dead (kill -0 fails), remove it and try once more; otherwise exit 3.
4. Read the previous version with readlink current. If it equals the requested version, log "nothing to do" and exit 0: the script is idempotent.
5. Fetch: copy myapp-VER.tar.gz and its .sha256 into mktemp -d, then sha256sum -c --quiet. A mismatch is exit 5 before any file under releases/ is touched. With --dry-run stop here and report what would happen.
6. Install: extract into releases/VER.partial, write ENVIRONMENT, then mv the directory to releases/VER. A crash mid-extract leaves a .partial directory that the next run deletes; it never leaves a half-filled releases/VER.
7. Switch: ln -sfn releases/VER current.new then mv -Tf current.new current. The mv is rename(2), which replaces the old link in one step.
8. Health-check up to HEALTH_RETRIES times with HEALTH_SLEEP between them. On success prune and exit 0. On failure switch current back to the previous version, re-run the health check so the log says whether the rollback worked, remove the bad release directory, exit 4.
9. Prune: list releases/ sorted by version descending (sort -rV), skip the current one, delete everything after the first KEEP_RELEASES - 1 others.
10. Run with no arguments: build a fake artifact store with versions 1.0.0 through 2.3.0 (1.2.0 unhealthy, 1.3.0 tampered), run the script as a child for each scenario, print check ok: lines and a pass/fail count.

Why the symlink switch must be a rename

The obvious code is rm current; ln -s releases/1.1.0 current. Between those two commands there is a window, microseconds on an idle box and much longer under load, in which current does not exist: a request that arrives then fails, and a systemd restart in that window cannot find the binary. ln -sfn releases/1.1.0 current is not guaranteed to be atomic either: POSIX allows ln -f to unlink the old name and then create the new one, two system calls, and busybox and the BSD ln do exactly that (current GNU coreutils happens to create a temporary link and rename it, which strace shows, but a deploy script should not depend on which ln is installed). The correct pattern is: create the link under a temporary name, then mv -Tf current.new current. mv on one file system is a single rename(2) call, and POSIX requires that it replaces the old name atomically, so any lookup of current sees the old target or the new one, never nothing. -T tells GNU mv to treat the destination as a file; without it mv would move current.new inside the directory current points to. A relative target keeps the tree movable. One more detail: a process that already opened files from the old release keeps using them, because open descriptors follow inodes, not names. New code runs only after the service restarts, which is why the health check comes after the switch and the restart.

Locking: mkdir versus flock, and the stale-lock problem

Two deploys at once leave current pointing at whichever finished last and an interleaved log. A lock must be one atomic test-and-set. mkdir is one on every POSIX file system, including NFS, which is why the script uses it. The alternative is flock: exec 9>/var/lock/deploy.lock; flock -n 9 || exit 3. The kernel holds that lock on the open descriptor and drops it when the process dies, so a crashed deploy never leaves a stale lock. Its cost: flock is a util-linux command, not POSIX, and advisory locks are unreliable on some network file systems. With mkdir, a process killed by SIGKILL (which cannot be trapped) leaves the directory behind, so the script stores the pid and checks kill -0 "$pid": signal 0 delivers nothing but reports whether the process exists. Pids are reused after a reboot, so a production script would also compare the process start time, or require a human to delete the lock. Name both options and the trade-off.

Traps, exit codes and what set -e really does

set -e exits on a failing simple command, but not inside if conditions, && or || chains, or a function called from one. set -u turns an unset variable into an error, which catches a typo in $APP_ROOT before it becomes rm -rf /releases. pipefail makes a pipeline fail when any stage fails. The EXIT trap runs for a normal return, for exit 5, for a set -e failure and after SIGINT or SIGTERM, so cleanup belongs there. Two rules make it safe: read $? in the trap's first statement and exit "$rc" at its end, so the caller sees the real code; and only remove the lock when this process created it (HAVE_LOCK=1), otherwise a run that exited 3 would delete another deploy's lock on its way out. The ERR trap only logs $LINENO, which saves minutes at 2 a.m.

Idempotency and dry runs

Deploy tools get run twice, by nervous people and by retrying automation. The script answers in three places: the version already current is a no-op; an existing release directory is reused; the .partial directory is removed before a retry. --dry-run goes as far as the checksum, the step most likely to fail for reasons unrelated to this box, then stops before writing under releases/. Until the prune step everything is reversible (the symlink) or additive (a new directory), and prune only removes releases that are neither current nor among the newest four others.

Tests. Run ./deploy.sh with no arguments. The self-test checks, in order: 1.0.0 deploys and current points at it; 1.1.0 deploys; a repeat of 1.1.0 exits 0 without changes; --dry-run of 2.0.0 leaves current and releases/ untouched; 1.2.0, whose health file says db connection refused, is rolled back to 1.1.0 with exit 4 and its directory removed; a tampered 1.3.0 exits 5 and changes nothing; a lock held by a live sleep process makes the next deploy exit 3; the same lock with a dead pid is broken and the deploy proceeds; the lock directory is gone afterwards; missing arguments and v2 exit 2; after 2.1.0, 2.2.0 and 2.3.0 exactly five releases remain and 1.0.0 was pruned; the app root contains only current, releases and shared. Fourteen checks, under 1 second.

Edge cases. First deploy on an empty box (no previous release: a health failure exits 4 with a clear message and no rollback target). Version strings with path characters are rejected before use. A .partial directory from a killed run is cleaned on the next attempt. sort -rV orders 2.10.0 after 2.9.0, which plain sort would not.

What a strong answer adds. Names rename(2) and -T; explains the difference between mkdir and flock locks and why a stale lock needs a pid check; reads $? first in the trap; separates "switch" from "restart" and knows open file descriptors keep the old binary alive; keeps the failed release for debugging in a real system; and says what changes for three VMs (run on one host at a time, stop at the first failure, or move to a rolling deploy with the load balancer draining each host).

• Two deploys overlap. Each thinks it owns current and the box ends in a mixed state. Mitigation: the mkdir lock; the second run exits 3 without touching anything.
• Lock left by a SIGKILLed run. Every later deploy exits 3 forever. Mitigation: store the pid, break the lock when kill -0 says the process is gone, and log that it happened.
• Half-extracted release becomes current. A crash mid-tar plus a later run that saw the directory and reused it. Mitigation: extract into VER.partial and mv only when complete.
• Rollback target is also unhealthy. The previous release depends on something that broke since. Mitigation: re-run the health check after rollback and log an explicit "page a human" line; exit non-zero either way.
• Checksum file missing or stale. The deploy would proceed on an unverified artifact. Mitigation: sha256sum -c on a missing file fails, and set -e plus the || guard turn that into exit 5.
• Prune deletes the running release. Happens when the prune list is sorted by mtime and a reused old directory looks fresh. Mitigation: skip the version readlink current reports, and sort by version, not mtime.
• Trap masks the real exit code. A command inside cleanup succeeds and the script exits 0 after a failed deploy. Mitigation: capture $? in the first line of the trap and exit "$rc" at its end.