Live bash exercises are common in DevOps screens: "count the top 10 IPs in this access log", "write a script that backs up a directory and fails loudly", "what does set -e do". Reviewers at product companies read your CI YAML and your Dockerfiles, which are bash underneath, and they notice unquoted variables. On call, the difference between five minutes and fifty is whether grep | awk | sort | uniq -c comes to your fingers. The Bandit wargame and the Missing Semester lectures exist because this is the skill most graduates arrive without.

The shell model: streams, exit codes, pipes

Every process starts with three open file descriptors (fds): 0 stdin (input), 1 stdout (normal output), 2 stderr (errors and diagnostics). The shell wires them for you. When a process exits it returns a number from 0 to 255, the exit code; 0 means success and anything else means failure. The shell stores the last one in $?.

Three processes joined by two pipes, each with stdin, stdout and stderr file descriptors, stderr going to the terminal, the kernel pipe buffers between them, and the three exit codes feeding PIPESTATUS and pipefail

A pipe a | b starts both processes at the same time and connects a's fd 1 to b's fd 0 through a kernel buffer (64 KiB on Linux). When the buffer is full, a blocks until b reads; that is back-pressure, and it is why head can stop a huge cat early. stderr is not in the pipe: it goes straight to the terminal unless you redirect it.

cmd > out.txt            # stdout to a file (truncate); >> appends
cmd 2> err.txt           # stderr to a file
cmd > all.txt 2>&1       # both to one file: redirect stdout first, then point fd 2 at fd 1
cmd 2>&1 > all.txt       # WRONG order: stderr goes to the old stdout (the terminal)
cmd 2>/dev/null          # drop errors
cmd < in.txt             # stdin from a file
cmd | tee log.txt        # stdout to the terminal and a file at once
cmd 2>&1 | tee log.txt   # the same, with errors included
find . -name '*.log' -print0 | xargs -0 gzip     # xargs turns lines into arguments; -0 survives spaces
seq 1 100 | xargs -n 10 -P 4 echo               # 10 args per call, 4 calls in parallel
a && b                   # run b only if a succeeded (exit 0)
a || b                   # run b only if a failed
a; b                     # run both regardless

Exit codes carry meaning by convention: 1 general error, 2 misuse of a builtin or bad arguments, 126 found but not executable, 127 command not found, 128 + N killed by signal N (so 137 is SIGKILL, 143 is SIGTERM). CI systems, systemd and Kubernetes all read exactly this number.

The text toolkit, one real example each

The operator's reflex is a chain. Top 10 client IPs from an Nginx access log, where the IP is the first field:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

Read it as a sentence: take the first column, sort so equal lines are adjacent, count each run, sort by count descending, keep ten. Every step is a process and the whole thing streams, so it works on a 10 GB file without loading it.

jq deserves ten minutes of practice because every cloud CLI speaks JSON. aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name, .PrivateIpAddress] | @tsv' turns a page of JSON into a table you can pipe into the tools above. -r prints raw strings without quotes; @tsv joins an array with tabs.

Bash scripting: variables and quoting

Bash has one data type, the string, and one source of most bugs, word splitting. After a variable expands, the shell splits the result on spaces, tabs and newlines and expands globs in each piece, unless the expansion is inside double quotes.

file="my report.txt"
rm $file          # runs: rm my report.txt  -> two arguments, deletes the wrong things
rm "$file"        # runs: rm "my report.txt" -> one argument, correct
echo "$HOME/bin"  # expands inside double quotes
echo '$HOME/bin'  # single quotes: nothing expands, literal text
dir="$(dirname "$file")"            # $(...) for command output; never backticks
n=$(( 7 * 6 ))                      # arithmetic
name="${1:-world}"                  # default when $1 is unset or empty
: "${API_URL:?set API_URL first}"   # abort with a message if unset
echo "${path##*/}"                  # strip the longest prefix matching */ -> the basename
echo "${file%.txt}"                 # strip the suffix

The rule is short: quote every expansion ("$var", "$(cmd)", "$@", "${arr[@]}") unless you can explain why you want splitting. $@ and $* differ exactly here: "$@" expands to each argument as its own word, spaces preserved, which is what you pass to another command; "$*" joins them into one string, which is only for printing.

Conditionals, loops, functions, arrays

if [[ -f "$cfg" && -r "$cfg" ]]; then echo "readable file"; fi    # [[ ]] is bash's safer test
if [[ "$env" == prod* ]]; then echo "glob match"; fi
if [[ "$n" =~ ^[0-9]+$ ]]; then echo "regex match"; fi
if (( n > 10 )); then echo "arithmetic"; fi
case "$1" in start|stop) echo ok ;; *) echo "usage: $0 start|stop" >&2; exit 2 ;; esac

for f in /var/log/*.log; do echo "$f"; done          # glob loop; never for f in $(ls)
for ((i = 0; i < 5; i++)); do echo "$i"; done
while IFS= read -r line; do echo "got: $line"; done < input.txt    # read lines safely
while ! curl -sf localhost:8080/healthz; do sleep 2; done           # wait for a service

deploy() {                           # functions take positional args; return an exit code
  local env="$1" tag="$2"            # local keeps variables out of the global scope
  [[ -n "$tag" ]] || return 2
  echo "deploying $tag to $env"
}
deploy prod v1.4.2 || echo "deploy failed with $?"

hosts=("web-1" "web-2" "db 1")       # indexed array; the third element has a space and survives
for h in "${hosts[@]}"; do ssh "$h" uptime; done
echo "${#hosts[@]}"                  # length: 3
declare -A port=([web]=80 [db]=5432) # associative array
echo "${port[db]}"

The safety header: set -euo pipefail, trap, mktemp

#!/usr/bin/env bash
set -euo pipefail
# -e           exit when a command fails, instead of carrying on with half the job done
# -u           an unset variable is an error, so a typo in $BACKUP_DIR cannot become rm -rf /
# -o pipefail  a pipeline's exit code is the last non-zero one, so `grep x f | sort` fails if grep does
IFS=$'\n\t'    # optional: word-split only on newline and tab, not space

tmp="$(mktemp -d)"                       # /tmp/tmp.X9fL3q: unique and not guessable
trap 'rm -rf "$tmp"' EXIT                # runs on exit, error and Ctrl-C; single quotes delay expansion
trap 'echo "failed at line $LINENO" >&2' ERR

Three things catch people. set -e does not fire inside if, while, && or || conditions, so grep -q x f || true is how you allow a failure on purpose. set -u makes "${arr[@]}" on an empty array an error on bash before 4.4, and $1 with no arguments an error everywhere; use "${1:-}". And pipefail means cmd | head -1 can fail with exit 141 (SIGPIPE) when head closes the pipe early; for that case, read all output into a variable first or accept the code.

Options, here-docs and exit codes in your own scripts

getopts parses short options; the code example shows it in full. A here-doc feeds a block of text to stdin, which is how you embed a config file or a SQL statement in a script:

cat > "$tmp/nginx.conf" <<EOF
server { listen ${PORT}; server_name ${HOST}; }      # variables expand
EOF
cat <<'EOF'
literal $text, nothing expands when the delimiter is quoted
EOF
psql "$DB" <<SQL
SELECT count(*) FROM orders WHERE created_at > now() - interval '1 hour';
SQL

Your script should exit with a meaningful code and print errors to stderr: echo "usage: $0 -e ENV" >&2; exit 2. A caller, human or CI, reads $? and the stderr, nothing else.

Safe scripting rules

Scheduling: cron and systemd timers

cron runs a command on a schedule from a crontab. Five fields: minute, hour, day of month, month, day of week.

crontab -e                                  # edit your crontab; crontab -l lists it
# m  h  dom mon dow  command
0    2  *   *   *    /opt/backup/run.sh >> /var/log/backup.log 2>&1       # daily at 02:00
*/5  *  *   *   *    /usr/local/bin/check-disk.sh                        # every 5 minutes
0    9  *   *   1-5  /opt/reports/weekday.sh                             # weekdays at 09:00

Cron's traps: it runs with a tiny PATH and no environment from your shell, so use absolute paths and set variables at the top of the crontab; a job that prints to stdout mails root or disappears, so redirect to a log; two overlapping runs of a slow job pile up, so use flock -n /run/backup.lock cmd. /etc/cron.d/ holds system crontabs with a user field.

A systemd timer does the same with logging, dependencies and catch-up for missed runs:

# /etc/systemd/system/backup.timer  (pairs with backup.service, same name)
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=*-*-* 02:00:00     # same grammar as systemd-analyze calendar
Persistent=true               # run on boot if the 02:00 run was missed
RandomizedDelaySec=15m        # spread load when 100 hosts share the schedule
[Install]
WantedBy=timers.target

systemctl enable --now backup.timer, systemctl list-timers shows next and last runs, journalctl -u backup.service has the output. The service unit it triggers has Type=oneshot and no Restart=.

ssh, scp, rsync and tmux

ssh -i ~/.ssh/id_ed25519 ubuntu@10.0.1.20 'sudo systemctl restart api && systemctl is-active api'
scp -r ./dist ubuntu@10.0.1.20:/opt/app/          # copy; fine for small things
rsync -avz --delete ./dist/ ubuntu@10.0.1.20:/opt/app/dist/   # only changed files; trailing slash means contents
rsync -avz --dry-run --exclude '.git' ./ host:/srv/site/       # see what would change first
ssh -L 5432:db.internal:5432 bastion                # local port 5432 tunnels to the database through the bastion

rsync beats scp for deploys and backups: it sends only the differences, keeps permissions and timestamps with -a, shows what it did with -v, and --delete removes files that no longer exist in the source, which scp never does.

sftp is scp with an interactive session (sftp host, then get, put, ls), over the same ssh connection. Plain FTP sends passwords in clear text; do not use it.

tmux keeps a shell session alive on the server when your laptop disconnects, which matters for anything that takes more than a minute over a flaky connection. tmux new -s deploy starts a named session, Ctrl-b d detaches, tmux attach -t deploy returns to it, Ctrl-b c opens a new window, Ctrl-b % splits the pane. Run every long migration and every rsync of a large tree inside tmux.

Vim survival on a server

The box you are paged onto will not have your editor, but it will have vi or vim. Know enough to fix one line and get out:

vim /etc/app/config.yaml   # open the file; you start in normal mode, keys are commands
# i        enter insert mode, type as usual
# Esc      back to normal mode
# :wq      write and quit
# :q!      quit without saving, when you made a mess
# /pattern search forward, n jumps to the next match
# dd       delete the current line
# u        undo the last change

Fifteen one-liners every operator uses

When to stop writing bash

Bash is the right tool for gluing commands together, for anything under about 100 lines, and for the first version of an automation. Switch to Python (or Go for anything that ships as a binary) when any of these is true:

• The script passes 100 lines or has more than two levels of nesting.
• It needs to parse JSON or YAML beyond a jq one-liner, or to call an HTTP API with retries and error handling.
• It needs real data structures: nested maps, objects, anything beyond a flat array.
• It needs tests. bats exists for bash, but a Python pytest suite is cheaper to write and read.
• It needs to run on macOS and Linux with the same behaviour (BSD vs GNU tools differ: sed -i, date -d).
• Another person will maintain it.

The Python and Go lesson picks up here. The rule in interviews: write the bash one-liner on the whiteboard, then say where you would switch.

A log question in a live screen. You are given a 2 GB Nginx access log and 10 minutes. The format is the default combined log: IP - - [time] "METHOD /path HTTP/1.1" STATUS BYTES "referer" "agent". The interviewer asks three things.

Step 1, "how many requests returned 5xx?" awk '$9 ~ /^5/' access.log | wc -l. Field 9 is the status because the quoted request line counts as three fields (method, path, protocol). Say that aloud; the interviewer wants to see you count fields rather than guess. Answer: 18,420.

Step 2, "which paths caused them?" awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head -5. Output:

  17902 /api/checkout
    311 /api/cart
    140 /health
     42 /api/search
     25 /

So 17,902 / 18,420 = 97.2 percent of the errors are one endpoint. That is a finding, not just a number.

Step 3, "when did it start?" Extract the hour and minute from field 4 [03/Nov/2026:09:14:27: awk '$9 ~ /^5/ {print substr($4, 14, 5)}' access.log | uniq -c | head. substr($4, 14, 5) takes 5 characters starting at position 14, which is 09:14. The counts jump from 2 to 3 per minute to 600 per minute at 09:14, so the problem began at 09:14 and the deploy log shows a release at 09:13. Total time: three commands, under four minutes.

Then the follow-up: "this works, but why not Python?" Answer: 2 GB streams through awk in about 20 seconds with no memory use, and the three commands are easier to type during an incident than a script. You would switch to Python when the question becomes "alert when 5xx exceeds 1 percent over 5 minutes", because that needs state, a config file, tests and an HTTP call to the alert API. That is problem 33, the bash log analyser, and problem 35, the Python health checker, in this roadmap.