Linux questions open most DevOps interviews in India, from TCS and Infosys to product startups: "what does a load average of 8 mean", "the disk is full but rm did not free space", "what is the difference between kill -9 and kill -15", "write a systemd unit". They are also where every Kubernetes incident ends: a pod is OOMKilled because of a cgroup limit, a node is NotReady because the disk filled, a container cannot start because of a file permission. An engineer who can read free, ss and journalctl on a node fixes in minutes what others escalate.

The filesystem: where things live

Linux has one tree starting at /; disks, devices and running processes all appear somewhere in it. The directories an operator opens every week:

A file on disk is an inode (the metadata: owner, permissions, size, where the data blocks are) plus a name in a directory that points to it. A hard link (ln a b) is a second name for the same inode; the data is deleted only when the last name is removed and no process has it open. A soft link or symlink (ln -s a b) is a small file containing a path; it can point across filesystems and it breaks if the target moves. ls -li shows the inode number and the link count.

Users, groups and permissions

Every process runs as a user ID (UID) and belongs to groups. Every file has an owner, a group and three permission sets: owner, group, others, each with read (4), write (2), execute (1). -rwxr-x--- is 750: owner everything, group read and run, others nothing. On a directory, execute means "may enter" and read means "may list".

ls -l /etc/ssh/sshd_config          # -rw-r--r-- root root: read by all, written by root
chmod 600 ~/.ssh/id_ed25519         # private key: owner only, or ssh refuses to use it
chmod 755 /opt/app/bin/server       # everyone may run it, only owner may change it
chown -R app:app /var/lib/app       # hand a directory to the service user
id app                              # uid, gid and groups of a user

Three special bits. setuid (chmod u+s, shown as s in the owner's execute slot) makes a program run with the file owner's UID instead of the caller's; /usr/bin/passwd is setuid root so a user can edit /etc/shadow through it. setgid on a directory makes new files inherit its group, for shared project directories. The sticky bit on /tmp (drwxrwxrwt) lets everyone create files but only the owner delete their own.

sudo runs one command as root (or another user with -u) and logs it to /var/log/auth.log. Who may use it is in /etc/sudoers and /etc/sudoers.d/, edited only with visudo because a syntax error there locks everyone out. A service runs as its own unprivileged user, never as root.

Processes: PIDs, fork, exec and signals

A process is a running program with a PID, a parent PID (PPID), a UID, open file descriptors, an environment and a working directory. New processes come from two system calls: fork copies the current process, exec replaces its program. When you type nginx in a shell, the shell forks and the child execs nginx. PID 1 is systemd, the ancestor of everything.

You talk to a process with signals. The three that matter:

Kubernetes and systemd both send SIGTERM, wait (30 seconds by default in Kubernetes, 90 in systemd), then SIGKILL. A service that ignores SIGTERM is killed mid-request on every deploy; that is the 12-factor disposability rule.

ps aux --sort=-%mem | head -5                # top memory users; STAT column: R running, S sleeping, D disk wait, Z zombie
top -o %CPU                                   # live view; press 1 to see each CPU, M to sort by memory
htop                                          # same, with colours and a tree view
pgrep -a java                                 # PIDs and commands matching a name
kill -TERM 1821; sleep 10; kill -0 1821 && kill -KILL 1821   # polite, then forceful; -0 only checks it exists
nice -n 10 ./backup.sh                        # lower priority; renice -n 5 -p PID changes a running one

A zombie (state Z) has exited but its parent has not read its exit status with wait. It uses no memory, only a PID, and you cannot kill it; you fix or kill its parent. Thousands of zombies in a container usually mean PID 1 is an app that never calls wait (use tini or --init).

systemd: keeping a service running

systemd is the init system on Ubuntu and every major distribution. It starts services in dependency order, restarts them when they fail, and collects their output. A unit is a thing it manages: a .service (a daemon), a .timer (a scheduled run), a .socket, a .mount, a .target (a group). Your unit files go in /etc/systemd/system/.

A complete unit for a Go or Node API:

# /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
After=network-online.target          # start after the network is up
Wants=network-online.target
StartLimitIntervalSec=60             # at most 5 restarts in 60 s, then give up and stay failed
StartLimitBurst=5

[Service]
Type=simple                          # ExecStart is the service; notify for apps that send READY=1
User=orders                          # never root
Group=orders
WorkingDirectory=/opt/orders
EnvironmentFile=/etc/orders/env      # PORT=8080, DATABASE_URL=... config in the environment
ExecStart=/opt/orders/bin/orders-api # for Node: /usr/bin/node /opt/orders/server.js
Restart=on-failure                   # restart on non-zero exit or a signal, not on a clean exit 0
RestartSec=2                         # wait 2 s between restarts (default is 100 ms)
TimeoutStopSec=20                    # SIGTERM, then SIGKILL after 20 s (default 90 s)
KillSignal=SIGTERM
MemoryMax=512M                       # cgroup hard limit; OOM killer acts inside this cgroup
CPUQuota=150%                        # at most 1.5 CPUs
TasksMax=512                         # cap threads and child processes
LimitNOFILE=65536                    # open file descriptors; default 1024 is too low
NoNewPrivileges=true                 # no setuid escalation from inside
ProtectSystem=full                   # /usr, /boot and /etc read-only for this service
PrivateTmp=true                      # its own /tmp

[Install]
WantedBy=multi-user.target           # "enable" links it into the normal boot target

The commands:

sudo systemctl daemon-reload                 # after editing any unit file
sudo systemctl enable --now orders-api       # start now and on every boot
systemctl status orders-api                  # active/failed, main PID, memory, last 10 log lines
sudo systemctl restart orders-api            # stop (SIGTERM, wait, SIGKILL) then start
systemctl list-units --type=service --state=failed
journalctl -u orders-api -f                  # follow this unit's log
journalctl -u orders-api --since "1 hour ago" -p err   # errors in the last hour
journalctl -b -1                             # the previous boot, for "why did the server reboot"
systemctl show orders-api -p MemoryCurrent   # what the cgroup says it uses now

Restart=on-failure with StartLimitBurst is the systemd version of CrashLoopBackOff: a crashing service restarts a few times, then systemd gives up and systemctl status shows failed with start-limit-hit; systemctl reset-failed clears it.

Packages and why you pin versions

Ubuntu uses apt with .deb packages; RHEL, Fedora and Amazon Linux use dnf with .rpm:

sudo apt update && sudo apt install -y nginx="$VERSION"   # pin the exact string that apt-cache policy shows
apt-mark hold nginx                                             # refuse upgrades
dpkg -L nginx | head                                            # files a package installed
dpkg -S /usr/sbin/nginx                                         # which package owns a file
sudo dnf install -y nginx-"$VERSION"                           # the dnf equivalent

You pin because an unpinned install gives you whatever is newest on the day the image or server is built: two nodes built a week apart run two versions, and the bug is on only one. Pin in Dockerfiles, Ansible and CI, and bump the pin on purpose. Security patches through unattended-upgrades are the exception you turn on deliberately.

Logs: /var/log and journald

Two log systems live side by side on Ubuntu. journald collects stdout and stderr of every systemd unit plus kernel messages into a binary, indexed store; journalctl reads it. rsyslog writes classic text files: /var/log/syslog (everything), /var/log/auth.log (logins, sudo), /var/log/kern.log. Applications with their own files use /var/log/<app>/, and logrotate (config in /etc/logrotate.d/) rotates and compresses them so they do not fill the disk. For a service you write, print to stdout and let journald or the container runtime collect it: 12-factor rule 11, and the reason kubectl logs works.

Disk: df, du, lsof, inodes, mounts

df -h                              # per filesystem: size, used, available, mount point
df -i                              # inodes: a disk can be "full" with free bytes if inodes run out
sudo du -xsh /var/* | sort -h | tail -5   # which directory is big; -x stays on one filesystem
sudo lsof +L1                      # deleted files still held open: the classic "rm freed nothing"
lsblk                              # disks, partitions and where they are mounted
findmnt /var/lib/docker            # is this its own mount, and from which device

The "disk is full but I deleted the file" case: a process still has the file open, so the inode and its blocks stay until the process closes it. lsof +L1 (link count less than 1) lists them. Fix: restart the process, or truncate the file in place with : > /proc/PID/fd/N. Never delete a log that is being written; truncate it.

Inode exhaustion looks like "No space left on device" while df -h shows free space; df -i shows 100 percent. The cause is millions of tiny files: session files, mail queues, a cache directory. Find them with find /var -xdev -type f | cut -d/ -f1-3 | sort | uniq -c | sort -n | tail.

/etc/fstab lists what is mounted at boot; one wrong line (a disk that no longer exists, without nofail) stops the boot at an emergency shell. Mount cloud data disks by UUID from blkid, never by /dev/sdX names, which can change between boots.

Memory: free, the page cache and the OOM killer

free -h                            # total, used, free, shared, buff/cache, available
head -3 /proc/meminfo              # the numbers free reads
vmstat 1 5                         # si/so columns: swapping, a sign of memory pressure
sudo dmesg -T | grep -i oom       # did the OOM killer run, and whom did it kill
cat /proc/1821/oom_score /proc/1821/oom_score_adj  # how likely this process is to be chosen

Two rules. First, free is not available. Linux uses spare RAM as a page cache for file data, so free is always small on a healthy server and that is good. The available column (MemAvailable in /proc/meminfo) is what a new process could get, including cache the kernel would drop. Alarm on available, not on free. Second, when available memory and swap are both gone, the kernel's OOM killer picks one process and sends it SIGKILL. The victim is the one with the highest oom_score, roughly its share of RAM plus oom_score_adj (from -1000, never kill, to +1000). So the biggest process dies, usually the database or the JVM you cared about most. Protect it with a negative oom_score_adj (systemd: OOMScoreAdjust=-900). Inside a container the same killer runs against the cgroup's memory limit; that is where OOMKilled in kubectl describe pod comes from.

CPU: load average versus utilisation

uptime prints three load averages: the average number of tasks that were runnable (wanting CPU) or in uninterruptible sleep (waiting on disk or NFS, state D) over the last 1, 5 and 15 minutes. It is a count, not a percentage, so it means nothing until you divide by the CPU count from nproc.

A load of 8 on 4 cores means that on average 8 tasks wanted to run and only 4 could: every CPU was busy and 4 tasks were queued. The same 8 on 16 cores is a quiet machine. The three numbers give the direction: 8, 6, 3 means the problem started minutes ago; 3, 6, 8 means it is easing. High load with low CPU usage in top means tasks in D state: a slow disk, a hung NFS mount. top shows the split: us user, sy kernel, wa waiting on I/O, st stolen by the hypervisor (a noisy neighbour, or a burstable instance out of credits).

Networking on the host, in one line each

The networking lesson goes deep; these are the commands you run on a node first.

ss -tlnp                          # listening TCP sockets with the process that owns each
ss -tan state established | wc -l # open connections right now
ip addr; ip route                 # addresses on each interface; the default gateway
dig +short api.internal           # what DNS answers, from this host's resolver
curl -sv --max-time 5 127.0.0.1:8080/healthz -o /dev/null   # is the service answering locally
sudo tcpdump -ni eth0 port 5432 -c 20      # the packets; -n no DNS, -c stop after 20

cgroups and namespaces: what a container is

User space processes above the kernel, with cgroups limiting CPU and memory, namespaces giving each container its own view, and the OOM killer choosing the process with the highest oom_score

A container is not a virtual machine. It is a normal Linux process (or tree of processes) that the kernel treats specially in two ways.

cgroups (control groups) limit what a group of processes may use: CPU time, memory, I/O bandwidth, number of PIDs. A Kubernetes memory limit of 512Mi becomes a memory.max file under /sys/fs/cgroup/ for that container, enforced by the OOM killer. CPU limits become cpu.max (a quota per period), which is why they show up as throttling, not a crash. systemd's MemoryMax= and CPUQuota= are the same mechanism for a plain service.

Namespaces limit what a process may see: its own PID tree (PID 1 inside is not PID 1 outside), network interfaces and ports, mount table and root filesystem, hostname, user IDs. kubectl exec enters those namespaces; nsenter -t PID -n ss -tlnp on the node does it by hand.

Add a filesystem image built from layers and a runtime (containerd) that sets this up, and you have Docker. What is shared is the kernel: a container cannot run a different one, and a kernel bug affects every container on the node.

SSH keys and config

ssh-keygen -t ed25519                                   # ~/.ssh/id_ed25519 and .pub; never share the private one
ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@10.0.1.20   # appends the public key to ~/.ssh/authorized_keys there

# ~/.ssh/config: so the long command becomes "ssh prod-web-1"
Host prod-web-*
    User ubuntu
    ProxyJump bastion.example.com
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 30

On the server, /etc/ssh/sshd_config should have PasswordAuthentication no and PermitRootLogin no; ~/.ssh must be 700 and authorized_keys 600 or sshd ignores them.

Symptom to first command

A 4-core web server at 9 am. You are paged: p99 latency on the API tripled. You SSH in and run three commands.

uptime prints load average: 8.12, 6.40, 3.05. nproc prints 4. So 8.12 / 4 = 2.03 tasks per CPU: every core is busy and on average about 4 tasks are waiting. The 15-minute value is 3.05, close to the core count, so the machine was fine 15 minutes ago; this is new.

top shows the CPU line us 31.0, sy 4.0, wa 61.5, st 0.0. CPU is mostly idle waiting on I/O (wa), not computing. High load with high wa means tasks are in D state waiting for a disk. The process list has three postgres backends in state D and a pg_dump started at 08:58 by a cron job.

free -h shows total 7.7 GiB, used 2.6 GiB, buff/cache 4.9 GiB, available 4.6 GiB. Memory is fine. The 4.9 GiB of cache is normal and is not the problem; if you had read the free column (280 MiB) you would have chased a ghost.

iostat -x 1 3 (from sysstat) shows the data disk at 99 percent utilisation with 40 ms average wait. The nightly dump was moved to 9 am by someone's cron edit, and it is reading the whole database at the same time the morning traffic arrives.

The fix path: sudo kill -TERM $(pgrep pg_dump) stops the dump cleanly (pg_dump handles SIGTERM; -9 would leave a half-written file). Within a minute, uptime shows 4.9, 6.1, 3.8 and falling. Then the proper fix: move the cron entry back to 02:00, run the dump with nice -n 19 ionice -c 3 so it yields to real traffic, and add an alert on wa above 30 percent for 5 minutes. Finally, write the timeline into the incident ticket: 08:58 cron ran, 09:02 latency alert, 09:07 cause found, 09:08 dump stopped, 09:09 latency back to normal. Time to diagnose: 5 minutes, because you divided load by cores and looked at wa before touching anything.