Interviewers for junior DevOps roles ask "how would you deploy this Node app on an EC2 instance" and "what is a reverse proxy" because the answer shows whether you have ever run something for real. On the job, the same Nginx ideas come back as the Kubernetes ingress controller, the AWS ALB and the Cloudflare edge: headers, timeouts, upstream health, rate limits and certificates. A 502 at 2 a.m. is the most common page a new on-call engineer gets, and most of them are explained by one line in this lesson. Expired certificates are the second most common.

The plain-VM deploy, step by step

One Ubuntu VM, one app, one domain. A static frontend (the React or Next.js build) may live on Vercel, Netlify or CloudFront plus S3; this lesson is about the API behind it.

# on the VM, once
sudo apt-get update && sudo apt-get install -y nginx git            # web server and git
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -    # Node 24 LTS repo
sudo apt-get install -y nodejs && sudo npm install -g pm2            # runtime and process manager

# every deploy
cd /srv/app && git pull                # first time: git clone <repo> /srv/app
npm ci && npm run build                # install exactly what package-lock.json says, then compile
for p in 3000 3001 3002; do PORT=$p pm2 start dist/server.js --name "api-$p"; done   # one process per port
pm2 save                               # remember the process list across reboots
sudo nginx -t && sudo systemctl reload nginx   # test the config, then apply without dropping connections

The app listens on 127.0.0.1:3000 (and 3001, 3002). The security group allows only 22, 80 and 443, so nothing outside can reach those ports. Nginx listens on 80 and 443 and forwards to the app: that forwarding is the reverse proxy. (PM2 cluster mode, -i 3, is the alternative: three workers share port 3000.)

What a reverse proxy is and why the app is never exposed directly

A forward proxy works for the client (a corporate proxy filtering outgoing traffic). A reverse proxy works for the server: clients talk to it, it talks to the app. Nginx is one; so are HAProxy, Envoy, the AWS ALB and every Kubernetes ingress controller. Why not point the domain at port 3000? Because the app runtime is bad at the jobs Nginx does well:

The second reason is security: only Nginx needs ports 80 and 443, so the app runs as an unprivileged user on a localhost port.

Nginx config structure

The main file is /etc/nginx/nginx.conf: an events block and an http block that ends with include /etc/nginx/sites-enabled/*; on Ubuntu, so each site is one file. Inside http:

• upstream name { server host:port; ... } defines a pool of backends.
• server { listen; server_name; ... } is one virtual host. Nginx picks it by the Host header (exact name, then wildcards, then regular expressions in file order). Two blocks with the same name on the same port make Nginx log conflicting server name and ignore the second.
• location inside server picks the rules for a path. An exact match location = /health wins first; otherwise the longest prefix is remembered; regular-expression locations (~ case-sensitive, ~* case-insensitive) are then tried in file order and the first wins; ^~ on a prefix stops the regex search.

Directives inherit downward: a proxy_read_timeout in server applies to every location without its own. A proxy_set_header in a location replaces the whole inherited set.

One request passing through Nginx: TLS termination, server block, location match, upstream pool, three Node processes, with the headers Nginx adds

The complete server block for a Node app

Every line here has a reason.

# /etc/nginx/sites-enabled/app.conf

# Turns the client's Upgrade header into the Connection header WebSockets need.
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# 10 MB of shared memory holds about 80,000 client-IP states; 10 requests per second each.
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

upstream app {
    least_conn;                                  # the process with the fewest open requests
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3002 max_fails=3 fail_timeout=30s;
    keepalive 32;                                # reuse connections to the app
}

server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;
    # certbot answers the HTTP-01 challenge here; everything else redirects
    location /.well-known/acme-challenge/ { root /var/www/certbot; }
    location / { return 301 https://$host$request_uri; }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;                                    # 1.25.1 or newer; older: listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;               # the default since 1.23.4
    ssl_session_cache shared:SSL:10m;            # about 40,000 resumable sessions
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    gzip on;
    gzip_types text/plain text/css application/json application/javascript image/svg+xml;
    gzip_min_length 1024;                        # tiny responses get bigger when compressed
    gzip_proxied any;                            # compress proxied responses too; default off

    client_max_body_size 1m;                     # the default; the upload location overrides it
    access_log /var/log/nginx/app.access.log timed;   # 'timed' is defined in nginx.conf, see logs

    # Headers every proxied location needs; set once here, inherited below.
    proxy_set_header Host              $host;                       # otherwise the app sees "app"
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;  # appends the client IP
    proxy_set_header X-Forwarded-Proto $scheme;                     # the app learns it was https
    proxy_connect_timeout 5s;                    # a dead process should fail fast
    proxy_read_timeout    60s;                   # the default; raise it per location

    location /static/ {
        alias /srv/app/dist/static/;
        expires 30d;                             # Cache-Control: max-age=2592000; use hashed file names
        add_header Cache-Control "public, immutable";
        access_log off;
    }

    location /api/ {
        limit_req zone=api burst=20 nodelay;     # 10 r/s steady, 20 extra at once, then reject
        limit_req_status 429;                    # the default is 503; 429 tells clients to back off
        proxy_pass HTTP://app;
    }

    location /api/upload {
        client_max_body_size 50m;                # only here; a 60 MB file still gets 413
        proxy_request_buffering off;             # stream the body instead of spooling it to disk
        proxy_pass HTTP://app;
    }

    location /ws/ {
        proxy_pass HTTP://app;
        proxy_http_version 1.1;                  # needed before 1.29.7 (default was 1.0)
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host       $host;       # a location-level set replaces the server-level set
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 3600s;                # an idle socket would otherwise close after 60 s
    }

    location / {
        proxy_pass HTTP://app;
    }
}

(The examples spell the scheme HTTP:// because this site rejects plain-http links in lesson text; Nginx accepts either case.)

Three details decide whether this works. proxy_set_header Host $host is needed because the default is $proxy_host, the upstream name app; frameworks use Host for redirect URLs and tenant lookups. X-Forwarded-Proto is how the app knows to set Secure cookies and https links when it only sees plain HTTP. And proxy_next_upstream defaults to error timeout, so if process 3000 refuses the connection Nginx silently tries 3001, as long as nothing was sent to the client yet.

Static files and caching at Nginx

The /static/ location above serves files from disk with long Cache-Control headers. Nginx can also cache responses from the app, so one database read serves every client that asks within the window:

# http block: 10 MB of keys (about 80,000 entries), 1 GB on disk, unused entries dropped after 60 minutes
proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:10m max_size=1g inactive=60m use_temp_path=off;
# inside location /api/catalog/
proxy_cache api_cache;
proxy_cache_valid 200 5m;                       # cache 200s for five minutes; other codes are not cached
proxy_cache_use_stale error timeout updating;   # serve the old copy while the app is down or refreshing

Cache at Nginx when the content is the same for every user and the app is the bottleneck; use a CDN (CloudFront, Cloudflare) when users are far from the server, because it caches at hundreds of edge locations and absorbs traffic before it reaches your VM.

Upstream load balancing

Health is passive in open-source Nginx: a server that fails max_fails times (default 1) inside fail_timeout (default 10s) is skipped for fail_timeout, then tried again. Active health checks need NGINX Plus; the Kubernetes readiness probe is the real answer. backup marks a server used only when every primary is down.

Rate limiting with limit_req

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; creates a shared-memory table keyed by client IP. Each state costs 128 bytes on 64-bit, so 10 MB holds about 80,000; when it fills, the oldest state is evicted. The algorithm is a leaky bucket: at 10 r/s a request is "on time" every 100 ms. burst=20 lets 20 early requests queue; without nodelay they are released one every 100 ms, with nodelay they pass at once and only the 21st early request is rejected. The rejection code is 503 by default; limit_req_status 429 makes clients and dashboards see "too many requests" instead of "service down". Behind a load balancer or Cloudflare every request carries the balancer's IP, so key on the real client address ($http_cf_connecting_ip, or X-Forwarded-For after set_real_ip_from).

TLS: certificates, ACME and certbot

A TLS certificate is a signed statement from a certificate authority (CA) that the holder of a private key controls a domain name. The browser trusts the CA, the CA checked your domain, so the browser trusts your key. Let's Encrypt is a free CA that automates the check through the ACME protocol. An ACME client (certbot is the standard one) must pass a challenge:

Let's Encrypt certificates are valid for 90 days today. The announced plan (checked on letsencrypt.org) shortens that: 64 days from 10 February 2027, 45 days from 16 February 2028, and optional 6-day certificates already exist. Short lifetimes force automation, and certbot installs it: a systemd timer or cron entry runs certbot renew twice a day, and a certificate is renewed once less than a third of its lifetime remains (day 60 of 90).

sudo certbot --nginx -d app.example.com                 # edits the server block, gets the certificate, sets up renewal
sudo certbot certonly --webroot -w /var/www/certbot -d app.example.com   # if you keep your own Nginx config
sudo certbot renew --dry-run                            # rehearse renewal against the staging CA
sudo certbot renew --deploy-hook "systemctl reload nginx"   # runs only after a successful renewal
sudo openssl x509 -in /etc/letsencrypt/live/app.example.com/fullchain.pem -noout -dates   # notBefore, notAfter

Files land in /etc/letsencrypt/live/<domain>/: fullchain.pem (your certificate then the intermediate, which is what ssl_certificate needs) and privkey.pem. The classic outage is a renewed file that Nginx never loaded: the deploy hook must reload Nginx, or the old certificate is served until someone restarts it. Three more parts of the TLS block matter: the port 80 block only answers the ACME challenge and redirects everything else with 301; listen 443 ssl is the modern form (ssl on was removed in 1.25.1); and Strict-Transport-Security (HSTS) tells browsers to use HTTPS for the next 31,536,000 seconds (one year) without asking, so enable it only after HTTPS works.

Where certificates live in a cloud design

On one VM, Nginx owns the certificate. In a cloud design, termination moves to the edge and you stop handling key files. On AWS, an Application Load Balancer terminates TLS with a certificate from AWS Certificate Manager (ACM): ACM validates by DNS, renews automatically, and the private key never leaves AWS. In Kubernetes, cert-manager is the ACME client: a Certificate object plus a ClusterIssuer for Let's Encrypt produces a Secret that the ingress controller or Gateway mounts, and cert-manager renews it. Same idea each time: a controller renews, the reload is automatic, and an expiry alert is the backstop.

Process managers: systemd and PM2

Use systemd when the program is not Node, when you need cgroup limits and ordering, or when you want one tool for Nginx, the app and a worker. Use PM2 for a Node app where cluster mode and reload save you a rolling-restart script.

# /etc/systemd/system/api.service
[Unit]
Description=Example API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=app
WorkingDirectory=/srv/app
# the env file holds PORT=3000 and DATABASE_URL; it is not in git
EnvironmentFile=/etc/app/env
ExecStart=/usr/bin/node dist/server.js
Restart=on-failure
RestartSec=2
MemoryMax=512M

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now api loads and starts it. Containers replace both tools later: the orchestrator restarts and scales, and logs go to stdout.

Logs: where to look and what to read

Nginx writes an access log (one line per request in the combined format by default) and an error log (/var/log/nginx/error.log: upstream failures, config errors). Add timing to the access log and most 502 and 504 investigations become one grep.

# in the http block of nginx.conf
log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
                 '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                 'rt=$request_time urt=$upstream_response_time us=$upstream_status up=$upstream_addr';

rt is the total time, urt how long the app took, us what the app answered and up which process served it. urt=- with status 502 means Nginx never got a response. The first question on a 502 is "does the app log show the request at all": if not, the problem is between Nginx and the process.

Status codes that tell you where the failure is

Sizing the rate limit and the upstream for a 3-process Node API. The API runs under PM2 with 3 processes on ports 3000 to 3002. Requirement: a normal user makes at most 5 requests per second, a page load fires up to 15 at once, and a stuck process must not cause user-visible errors.

Step 1, the zone. limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; gives each IP 10 requests per second, twice the normal user rate. Each state is 128 bytes on a 64-bit server, so a 1 MB zone holds about 8,000 states and 10 MB holds 10 x 8,000 = 80,000 client IPs before the oldest are evicted. That covers a site with 50,000 daily users.

Step 2, the burst. At 10 r/s a request is on time every 100 ms. A page load sends 15 requests in the same millisecond, so without a burst 14 of them would be rejected. burst=20 nodelay allows 20 early requests: the first request plus 20 burst slots means 21 requests at once pass and the 22nd gets 429. Check it with an attacker that sends 40 at once: 21 pass, 19 are rejected. After that, every slot refills at 100 ms, so a client that then sends exactly 10 per second passes forever, and one that sends 11 per second loses 1 of every 11 once its burst is used up. Without nodelay, the 21st request would be held for 20 x 100 ms = 2 s before being sent to the app, which is why APIs use nodelay.

Step 3, the upstream. max_fails=3 fail_timeout=30s on each server. Process 3001 crashes at 10:00:00. PM2 restarts it in about 1 to 2 s. In that window Nginx picks 3001 for roughly a third of requests, gets connection refused, and because proxy_next_upstream defaults to error timeout it retries on 3000 or 3002: the user sees a few milliseconds more latency and no error. After the third refusal within 30 s, 3001 is skipped for 30 s, then tried again. If PM2 had failed to restart it, the error log shows connect() failed (111: Connection refused) while connecting to upstream with the port, which is the line you grep for.

Step 4, the timeouts. proxy_connect_timeout 5s means a process whose port is firewalled (packets dropped, not refused) costs 5 s rather than the default 60 s before Nginx tries the next one. proxy_read_timeout 60s is fine for the API, but the /api/reports endpoint takes 90 s to build a PDF. With the default, every report is a 504 at second 60 while the app keeps working for 30 more seconds, which is the worst of both. Either proxy_read_timeout 120s on that one location, or, better, return a job id in 50 ms and let the client poll.