"What is an SLO, and what is the difference from an SLA" opens most SRE interviews. The follow-ups are numeric: how many minutes is 99.9 percent, what does a burn rate of 14.4 mean, why page on burn rate instead of on error count. Then come the scenario questions: you are paged at 2 a.m., what are your first five minutes; how do you run a postmortem without blame. The same material is what separates a team that ships calmly from one that is always on fire, and it is the language of every reliability conversation in a platform job.

What SRE is

SRE is what happens when you treat operations as a software problem. The team that keeps a service up has a budget for manual work, writes code to remove it, and measures reliability as a number negotiated with product teams rather than a feeling. Three ideas carry the practice: measure the user's experience (SLIs and SLOs), spend the allowed failure deliberately (error budgets), and learn from every incident without blame (postmortems).

Toil is operations work that is manual, repetitive, automatable, tactical, without lasting value, and that grows with the service: restarting pods by hand, rotating a certificate by ticket, approving the same quota request weekly. The SRE book's rule is to keep toil below 50 percent of each engineer's time; the rest goes into engineering that removes toil. A team over the line stops taking on new services until it is back under.

SLI, SLO, SLA

The SLA is always looser than the internal SLO, so you notice and fix a problem before it costs money. Four kinds of SLI cover most services:

• Availability: good = request got a successful response. sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m])).
• Latency: good = request finished within a threshold. From a histogram: rate(http_request_duration_seconds_bucket{le="0.3"}[5m]) / rate(http_request_duration_seconds_count[5m]). Note this is a ratio, not a percentile: "99 percent of requests under 300 ms" is an SLO you can budget; "p99 under 300 ms" is not.
• Correctness: good = response had the right content (a canary query with a known answer, a checksum on a batch output).
• Freshness: good = data served was updated within N minutes (a search index, a recommendation table, a replica).

How to pick one: measure where the user is, not where it is convenient. Count at the load balancer or in a probe from outside, not inside the pod, because a dead pod reports nothing. Count what users do, not what the system does: "checkout requests good" beats "CPU under 80 percent". Keep two or three SLIs per service; more and nobody can say what it promises.

The error budget

The error budget is 1 - SLO: the fraction of events you are allowed to fail. At 99.9 percent, 0.1 percent of requests may fail, which at 10 million requests a day over 30 days is 300,000 requests. In time, if every failure were a full outage, 0.1 percent of 30 days is 43.2 minutes a month.

Each extra nine costs roughly ten times more engineering, and no user notices the difference between 99.99 and 99.999 on a mobile network that is itself at 99 percent. Pick the lowest SLO that keeps users happy; the budget is for spending on releases, migrations and experiments.

The error budget policy is the written agreement about what happens as the budget is spent. A usual one: above 50 percent left, ship freely; under 50 percent, risky changes need a reviewer; at zero, feature releases stop and the team works only on reliability until the rolling window recovers; and the policy names who can override it (a director, in writing). Without the policy, the budget is a dashboard nobody acts on.

Burn rate and multi-window alerts

Burn rate is how fast you are spending the budget relative to the pace that would spend it exactly at the end of the window. For a 99.9 percent SLO the allowed error rate is 0.1 percent; an error rate of 0.1 percent is burn rate 1 (budget gone in 30 days), 1 percent is burn rate 10 (gone in 3 days), 100 percent is burn rate 1,000 (gone in 43 minutes).

Chart of remaining error budget over a 30 day window with burn rate 1, 6 and 14.4 reference lines, two incidents draining it, and the SRE workbook alert table beside it

Alerting on burn rate instead of a raw error count makes every alert mean the same thing: "at this pace, how much of the month is this hour costing". The SRE workbook's recommended configuration for a 99.9 percent SLO over 30 days:

Two windows per rule, both of which must exceed the threshold. The long window gives precision: an hour at 14.4x is something real, not a 30 second blip. The short window gives a fast reset: without it, a one-hour alert keeps firing for up to an hour after the fix. The workbook's rule of thumb is short window = long window / 12. As Prometheus rules on recorded error ratios:

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetBurn
        expr: |
          (
            service:http_errors:ratio_rate1h{service="checkout"} > (14.4 * 0.001)
            and
            service:http_errors:ratio_rate5m{service="checkout"} > (14.4 * 0.001)
          )
          or
          (
            service:http_errors:ratio_rate6h{service="checkout"} > (6 * 0.001)
            and
            service:http_errors:ratio_rate30m{service="checkout"} > (6 * 0.001)
          )
        labels:
          severity: page
        annotations:
          summary: "checkout is burning error budget at {{ $value | printf \"%.3f\" }} error ratio"
          runbook_url: https://runbooks.example.internal/checkout-slo-burn
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          service:http_errors:ratio_rate3d{service="checkout"} > 0.001
          and
          service:http_errors:ratio_rate6h{service="checkout"} > 0.001
        labels:
          severity: ticket

Symptoms, not causes, and what makes a good alert

A symptom-based alert fires on what users feel: error ratio, latency, missing data. A cause-based alert fires on something that might lead to that: CPU at 90 percent, a disk at 80 percent, one pod restarting. Causes are many, change with the architecture, and often hurt nobody (a pod restarting behind a healthy Service). Symptoms are few, stable and always matter. Alert on symptoms; keep cause signals on the dashboard you open after the page.

Alert fatigue is when the on-call engineer ignores pages because most are noise, then misses the real one. The rules for a page: it is actionable (a human must do something now), urgent (it cannot wait until morning; otherwise it is a ticket), and it carries a runbook link and the labels needed to start (service, namespace, cluster). Review every page weekly: if it needed no action, delete or demote it. A healthy rotation sees a few pages a week, not a few per night.

Capacity planning and load testing

Capacity planning answers "will we survive the festival sale". Start from the demand forecast (peak requests per second, often 3 to 5 times the daily average around Indian sale events), measure what one replica handles at the latency SLO (not at 100 percent CPU), and add headroom for one zone failing (N+1 across 3 AZs means running at most 66 percent of total capacity). Load testing measures that per-replica number: k6 scripts in a CI job ramp to the target rate against a staging copy and fail the build if p95 crosses the SLO threshold. Test before the sale, and test the whole path including the database.

Incident response

An incident is any event where the SLO is in danger and more than one person must coordinate. Three things make it go well: a severity scale agreed in advance, named roles, and the habit of mitigating before understanding.

Roles, from the SRE book's incident management chapter: the incident commander holds the state of the incident, decides, delegates and keeps the live incident document; operations is the only group that changes the system; communications posts updates to stakeholders and the status page on a fixed cadence so engineers are not interrupted; planning files bugs, tracks every change for later revert and arranges handoffs. On a small team one person holds several roles, but the commander should not be the one typing commands.

The first five minutes after a page:

1. Acknowledge the page so it does not escalate. Open the runbook linked in the alert.
2. Confirm impact from the user's side: the SLO dashboard, the status of the external probe, the error ratio by route. Decide the severity.
3. Say so: open the incident channel, post "investigating: checkout error ratio 8 percent since 02:10, impact: payments failing, IC: me". If SEV1 or SEV2, update the status page.
4. Ask the question that resolves most incidents: what changed? Deploys in the last hour (kubectl rollout history, the Argo CD timeline), config, feature flags, certificate expiries, a dependency's status page, traffic shape.
5. Mitigate first, understand later. If a deploy matches the timing, roll it back now and read the diff afterwards. Other standard mitigations: fail over to another region or replica, turn the feature flag off, scale up, shed load with a rate limit, drain a bad node. Root cause work happens after users are fine.

Then keep a timeline in the incident document: timestamp, who, what was observed or done. Memory of a 3 a.m. incident is gone by 9 a.m., and the timeline is the postmortem's raw material. Communications posts on a cadence ("next update in 30 minutes", and then at 30 minutes) using a template: what is affected, what users see, what we are doing, when the next update comes. Escalate when you are 15 minutes in with no idea, or the fix needs access you do not have; asking for help is not a failure, sitting alone is.

Postmortems

A postmortem is written for any incident that crossed a threshold: user-visible impact beyond a limit, data loss of any kind, on-call intervention such as a rollback, a long time to resolve, or a monitoring failure where a human found the problem before the alerts did. It is blameless: it looks for the conditions that let a reasonable person, acting on what they knew, cause or miss the problem, and never names an individual as the cause. Once people fear a postmortem they hide what happened, and the organisation stops learning.

The template:

Title: Checkout payments failing, 2026-10-14, SEV2
Owner: on-call SRE for checkout     Reviewers: checkout tech lead, payments SRE
Impact: 02:10 to 02:31 IST, 21 min. 8 percent of checkout requests returned 500,
        about 11,700 failed payments, 3.9 percent of the monthly error budget.
Detection: burn-rate page at 02:14, 4 min after impact. No customer tickets first.
Timeline:
  02:06  checkout v2.31 rolled out via Argo CD auto-sync
  02:10  error ratio rises from 0.1 to 8 percent on POST /pay
  02:14  page: CheckoutErrorBudgetBurn, 1h and 5m windows at 14.4x
  02:16  on-call acknowledges, opens incident channel, IC assigned
  02:21  what changed: v2.31 is the only change in 6 hours
  02:24  rollback to v2.30 started
  02:31  error ratio back to 0.1 percent; status page updated
  02:58  stable for 25 minutes; incident closed
Root cause: v2.31 cut the payments client timeout from 5 s to 500 ms; the provider's
        p99 is 800 ms, so 8 percent of calls timed out.
Contributing factors: timeout lives in a config file with no review; staging uses a
        mock provider with 10 ms latency; the canary stage was skipped.
What went well: page 4 min after impact; rollback took 7 min.
What went badly: 10 min from page to rollback start, spent reading logs, not asking what changed.
Where we got lucky: 02:00 is low traffic in India; at 20:00 the impact would be 10x.
Action items (owner, due, ticket):
  - Canary stage with a 10 min bake on the checkout rollout (platform, 2 weeks, PLAT-412)
  - Alert annotation links to the recent-changes dashboard (SRE, 1 week, SRE-88)
  - Staging mock replays the provider's real latency distribution (checkout, 3 weeks, CHK-901)

Five whys (ask why five times until you reach a process cause) is a useful warm-up and a poor final answer: it produces one straight line of causes, while real incidents have several interacting ones, and it stops wherever the asker gets tired. Write root cause and contributing factors, plural.

Two short readings every SRE should know. How Complex Systems Fail argues that complex systems always run with latent faults and partial failures, and that an accident needs several of them to line up, so there is never one root cause and the people operating the system are mostly its defence, not its weakness. Ironies of Automation observes that automating the normal case leaves the human with only the abnormal, rare cases, exactly when they are least practised, so automation must keep operators in the loop and make the system's state visible. Both are why "human error" is never an acceptable root cause.

On-call, runbooks, chaos and DORA

On-call is a rotation (one primary, one secondary, a week each, 6 to 8 people so each is on about one week in two months), a documented handoff at the boundary (open incidents, silences, known risks, what changed this week), paging rules (what pages, what tickets), and compensation: time off or pay for the hours, and a cap on pages per shift above which the rotation itself is a bug. A rotation under 4 people burns out.

A runbook is the page the alert links to. Structure: what the alert means in one sentence, two queries to confirm impact, the three most likely causes with the command that checks each, the mitigation for each with the exact command (kubectl rollout undo deployment/checkout -n shop, the flag name to flip), when to escalate and to whom, and the date it was last tested. Write it the week you create the alert; one written during the incident is too late.

Chaos engineering, as the Chaos Engineering paper describes it, is running controlled experiments to find weaknesses before they find you: define steady state (the SLI), form a hypothesis ("losing one AZ keeps availability above 99.9 percent"), inject the failure with a small blast radius, watch the SLI, and stop the moment it degrades. A game day is the rehearsal: the team plays through an incident (a region loss, an expired certificate, a full disk) with the real on-call tooling and timeline, then writes the same postmortem it would after a real one. Tools: Chaos Mesh or LitmusChaos on Kubernetes, AWS Fault Injection Service for the cloud layer.

The DORA metrics are the delivery-side counterpart of SLOs: deployment frequency and change lead time (throughput), change fail rate and failed deployment recovery time (stability), plus deployment rework rate in the current five-metric model. A slow rollback shows up as a bad recovery time and as a drained error budget; small, frequent, reversible deploys with canaries fix both.

The service. Checkout handles 10 million requests a day and promises 99.9 percent availability over a rolling 30 days.

Step 1: the budget in requests. Requests in the window: 10,000,000 x 30 = 300,000,000. Error budget = (1 - 0.999) x 300,000,000 = 300,000 failed requests a month.

Step 2: the budget in minutes. If failures came as full outages, the budget is 0.001 x 30 days x 24 x 60 = 43.2 minutes of downtime a month. Compare: 99.99 percent would be 4.3 minutes, which no rollback in Kubernetes can meet without automation that triggers in under a minute.

Step 3: a 1-hour burn at 14.4x. Burn rate 14.4 means an error rate of 14.4 x 0.1 percent = 1.44 percent. Requests per hour: 10,000,000 / 24 = 416,667. Errors in that hour: 416,667 x 0.0144 = 6,000. Fraction of the monthly budget: 6,000 / 300,000 = 2 percent. The same answer from the formula: 14.4 x 1 hour / 720 hours = 0.02. That is exactly the workbook's first row: the 1-hour page fires when one hour has cost 2 percent of the month, and the 5-minute window must agree so it resets quickly after the fix. At this pace the whole budget is gone in 720 / 14.4 = 50 hours, which is why it is a page and not a ticket.

Step 4: the other two rows. 6x over 6 hours: 6 x 6 / 720 = 5 percent of the budget, gone in 5 days. 1x over 3 days: 1 x 72 / 720 = 10 percent, and at burn rate 1 the budget lasts the whole 30 days, which is exactly on the SLO line: not urgent, hence a ticket.

Step 5: a real incident. A bad deploy pushes the error rate to 3 percent (burn rate 30) for 90 minutes. Budget spent: 30 x 1.5 / 720 = 6.25 percent, or 18,750 failed requests out of 300,000. The 1-hour window crosses 14.4 after 14.4 / 30 of an hour, about 29 minutes, so the page arrives then (the code example shows 28 minutes on generated data). A 5-minute-only alert would page at minute 1 but also on every 5-minute blip; the two-window rule trades 28 minutes of detection time for a quiet pager. After the rollback, the 5-minute window drops under threshold within minutes and the alert resolves, while the 1-hour average alone would have stayed above 14.4 for most of an hour.