Troubleshooting is a learnable skill, not a talent. This chapter writes down the method that experienced engineers use without noticing and shows where novices go wrong. It frames debugging as the scientific method applied to a running system: observe, hypothesise, test, repeat, and it stresses that in production your first job is to make the system work again, not to find out why it broke.

The chapter uses a worked example, a Shakespeare search service returning strange results, and closes with how to design systems so that they are easy to debug in the first place.

In one minute

• Troubleshooting needs two things: a generic method, and knowledge of the particular system. The method is the hypothetico-deductive loop: from observations and a model of how the system should behave, form hypotheses, test them, and narrow down.
• The process has stages: problem report, triage, examine, diagnose, test and treat, cure.
• Common pitfalls: chasing irrelevant symptoms, misunderstanding how a change will affect the system, inventing improbable theories, latching on to the cause of a past incident, and mistaking correlation for causation.
• Stop the bleeding first. Divert traffic, roll back, degrade gracefully, add capacity. Preserve evidence (logs, a bad task) for root-causing, but restore service before you root-cause.
• Examine with monitoring (time-series dashboards), logging (with adjustable verbosity and structured output) and tracing (Dapper). Compare what each component is doing to what it should be doing.
• Diagnose by simplify and reduce (feed known inputs, check outputs), divide and conquer (bisect the stack or the deployment), what, where and why (what is the system doing, where is it spending resources, why), what touched it last, and specific diagnoses built into the service.
• Test hypotheses one at a time, cheapest and most likely first; note what you tried and what happened, including negative results, which are valuable and should be published.
• Make systems debuggable: observability built in from the start, well-understood and documented interfaces, and the ability to change verbosity at runtime.

Theory

Given a set of observations about a system and a model of how it is supposed to work, troubleshooting is iterating between forming a hypothesis about the failure and testing it. A test is either passive, comparing observed state to what the hypothesis predicts, or active, changing the system in a controlled way and seeing what happens. Each result refines the model and the next hypothesis. Experts are faster because they have better models and have seen more failures, not because they skip the steps.

The pitfalls are failures of method. Wild goose chases happen when you look at symptoms that are not related to the problem. Misunderstanding how your change will affect the system leads to active tests that mislead you. Improbable theories are cheap to generate and expensive to test; when you hear hoofbeats, think horses, not zebras. Assuming the current incident is the same as the last one with similar symptoms is especially seductive after a run of similar pages. Correlations in time are often coincidences or share a hidden common cause.

In practice

Problem report. A good report says what the expected behaviour was, what actually happened, and how to reproduce. Reports should go to a tracker, not to a person, so they are visible and searchable, and so that the person's absence does not lose the report.

Triage. Assess severity, then stabilise. A system that is serving errors should be made to serve something, even if degraded, before anyone investigates why. The analogy is to an emergency room: stop the bleeding, then diagnose. Novices resist this because they want to find the bug; in production the users do not care about the bug.

Examine. Monitoring is the first stop: when did the metrics change, and in which component. Logs are second: structured, with request identifiers, and with verbosity that can be raised on a live task without restarting it. Tracing with something like Dapper shows how one request flowed across services and where it spent time. Exposing the current state of a server (its RPC log, configuration, and queue lengths) through a debug endpoint is cheap and pays off constantly.

Diagnose. Several complementary techniques. Simplify and reduce: build a minimal reproduction or feed a known input and check where the output goes wrong. Divide and conquer: in a layered system, start at one end and work through, or bisect, checking the connection between halves, which turns a search over n components into log n. Ask what the system is doing, where it is spending resources, and why it was asked to do that; a malfunctioning system is still doing something, and understanding what explains a lot. Ask what touched it last, because most incidents follow a change: a push, a config change, a traffic shift. And build specific diagnostics into the service, such as a handler that reports the health of each dependency.

Test and treat. Rank hypotheses by probability and cost of testing, test the cheap and likely ones first, test one variable at a time, and document the test and the result so the next person does not repeat it. Some tests have side effects (restarting a task) and must be ordered carefully; some have no clean result and only shift the probabilities.

Negative results are magic

A negative result is an experiment where the expected outcome did not happen. These are valuable: they rule out hypotheses, they save others from trying the same thing, they tell you about the system's actual limits, and publishing them improves everyone's model. Google encourages writing them up in postmortems and design documents rather than discarding them as failures.

Cure

Once a fix is found, prove it: show that the hypothesis explains all the observed symptoms and that the fix removes them. In production you often cannot fully reproduce an incident, so the proof is probabilistic and the postmortem records the uncertainty.

Case study and making troubleshooting easier

The Shakespeare example walks through a search service returning results for the wrong plays, traced via dashboards and logs to a corrupted index produced by a pipeline run that read a partial input. The general lesson is that the time to make a system debuggable is when you design it: build observability in, document the interfaces and expected behaviour, keep dependencies and data flow legible, and make sure every log line can be tied to a request. Good debuggability is also an argument for simplicity (chapter 9).