Every SRE and platform interview has a monitoring round. The questions are specific: "write the PromQL for p95 latency", "why does rate() need a window longer than the scrape interval", "why is user_id a bad label", "how would you find which of six services made this request slow". In the job, the Prometheus and Grafana stack (100xDevs weeks 31 and 32) is what you look at during every incident, and OpenTelemetry is the standard every new service is instrumented with. Knowing the queries cold is the difference between a 5 minute and a 50 minute outage.

Monitoring versus observability

Monitoring asks questions you wrote in advance: is CPU above 80 percent, is the health check passing. Observability lets you answer questions you did not plan for, from data the system already emits: "why are only checkout requests from the Android app slow since 14:10". That needs context (labels, trace ids) you can slice later.

Three signals do the work. Metrics are numbers sampled over time: cheap, aggregated, the basis of alerting. Logs are timestamped events with detail. Traces follow one request across services and show where the time went. A fourth, profiling (which function burns CPU), is an OpenTelemetry signal still in development.

Telemetry pipeline from an app with the OpenTelemetry SDK through a collector to Prometheus, Loki and Tempo, with Grafana querying all three and Alertmanager paging on-call

The Prometheus model

Prometheus pulls. Every scrape_interval (default 1m; most clusters set 15s or 30s) it sends an HTTP GET to each target's /metrics path and stores the result with a timestamp. A target that does not answer within scrape_timeout (default 10s) gets up = 0, the first alert to write.

Targets come from service discovery: on Kubernetes, kubernetes_sd_configs lists pods, services or endpointslices from the API server. In practice you install the Prometheus Operator (the kube-prometheus-stack chart) and declare targets with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: shop
  labels:
    release: kube-prometheus-stack   # must match the Prometheus CR's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: checkout                  # selects the Service, not the pods
  endpoints:
    - port: http                     # the Service's named port
      interval: 15s

The Operator watches ServiceMonitors (and PodMonitors, which select pods directly), generates the scrape config and reloads Prometheus. The classic mistake is a ServiceMonitor whose labels miss the serviceMonitorSelector: nothing is scraped, nothing complains. Each unique combination of name and labels is one time series. Names use base units (_seconds, _bytes); counters end in _total.

Metric types and when to use each

Histogram versus summary is a standard question. A histogram is a set of counters, so the server can add buckets across 50 pods and compute any quantile. A summary computes the quantile inside the process, and p99 values cannot be averaged across pods. Choose histograms; the price is picking buckets in advance.

Labels and cardinality

Labels are what make PromQL useful: sum by (status) or {namespace="shop"}. The cost: every distinct label value set is a new series held in memory. The Prometheus docs say it plainly: no labels for unbounded sets such as user IDs or email addresses.

A latency histogram with 15 buckets exposes 17 series (buckets plus _sum and _count) per label combination. With method (5), route (20) and status (5): 500 combinations, 8,500 series. Add user_id with one million users: 17 million series from one metric. Prometheus falls over at a few million series on a normal machine, during the incident you need it for. Raw request paths (/orders/8812 instead of /orders/{id}) do the same.

The four golden signals, RED and USE

The SRE book's four golden signals for any service are latency, traffic, errors and saturation. Two shorthand methods map them to dashboards: RED for request-driven services (Rate in requests per second, Errors as a ratio, Duration as p50, p95, p99), the dashboard every service should have, and USE for resources such as CPU or a connection pool (Utilisation, Saturation meaning work waiting, Errors). Alert on RED, the symptoms users feel; look at USE to find the cause.

PromQL: the queries every on-call engineer needs

A metric name alone returns an instant vector, the latest value of every matching series; name[5m] returns a range vector, the samples of the last 5 minutes, which rate takes.

# Requests per second per service. rate() undoes counter resets and
# extrapolates to the window edges.
sum by (service) (rate(http_requests_total[5m]))

# Error ratio: 5xx over everything, both sides aggregated the same way.
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by (service) (rate(http_requests_total[5m]))

# p95 latency: aggregate the buckets first, keep le.
histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# Requests in the last hour: rate() times the window, so it can be a non-integer.
increase(http_requests_total{service="checkout"}[1h])

# A dead exporter produces no series, not a zero: absent() returns 1 then.
absent(up{job="checkout"})

The rate window versus the scrape interval is the subtlety interviewers test. rate() needs at least two samples in the window. With a 15 s scrape, rate(x[15s]) usually holds one sample and returns nothing, and rate(x[30s]) breaks whenever a scrape is late. Use at least four scrape intervals ([1m] at 15 s) and [5m] for alerts. irate() uses only the last two samples: for graphs, never alerts. And always rate() first, then sum: a sum of counters drops whenever one pod restarts, and its rate is garbage.

Exporters and instrumenting your own app

An exporter turns something that does not speak Prometheus into a /metrics page. In every cluster: node_exporter (CPU, memory, disk, network per node), kube-state-metrics (object state: replicas desired versus available, pod phase, restarts, requests and limits), cAdvisor (inside kubelet: per-container CPU and memory), and blackbox_exporter (probes an HTTP, TCP, DNS or ICMP endpoint from outside; an uptime monitor starts here).

Your own service uses a client library (Go client_golang, Python prometheus_client, or the OpenTelemetry SDK) and exposes /metrics:

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter("http_requests_total", "Requests", ["method", "route", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Latency", ["route"],
                    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5))

def handle(method, route):
    with LATENCY.labels(route=route).time():
        status = do_work()
    REQUESTS.labels(method, route, str(status)).inc()

start_http_server(9100)   # serves /metrics

Recording rules, alerting rules and Alertmanager

A recording rule precomputes a query on a schedule and stores the result as a new series, named level:metric:operations, so dashboards and alerts read one cheap series instead of summing 2,000 pods. An alerting rule evaluates an expression every evaluation_interval and, once it has held for the for duration, sends the alert to Alertmanager; until then it is pending.

groups:
  - name: checkout
    interval: 30s
    rules:
      - record: service:http_errors:ratio_rate5m
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m]))
      - alert: CheckoutHighErrorRatio
        expr: service:http_errors:ratio_rate5m{service="checkout"} > 0.05
        for: 5m                     # a shorter blip never fires
        labels:
          severity: page
        annotations:
          summary: "checkout error ratio is {{ $value | humanizePercentage }}"
          runbook_url: https://runbooks.example.internal/checkout-errors

Alertmanager decides who hears about an alert. The routing tree matches on labels (severity: page goes to PagerDuty, ticket to Slack or Jira). Grouping (group_by: [alertname, namespace]) folds 200 pod alerts into one notification; group_wait (default 30s) lets the group fill first, group_interval (default 5m) spaces updates, repeat_interval (default 4h) re-sends an unresolved alert. Inhibition mutes alerts while a bigger one fires: ClusterUnreachable suppresses every TargetDown in that cluster (inhibit_rules with source_matchers, target_matchers and equal). Silences mute matching alerts for a window, set before maintenance. Run 3 replicas.

Long-term storage

One Prometheus keeps data locally in 2 hour blocks behind a write-ahead log, for --storage.tsdb.retention.time (default 15d). It does not replicate or shard. For many clusters, a year of history, or one global view, you remote-write to Thanos or Grafana Mimir, which speak PromQL and keep blocks in object storage. The Google ancestor is Monarch, its planet-scale in-memory time series database: regional ingestion, a global query layer, a hard cardinality limit per tenant.

Grafana

Grafana queries all three stores and draws the panels. The rules:

• Dashboards as code. A dashboard is JSON. Keep it in Git and load it by provisioning (a ConfigMap labelled grafana_dashboard: "1" with the kube-prometheus-stack sidecar, or Terraform's Grafana provider). A dashboard edited in the browser was never reviewed.
• Variables. $service from label_values(http_requests_total, service), so one dashboard serves 200 services.
• The dashboard every service gets: a top row of RED panels (requests per second, error ratio, p50/p95/p99), a second row of saturation (CPU and memory against limits, restarts, replicas ready), a third row of dependencies. Generated from one template.
• Avoid sprawl. One dashboard per service from the template, one per platform component, each with an owner in the title; delete the rest.

Structured logging

A structured log is one JSON object per line, with every fact as a key:

{"ts":"2026-11-04T14:10:07.412Z","level":"error","msg":"payment declined","service":"checkout","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","request_id":"c1f4e0a9","duration_ms":212,"order_id":8812}

Compare ERROR payment declined for order 8812 after 212ms. The text line has no fields, so Loki's | json | order_id = 8812 has nothing to match, a query cannot group by service, and duration_ms > 1000 becomes a regex that breaks when someone rewords the message. Correlation breaks too: without trace_id and request_id as keys, the 40 lines of one request across six services cannot be pulled out of a million, and Grafana cannot link the line to its trace. Use log/slog in Go or a JSON formatter for Python's logging, and put the request's fields as keys, never pasted into msg.

Levels: debug off in production, info at most one line per request, warn for what will become a problem, error for failed work. info inside a loop is how teams reach 1 TB per day.

Shipping and querying logs

The app writes to stdout, the runtime writes it to a file on the node, a DaemonSet agent (the OpenTelemetry Collector with the filelog receiver and k8sattributes processor, or Fluent Bit) reads the files, adds namespace and pod labels and pushes to the store. Loki indexes only the labels, not the text, and keeps lines compressed in object storage: cheap, and you "select a stream by labels, then grep". The ELK stack (Elasticsearch, Logstash, Kibana) indexes every field: fast queries, expensive cluster.

Two LogQL queries (Loki's language):

# Lines containing "error", parsed as JSON, filtered to one order.
{namespace="shop", app="checkout"} |= "error" | json | order_id = 8812

# Error lines per second by pod: a metric query over logs, drawn in Grafana.
sum by (pod) (rate({namespace="shop", app="checkout"} |= "error" [5m]))

Retention is a cost decision: 7 to 14 days for most services, 90 days or more for audit logs in a cheaper tier.

Traces

A trace is the tree of spans produced by one request: each span is one unit of work (an HTTP handler, a database query) with a name, start time, duration, attributes and a parent. Six services calling each other give one trace of at least six spans, drawn as a waterfall: 420 ms total, of which 380 ms was the inventory service waiting on one slow SQL query. Logs from six services cannot show that: no shared clock, no parent. This is Google's Dapper design: a trace id made at the edge, propagated in every call, spans sampled and collected out of band.

Context propagation is how the id travels. The W3C traceparent header carries version-traceid-parentid-flags, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01: a 32 hex character trace id, a 16 hex character parent span id, and 01 meaning sampled. Instrumented HTTP libraries read and write it automatically; for Kafka and SQS the same values ride in message headers.

Sampling keeps cost sane. Head sampling decides at the root (keep 10 percent of trace ids): cheap, but it cannot know the trace will contain an error. Tail sampling decides once the whole trace reaches a collector (keep every trace with an error or over 1 s, plus 5 percent of the rest): better, but every span of a trace must reach the same collector instance, which costs memory and a load-balancing step. Stores: Grafana Tempo, Jaeger.

OpenTelemetry: the standard

OpenTelemetry (OTel) is the CNCF project that replaced the vendor SDKs. It defines the API and SDK for every language, the OTLP wire protocol (gRPC on 4317, HTTP on 4318), semantic conventions (http.response.status_code, k8s.pod.name), and the Collector. Auto-instrumentation (a Java agent, a Python wrapper, injected into pods by the OTel Operator) produces spans and metrics for HTTP, gRPC, SQL and queue libraries with no code change; manual spans go only around your own logic.

The Collector is a pipeline of receivers (accept data), processors (transform it) and exporters (send it on), wired under service.pipelines, one per signal. Nothing is active until a pipeline names it:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:                  # first: refuse data before the collector OOMs
    check_interval: 1s
    limit_mib: 800
    spike_limit_mib: 200
  k8sattributes: {}                # adds k8s.namespace.name, k8s.pod.name
  batch: {}                        # last: fewer, larger exports

exporters:
  prometheus_remote_write:
    http:
      endpoint: https://mimir:9009/api/v1/push
  otlphttp/loki:                   # type/name: several exporters of one type
    endpoint: https://loki:3100/otlp
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true               # inside the cluster only

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [prometheus_remote_write]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp/loki]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp/tempo]

Run it as a DaemonSet (one agent per node) forwarding to a Deployment (gateway) that tail-samples and talks to the backends.

Buy or build: APM products

Datadog, New Relic, Dynatrace and Grafana Cloud sell the whole pipeline as a service: install an agent, get dashboards, APM traces, logs and alerting the same afternoon (100xDevs week 30.2 uses New Relic). A team buys when it is small, has nobody to run Prometheus at 3 a.m., and the bill stays under one engineer's cost. It builds when per-GB and per-host pricing passes that line, when data must stay in-region, or when a platform team already runs Kubernetes. Either way, instrument with OpenTelemetry: switching is then one exporter change.

What observability costs

• Metrics: cost is series, not samples. At 1 to 2 bytes per sample, 10,000 series at 15 s for 30 days is 1.7 billion samples and 1.7 to 3.5 GB (worked example below). With a user_id label it is 17 million series and fits no machine.
• Logs: 2,000 pods x 20 lines per second x 250 bytes is 10 MB per second, 864 GB per day raw, about 86 GB per day compressed (assuming 10 to 1), 2.6 TB for 30 days. Hosted products charge per GB ingested: at an assumed 0.50 USD per GB, 432 USD per day.
• Traces: 10,000 requests per second x 8 spans x 500 bytes is 40 MB per second before sampling; at 10 percent, 346 GB per day.

Metrics are the cheapest signal, so alerting is built on them; logs and traces are what you open after the page.

Part 1: error ratio and p95 by hand. Over the last 5 minutes, after rate() and sum by (le), the checkout latency histogram has these cumulative bucket counts (per-second rates scaled to counts for simplicity): le=0.1: 600, le=0.25: 860, le=0.5: 960, le=1: 990, le=+Inf: 1000. The error counter rate, scaled the same way, is 12.

Error ratio = 12 / 1000 = 0.012, so 1.2 percent of requests failed. Against a 1 percent threshold this alert fires (if it holds for the for duration).

p95: the rank is 0.95 x 1000 = 950. The first bucket whose cumulative count reaches 950 is le=0.5 (960; the one below, le=0.25, has only 860). So p95 lies between 0.25 s and 0.5 s. That bucket holds 960 - 860 = 100 observations and we need the 950 - 860 = 90th of them. Linear interpolation: 0.25 + (0.5 - 0.25) x 90 / 100 = 0.25 + 0.225 = 0.475 s. That is exactly what histogram_quantile(0.95, ...) returns. The true p95 could be anywhere in (0.25, 0.5]; the answer is only as precise as the bucket. p50 by the same method: rank 500 lands in the first bucket (600), lower bound 0, so 0 + 0.1 x 500 / 600 = 0.083 s. If the rank had landed in the +Inf bucket, Prometheus would return 1.0, the upper bound of the highest finite bucket, which is why a p99 that sits flat at your top bucket means "add bigger buckets", not "latency is exactly 1 s".

Part 2: sizing metrics storage. 200 services x 50 series each = 10,000 series. At a 15 s scrape, that is 10,000 / 15 = 667 samples per second. Over 30 days: 667 x 86,400 x 30 = 1.73 billion samples. At the Prometheus rule of thumb of 1 to 2 bytes per sample, disk is 1.7 to 3.5 GB; using the docs formula retention_time_seconds x ingested_samples_per_second x bytes_per_sample with 1.5 bytes: 2,592,000 x 667 x 1.5 = 2.6 GB. Tiny. Now the realistic case: each service has 3 histograms with 15 buckets across 5 routes and 5 statuses, plus the kube-state-metrics and cAdvisor series for its pods, about 2,000 series per service. 400,000 series, 26,667 samples per second, 69 billion samples in 30 days, about 104 GB at 1.5 bytes. Still fine on one disk; what breaks first is memory, since every active series costs a few KB in the head block, so 400,000 series needs on the order of a few GB of RAM, and 17 million (the user_id mistake) needs a machine that does not exist.