Interviews for DevOps and SRE roles now include a coding round. The common tasks are "write a health checker with retries", "parse these logs and find the top errors", "call this API with pagination and summarise". Python answers them fastest. Platform and CNCF-adjacent roles add Go: "write a worker pool", "what is a goroutine leak", "how would you read client-go". LFX Mentorship and GSoC projects in Kubernetes, Argo, Prometheus or Cilium require Go on day one. Knowing which language to reach for, and the standard shape of a program in each, is what the interviewer is checking.

Why two languages

The honest division of labour: write the health check, the cost report and the log parser in Python. Write the controller, the admission webhook, the kubectl plugin and the agent in Go. If a Python script grows a long-running loop and a concurrency problem, that is the signal to rewrite it in Go.

The shape of a good Python ops script

Every ops script you write in Python should have the same skeleton: argparse for flags including --dry-run, logging to stderr so stdout stays parseable, subprocess.run with check=True and a timeout, HTTP calls with retries and backoff, pathlib for files, and explicit exit codes. The reader of your script is the on-call engineer at 3 a.m., and the shape tells them where to look.

#!/usr/bin/env python3
"""Drain a node safely: cordon, wait for pods to leave, with dry-run and timeouts."""
import argparse
import json
import logging
import subprocess
import sys
import time
import urllib.request
import urllib.error
from pathlib import Path

log = logging.getLogger("drain")


def run(cmd, timeout=30, dry_run=False):
    """Run a command; raise on non-zero exit; never hang forever."""
    log.info("run: %s", " ".join(cmd))
    if dry_run:
        return ""
    result = subprocess.run(cmd, check=True, capture_output=True, text=True, timeout=timeout)
    return result.stdout


def get_json(url, retries=3, backoff=0.5, timeout=5):
    """GET with retries and exponential backoff; retry only on 5xx and network errors."""
    for attempt in range(retries):
        try:
            with urllib.request.urlopen(url, timeout=timeout) as resp:
                return json.load(resp)
        except urllib.error.HTTPError as e:
            if e.code < 500:
                raise  # 4xx is our bug, do not retry
            log.warning("attempt %d: HTTP %d", attempt + 1, e.code)
        except (urllib.error.URLError, TimeoutError) as e:
            log.warning("attempt %d: %s", attempt + 1, e)
        time.sleep(backoff * 2 ** attempt)
    raise RuntimeError(f"gave up on {url} after {retries} attempts")


def main(argv=None):
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("node")
    p.add_argument("--timeout", type=int, default=300, help="seconds to wait for pods to leave")
    p.add_argument("--dry-run", action="store_true")
    p.add_argument("-v", "--verbose", action="store_true")
    args = p.parse_args(argv)
    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO,
                        format="%(asctime)s %(levelname)s %(message)s", stream=sys.stderr)
    try:
        run(["kubectl", "cordon", args.node], dry_run=args.dry_run)
        run(["kubectl", "drain", args.node, "--ignore-daemonsets", "--delete-emptydir-data",
             f"--timeout={args.timeout}s"], timeout=args.timeout + 10, dry_run=args.dry_run)
    except subprocess.CalledProcessError as e:
        log.error("command failed (%d): %s", e.returncode, e.stderr.strip())
        return 2
    except subprocess.TimeoutExpired:
        log.error("drain did not finish in %d s", args.timeout)
        return 3
    Path("/tmp/last-drained").write_text(args.node)
    print(json.dumps({"node": args.node, "drained": not args.dry_run}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Things to notice. check=True turns a failed command into an exception instead of a silent zero. timeout means a hung kubectl cannot hang your CronJob. The retry loop retries only on 5xx and network errors, because retrying a 403 just wastes time. Logs go to stderr and the one line of real output goes to stdout as JSON, so the next tool in the pipe can read it. Exit codes are distinct, so the caller can tell "command failed" from "timed out". For YAML, import yaml and yaml.safe_load (never yaml.load without a Loader) cover Kubernetes manifests and Helm values; requests with a Session and a Retry adapter is the usual upgrade from urllib when the package is available.

Calling cloud APIs from Python

boto3 is the AWS SDK. Three rules. First, never put keys in code or in the repository: boto3 reads credentials from the environment, then from the pod's web identity token (IRSA on EKS), then from an SSO session, then from ~/.aws/credentials, and last from the container or instance role (Pod Identity on EKS, an EC2 instance profile), in that order. Second, every list call is paginated and the default page hides the rest, so use a paginator. Third, pass a region explicitly in tooling that runs in CI.

import boto3

ec2 = boto3.client("ec2", region_name="ap-south-1")
paginator = ec2.get_paginator("describe_instances")
untagged = []
for page in paginator.paginate(Filters=[{"Name": "instance-state-name", "Values": ["running"]}]):
    for reservation in page["Reservations"]:
        for inst in reservation["Instances"]:
            tags = {t["Key"]: t["Value"] for t in inst.get("Tags", [])}
            if "owner" not in tags:
                untagged.append(inst["InstanceId"])
print(len(untagged), "running instances have no owner tag")

Testing scripts like these is where most candidates stop, and where you can stand out. pytest plus unittest.mock lets you test the logic without a cluster:

from unittest.mock import patch
import subprocess
import drain  # the script above, importable because the logic is in main()


def test_dry_run_does_not_call_kubectl():
    with patch("drain.subprocess.run") as fake:
        assert drain.main(["node-1", "--dry-run"]) == 0
        fake.assert_not_called()


def test_failed_drain_returns_2():
    err = subprocess.CalledProcessError(1, ["kubectl"], stderr="node not found")
    with patch("drain.subprocess.run", side_effect=err):
        assert drain.main(["node-1"]) == 2

Mock the boundary (subprocess, HTTP, boto3 client), not your own functions. For boto3, botocore.stub.Stubber replays canned responses.

Contracts first: OpenAPI specs and generated clients

An OpenAPI spec is a YAML or JSON document that describes an HTTP API: every path, method, parameter, request body, response schema and status code. It is the contract between the team that owns the service and everyone who calls it. Ops tooling should not hand-write requests.get(f"{base}/v1/nodes/{name}") calls against such an API. Generate a client from the spec instead: oapi-codegen produces Go client code, server stubs and typed models from an OpenAPI 3.0 or 3.1 document, and openapi-python-client produces a typed Python package with one function per endpoint and a class per schema. A generated client gives you the field names, the types and the enum values at compile time or in your editor, and when the service changes its spec, regenerating the client shows you every call that broke, instead of a 400 at runtime. The same spec is what an API gateway or a middleware validates incoming requests against: a body with a missing required field is rejected at the edge, before your handler runs. Keep the spec in the repo, lint it in CI and treat a spec change like a schema migration.

Go in one page for a Python person

Go is small. Everything you need for ops tooling fits in one page of differences.

package main

import (
	"errors"
	"fmt"
	"os"
)

// A struct is a typed record. Fields that start with a capital letter are exported.
type Target struct {
	Name string
	URL  string
	Tags map[string]string // maps must be made before use: make(map[string]string)
}

// Methods attach to a type. The receiver (t Target) is like self.
func (t Target) String() string { return t.Name + " -> " + t.URL }

// An interface is satisfied by any type that has the methods. No "implements" keyword.
type Checker interface {
	Check(t Target) error
}

// Errors are values. The caller must look at them; nothing is thrown.
var ErrNoURL = errors.New("target has no URL")

func validate(t Target) error {
	if t.URL == "" {
		return fmt.Errorf("validate %s: %w", t.Name, ErrNoURL) // %w wraps, errors.Is can find it
	}
	return nil
}

func main() {
	targets := []Target{{Name: "api", URL: "https://api.example.com"}, {Name: "bad"}} // a slice
	targets = append(targets, Target{Name: "db", URL: "tcp://db:5432"})

	f, err := os.CreateTemp("", "report")
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer os.Remove(f.Name()) // defer runs when main returns, in reverse order
	defer f.Close()

	for i, t := range targets { // range gives index and copy of the value
		if err := validate(t); err != nil {
			if errors.Is(err, ErrNoURL) {
				fmt.Printf("%d: skipped: %v\n", i, err)
			}
			continue
		}
		fmt.Fprintln(f, t) // uses String() because Target satisfies fmt.Stringer
	}
}

The translation table: list is a slice ([]T, grows with append), dict is a map (map[K]V, must be maked), class is a struct plus methods, duck typing is an interface (satisfied implicitly), try/except is if err != nil, with is defer, and None is nil. Unused imports and variables are compile errors, gofmt decides the formatting, and there are no exceptions, no inheritance and no generics tricks you need for tooling. Go has had generics since 1.18, and you will see [T any] in library code, but ops tools rarely need to write them.

Goroutines, channels, context and the worker pool

A Go worker pool: a producer sends jobs into a channel, four worker goroutines read them, each check honours a context deadline, results flow through a buffered channel to the main goroutine which waits on a WaitGroup

A goroutine is a function running concurrently, started with the keyword go. It costs a few kilobytes, so ten thousand of them are fine. A channel is a typed queue that goroutines use to hand values to each other; sending on an unbuffered channel blocks until someone receives. A WaitGroup counts running goroutines so main can wait for them. A context carries a deadline and a cancel signal down every call; every function that might block takes ctx context.Context as its first argument and gives up when ctx.Done() fires.

The worker pool is the pattern you will write most: N workers read jobs from one channel and write results to another. It bounds concurrency (you do not open 10000 connections to the API server), it collects results without locks, and with a context it has a hard deadline. The code example implements it with four workers over twelve checks, and the standard library net/http client takes the same context through http.NewRequestWithContext, so a real version is a small change.

The standard library, modules and building

Go's standard library covers most of an ops tool: net/http for clients and servers (http.ListenAndServe and http.HandleFunc give you a /healthz endpoint in five lines), encoding/json, os/exec for running commands with a context (exec.CommandContext), flag for simple flags, log/slog for structured logs, time, sync. A module is a directory with a go.mod file naming the module path and the Go version; go mod tidy records every dependency with its exact version and hash in go.sum, so builds are reproducible.

go mod init github.com/you/checker        # creates go.mod
go build -o checker .                     # a static binary for this OS and CPU
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" -o checker-linux-arm64 .   # cross-compile, stripped
go vet ./... && go test ./...             # the two commands every CI job runs

CGO_ENABLED=0 makes the binary fully static, which is what lets it run FROM scratch or distroless in a container. A real CLI with sub-commands, help text and shell completion is usually built with the cobra library (the one behind kubectl, helm and gh): a root command, AddCommand for each sub-command, flags bound to variables, and a RunE function that returns an error. You do not need it for a single-purpose tool; flag is enough.

Measure before you argue: benchmarking languages

"Go is faster than Python" is a claim, not a number. Write the same hello handler (return {"ok":true} with a 200) in Node, Python and Go, run each on the same machine, and point one load generator at them in turn: wrk -t4 -c100 -d30s --latency against the local port prints requests per second and a latency distribution, and k6 reports http_req_duration with a threshold such as p(99)<300. Record requests per second and the p99 for each, with the CPU count, the framework and the flags used. Any number you quote in an interview or a design doc is from your own run and labelled as such ("on my 4-core laptop, Go stdlib served N requests per second"). Numbers copied from a blog post describe someone else's machine.

Table-driven tests

Go tests live in _test.go files, use the testing package, and are run with go test. The idiom is a slice of cases and one loop:

func TestParseCIDR(t *testing.T) {
	cases := []struct {
		name    string
		in      string
		wantN   int
		wantErr bool
	}{
		{"slash24", "10.0.0.0/24", 256, false},
		{"slash28", "10.0.0.0/28", 16, false},
		{"garbage", "not-a-cidr", 0, true},
	}
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			n, err := countAddresses(tc.in)
			if (err != nil) != tc.wantErr {
				t.Fatalf("err = %v, wantErr %v", err, tc.wantErr)
			}
			if n != tc.wantN {
				t.Errorf("got %d, want %d", n, tc.wantN)
			}
		})
	}
}

go test -race ./... runs the race detector, which finds the data races that only show up under load in production. Use it in CI for anything with goroutines.

Common Go mistakes in ops tools

Reading client-go

Kubernetes controllers in Go all look the same once you know the shape: a clientset (or a controller-runtime Client) talks to the API server, an informer keeps a local cache of objects and calls your handler on add, update and delete, a work queue holds the keys of objects to reconcile, and a reconcile function reads the current object, compares it with the desired state, and makes one change. The reconcile function must be idempotent, because the same key is delivered again after errors and on every resync. The go-reconcile-loop problem has you write that function without client-go; lesson 25 shows the real thing with controller-runtime and Kubebuilder.

Sizing a worker pool for a health checker. You must check 1200 endpoints every minute from one pod. Each check is an HTTP GET with a p50 of 150 ms and a p99 of 2 s, and 3 percent of endpoints are down and take the full timeout.

Step 1, serial time. 1200 checks at an average of about 200 ms (the mean sits above the median because of the slow tail) is 240 s. A 60 s cycle is impossible serially; you need at least 4 times parallelism just to break even.

Step 2, choose the timeout. The p99 is 2 s, so a 3 s context timeout loses almost nothing. The 36 down endpoints (3 percent of 1200) will each take 3 s.

Step 3, workers. With N workers the cycle takes roughly (total work) / N. Total work is 1164 x 0.2 s plus 36 x 3 s, which is 232.8 s plus 108 s, which is 340.8 s. For a 30 s cycle (half the budget, leaving room for jitter) N must be at least 340.8 / 30, which is 11.4, so 12 workers. Choose 16 for headroom.

Step 4, the channels. The jobs channel can be unbuffered: the producer blocks until a worker is free, which is the back-pressure you want. The results channel is buffered to 1200, so a worker never blocks on sending while main is still feeding jobs; a deadlock here is the classic leak. One WaitGroup counts 16 workers; main closes jobs after the last send, waits on the group, then closes results and reads them.

Step 5, memory and connections. 16 goroutines at about 8 KB each is trivial. 16 in-flight HTTP connections is fine for any target. If the targets share a few hosts, set http.Transport.MaxConnsPerHost so you do not become the DDoS.

Step 6, the Python comparison. The same design in Python uses concurrent.futures.ThreadPoolExecutor(max_workers=16), which works because the checks are I/O bound and release the GIL. It is a fine choice if the checker runs as a CronJob. It becomes the wrong choice when the checker also exposes /metrics, keeps state across cycles and runs for months: then the Go version's static binary, race detector and goroutine metrics pay for themselves. The code example runs the Go design at small scale and proves the goroutine count returns to its starting value.