SREs are engineers, and the engineering half of the job has to produce something. This chapter is about what that looks like when it works: an SRE team identifies a problem that only operations people really see, builds a tool for it, and the tool becomes infrastructure used across the company. The case study is Auxon, which replaced hand-made capacity plans with an optimiser fed by declared intent.

The second half is organisational: how to start such a project, how to staff it, and how to make it survive.

In one minute

• SRE is well placed to write production software because SREs understand the whole stack, see the failure modes, and feel the pain of operational toil directly.
• Traditional capacity planning is a loop of forecast, allocate, review, deploy, repeat. It is tedious, fragile when assumptions change, and results in requests like "X cores in cluster Y" that are inflexible and often wrong.
• Intent-based capacity planning declares requirements ("serve 100k QPS per continent with N+2 redundancy, at latency L") and lets a solver produce the allocation. Intent is the why behind a request; the solver handles the how.
• Intent precedence: from the least useful "I want 50 cores in cluster X" through "I want to run in clusters X, Y, Z", "I want to run in any 3 clusters in region R", to "I want 5 nines in geography G", which is the most useful because it leaves the most freedom to the solver.
• Auxon takes as inputs the per-service demand forecast, performance data (how much resource per unit of demand), resource supply, resource pricing, and an intent config, and uses mixed-integer programming to produce an allocation plan.
• The project started as a side project by a few SREs who noticed the pain, built a prototype, got early adopters, and grew; its success depended on a clear product vision, fuzzy requirements handled by designing for change, and launching early to get feedback.
• Fostering software engineering in SRE requires management support for time away from operations, treating the work like any product (reviews, design docs, launches), hiring a mix of skills, and accepting that not every project will succeed.

Why is software engineering within SRE important?

SREs have a unique vantage point: they see the common failure patterns, the places where humans intervene, and the mismatch between what the system was designed for and how it is actually used. When they build tools, the tools solve the real problems and are designed with scalability, graceful degradation and operability in mind from the start. The guiding principle is that team size should not scale with service growth; software is how that is achieved.

Auxon case study: project background and problem space

At Google, capacity planning was a quarterly ritual. Each team forecast demand, translated it into machine requests, negotiated with the capacity planners, and then watched the plan become wrong as launches slipped and budgets changed. Requests were expressed in implementation terms (so many machines in such a cluster), which left the planners no room to optimise and made every change a renegotiation.

The insight was to separate intent from implementation. A service owner knows what they need to achieve: a level of redundancy, a latency target, a set of regions. Given that intent plus data about how the service consumes resources, a program can bin-pack far better than humans, and can redo the whole plan in minutes when an input changes.

Introduction to Auxon

Auxon's components: the performance data describes how a service scales (per unit of load, how many of each resource); the demand forecast data gives expected load over time per region; the resource supply says what exists and when; resource pricing supplies the cost function to minimise; and the intent configuration says how services depend on each other and what constraints must hold. The solver, built on mixed-integer linear programming, produces a plan listing which resources go to which service in which cluster. The plan is then reviewed and turned into actual allocations.

Because requirements were fuzzy and would change, Auxon was designed in layers so that the solver, the input formats and the user-facing config language could evolve independently. Launching early with a subset of users surfaced the real requirements faster than any design review could.

Raising awareness and driving adoption

The team spent effort on evangelism: presentations, documentation, finding a few teams whose pain was acute and getting them onto the system first. Their success stories brought in the next adopters. Setting expectations mattered: Auxon was not magic, and it was explicit about which inputs it needed and how good the forecast had to be.

Team dynamics

The core team was small and combined people who had deep SRE experience with people who had a statistics and optimisation background, which the SREs did not. The chapter is candid that the project needed senior leadership support to protect time for it, and that it faced the usual pressure to go back to operations whenever things got busy.

Fostering software engineering in SRE

Ideas for projects come from the toil the team already feels. Pick projects that are important and that SRE is uniquely placed to solve. Treat them as products: write design documents, do code reviews, plan launches, measure adoption. Staff them with people who want to do software work and make sure they get the career credit for it. Accept that some projects fail, and make sure that lesson is cheap by starting small. The result, when it works, is a team whose operational load shrinks while its influence grows.