The last chapter of DDIA is Kleppmann's argument about how data systems should be built. The preceding eleven chapters established that no single tool fits every access pattern, that distributed transactions across heterogeneous systems are painful, and that logs of immutable events are a sound way to replicate and derive data. Chapter 12 combines these into a design philosophy: treat the whole organisation's data as a set of derived datasets flowing through ordered logs, build correctness end to end rather than trusting any one component, and take responsibility for what the resulting systems do to people.

It is the chapter most often skipped and the one most useful in a senior system-design interview, because it supplies the vocabulary for explaining why you chose a Kafka pipeline over a two-phase commit, how you prevent duplicate charges without XA, and when "eventually consistent" is acceptable.

In one minute

• No tool fits all uses, so every realistic architecture combines specialised systems; the problem is data integration: keeping the copies consistent.
• Route all writes through one totally ordered log and have every other system consume it; this gives deterministic derivation without distributed transactions, at the cost of asynchrony.
• Total order has limits: it cannot span datacenters or microservices cheaply and it ignores causality that bypasses the log. Capturing causal dependencies without a global order remains open.
• Batch and stream are the same idea at different latencies; the lambda architecture ran both in parallel, and newer engines (Spark, Flink, Beam) unify them.
• Unbundling the database: a database is an index, a log, a cache and a query engine fused together; the same components can be composed across systems with stream processors as the glue, like Unix tools with pipes.
• Design applications around dataflow: application code is a derivation function from input events to output state, and derived state (indexes, caches, notifications) is pushed to clients as a stream rather than polled.
• Correctness needs the end-to-end argument: deduplication, uniqueness and atomicity must be enforced by the application using operation identifiers, because no transport layer can guarantee exactly-once across the whole path.
• Uniqueness constraints in log-based systems come from partitioning by the constrained value and processing serially within the partition; multi-partition constraints decompose into per-partition steps.
• Timeliness (users see up-to-date data) and integrity (no corruption or loss) are different; stream systems give up some timeliness but can keep integrity through idempotence and immutable logs, and many constraints can be enforced loosely with compensation.
• Trust but verify: auditing, checksums and periodic reconciliation catch bugs that no proof prevents.
• Data systems shape people's lives; predictive analytics, surveillance and data retention are ethical choices engineers make, not externalities.

Data integration

Combining specialised tools

The recurring lesson of the book is that every choice is a trade-off: an OLTP store, a search index, an analytics warehouse, a cache, a recommendation model, each is optimised for one access pattern and is poor at others. A real application uses several, so the same data is copied into several representations: the user table in PostgreSQL, the same users in Elasticsearch for search, in Redis for sessions, in the warehouse for reporting. The hard part is data integration: making sure all those copies agree, and knowing which one is right when they do not.

Reasoning about dataflows

Chapter 11 showed the two choices. Dual writes from application code are wrong: concurrent writers can apply changes to two systems in different orders, and partial failures leave them inconsistent. The alternative is to decide where each kind of write happens first (the system of record) and derive everything else from its change log, through CDC or event sourcing. With a single leader and a total order of writes, all derived systems apply the same changes in the same order and converge, and a derived system can be rebuilt from the log at any time. This is exactly the state machine replication principle from Chapter 9, applied across heterogeneous systems rather than replicas of one database.

Derived data versus distributed transactions

Distributed transactions (XA/2PC) also keep heterogeneous systems consistent, so why prefer logs? The two approaches order writes differently. Transactions use locks and atomic commit, giving linearizability and the useful guarantees that follow (read your own writes immediately). Log-based derivation uses a log for ordering and processes changes asynchronously, giving no timeliness guarantee: derived systems are eventually consistent. In return, derived systems are decoupled: a slow or failed consumer does not block the system of record, retries are safe, and each system can be replaced or rebuilt independently.

XA's practical problems (Chapter 9) mean most heterogeneous deployments cannot use it anyway. Kleppmann's view is that for integration between different systems, log-based derived data is the most promising approach, and that the lack of good consensus-based transaction protocols across heterogeneous storage is a gap, not a reason to return to dual writes.

Limits of total ordering

A single totally ordered log is powerful, but it has limits as systems scale:

• A total order needs a single leader (or a consensus group), so throughput is bounded by one node and the log cannot be partitioned without losing cross-partition ordering.
• Across geographically distributed datacenters, a single leader means a round trip to another continent on every write; in practice each datacenter has its own leader, so there is no global order.
• Microservices each with their own durable state have no shared order between services.
• Clients with local state (offline-capable apps) perform operations that the server sees much later.

In each case, events in different partitions, services or regions have no defined order relative to each other. Total order broadcast is equivalent to consensus, and consensus across thousands of nodes or across continents is slow or infeasible.

Ordering events to capture causality

Without a total order, causality can be violated. A user removes a friend, then posts a private message; if the "unfriend" and the "post" travel through different partitions, the ex-friend's notification service may see the post first and deliver it. The events have a causal dependency that the system failed to capture. Possible mitigations: give the event a logical timestamp (Lamport clock) so consumers can at least detect reordering; have the writing node record the state it read (the friend list version) and attach that identifier to the event so consumers can check; or use conflict-free data types. None is a complete solution, and the chapter is honest that cheap causal ordering without a central sequencer remains an open research problem.

Batch and stream processing

The same thing at different speeds

Data integration's output is derived views kept up to date; batch and stream processing are the two tools for producing them. Chapter 10 and 11 showed they share ideas: immutable inputs, deterministic derivation functions, outputs written to new locations, and retries safe because the inputs are not consumed. The main differences are that stream processors handle unbounded data and must reason about time, while batch processors can sort and see the whole input.

Derived views also give gradual evolution: to change a schema or a derivation, build the new view alongside the old one from the same log, let clients migrate, then drop the old one. No risky in-place migration is needed, because the old view is never mutated.

The lambda architecture

Nathan Marz's lambda architecture proposed running a batch system (Hadoop) and a stream system (Storm) in parallel over the same immutable event log. The stream layer produces approximate, fast views; the batch layer periodically reprocesses the entire history to produce accurate views that replace the stream output. Its strengths are the immutable log and the ability to rebuild from history. Its weaknesses: the same logic is written twice in two frameworks, the two outputs must be merged for queries, and reprocessing the whole dataset every time is expensive, so people batch incrementally and reintroduce the complexity of windows and late data they wanted to avoid.

Unifying batch and stream

Newer systems remove the duplication. The goal is one engine that can replay historical events through the same code as live events, which requires: replaying history from a log-based broker or a filesystem (Kafka retention, HDFS archives), exactly-once semantics in the stream engine (so reprocessing gives batch-quality results), and windowing by event time rather than processing time, so the result does not depend on when the job ran. Apache Beam's model, implemented by Flink, Spark and Google Cloud Dataflow, treats a bounded dataset as a special case of a stream. Flink runs batch jobs as streams with a known end; Spark runs streams as microbatches; both expose one API.

Unbundling databases

Composing data storage technologies

A database is a bundle of features: secondary indexes, materialized views, replication logs, full-text search. Each is a derived dataset maintained by the database from the primary data, and each also exists as a separate product outside the database:

Inside a database	Outside, as a separate system
Secondary index: CREATE INDEX scans the table then keeps it updated on every write	A stream processor consumes the change log and maintains a derived view
Materialized view	A cache or precomputed aggregate updated from a stream
Replication log	A log-based broker (Kafka)
Full-text index	Elasticsearch fed by CDC

Creating an index in a database is itself a derivation: snapshot the table, process the snapshot, then apply the changes that happened meanwhile from the log. This is the same procedure a new follower uses when joining a replication group and the same procedure a stream processor uses to bootstrap a view.

Federated versus unbundled

Two ways to compose storage systems reflect the two Unix-and-database traditions:

• Federated databases (polystore) offer a unified query interface over many backends, such as PostgreSQL foreign data wrappers. They solve the read side: one query language across systems.
• Unbundled databases solve the write side: synchronising writes across systems so that each component gets a consistent view. Unbundling means composing the index, the log, the cache and the query engine from different products, with a stream processor as the glue.

Unbundling is the Unix approach: small, specialised tools communicating through a uniform interface (the event log). Keeping writes consistent across heterogeneous systems is traditionally done with distributed transactions, but the unbundled approach uses asynchronous event logs plus idempotent consumers. The log works because it is loose coupling at both the system level (a failed consumer does not block others; the log buffers) and the human level (different teams own different consumers and need not coordinate deploys).

What is missing

A single integrated database will often outperform a composition of parts for a workload the database supports; the unbundled approach wins when no single product covers the requirements. What is missing today is a "Unix shell" for data systems: a declarative way to say mysql | elasticsearch that creates and maintains an index of a table in a search server with the same ease as CREATE INDEX. Kafka Connect and dbt-style tools gesture at this but the high-level language does not yet exist.

Designing applications around dataflow

Application code as a derivation function

Viewed through this lens, application code is a derivation function: a secondary index is a simple one; a full-text index applies tokenisation and stemming; a machine learning model is a complex one over training data; a cache is a materialized view of a query. Databases offer triggers and stored procedures to run user code, but they are awkward. The unbundled approach runs application code in a stream processor, which gives the code the database's data model but a normal programming language and deployment story.

Separation of application code and state

The common microservice pattern keeps state in a database and stateless application servers in front; services call each other synchronously, and a slow dependency slows the caller. Dataflow proposes an alternative: a service subscribes to the change stream of the data it needs and keeps a local copy (a stream-table join). A purchase service that needs exchange rates does not call the currency service per request; it maintains a local table from the rate change stream, so each request is served locally. Reads are faster and more fault-tolerant (the rate service can be down); the cost is that the local copy may be a few seconds stale and must be sized to fit.

Observing derived state

The write path builds derived state eagerly when data changes; the read path serves queries lazily from it. A derived view is where the two meet, and the boundary can move: a search index precomputes the posting lists (write path) and intersects them at query time (read path); caches precompute more; databases without indexes do everything on read.

Traditionally the read path is request/response: a client asks, the system answers, and the client polls for changes. End-to-end event streams extend dataflow to the client: a change is pushed through the server to the browser or app as a stream (WebSockets, server-sent events), and the client's local state is itself a derived view that updates. Offline-first clients hold state locally and sync later, which is a replica with replication lag. The tools exist (React, Flux, Elm, Redux manage client state as reductions of event streams), but the server-to-client channel is still mostly request/response rather than subscription.

Reads as events. Writes are recorded as events and processed; reads could be too. A read request becomes an event in the log, the stream processor joins it with its state and emits the response to an output stream. This makes reads part of the causal record (useful for auditing "what did the user see when they decided?"), enables stream-table joins with reads, and allows multi-partition data processing: a query that needs several partitions is routed through the log so the ordering of reads and writes is deterministic. The cost is latency and a log full of read traffic; it is a design option, not a default.

Aiming for correctness

The end-to-end argument

Chapters 7 to 9 covered the guarantees a database can give. Yet correct applications are built on top of databases with weak isolation, by careful engineers. Kleppmann argues that correctness should be reasoned about end to end.

The end-to-end argument (Saltzer, Reed and Clark, 1984) says that a function like duplicate suppression or integrity checking can only be completely and correctly implemented with the knowledge and help of the application at the endpoints; implementing it in a lower layer is at best a performance optimisation. TCP deduplicates packets, but it cannot stop a user double-clicking "pay" and sending two independent requests, or a client retrying after a timeout and a successful first request, or a database transaction that committed but whose acknowledgment was lost. Each layer deduplicates within itself and still the end-to-end flow produces duplicates.

The remedy is an operation identifier generated at the source: the client creates a unique ID per intended operation (a UUID in a hidden form field, a request key), and every layer that might duplicate the request can check it. In the database, store the ID in a table with a uniqueness constraint; a retried request violates the constraint and is rejected. The write then becomes idempotent. This is why payment APIs expose idempotency keys. Database transactions alone cannot provide it, because they do not know about the retry.

Uniqueness constraints in log-based systems

Uniqueness is the canonical constraint needing linearizability (Chapter 9). In an unbundled system without a global lock, it can still be enforced:

1. Partition the log of requests by the value that must be unique: a username claim goes to the partition for hash(username), so every claim on a given name lands in one partition in a fixed order.
2. A stream processor consumes each partition sequentially, keeps a local table of claimed names, and for each request emits "success" or "rejected" to an output stream.
3. The client waits for its decision on the output stream (correlated by request ID).

This is total order broadcast within a partition and scales linearly with partitions, since each is independent. It generalises to any constraint on one key: account balances (partition by account), stock levels (by product), seat booking (by flight).

Multi-partition request processing

A bank transfer touches two accounts, likely in different partitions, and must not credit without debiting. Without atomic commit across partitions, decompose it:

1. The client appends a single transfer request with a unique ID to a log partitioned by request ID (so the whole request is atomically present or not).
2. A processor reads the request and emits two events: a debit instruction to A's partition and a credit instruction to B's partition, each tagged with the request ID.
3. Processors for A and B apply their instruction idempotently, using the request ID to ignore duplicates.

Because the request was durable before being split, a crash between steps 2 and 3 is recovered by replay; the request ID deduplicates. There is never a moment when both half-operations are committed inconsistently with the log. The trade-off is that the system is eventually consistent across the partitions, and a reader could briefly see the debit without the credit. For many constraints that is acceptable; for a hard constraint it is not.

Timeliness and integrity

The word "consistency" conflates two requirements:

• Timeliness: users observe the system in an up-to-date state. Violations (stale reads) are inconvenient but temporary; waiting fixes them. Read-after-write and linearizability are timeliness guarantees.
• Integrity: no corruption, no loss, no contradictory data; an index agrees with its table, debits equal credits. Violations are permanent until explicitly repaired and usually undetected. ACID atomicity and durability are integrity guarantees.

Kleppmann's claim is that integrity matters far more than timeliness, and that log-based dataflow systems are good at integrity even while relaxing timeliness: durable, immutable events; deterministic derivation; idempotent consumers; end-to-end request IDs. Together these give exactly-once effects without distributed transactions. ACID transactions provide both at once; dataflow decouples them, so that violated timeliness causes no violated integrity.

Loosely interpreted constraints

Many constraints need not be enforced strictly in advance. Overselling a product by a few units can be handled by apologising and compensating; overbooking an aircraft is standard practice; a bank allows an overdraft and charges a fee; two users getting the same username can be resolved by asking one to choose another. In each case the cost of a compensating transaction after the fact is lower than the cost of coordination before it. The business processes around a system already tolerate mistakes, so the system can be designed to detect and fix integrity violations rather than prevent all of them synchronously.

Coordination-avoiding systems

Combining these ideas yields systems that keep integrity with far less coordination than a serializable, linearizable database:

1. Dataflow systems can maintain integrity of derived data without atomic commit, linearizability or synchronous cross-partition coordination.
2. Strict uniqueness constraints require timeliness and coordination, but many constraints can be loosened to be checked asynchronously and repaired, which restores availability and reduces latency.

Such coordination-avoiding designs can replicate across datacenters asynchronously and stay available under partitions, where a linearizable system cannot. The cost is occasional compensations and the engineering of idempotence; the gain is throughput and resilience. Where synchronous coordination is truly needed (a hard safety constraint), it can be confined to the small part of the system that needs it rather than applied everywhere.

Trust, but verify

All of this reasons under a system model: disks persist, memory is reliable, software is correct. Reality violates each. Disks corrupt data silently, memory flips bits, operating systems have bugs, and databases have had isolation bugs for years (MySQL and PostgreSQL both shipped versions with serializability bugs that went unnoticed because the anomalies were rare). Software bugs are the dominant source of data corruption, and no ACID guarantee protects against application code that writes wrong data.

The answer is auditing: verifying integrity instead of assuming it. Checksums on every write (as HDFS and ZFS do), periodic comparison of derived views against their source (reconciling a search index against the database it should mirror), recomputing aggregates from the raw log and diffing, and end-to-end audits of money flows as accountants have always done. Dataflow architectures help because the log is a complete record of inputs: a derived view is auditable by rederiving it. Cryptographic auditing (Merkle trees, certificate transparency, blockchain-style ledgers) extends this to parties who do not trust each other. A system that has never been audited has unknown integrity, however many transactions it offers.

Doing the right thing

Predictive analytics and algorithmic decisions

Systems increasingly make or inform consequential decisions: who gets a loan, who is screened at a border, who is shown a job ad. A model trained on historical data reproduces historical bias and gives it the appearance of objectivity; a feedback loop then makes its predictions self-fulfilling (deny credit to a group, the group's defaults rise). Decisions are hard to contest because no human made them and the model's reasoning is opaque. The engineer who builds the pipeline bears responsibility for these outcomes: "the algorithm did it" is not an excuse.

Privacy and tracking

Services that collect behavioural data are not only responding to user requests but tracking them, and much of the tracking exists to serve advertising or to train models rather than to serve the user who generated the data. Consent is rarely informed: privacy policies are not read, and users cannot see what is inferred from their data. Data collected for one purpose is routinely reused for others, breached, or subpoenaed. Surveillance is the accurate word for pervasive behavioural logging, and reasoning about a feature with that word substituted is a useful honesty check.

Data as assets and liabilities

The industry treats data as an asset ("data is the new oil"). Kleppmann argues it is also a liability: every record retained is a record that can be leaked, misused, or demanded by a hostile future government. Privacy is not secrecy but the right to choose what to reveal to whom, and users have lost that choice. The design principles of this book cut both ways: immutable logs and derived views make deletion hard, which is good for integrity and bad for a user's right to be forgotten.

Legislation and self-regulation

Data protection law (the GDPR in the EU, and successors elsewhere) tries to restore some balance with purpose limitation, data minimisation and the right to deletion. Legislation lags technology, so the chapter calls for a culture shift inside engineering: collect less, encrypt more, expire data by default, give users real access to and control over what is known about them, and treat the people whose data flows through the system as stakeholders rather than resources. The final argument of the book is that building reliable, scalable and maintainable systems is necessary but not sufficient; they must also do right by the people whose lives they touch.

Design trade-offs

Decision	Option	Choose when
Keeping systems in sync	Distributed transactions (XA)	Legacy systems that cannot be restructured and need linearizable consistency
	Log-based derivation (CDC, event sourcing)	Any new integration; tolerable staleness; need for replay and independent rebuild
Ordering	Single global log	Modest scale, one region; strong ordering needs
	Per-partition or per-region logs	High throughput, multi-region, microservices; accept lost cross-partition order
Batch vs stream	Lambda (parallel batch and stream)	Historical; avoid in new designs
	Unified engine (Flink, Beam, Spark)	Replaying history with the same code as live
Enforcing constraints	Synchronous, linearizable	Hard safety constraints; the cost of a violation is unacceptable
	Partitioned log with serial processor	Single-key uniqueness or balances at scale
	Loose constraint with compensation	Violations are cheap and business processes already tolerate them
Deduplication	Rely on transport layer or transaction	Never sufficient on its own
	End-to-end operation IDs	Always, for any operation that must not repeat

Interview angle

• Why does Kleppmann prefer log-based derived data over distributed transactions for integrating systems? Transactions give linearizability but XA across heterogeneous systems blocks on coordinator failure, holds locks, and couples the availability of every participant. A single ordered log gives all consumers the same changes in the same order, so they converge deterministically, can be rebuilt by replay, and fail independently. The price is asynchrony: derived views lag the source.

• A user double-clicks "Pay" and gets charged twice even though the database uses serializable transactions. What went wrong, and how do you fix it? The two clicks are two independent requests; the transaction isolation protects each one, not their relationship. Apply the end-to-end argument: generate an idempotency key on the client for the intended payment, store it with a uniqueness constraint in the database, and reject a second request with the same key. Deduplication in TCP, in the load balancer or in the database alone cannot know the two requests are the same intent.

• How do you enforce unique usernames in a system built on Kafka without a global lock? Partition the claim log by hash(username) so all claims on a name land in one partition in a fixed order. A single-threaded consumer per partition keeps a table of taken names and emits accept or reject per request to an output topic; the client awaits its decision by request ID. This is per-partition total order broadcast; it scales with partitions because names are independent.

• Explain timeliness versus integrity, with an example of each failing. Timeliness: a user updates their profile and briefly sees the old version from a lagging replica; waiting fixes it. Integrity: a payment debited one account but the credit was lost, or an index points to a deleted row; it never fixes itself. Dataflow systems trade some timeliness for availability but can keep integrity through durable events, deterministic derivation and idempotent, request-ID-deduplicated consumers.

• Critique the lambda architecture. Its immutable event log and ability to recompute from history are right. Its parallel batch and stream layers duplicate logic in two frameworks, require merging two outputs at query time, and make full reprocessing expensive, so teams add incremental batching and inherit the stream problems anyway. Unified engines with event-time windows and exactly-once semantics let one codebase handle both replay and live data.

• When should a constraint be enforced loosely instead of strictly? When the cost of a violation plus compensation is lower than the cost of coordination: overselling by a few units, overbooking, overdrafts. Model the violation explicitly, detect it asynchronously and repair with a compensating action. Keep strict synchronous enforcement for constraints where violation is unacceptable, and confine it to the smallest part of the system.

• What does "trust but verify" mean for a data pipeline? Assume disks, networks, databases and your own code can corrupt data silently, and check: checksum stored data, reconcile derived views against their sources on a schedule, recompute aggregates from the raw log and diff, and alert on discrepancies. Immutable logs make this cheap because any view can be rederived and compared.

Pitfalls and misconceptions

• "Exactly-once delivery solves duplicates." Transport-level deduplication cannot see application-level retries or user double submissions. Only an end-to-end operation ID does.
• Treating a single Kafka cluster as a global total order. Order holds within a partition only; across partitions, regions and services there is no order, and causal dependencies can be violated.
• Thinking eventual consistency means eventual integrity. Timeliness heals with time; integrity violations do not. Design for integrity first.
• Using a database's transactions as proof of correctness. Transactions protect against concurrency anomalies inside the database, not against wrong application logic, lost acknowledgments or corrupted disks. Audit.
• Enforcing every constraint strictly. Most business constraints already tolerate violations in the real world; synchronous enforcement everywhere buys little and costs availability.
• Unbundling for its own sake. A single integrated database often beats a composition of six systems for a workload it supports; unbundle when no single product covers the requirements.
• Treating ethics as someone else's job. The engineer who designs the retention policy and the model features decides who is harmed by them.

Key terms

Term	Meaning
Data integration	Keeping the many copies of the same data in different systems consistent
System of record	The authoritative source from which other copies are derived
Derived dataset	Data computed from a source dataset, rebuildable by reprocessing
Total order	A single sequence in which every event has a defined position; needs a single leader or consensus
Lambda architecture	Parallel batch and stream layers over one immutable log, merged at read time
Unbundled database	Composing index, log, cache and query engine from separate products, glued by stream processors
Federated database	A unified query interface over many storage backends
Derivation function	Application code viewed as a function from input events to output state
End-to-end event stream	Pushing state changes through the server to the client as a subscription rather than polling
Reads as events	Logging read requests into the stream so they are ordered and auditable alongside writes
End-to-end argument	Correctness properties such as deduplication must be implemented at the endpoints with application knowledge
Operation identifier (idempotency key)	A client-generated unique ID per intended operation, used to suppress duplicates at every layer
Timeliness	Users see an up-to-date state; violations are temporary
Integrity	No corruption, loss or contradiction; violations are permanent until repaired
Compensating transaction	An after-the-fact correction for a loosely enforced constraint
Coordination-avoiding system	A design keeping integrity with minimal synchronous cross-node coordination
Auditing	Verifying integrity (checksums, reconciliation, rederivation) rather than assuming it
Data as liability	The view that stored data is a risk (breach, misuse, compelled disclosure) as well as an asset

Further reading

• DDIA, Chapter 12: "The Future of Data Systems"; Kleppmann's references at https://github.com/ept/ddia-references
• J. H. Saltzer, D. P. Reed and D. D. Clark, "End-to-End Arguments in System Design," ACM TOCS, 1984
• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction," 2013: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
• Jay Kreps, "Questioning the Lambda Architecture," O'Reilly Radar, 2014
• Peter Bailis et al., "Coordination Avoidance in Database Systems," VLDB 2015
• Pat Helland, "Immutability Changes Everything," CIDR 2015
• Martin Kleppmann, "Turning the database inside-out with Apache Samza," 2015: https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html
• Cathy O'Neil, Weapons of Math Destruction, 2016
• Bruce Schneier, Data and Goliath, 2015