A batch job takes a bounded input and a day later produces an output. Most data is not bounded: users keep clicking, sensors keep reporting, orders keep arriving. Running a batch job every few seconds is one answer, but a cleaner one is to process each event as it happens. That is stream processing, and this chapter is about the infrastructure for it: how events travel from producers to consumers, how databases and streams relate, and how to compute over a never-ending input without losing correctness when nodes fail.

The ideas here are the backbone of modern backend architecture. Kafka-style logs, change data capture, event sourcing, windowed aggregations and exactly-once processing show up in nearly every system-design interview that involves analytics, notifications, caches or search indexes kept up to date.

In one minute

• An event is an immutable record that something happened at a time; a stream is an unbounded, incrementally processed sequence of them, grouped into topics.
• Messaging systems answer two questions: what happens when producers outrun consumers (drop, buffer, backpressure) and what happens when a node crashes (lose messages or persist them).
• Traditional message brokers (RabbitMQ, ActiveMQ, JMS/AMQP) delete a message once acknowledged; delivery order breaks under redelivery and the broker is a transient buffer.
• Log-based brokers (Kafka, Kinesis, Pulsar) append messages to partitioned, durable logs; consumers track an offset per partition, so they can replay, fall behind without harming producers, and preserve order within a partition.
• A database's replication log is a stream. Change data capture (CDC) exposes it to other systems so search indexes, caches and warehouses stay in sync as derived data without dual writes.
• Event sourcing stores application-level events as the source of truth and derives current state; a command is a request that may be rejected, an event is a fact.
• Stream processing uses include complex event processing (pattern matching), streaming analytics (windowed aggregates), maintaining materialized views, and search on streams.
• Distinguish event time from processing time; late events force a choice between waiting and emitting corrections. Windows are tumbling, hopping, sliding or session.
• Stream joins are stream-stream (windowed), stream-table (enrichment against a CDC-maintained table) and table-table (materialized view of a join); all are time-dependent.
• Fault tolerance uses microbatching or checkpointing plus either atomic commit of output and offsets or idempotent writes, so that effects appear exactly once despite retries.

Transmitting event streams

Events, producers and consumers

In streaming the unit is an event: a small, self-contained, immutable record of something that happened, with a timestamp, encoded as JSON or binary (Avro, Protocol Buffers). A producer generates events and one or more consumers process them; related events form a topic. A database table could be the transport, with consumers polling for new rows, but frequent polls waste work and infrequent ones add latency, and databases are poor at notifying clients of new data (triggers and LISTEN/NOTIFY are limited). Hence specialised messaging systems.

Messaging systems

A messaging system pushes events from producer to consumer. Any such system must answer two questions:

1. Producers outrun consumers? Drop messages, buffer them (in memory, then on disk), or apply backpressure, blocking the producer as Unix pipes and TCP do.
2. A node crashes? Lose messages (tolerable for some metrics) or persist and replicate them, at a cost in latency and throughput.

Direct messaging skips intermediaries: UDP multicast for market data feeds, brokerless libraries like ZeroMQ, StatsD metrics over UDP, or HTTP/RPC calls to consumers (webhooks). It is fast but assumes both parties are online; loss handling falls on the application, and a consumer that is down misses messages unless producers retry.

Message brokers

A message broker (message queue) is a server acting as a database for messages: producers write, consumers read, and the broker handles durability, buffering and consumer failure. Because clients come and go, the broker is a buffer that tolerates slow and absent consumers, moving the "what if the consumer crashed" problem onto itself. Queuing means consumption is asynchronous: a producer learns only that the broker has buffered the message, not that a consumer has processed it.

Compared with a database, a traditional broker (RabbitMQ, ActiveMQ, IBM MQ, Azure Service Bus; JMS and AMQP are the standards) deletes a message once delivered, assumes a small working set, selects messages by topic subscription rather than arbitrary query, and pushes changes rather than answering point-in-time queries.

When several consumers read one topic, two patterns are used, sometimes combined:

• Load balancing: each message goes to one consumer in a group; consumers share the work. Useful when messages are expensive to process.
• Fan-out: each message goes to every consumer; independent consumers each get the full stream, like several batch jobs reading one input file.

Acknowledgments and redelivery

Consumers can crash mid-message, so brokers use acknowledgments: the broker removes a message only when the consumer confirms it processed it. If the connection drops first, the broker redelivers, possibly to another consumer. Lose the ack on the network and the message is processed twice, which is where atomic commit and idempotence (later in the chapter) come in.

Redelivery combined with load balancing breaks ordering. If consumer 2 crashes holding message m3 while consumer 1 processes m4, m3 is redelivered after m4 completed. Under JMS and AMQP that is allowed; it matters whenever messages are causally related, and the fix is to use a separate queue per consumer, giving up load balancing.

Partitioned logs

Traditional brokers are transient: once acknowledged, a message is gone, so a new consumer sees nothing before it joined, and a buggy consumer cannot be rerun. Databases and files are the opposite: durable and re-readable. Log-based message brokers combine the two.

A log is an append-only sequence of records on disk, the structure already seen in Chapter 3 (storage engines) and Chapter 5 (replication). Producers append to the log; consumers read it sequentially and, at the end, wait for new appends, like tail -f. To scale beyond one disk, the log is partitioned: each partition is an independent append-only file on its own machine(s), and a topic is a group of partitions carrying the same kind of message. Within a partition the broker assigns each message a monotonically increasing offset; there is no ordering guarantee across partitions.

Apache Kafka, Amazon Kinesis and Apache Pulsar work this way. Throughput is high because appends and sequential reads are cheap, partitions are replicated for durability, and a topic can span many partitions on many machines.

Logs compared with traditional messaging

Fan-out is trivial: several consumer groups read the same log independently. Load balancing is done by assigning whole partitions to consumers in a group, so each consumer reads all messages of its partitions in order, single-threaded. Two consequences: parallelism is bounded by the partition count, and a slow message blocks everything behind it in its partition (head-of-line blocking). Rule of thumb: expensive per-message work with order unimportant suits a JMS/AMQP broker; high throughput with per-key ordering suits a log, with the key hashed to a partition.

Consumer offsets replace per-message acks: everything below the committed offset has been processed, everything above has not. The offset is the log sequence number of single-leader replication with the broker as leader and the consumer as follower. A failed consumer's partitions move to another member of the group, which resumes from the last committed offset; messages between that offset and the crash are processed twice, so idempotence or atomic commit is needed.

Disk space and lag. The log is split into segments and old segments are deleted or archived, making it a large ring buffer; retention is typically days to weeks, so a consumer can be offline that long and catch up, and performance does not degrade as the log fills. Of the three responses to a slow consumer (drop, buffer, backpressure), log brokers buffer: a consumer that falls behind by more than the retention loses messages, but only that consumer is affected. Monitor consumer lag and alert well before retention is exceeded.

Replaying old messages. Reading a log is a read-only operation that changes only the consumer's offset. A consumer can rewind to yesterday's offset and reprocess, run a new version of code alongside the old, or bootstrap a fresh derived view from history. This is the batch-processing property (immutable input, repeatable job) brought to streams, and it is what makes log-based brokers suitable for integrating systems rather than only for notifications.

Databases and streams

Keeping systems in sync

A replication log is a stream of change events. The same idea applies to the heterogeneous systems a company runs: an OLTP database, a cache, a full-text index, a warehouse, each needing a copy of the same data in its own format. Nightly ETL is too slow for caches and indexes, which tempts people into dual writes: the application writes to the database and then updates the cache and index itself.

Dual writes fail two ways. Race conditions: two clients update the same record concurrently; the database applies A then B, the index B then A, and they disagree permanently with no error raised. Partial failure: one write succeeds and the other fails, leaving inconsistency unless you use XA with all its Chapter 9 problems. The root cause is that there is no single leader deciding the order of writes.

Change data capture

The fix is to make one system the leader and make the others followers that consume its change log. Change data capture (CDC) observes every write to a database, in commit order, and exports it as a stream to other systems. The database is the system of record; the search index, cache and warehouse are derived data consuming the log, with a log-based broker in between to buffer and fan out.

Implementation options are database triggers (fragile and slow) or parsing the replication log, which is what Debezium, LinkedIn's Databus, Facebook's Wormhole and Oracle GoldenGate do; PostgreSQL exposes logical decoding and MongoDB change streams for the purpose. CDC is asynchronous, so a slow consumer does not slow the database, at the cost of replication lag and its anomalies (Chapter 5).

Initial snapshot. Replication logs are truncated, so a new consumer cannot rebuild state from the log alone. It starts from a consistent snapshot of the database that corresponds to a known log position, then consumes from that position. Some CDC tools do this automatically; otherwise it is a manual step.

Log compaction. The alternative to snapshots is to keep the log but discard superseded records. Kafka's compacted topics keep only the latest value for each key (and drop keys with a tombstone null value), so the log is bounded by the size of the database's current state rather than its history. A consumer can rebuild a full copy of a table by reading a compacted topic from offset zero; this is how Kafka Streams' tables and many CDC pipelines bootstrap.

Event sourcing

Event sourcing, from the domain-driven design community, stores the history of changes to the application state as a log of immutable events written by the application itself, and derives the current state from them. It differs from CDC in the level of abstraction: CDC captures low-level row changes from a database that is used mutably, and the application is unaware of it; event sourcing records domain-level facts ("student cancelled enrolment in course") that explain intent, and the application is built around them.

Advantages: a complete audit trail, bugs in derived state fixed by replaying through corrected code, new views added later for unanticipated questions, and events that capture why something changed, which a row's final state (enrolled = false) cannot.

Deriving current state. Users want state, not events, so the application replays the log through a deterministic function and caches snapshots to avoid replaying from the beginning. Unlike CDC, where the latest row change supersedes earlier ones and compaction applies, event-sourced state depends on every event, so snapshots are the optimisation rather than compaction.

Commands versus events. A command is a request from a user that may be rejected (reserve a seat that is already taken). Validation happens synchronously, with the current state. Once a command is accepted, it becomes an event: a durable, immutable fact that cannot be rejected afterwards. Any consumer of the event stream can rely on it; if a seat later turns out to be double-booked, that is a new compensating event, not a retraction. The validation step must therefore be done before the fact is published, which may require serializable or linearizable checks against the current state, and is where uniqueness constraints in streaming systems get difficult (Chapter 12).

State, streams and immutability

The unifying observation: a database's current state is the result of applying its change log, and the change log can be derived from the sequence of states. Mutable state and an append-only log of changes are two sides of one coin. Accountants have worked this way for centuries: a ledger is append-only, and mistakes are fixed by adding correcting entries, not erasing. Immutability pays in the same way for software:

• Mistakes are recoverable, including bugs in derived views.
• The log records more than the final state: a customer who added then removed a cart item is invisible in the cart table but visible in the log.
• One log can feed many read-optimised views, separating the form data is written in from the forms it is read in. That is command query responsibility segregation (CQRS), and it dissolves much of the normalise-or-denormalise debate: write normalised events, read denormalised views.
• Asynchronous derivation makes views stale; fix it by reading your own writes through the stream or by updating the view synchronously within the same partition.

Limitations: a very high write rate makes a full immutable history impractical, and regulatory deletion (GDPR) requires actually rewriting history, which Datomic calls excision; truly deleting data is hard in any replicated, backed-up system.

Processing streams

What to do with a stream

Once you have a stream you can write it to a database, cache or index (the derived-data path above), push it to users as notifications or dashboards, or process input streams into output streams through a pipeline of operators, the streaming analogue of MapReduce jobs. The batch philosophy carries over (read-only input, append-only output), with two differences: the input is unbounded, so sorting it is impossible, and fault tolerance cannot be "restart from the beginning."

Uses of stream processing

Complex event processing (CEP) searches for patterns across events: three failed logins within a minute, a price crossing a moving average. Queries are stored long-term and events flow past them; a match emits a complex event. The relation between queries and data is inverted relative to a database: queries are persistent, data is transient (Esper, TIBCO StreamBase, Flink CEP).

Stream analytics computes aggregates over windows of time: events per second, rolling averages, anomaly detection. Approximate algorithms (Bloom filters, HyperLogLog, percentile sketches) are common for speed, but approximation is a choice, not a requirement. Storm, Spark Streaming, Flink, Samza, Kafka Streams and Google Cloud Dataflow are the frameworks.

Maintaining materialized views is CDC and event sourcing seen from the processor's side: a derived view that must be updated incrementally and forever. Unlike analytics windows, it needs the full history, which Samza and Kafka Streams support by treating a compacted log as a table.

Search on streams is CEP with full-text queries: Elasticsearch percolator indexes stored queries and runs each document against them, scaling to many standing queries by indexing the queries rather than the documents.

Actor frameworks (Akka, Orleans, Erlang/OTP) also pass messages, but for concurrency and service communication, usually without durability; they are not stream processors.

Reasoning about time

Event time versus processing time

Batch jobs use the timestamp embedded in each event, so a rerun gives identical results. Stream frameworks sometimes use the processing machine's local clock, which is simple but wrong whenever processing lags the event: a restart after a deployment pushes a burst of old events through at once, and a chart of "events per minute by processing time" shows a spike that never happened. Event time is when the event occurred; processing time is when the operator sees it. Queuing, network faults, consumer restarts and reprocessing all open a gap between the two, and events may arrive out of order.

Knowing when you are ready

With event time, an operator counting events per minute never knows for sure that it has seen every event for a minute: stragglers may still be in a queue somewhere. The options are to ignore late events (tracking how many are dropped as a metric), or to publish a result and then publish a correction when a straggler arrives. Some systems have producers emit a special marker when they have finished a window, so consumers can trigger on it, but with many producers (mobile apps come and go) this is not always possible. Flink's watermarks are the generic form: an estimate, flowing through the stream, that no events older than time t are still to come.

Whose clock

A mobile app buffering offline events sends them hours later, and its clock may be wrong. A pragmatic fix logs three timestamps: event time and send time by the device clock, receive time by the server clock. Send time minus receive time estimates the device's clock offset (assuming small network delay), which is then applied to the event time. Do not trust a clock you do not control, but you can estimate its offset.

Types of windows

Window	Definition	Example
Tumbling	Fixed length, non-overlapping; each event in exactly one window	Per-minute counts: 10:03:00 to 10:03:59
Hopping	Fixed length, overlapping by a fixed hop; smooths a tumbling window	5-minute windows every minute: 10:03:00 to 10:07:59, then 10:04:00 to 10:08:59
Sliding	All events within some interval of each other, regardless of fixed boundaries	Events within 5 minutes of each other; implemented by a buffer sorted by time, expiring old events
Session	No fixed duration; groups events of the same user close together in time, ending after a gap of inactivity	Website sessionization: ends after 30 minutes without activity

Hopping windows are usually implemented as several tumbling windows aggregated together. Session windows are the most common in analytics and have no fixed size, which makes them awkward in batch and natural in streaming.

Stream joins

Joins in a stream processor are harder than in batch because new events can arrive at any time; three types appear:

Stream-stream join (window join)

Search analytics: a search event and a click event share a session ID, and click-through rate needs them joined, but the click may come seconds or weeks later, or never. The processor keeps state: an index of recent searches and clicks by session ID within a window (say an hour). A new search looks for its click, a click looks for its search, and expired searches are emitted as no-click. Both inputs are streams and both are indexed.

Stream-table join (stream enrichment)

Activity events need enriching with user profile data. Querying the database per event is slow and nondeterministic, so the processor keeps a local copy of the table, kept current by consuming the database's CDC stream, and joins each event against it. It resembles a batch broadcast hash join except that the table side changes over time; the "table" is really a changelog stream materialized as state (Kafka Streams' KTable).

Table-table join (materialized view maintenance)

A Twitter-style timeline cache: a new tweet is added to every follower's timeline, a deleted tweet removed, a follow adds the followee's recent tweets, an unfollow removes them. The processor consumes the tweet and follow streams and maintains tweets JOIN follows incrementally: a continuously updated materialized view of a join.

Time dependence of joins

All three joins keep state keyed by the join attribute and are sensitive to ordering. If a user updates their profile and tweets at nearly the same time, the enriched tweet shows the old or the new profile depending on which stream's event was processed first; across partitions there is no defined order, so results are nondeterministic and a rerun may differ. Data warehouses call this a slowly changing dimension (SCD) and solve it with versioned IDs: every profile version gets a new ID and the activity event references the version it saw, making the join deterministic and defeating compaction (every version must be kept).

Fault tolerance

The problem

Batch gets exactly-once semantics cheaply: a failed task is retried, its partial output discarded, and the final output looks as if every record was processed once. A stream never finishes, so "discard and restart" does not apply, yet retries are mandatory and a retry that repeats a side effect (a counter incremented twice, a second email) is a bug. The goal is effectively-once: the visible effects of processing with retries equal those of processing each event exactly once.

Microbatching and checkpointing

Microbatching (Spark Streaming) divides the stream into small blocks, typically a second, and runs each as a tiny batch job. Fault tolerance is batch fault tolerance; output is at the end of each batch; latency is bounded below by the batch size. It implicitly provides a tumbling window equal to the batch, which may not match the window you want.

Checkpointing (Apache Flink, following Chandy-Lamport) injects barriers into the stream periodically; when a barrier passes an operator, that operator writes its state to durable storage. On failure, every operator restores from the last checkpoint and input is replayed from the corresponding offsets. Output is not held back by batch boundaries.

Both handle state inside the framework. Neither prevents duplicated external side effects: once an output has been written to a database or an email sent, restarting from a checkpoint repeats it.

Atomic commit revisited

All effects of processing an event (output messages, database writes, state changes, the offset commit) must happen atomically or not at all. XA across heterogeneous systems is one route, with Chapter 9's problems. Google Cloud Dataflow, VoltDB and Kafka transactions do it internally: a Kafka producer writes output records and consumer offsets in one transaction, and read_committed consumers see both or neither. It works because the transaction stays inside one system.

Idempotence

The other route makes retries harmless. An idempotent operation produces the same result when applied many times: setting a key to a value is idempotent, incrementing a counter is not. Non-idempotent operations can be made idempotent with metadata: include the Kafka offset of the event with each database write and have the database skip updates whose offset is not newer than the one it already stored. Storm's Trident does this. Idempotence assumes that redelivered events come in the same order and that a replacement consumer behaves like the original, which requires fencing against a failed node that is still writing (Chapter 8).

Rebuilding state after failure

Window aggregates, join indexes and tables used for enrichment all live in operator state and must survive crashes. Options: keep state in a remote datastore (slow per lookup), keep it local and periodically persist it (Flink snapshots to HDFS; Samza and Kafka Streams write every state change to a compacted changelog topic that is replayed on recovery), or recompute it from input if the window is short. Which is best depends on network and disk speeds, which keep shifting.

Design trade-offs

Decision	Option	Choose when
Transport	Direct messaging (UDP, ZeroMQ, webhooks)	Lowest latency; occasional loss acceptable; both parties online
	Traditional broker (RabbitMQ, SQS)	Per-message load balancing; expensive processing; no replay needed
	Log-based broker (Kafka, Kinesis, Pulsar)	High throughput; order within a key; replay; many independent consumers
Keeping systems in sync	Dual writes	Never; race conditions and partial failure
	CDC from the database	Existing database is the source of truth; consumers need row-level changes
	Event sourcing	Domain intent matters; audit trail; many future views anticipated
Time	Processing time	Only when lag is irrelevant and reprocessing never happens
	Event time with watermarks	Correct analytics; late data must be handled
Fault tolerance	Microbatching	Simple; latency of a second is fine
	Checkpointing with idempotent sinks	Low latency; sinks can dedupe by offset
	Transactional output (Kafka transactions)	Output stays inside the log system

Interview angle

• Explain why Kafka is called a log and how that differs from RabbitMQ. Kafka appends messages to partitioned, durable logs and consumers track an offset per partition; messages are retained for a configured period regardless of consumption, so they can be replayed and read by many consumer groups. RabbitMQ deletes a message after acknowledgment and tracks each message individually, allowing per-message load balancing but no replay. Kafka preserves order within a partition; RabbitMQ with multiple consumers does not.

• How do you keep a search index consistent with a primary database without dual writes? Use CDC: parse the database's replication log (Debezium, PostgreSQL logical decoding) and publish each committed change to a log broker partitioned by key. An indexer consumes in order and applies the change. Bootstrap from a consistent snapshot at a known log position, or from a compacted topic. The database is the single leader; the index is a follower with lag but no divergence.

• What goes wrong if a stream processor measures events per minute by processing time? When the processor restarts after a deployment, it replays a backlog in seconds and the chart shows a spike that never occurred in reality, followed by a gap. Use event time, and handle late events with watermarks and either dropping with a metric or emitting corrections.

• Design sessionization for a web analytics pipeline. Partition the click stream by user ID so each user's events land in one partition and arrive in order. Each operator keeps per-user state: current session start and last event time. A new event within 30 minutes of the last extends the session; otherwise close the old session, emit it, and start a new one. Use event time with a watermark to decide when to close idle sessions; store state locally with a changelog topic for recovery.

• How do you get exactly-once semantics in a stream job that writes to a database? Exactly-once is about effects, not delivery. Make the write idempotent: store the event's offset with the row and reject writes with an older or equal offset, or use a unique operation ID and upsert. Alternatively keep the output inside the log system and use Kafka transactions to commit output and offsets atomically. Fence replaced consumers so a zombie cannot keep writing.

• Compare event sourcing with CDC. CDC captures low-level row changes from a mutable database the application uses normally; it is for synchronising other systems. Event sourcing makes application-level, immutable domain events the source of truth and derives all state from them; it captures intent and supports arbitrary new views, at the cost of more application design and no compaction (snapshots instead). Both end up as a log consumed by stream processors.

• A stream-table join gives different results on rerun. Why, and how do you fix it? The table side changes over time and the relative order of a profile update and an activity event is not defined across partitions, so the enriched output depends on timing. Fix by versioning the table rows (slowly changing dimensions): each event references the specific version it saw, making the join deterministic. The cost is that no version can be compacted away.

Pitfalls and misconceptions

• "At-least-once plus idempotence is a hack; I want exactly-once delivery." Exactly-once delivery is impossible in general over an unreliable network; exactly-once processing effects are achievable with idempotence or transactional output. Interviewers test this distinction.
• Dual writes are fine if I write in a try/catch. They are not: concurrent writers can apply updates in different orders to the two systems with no error raised. Only a single ordered log removes the race.
• Adding consumers always adds throughput. In a log-based broker, parallelism is capped by partition count, and repartitioning changes key-to-partition mapping, which breaks per-key ordering during the transition.
• Ordering across partitions. There is none. If two events must be ordered, give them the same key.
• Processing time is a fine proxy for event time. It silently breaks under backlog, replays and mobile offline buffering.
• Event sourcing means no current-state database. It means state is derived; you still need read models, snapshots and synchronous validation of commands before they become events.
• Checkpointing alone gives exactly-once. It restores internal state, but external side effects already performed are repeated unless the sink is idempotent or transactional.

Key terms

Term	Meaning
Event	Immutable record of something that happened, with a timestamp
Topic / partition / offset	A named stream; an independent ordered log within it; a message's position within a partition
Backpressure	Slowing the producer when the consumer cannot keep up, as pipes and TCP do
Message broker	Server that buffers and routes messages between producers and consumers
Consumer group	A set of consumers that share a topic's partitions and a committed offset per partition
Consumer lag	Distance between a consumer's offset and the head of the partition
Log compaction	Retaining only the latest record per key (and dropping tombstoned keys) so a log stays bounded
Change data capture	Extracting committed changes from a database's log and publishing them as a stream
Event sourcing	Storing domain events as the source of truth and deriving state from them
Command vs event	A request that may be rejected, versus an accepted fact that cannot be
Event time vs processing time	When something happened versus when the operator sees it
Watermark	A signal that no events older than a given time are expected
Tumbling / hopping / sliding / session window	Fixed non-overlapping; fixed overlapping; interval-based; gap-terminated
Slowly changing dimension	Versioned table rows so that joins against historical state are deterministic
Microbatching	Treating the stream as a series of small batch jobs
Checkpointing	Periodically snapshotting operator state at barriers for recovery
Idempotence	An operation whose repeated application has the same effect as a single one
Effectively-once	Processing with retries whose visible effects equal exactly-once processing

Further reading

• DDIA, Chapter 11: "Stream Processing"; Kleppmann's references at https://github.com/ept/ddia-references
• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction," 2013: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
• Tyler Akidau, "The world beyond batch: Streaming 101" and "Streaming 102," O'Reilly Radar, 2015
• Tyler Akidau et al., "The Dataflow Model," VLDB 2015
• Paris Carbone et al., "Lightweight Asynchronous Snapshots for Distributed Dataflows," 2015: https://arxiv.org/abs/1506.08603
• Martin Kleppmann, "Turning the database inside-out with Apache Samza," 2015: https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html
• Apache Kafka documentation (design section, including log compaction and transactions): https://kafka.apache.org/documentation/
• Debezium documentation: https://debezium.io/documentation/