Applications change: a feature needs a new field, a column is renamed, a single value becomes a list. Every change has to be mirrored in the data the application stores and sends, and that data is encoded in some format. Chapter 4 is about choosing an encoding whose meaning survives change, so a system can be upgraded piece by piece while old and new code run side by side and read each other's output.

That is the normal state of a deployed system: a rolling upgrade has servers on two versions at once, a mobile update takes weeks to reach everyone, and a row written five years ago is still in the database. Encoding is where "we changed the code" meets "the data is already out there", and getting it wrong produces silent corruption rather than a loud failure.

In one minute

• Backward compatibility: newer code reads data written by older code. Forward compatibility: older code reads data written by newer code. Rolling upgrades and long-lived data demand both.
• Language-built-in serialization (Java Serializable, Python pickle, Ruby Marshal) locks you into one language, is a remote-code-execution risk, ignores versioning and is slow.
• JSON, XML and CSV are readable and universal but ambiguous about numbers (ints vs floats, precision above 2^53), lack binary strings, and their optional schema languages are rarely used. Binary JSON variants such as MessagePack save little because field names are still spelled out.
• Thrift and Protocol Buffers encode each field as a numeric tag plus type and value. Never change or reuse a tag; new fields need fresh tags and must be optional or defaulted; only optional fields may be removed.
• Avro encodes only values in schema order, so it is the most compact. Decoding needs the writer's schema; schema resolution reconciles it with the reader's schema by field name. Fields may be added or removed only if they have a default.
• Schemas buy compactness, documentation that cannot drift, compile-time checks and a compatibility check before deployment.
• Data flows through databases (data outlives code; preserve unknown fields), services (REST or RPC; requests need backward compatibility, responses forward compatibility) and message brokers (RabbitMQ, Kafka; the actor model).
• RPC cannot make a network call behave like a local call: timeouts, retries, duplicate execution and cross-language types leak through. gRPC, Finagle and Rest.li make the remoteness explicit with futures and streams.

Compatibility and rolling upgrades

Programs keep data in two forms. In memory it lives in objects, lists, hash tables and trees full of pointers. On disk or on the wire there are no pointers, only a self-contained byte sequence. Translating from memory to bytes is encoding (serialization, marshalling); the reverse is decoding (parsing, unmarshalling). The book says encoding rather than serialization to avoid clashing with the transaction-isolation meaning of the word.

When the code that produces the bytes changes, the bytes may change too. In a rolling upgrade you deploy a new version to a few nodes, check it, then move on, so a release never takes the service down and can be rolled back cheaply. During that window old and new code coexist, and client software may stay old for months.

Property	Who reads	Who wrote	Why it is hard
Backward compatibility	New code	Old code	Easy in principle: the new code's author knows the old format and can handle it explicitly
Forward compatibility	Old code	New code	Hard: old code must tolerate additions it has never seen, ideally ignoring them without destroying them

A format with both properties lets you upgrade in any order, roll back, and keep years of stored records readable.

Formats for encoding data

Language-specific encodings

Most languages ship a one-line way to dump objects to bytes: Java's java.io.Serializable, Ruby's Marshal, Python's pickle, third-party variants such as Kryo. The book's advice is to use them only for very transient purposes.

• Language lock-in. The bytes encode one language's object model; a Go service cannot read a Java-serialized blob.
• Security. To rebuild objects the decoder must instantiate arbitrary classes. If an attacker controls the bytes, they choose the classes, which has repeatedly led to remote code execution.
• Versioning is an afterthought. Backward and forward compatibility are usually neglected.
• Efficiency. Java's built-in serialization is notorious for slow CPU performance and bloated output.

JSON, XML and CSV

Standardized textual formats fix lock-in: JSON, XML and CSV can be read in any language. JSON won on browser support and simplicity relative to XML; CSV is powerful despite being nearly schema-less. All three share problems that surface once the data gets interesting.

Numbers are ambiguous. XML and CSV cannot distinguish a number from a string of digits without an external schema. JSON separates strings from numbers but not integers from floating point, and specifies no precision. Integers above 2^53 cannot be represented exactly in an IEEE 754 double, so JavaScript parsers round them. Twitter hit this with 64-bit tweet IDs and now returns each ID twice, as a JSON number and as a decimal string.

No binary strings. JSON and XML carry Unicode text but not raw bytes. The workaround is Base64 at a 33 percent size penalty.

Schema support is optional. XML Schema and JSON Schema exist, but many consumers skip them, so whether "2024-01-01" is a date or a string is hard-coded in each application. CSV has no schema at all, and many parsers mishandle commas and newlines inside values.

Even so, they are good enough for much work, especially between organizations, where agreeing on any format is the hard part.

Binary encodings of JSON and XML

Inside one organization you can pick something smaller, which has produced binary JSON variants (MessagePack, BSON, BJSON, UBJSON, BISON, Smile) and binary XML variants (WBXML, Fast Infoset). Some extend the data model (BSON adds dates and binary strings), but all still embed every field name in the bytes.

The book traces one record through every format, a user with a name, a favourite number and two interests, so the sizes are comparable:

Encoding	Example record	What the bytes contain
JSON (whitespace removed)	81 bytes	Field names, values, punctuation
MessagePack	66 bytes	Type markers, lengths, field names, values
Thrift BinaryProtocol	59 bytes	Field type, field tag, length, value
Thrift CompactProtocol	34 bytes	Tag and type packed into one byte, varints
Protocol Buffers	33 bytes	Tag and wire type packed, varints
Avro	32 bytes	Values only, in schema order

MessagePack saves fifteen bytes over JSON, a poor trade for losing readability. The real gains come from not repeating field names, and that requires a schema.

Thrift and Protocol Buffers

Apache Thrift (Facebook) and Protocol Buffers (Google), both open-sourced around 2007–2008, share one idea: a schema in an interface definition language describes fields and types, and a code generator emits classes that encode and decode records.

message Person {
  required string user_name       = 1;
  optional int64  favorite_number = 2;
  repeated string interests       = 3;
}

The numbers are field tags, the heart of the design. The encoded bytes never contain a field name, only its tag, a type indicator, a length where needed, and the value. The decoder maps tag 1 back to user_name using the schema.

Thrift's BinaryProtocol spends a byte on the type and two on the tag per field. CompactProtocol packs type and tag into one byte for small tags and encodes integers as variable-length varints, so 1337 takes two bytes rather than eight. Protocol Buffers has a single binary format that works much like CompactProtocol.

required changes nothing in the encoding; it only enables a runtime check that the field is present. It does nothing for compatibility, and since a required field can never be added or removed safely, proto3 dropped the keyword.

Field tags and schema evolution

Because the bytes refer to fields by tag, the rules follow directly.

• Renaming is free. The name exists only in the schema.
• Adding a field means a new tag. Old code meets an unknown tag and skips it, using the type indicator to know how many bytes to skip: forward compatibility. New code reading old data must cope with the field being absent, so it must be optional or have a default: backward compatibility.
• Removing a field is the mirror image: only optional fields, and the tag must never be reused, because old data still carries it with the old meaning.
• Changing a type is sometimes possible but lossy. Widening int32 to int64 works for new code reading old data, but old code reading a 64-bit value into a 32-bit variable truncates.

Protocol Buffers has no list type; a repeated field writes the same tag several times. So an optional field can become repeated: new code reading old data sees a list of zero or one element, old code reading new data keeps the last occurrence. Thrift has a dedicated list type, which forbids that trick but allows nested lists.

Avro

Apache Avro began in 2009 as a Hadoop subproject because Thrift did not fit Hadoop's use cases. Its schema is written in Avro IDL (for humans) or JSON (for machines):

record Person {
  string               userName;
  union { null, long } favoriteNumber = null;
  array<string>        interests;
}

There are no tag numbers. The encoded record is the values concatenated: a length-prefixed string, a varint, a length-prefixed array. Nothing says which field is which or what type it is; the decoder walks the schema in field order and reads each value as the schema dictates. The result is 32 bytes, and it can only be decoded with the exact schema the writer used. So how can the schema evolve?

Writer's schema and reader's schema

Avro separates two schemas. The writer's schema is whatever version the encoding application had. The reader's schema is whatever version the decoding application was built with. They need not be identical, only compatible.

With both schemas side by side the library performs schema resolution: fields are matched by name, so order may differ; a field only in the writer's schema is read and discarded; a field only in the reader's schema is filled with the reader's default value; types are converted where Avro knows how (int to long, float to double).

Schema evolution rules in Avro

• You may add or remove a field only if it has a default. Adding without a default breaks backward compatibility; removing without one breaks forward compatibility.
• There is no optional. A field may be null only if its type is a union containing null, and the default must match the union's first branch, hence union { null, long } x = null.
• Changing a type is allowed when Avro can convert.
• Renaming works through aliases in the reader's schema. That is backward but not forward compatible, since old readers do not know the new name. Adding a branch to a union has the same asymmetry.

How the reader learns the writer's schema

Shipping the writer's schema with every record would erase Avro's size advantage, since the schema is usually larger than the record. The answer depends on context:

Context	Where the writer's schema lives
Large file with many records (Hadoop)	Once at the start of the file: Avro's object container file format
Database of individually written records	Each record carries a schema version number; a registry maps versions to schemas. LinkedIn's Espresso works this way
Records over a network connection	Negotiated at connection setup and used for the connection's lifetime (Avro RPC)

A registry of schema versions is worth having with any format: it documents what has been stored and lets you check a change for compatibility before deploying it.

Dynamically generated schemas

Avro's tagless design is friendlier to schemas generated by a program. Dumping a relational table to a file, you generate an Avro schema from the columns; if the table gains a column, you regenerate the schema and readers resolve by name. With Thrift or Protobuf someone must assign tag numbers by hand and remember never to reuse one, exactly the bookkeeping that goes wrong.

Code generation and dynamically typed languages

Thrift and Protobuf lean on code generation, which gives statically typed languages (Java, C++, C#) typed structures, compile-time checks and autocompletion. In dynamic languages (JavaScript, Ruby, Python) there is no compile-time check to gain, and generated code is often an obstacle.

Avro's code generation is optional. An object container file is self-describing because it embeds the writer's schema, so a tool can open it and read the records knowing nothing in advance, as it would a JSON file. Apache Pig reads and writes Avro this way.

The merits of schemas

The idea is old: ASN.1 (1984) uses tag numbers like Thrift and Protobuf, and its DER encoding still carries every X.509 certificate. Compared with JSON, XML and CSV, schema-based formats offer compactness (no repeated field names), documentation that cannot drift (the schema is required for decoding, so it is always current), a pre-deployment compatibility check when you keep a registry, and static typing through generated code. They keep the flexibility of schemaless JSON through schema evolution. "Schema" does not mean "rigid".

Dataflow through databases

Compatibility only matters because data moves between processes. The first route: a process writes to the database and later a process, maybe the same one, reads it back. That is sending a message to your future self. Backward compatibility is clearly needed. Forward compatibility is needed too: during a rolling upgrade a newer process may write a record that an older process then reads.

Preserving unknown fields

Here is a trap the encoding alone cannot fix. New code adds a field and writes a record with it. Old code reads the record, changes an unrelated field, and writes it back. If the old code decoded into a model object with no slot for the new field, the field is gone. The formats above can all preserve unknown fields when used directly, but an ORM or mapping layer may drop them. The read-modify-write path has to be written with this in mind.

Data outlives code

A deployment replaces the code in minutes. The data it stored is still there years later, in whatever format was current when each record was written: data outlives code. Rewriting every record into the new schema is possible but expensive, so most databases avoid it. Adding a column with a null default is a metadata change in most relational databases, and old rows read as null. LinkedIn's Espresso uses Avro schema evolution for the same purpose. The database presents one schema even though the bytes underneath span many versions.

Archival storage

Snapshots for backup or for a data warehouse are the natural place to consolidate. The dump is written at one moment by one program, so everything can use the latest schema even if the live database holds a mix. Avro object container files fit well, and a column-oriented format such as Parquet is often better for analytics (Chapter 10).

Dataflow through services: REST and RPC

The second route is a network request from a client to a server exposing an API, a service. Servers are also clients of other services, which is the basis of service-oriented architecture and microservices: decompose an application into services by functionality so each can be deployed and evolved independently by its own team. That works only if old and new versions of clients and servers can talk. A service is a database with a deliberately narrow interface: it exposes only the operations its API allows.

Web services: REST versus SOAP

	REST	SOAP
Nature	A design philosophy built on HTTP principles	An XML protocol that uses HTTP mostly as transport
Description	URLs identify resources; HTTP verbs, caching and content negotiation used as designed; OpenAPI (Swagger) optionally describes the API	WSDL, a machine-readable XML document; extended by the sprawling WS-* standards
Tooling	Optional; curl is enough to debug	Effectively mandatory: WSDL is not human-friendly, so clients rely on code generation
Where found	Public APIs, microservices	Large enterprises and legacy integration; declining

SOAP promised that generated local classes would hide the remote service. Interoperability between vendors' stacks was poor, and most organizations outside large enterprises have abandoned it.

The problems with remote procedure calls

RPC predates web services: make a remote request look like a local function call, a property called location transparency. The book argues this is fundamentally flawed, because a network call is nothing like a local one:

• A local call succeeds or throws predictably. A network request may be lost, or the remote machine may be slow or down, and the client cannot tell which.
• A call may time out with no result. You do not know whether the request got through.
• Retrying may execute the action twice if the first attempt succeeded but its response was lost. Retries are safe only if the operation is idempotent.
• Latency varies from microseconds to seconds; a local call has nearly constant cost.
• Parameters must be encoded. Primitives are fine; objects with references to other objects are not.
• Client and server may use different languages with mismatched types, such as JavaScript's integers above 2^53.

Many RPC systems were built anyway: Java RMI and EJB (Java only), DCOM (Microsoft only), CORBA (complex, no backward or forward compatibility). REST is attractive partly because it does not pretend a network call is local.

Current directions for RPC

RPC did not die; it grew up. Thrift and Avro ship RPC support, gRPC is RPC over Protocol Buffers, Finagle (Twitter) uses Thrift, Rest.li (LinkedIn) uses JSON over HTTP. The new generation is explicit about the network: Finagle and Rest.li return futures (promises) for calls that may fail; gRPC supports streams, a series of requests and responses over time; several frameworks include service discovery.

Binary RPC beats JSON over REST on performance; REST wins on debuggability, ecosystem and language support. So REST dominates public APIs, while RPC frameworks mostly serve requests between services inside one organization.

Data encoding and evolution for RPC

The usual assumption is that servers are updated first, then clients, so a service needs only backward compatibility on requests (new server reads old client's request) and forward compatibility on responses (old client reads new server's response). These properties come from the encoding: Thrift, gRPC and Avro RPC follow their format's rules; SOAP uses XML Schema; REST APIs usually use JSON, where adding optional request parameters and new response fields is normally compatible.

A provider often cannot force clients to upgrade; public APIs are used by other organizations for years, so it must maintain several API versions for a long time. There is no agreed way to version a REST API. Common choices: a version in the URL (/v2/users), a version in the Accept header, or an API key identifying the client with its version stored server-side.

Message-passing dataflow

The third route sits between RPC and databases. Like RPC, a client sends a message to another process with low latency. Like a database, the message goes through an intermediary, a message broker (message queue), that stores it temporarily. Compared with direct RPC this gives a buffer when the recipient is unavailable or overloaded, redelivery after a crash, no need for the sender to know the recipient's address, delivery of one message to several recipients, and decoupling of sender from receiver.

The defining difference from RPC is that message passing is asynchronous and one-way. The sender does not wait for delivery; a reply, if any, arrives on a separate channel.

Message brokers

Commercial brokers (TIBCO, IBM WebSphere MQ, webMethods) have given way to open source ones: RabbitMQ, ActiveMQ, HornetQ, NATS and Apache Kafka. Chapter 11 compares them; the shape is shared. A producer sends a message to a named queue or topic; the broker delivers it to one or more consumers. A topic is one-way, but a consumer may publish to another topic (building a pipeline) or to a reply queue the original sender consumes.

Brokers enforce no data model; a message is bytes plus metadata, so any encoding works. With a backward- and forward-compatible encoding, publishers and consumers deploy independently. The unknown-field trap returns: a consumer that republishes to another topic must preserve fields it does not understand.

Distributed actor frameworks

The actor model is a concurrency model for a single process. Instead of threads sharing mutable state, logic is encapsulated in actors, each with private state, communicating only through asynchronous messages and processing one message at a time. Delivery is not guaranteed.

A distributed actor framework extends this across machines, encoding messages for the network. Location transparency works better here than in RPC because the model already assumes messages can be lost. Rolling upgrades still need care, since a new node may message an old one:

Framework	Default encoding	Rolling upgrade story
Akka (JVM)	Java serialization, with no forward or backward compatibility	Swap in Protocol Buffers
Orleans (.NET)	A custom encoding that does not support rolling upgrades	Deploy a new cluster and shift traffic, or use custom serialization
Erlang OTP	Erlang term format	Hot code swapping is a selling point, but record schema changes are hard; the newer maps datatype may ease this

Design trade-offs

Question	Textual (JSON, XML, CSV)	Thrift / Protocol Buffers	Avro
Schema needed to decode	No	Yes, compiled in	Yes, writer's and reader's
Field identity in bytes	Field name	Numeric tag	Position in writer's schema
Example record size	81 bytes	33–59 bytes	32 bytes
Add a field	Readers ignore unknown keys	New tag; optional or defaulted	Must have a default
Remove a field	Stop writing it	Optional fields only; never reuse the tag	Must have had a default
Rename a field	Breaks readers	Free	Alias; backward but not forward compatible
Generated schemas	Not applicable	Awkward (manual tags)	Natural
Code generation	None	Central	Optional; self-describing files
Best fit	Public APIs, debugging, cross-organization exchange	Internal RPC between typed services	Bulk files, Kafka with a schema registry, database dumps

Public APIs: JSON over HTTP, with 64-bit integers sent as strings. Internal RPC between typed services: Protocol Buffers with gRPC. Event streams and bulk files with many independent producers and consumers: Avro with a registry. In every case, test compatibility before deploying and make sure read-modify-write paths preserve unknown fields.

Interview angle

• Define backward and forward compatibility. Which is harder? Backward: new code reads old data. Forward: old code reads new data. Forward is harder because old code must tolerate things that do not yet exist, ignoring unknown fields without discarding them; the new code's author can at least see the old format.
• Why do rolling upgrades need both? During a rollout some nodes run old code and some new. A new node writes data an old node reads (forward), and old nodes write data new nodes read (backward). If either direction fails, you cannot roll out gradually or roll back.
• How does Protocol Buffers support schema evolution, and what goes wrong? Fields are identified by tag, so renaming is free. New fields take new tags and must be optional or defaulted; unknown tags are skipped using the wire type. A removed field's tag must never be reused because old data still carries it. Widening int32 to int64 truncates when old code reads new data.
• Compare Avro and Protocol Buffers. Avro writes only values in schema order, no tags or type markers, so it is smallest. The cost is that the reader must know the writer's schema exactly; Avro recovers flexibility through schema resolution by field name with defaults. That also makes Avro the better fit for dynamically generated schemas such as database dumps.
• What is wrong with treating RPC like a local call? A network call can be lost, time out with no result, or succeed while its response is lost, so a retry may run the action twice unless it is idempotent. Latency is wildly variable, parameters must be serialized, and the two sides may use languages with mismatched types.
• A field added by new code vanishes after old code edits a record. What happened? The old code decoded the record into a model object without the new field, modified something else, and re-encoded, dropping the unknown field. The encoding could preserve it; the application mapping layer did not.
• How would you version a public REST API? A version in the URL path, in the Accept header, or a per-client API key with the version recorded server-side. Additive changes should not need a new version, and old versions must live a long time because external clients will not upgrade on your schedule.

Pitfalls and misconceptions

• "JSON is flexible, so schema is not a problem." Integers above 2^53 are rounded by JavaScript, binary data needs Base64, and field meaning lives in undocumented application code. Flexibility is not evolvability.
• "required makes Protobuf data safer." It adds only a runtime check; the bytes are identical. Required fields can never be added or removed compatibly, which is why proto3 dropped the keyword.
• "I removed the field, so its tag is free." Never reuse a tag. Old records and old clients still send it with the old meaning.
• "Avro has no tags, so it cannot evolve." It evolves through schema resolution by field name. The constraints are that added or removed fields need defaults and the reader must find the writer's schema.
• "Our encoding preserves unknown fields, so we are fine." The encoder may; your ORM or model classes may not.
• "Servers need forward compatibility, clients backward." Reversed. Servers upgrade first, so servers need backward compatibility on requests and clients need forward compatibility on responses.
• "A schema change means rewriting every row." Most relational databases treat adding a nullable column as metadata and return null for old rows. Rewriting is the expensive exception.

Key terms

Term	Meaning
Encoding / decoding	Translating in-memory structures to a self-contained byte sequence and back; also serialization/deserialization, marshalling/unmarshalling
Backward compatibility	Newer code can read data written by older code
Forward compatibility	Older code can read data written by newer code
Rolling upgrade	Deploying a new version a few nodes at a time, so old and new versions run concurrently
Field tag	The small integer identifying a field in Thrift and Protocol Buffers encodings instead of its name
Writer's schema	The schema version the encoding application used (Avro)
Reader's schema	The schema version the decoding application expects (Avro)
Schema resolution	Avro's reconciliation of writer's and reader's schema: match by name, drop unknown fields, fill defaults, convert types
Object container file	Avro's file format with the writer's schema once in the header, making the file self-describing
Schema registry	A service or table mapping version numbers to schemas, so records carry a short version rather than a full schema
Data outlives code	Stored records persist across many code versions, so a database holds a mix of historical encodings
REST	An HTTP API style: resources identified by URLs, HTTP verbs and headers used as designed, simple data formats
RPC	Remote procedure call: making a network request look like a local function call (location transparency)
Idempotent	An operation with the same effect whether performed once or many times, making retries safe
Message broker	Middleware that stores messages sent to a queue or topic and delivers them to consumers (RabbitMQ, Kafka)
Actor model	A concurrency model where actors hold private state and communicate only through asynchronous, possibly lost, messages

Further reading

• DDIA, Chapter 4, "Encoding and Evolution"; the chapter's reference list: https://github.com/ept/ddia-references
• Martin Kleppmann, "Schema evolution in Avro, Protocol Buffers and Thrift": https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
• Protocol Buffers encoding reference: https://protobuf.dev/programming-guides/encoding/
• Apache Avro specification, including schema resolution: https://avro.apache.org/docs/current/specification/
• Andrew D. Birrell and Bruce Jay Nelson, "Implementing Remote Procedure Calls", ACM Transactions on Computer Systems, 1984
• Jim Waldo, Geoff Wyant, Ann Wollrath and Sam Kendall, "A Note on Distributed Computing", Sun Microsystems Laboratories, 1994
• Roy Fielding, "Architectural Styles and the Design of Network-based Software Architectures", PhD thesis, 2000: https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
• gRPC documentation: https://grpc.io/docs/