Skip to main content

Formats for Encoding Data

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 4 — rewritten and condensed for teaching.

Applications never stand still — features get added, requirements shift, and almost every one of those changes drags a change to stored data along with it. But code changes don't roll out instantly: server-side deployments usually go out as a rolling upgrade (a few nodes at a time, to avoid downtime), and client-side apps update whenever the user gets around to it. That means old code, new code, old data, and new data can all coexist in the same running system, and it needs to keep working anyway. That requirement has two names:

  • Backward compatibility — newer code can read data written by older code.
  • Forward compatibility — older code can read data written by newer code.

Backward compatibility is usually the easy one — you're the one writing the new code, so you already know what the old format looks like. Forward compatibility is the hard direction: it requires old code to gracefully ignore things a newer version added, which it obviously wasn't written expecting.

This lesson looks at the encoding formats that determine how easy or hard that actually is — JSON, XML, and their binary cousins, then the schema-driven formats (Thrift, Protocol Buffers, and Avro) purpose-built to make both compatibility directions easy.

Two Representations, One Translation Problem​

Programs work with data in two fundamentally different shapes:

  1. In memory — objects, structs, lists, hash tables, often linked together with pointers, optimized for the CPU to manipulate quickly.
  2. As bytes — whatever you write to a file or send over a network. Pointers are meaningless to another process, so this representation has to look completely different from the in-memory one.

Converting in-memory → bytes is encoding (also serialization or marshalling); the reverse is decoding (parsing, deserialization, unmarshalling).

A naming collision worth knowing

"Serialization" also means something completely different in the context of database transactions (covered later in this series). To avoid confusion, this lesson sticks to encoding/decoding throughout.

Language-Specific Formats: Convenient, but a Trap​

Most languages ship a built-in encoder — Java's Serializable, Ruby's Marshal, Python's pickle. They're tempting because they need almost no extra code. They're also a bad long-term choice, for four compounding reasons:

  1. Lock-in. The format is tied to one language — reading the data from anything else ranges from painful to impossible.
  2. A real security hole. Decoding often means instantiating arbitrary classes from untrusted bytes — if an attacker controls the byte sequence you decode, they can potentially get your process to instantiate anything, sometimes escalating all the way to remote code execution.
  3. Compatibility is an afterthought. These formats were built for "quickly save and restore an object," not for the forward/backward compatibility this lesson is actually about.
  4. Performance is often an afterthought too — Java's built-in serialization, in particular, is notorious for bloated output and slow encode/decode.

The rule of thumb: fine for a quick, transient, single-process cache — a bad idea for anything that needs to outlive one version of one program.

Check: encoding fundamentals​

Quiz

Encoding, decoding, and language-specific formats

1 / 2

Why is forward compatibility ("older code can read data written by newer code") generally harder to achieve than backward compatibility?

Pick an answer to see if you are right.

JSON, XML, and Their Binary Variants​

Standardized, cross-language text formats — JSON, XML, CSV — fix the language lock-in problem, but they carry their own subtler issues:

  • Number ambiguity. XML and CSV can't distinguish a number from a numeric-looking string without an external schema. JSON distinguishes strings from numbers, but not integers from floats, and specifies no precision — a real problem once you exceed 2^53, since that's the largest integer a IEEE-754 double can represent exactly. (Twitter's API famously includes each tweet's 64-bit ID twice — once as a JSON number, once as a string — purely to work around JavaScript mangling the numeric version.)
  • No native binary strings. Both formats handle Unicode text well but have no concept of a raw byte string — so people Base64-encode binary data as text, which is a real 33% size penalty.
  • Optional, rarely-used schemas. Schema languages exist for both, but plenty of JSON tooling skips them entirely, which means encoding decisions (is this a number or a string that merely looks like one?) end up hardcoded into application logic instead of declared once.
  • CSV has no schema at all — column meaning is entirely implicit, added columns require manual handling everywhere, and its escaping rules (quoted commas, embedded newlines) are formally specified but inconsistently implemented.

None of this stops JSON/XML/CSV from being genuinely good choices, especially as interchange formats between organizations — when the real difficulty is getting two separate companies to agree on anything, a format's raw efficiency stops being the binding constraint.

Binary JSON: A Smaller Win Than You'd Expect​

Internal-only data doesn't need a lowest-common-denominator format, which is why a small zoo of binary JSON encodings exists — MessagePack, BSON, BJSON, UBJSON, Smile, and others. None displaced textual JSON/XML, and once you see why, it makes sense: since they keep JSON's schema-less data model, they still have to embed every field name in every encoded record.

Take this record:

{
  "userName": "Martin",
  "favoriteNumber": 1337,
  "interests": ["daydreaming", "hacking"]
}

MessagePack encodes it in 66 bytes — barely smaller than the 81-byte whitespace-stripped JSON. The first few bytes show why field names still dominate the payload:

Byte(s)Meaning
0x83An object (top nibble 0x80) with 3 fields (bottom nibble 0x03)
0xa8A string (top nibble 0xa0) that's 8 bytes long
next 8 bytesuserName, spelled out in ASCII
0xa6 + 6 bytesThe value Martin
…(and so on for the remaining two fields)

Every field name gets fully spelled out, every single time. The formats covered next fix this by moving the field names out of the encoded data entirely — into a schema shared once, in advance, between writer and reader.

Check: JSON, XML, and binary variants​

Quiz

Textual formats and their binary cousins

1 / 2

Why can integer tweet IDs larger than 2^53 be silently corrupted by a JavaScript application parsing Twitter's JSON API?

Pick an answer to see if you are right.

Practice: build a length-prefixed encoder​

The core trick every binary format in this lesson uses — MessagePack, Thrift, Protobuf, Avro alike — is a length-prefixed string: store how many bytes are coming, then the raw bytes, with no delimiter and no escaping needed. Implement a tiny version of this for a flat record of string fields.

Thrift and Protocol Buffers: Schemas With Field Tags​

Thrift (Facebook) and Protocol Buffers (Google) both require a schema up front, and both compile that schema into generated code for whatever language you're using. A Thrift schema for our example record:

struct Person {
  1: required string userName,
  2: optional i64 favoriteNumber,
  3: optional list<string> interests
}

Protocol Buffers looks almost identical:

message Person {
  required string user_name       = 1;
  optional int64  favorite_number = 2;
  repeated string interests       = 3;
}

With a schema in hand, the encoded bytes no longer need field names at all — just a field tag (the numbers 1, 2, 3), a type marker, and the raw value. That shrinks our example dramatically: Thrift's compact binary format packs it into 34 bytes, and Protocol Buffers into 33 — roughly half of MessagePack's 66, purely from dropping repeated field names and using variable-length integer encoding (small numbers cost fewer bytes; 1337 needs only 2 bytes instead of a fixed-width 8).

required vs. optional changes nothing about the bytes

Marking a field required only adds a runtime check that fails if it's missing — the wire encoding itself is identical either way. This matters a lot for the compatibility rules below.

Field Tags Are the Whole Compatibility Story​

An encoded record is just its fields concatenated, each tagged with a number. This single fact is what makes schema evolution work:

  • You can rename a field freely — the encoded bytes never mention field names, only tag numbers.
  • You can never change a field's tag — every already-encoded record on disk becomes unreadable the instant you do, since the tag is the field's identity on the wire.
  • Adding a field is safe as long as it gets a brand-new tag number. Old code reading data written by new code simply doesn't recognize the new tag — the type marker tells it exactly how many bytes to skip, so it ignores the field and moves on. That's forward compatibility, achieved for free.
  • New code reading old data (backward compatibility) works automatically too, provided you follow one rule: any field added after initial deployment must be optional or carry a default. A required field on new data that old writers never populated would fail validation the moment new code tries to read what old code wrote.
  • Removing a field is the mirror image of adding one: you may only remove an optional field, and its old tag number can never be reused — some already-written data somewhere might still carry it, and new code has to know to just ignore it.

Changing a field's datatype is riskier — always technically possible, but truncation or precision loss can sneak in (widening a 32-bit int to 64-bit is safe for old→new; new→old can silently truncate a value that no longer fits).

One quirk specific to Protocol Buffers: it has no dedicated list type at all — a repeated field is just the same tag number appearing multiple times in the encoded record. That has a genuinely nice side effect: an optional (single-valued) field can be turned into repeated later with zero compatibility break — old code reading new data just sees the last value written under that tag, and new code reading old data just sees a list of zero or one elements. Thrift, by contrast, has a dedicated (and separately-typed) list construct — no such free upgrade path, but nested lists work naturally.

Check: Thrift and Protocol Buffers​

Quiz

Field tags and schema evolution

1 / 3

Why can you safely rename a field in a Thrift or Protocol Buffers schema, but never change its tag number?

Pick an answer to see if you are right.

Avro: No Tags, Just Two Schemas​

Avro takes a genuinely different approach. Its schema has no field tags at all:

record Person {
  string userName;
  union { null, long } favoriteNumber = null;
  array<string> interests;
}

Encoded, our example record shrinks to just 32 bytes — the smallest of every format covered here — because the encoding is nothing but the values, concatenated in schema-declared order. There's no type marker, no field name, no tag anywhere in the bytes: a string is just a length prefix and raw UTF-8, and nothing in the data itself says "this is a string." The only way to correctly parse it is to already know the exact schema it was written with, field by field, in order.

That constraint sounds worse than it is, because Avro makes an important distinction:

  • The writer's schema — whatever schema the code that encoded this specific data was using at the time.
  • The reader's schema — whatever schema the code doing the decoding currently expects.

These two don't have to match — they only need to be compatible. Avro's library resolves the difference by matching fields up by name between the two schemas: a field present in the writer's schema but missing from the reader's is silently dropped; a field the reader expects but the writer never wrote gets filled in from a default value declared in the reader's schema.

Avro's Evolution Rule Is Refreshingly Simple​

Forward compatibility means a newer writer, older reader; backward compatibility means a newer reader, older writer. Avro collapses both into one rule: you may only add or remove a field that has a default value.

  • Add a field with a default → old data missing it just gets the default filled in. Safe both ways.
  • Add a field without a default → a new reader trying to read old data has no way to fill the gap. Backward compatibility breaks.
  • Remove a field without a default → an old reader expecting it, reading new data that no longer writes it, breaks. Forward compatibility breaks.

Because there's no universal null default the way some languages allow for any variable, Avro requires an explicit union type (union { null, long }) before null can be used as a default — more ceremony than "everything's nullable by default," but it forces you to be explicit about exactly what actually can be absent.

Changing a field's name is possible via reader-side aliases that map old writer names onto new reader names — but that's backward-compatible only, not forward-compatible (an old reader has no idea about a new alias). Adding a branch to a union type follows the identical asymmetry.

Where Does the Reader Get the Writer's Schema From?​

You obviously can't attach the entire schema to every single record — it'd dwarf the actual data. Avro's answer depends on context:

ContextHow the reader finds the writer's schema
One big file of millions of records (common in Hadoop)Write the schema once, at the top of the file (Avro's object container file format)
Records written individually to a database over timeStore a schema version number with each record; look the matching schema up from a schema registry
Two processes talking over a live connectionNegotiate the schema once, at connection setup, and reuse it for the connection's lifetime

The Payoff: Schemas That Write Themselves​

Because Avro's encoding carries no tag numbers, it plays unusually well with dynamically generated schemas. Imagine dumping an entire relational database table to Avro: generate one Avro field per column, using the column name directly as the Avro field name. If the underlying table later gains or loses a column, you just regenerate the Avro schema from the new table definition and export again — nothing else needs to change, because fields are matched by name, and names come straight from the database.

With Thrift or Protocol Buffers, the exact same scenario needs someone (or some careful piece of tooling) to hand-assign — and never accidentally reuse — a tag number for every column, every time the table's schema changes. That bookkeeping simply isn't a problem Avro's design has to solve.

This same tag-free property is also why Avro needs no code generation to be useful: since an Avro object container file already embeds its own writer's schema, you can open one and inspect its contents the same way you'd casually open a JSON file — genuinely useful for dynamically-typed languages and ad hoc data tools (like Apache Pig) that would find generated classes more of an obstacle than a convenience.

Check: Avro​

Quiz

Avro's writer/reader schema model

1 / 3

Why must Avro's reader know the EXACT schema a piece of data was encoded with, field by field, in order — unlike Thrift or Protocol Buffers?

Pick an answer to see if you are right.

Practice: resolve a reader's schema against a writer's​

Implement Avro's core trick directly: given data written under one schema (a list of (field_name, value) pairs) and a different reader's schema (a list of expected field names, each with a default), produce the record the reader expects — matching by name, filling in defaults for anything missing, and dropping anything the reader doesn't ask for.

Key takeaways
  • Backward compatibility (new code reads old data) is usually straightforward; forward compatibility (old code reads new data) is the genuinely hard direction, since it requires code that was already deployed to gracefully skip things it was never written to expect.
  • Language-specific serialization (pickle, Java Serializable) is convenient but risky — locked to one language, weak on compatibility, and a real security liability when decoding untrusted bytes.
  • JSON/XML/CSV are excellent cross-organization interchange formats but carry subtle gaps — ambiguous numeric precision, no native binary strings, and often-skipped schemas.
  • Thrift and Protocol Buffers replace field names with numeric field tags, unlocking compact encoding and a small, precise set of evolution rules: never reuse or change a tag, only add fields as optional/defaulted, only remove optional fields.
  • Avro drops tags entirely, matches fields by name via a writer's-schema/reader's-schema resolution step, and collapses compatibility down to one rule: only add or remove fields that carry a default — which also happens to make it uniquely well suited to schemas generated dynamically from something like a database table.

Further Reading​