Encoding & Evolution

Chapter 1: In-Memory vs On-Wire

Before we compare JSON vs Protobuf vs Avro, we need to understand why encoding exists at all. The answer comes down to two fundamentally different representations of data.

Two Worlds

In memory, data lives as objects, structs, hash maps, arrays, and trees. These structures are optimized for CPU access: pointers let you jump to any field in O(1), data can reference other data via shared pointers, and the layout is designed for the memory hierarchy (cache lines, page alignment). A Python dict, a Java HashMap, a Go struct — these are all in-memory representations.

On the wire (or on disk), data must become a flat sequence of bytes. You cannot send a pointer over the network — the receiving process has a completely different memory space. The byte sequence must be self-contained: it carries all the information needed to reconstruct the original data, with no external references.

The translation from in-memory to bytes is called encoding (also called serialization or marshalling). The reverse — bytes back to in-memory objects — is decoding (also called deserialization, parsing, or unmarshalling).

Language-Specific Formats: A Cautionary Tale

Many programming languages offer built-in serialization: Java has java.io.Serializable, Python has pickle, Ruby has Marshal. These are convenient — you can serialize any in-memory object in one line of code. But they have severe problems:

The Encoding Pipeline

The simulation below shows an in-memory object being encoded into bytes, transmitted, and decoded back into an in-memory object. Notice the information that must be preserved (field names, types, values) and the information that is lost (memory addresses, method definitions, private internal state).

What Gets Preserved, What Gets Lost

The encoding format you choose determines which columns say "Yes" and which say "Depends." JSON preserves field names as strings but loses precise types (is 42 an int32 or an int64?). Protobuf preserves field tags (numbers, not names) and precise types, but requires a schema to decode. Avro preserves field names and types but requires the writer's schema at read time. Each choice has consequences for compatibility, performance, and usability.

Chapter 2: JSON, XML, CSV

If language-specific formats are a trap, what should you use? The first answer most developers reach for is a human-readable text format: JSON, XML, or CSV. These formats are widely supported, easy to debug (you can open them in a text editor), and language-independent. But they each have subtle problems that bite you at scale.

JSON: The Lingua Franca

JSON (JavaScript Object Notation) is the dominant format for web APIs. Its strengths are real: every language has a JSON library, it is human-readable, and it supports nested objects and arrays. But JSON has three landmines that every systems engineer must know about.

Landmine 1: Number ambiguity. JSON does not distinguish between integers and floating-point numbers. The value 42 might be parsed as a 32-bit int, a 64-bit int, or a 64-bit float — it depends entirely on the parser implementation. This matters enormously for large integers.

Landmine 2: No binary support. JSON can only represent text. If you need to embed binary data (images, encrypted blobs, compressed data), you must encode it as Base64 — which inflates the data by ~33%. A 1 MB image becomes ~1.37 MB in the JSON payload.

Landmine 3: No schema. JSON is schemaless by default. A field can be a string in one record and a number in another. There is no enforcement mechanism built into the format. JSON Schema exists but it is a documentation tool, not a wire-format enforcement tool — there is nothing in the JSON bytes that says "this field must be an integer."

XML: Powerful but Verbose

XML (Extensible Markup Language) was the dominant data format before JSON took over. It supports schemas (XSD), namespaces, attributes vs. elements, and XSLT transformations. But it is verbose — every value is wrapped in opening and closing tags — and the schema language (XSD) is notoriously complex. A simple user profile in XML is roughly 2-3x the size of the equivalent JSON.

CSV: Fragile at Best

CSV (Comma-Separated Values) is barely a format at all. There is no official standard (RFC 4180 is a best-effort attempt). There is no schema. There is no type information. If a value contains a comma, you quote it. If a value contains a quote, you escape it. If a value contains a newline... it depends on the parser. CSV is fine for exporting spreadsheets. It is not a serious option for inter-service communication.

Size Comparison

The simulation below encodes the same data in JSON, XML, and CSV. Watch the byte counts. Also notice where JSON silently corrupts a large integer.

Worked Example: Encoding a User Profile

json
// JSON: 67 bytes
{"userId":42,"name":"Alice","email":"alice@example.com"}

xml
<!-- XML: 138 bytes -->
<user>
  <userId>42</userId>
  <name>Alice</name>
  <email>alice@example.com</email>
</user>

csv
# CSV: 40 bytes (but no schema, no types, no nesting)
42,Alice,alice@example.com

CSV is smallest, but it cannot represent nested data, has no type information, and breaks if any field contains a comma. XML is largest because of the tag overhead. JSON is the sweet spot for human-readability, but still suffers from the type ambiguity and size overhead we discussed.

Chapter 3: Binary Encoding

JSON is human-readable but wastes bytes. Every field name is repeated in full text for every single record. The string "favoriteNumber" is 16 bytes — and you pay that cost in every message, every record, millions of times per day. Binary encoding formats solve this by replacing text with compact binary representations.

MessagePack: Binary JSON

MessagePack is the simplest binary format. It is essentially JSON with a binary wire format: field names are still stored as strings, but values are encoded more compactly (integers use variable-length encoding, strings carry explicit lengths instead of quote delimiters). A JSON object like {"userName":"Martin","favoriteNumber":1337} shrinks from ~45 bytes to ~33 bytes.

MessagePack is a quick win — smaller than JSON, faster to parse — but it still carries full field names. The real savings come from formats that eliminate field names entirely.

Protocol Buffers and Thrift: Field Tags Replace Names

Protocol Buffers (Protobuf, created at Google) and Apache Thrift (created at Facebook) take a different approach. Instead of encoding field names, they assign each field a numeric tag in a schema definition. The schema is shared between sender and receiver ahead of time. The wire format only carries the tag numbers, not the names.

protobuf
// Protocol Buffers schema (.proto file)
message Person {
  string user_name = 1;      // tag 1
  int64 favorite_number = 2;  // tag 2
}

thrift
// Thrift schema (.thrift file)
struct Person {
  1: string userName,
  2: i64 favoriteNumber,
}

When this message is encoded, the bytes contain (tag=1, type=string, value="Martin") and (tag=2, type=varint, value=1337). The tag is a small integer (1-2 bytes), not a 14-character string. This saves enormous amounts of space when you have millions of records.

Varint Encoding: How Protobuf Packs Integers

Protobuf does not use fixed-width integers. The number 1 should not cost 8 bytes just because the schema says int64. Instead, Protobuf uses varint encoding: each byte uses 7 bits for data and 1 bit (the MSB, most significant bit) as a continuation flag. If the MSB is 1, more bytes follow. If it is 0, this is the last byte.

For signed integers, Protobuf uses ZigZag encoding to map signed values to unsigned varints: 0 → 0, -1 → 1, 1 → 2, -2 → 3, 2 → 4, and so on. This keeps small negative numbers small (without ZigZag, -1 as a 64-bit two's complement would encode as a 10-byte varint).

Thrift: Two Wire Formats

Thrift offers two encoding formats with different trade-offs:

CompactProtocol packs the field type and tag delta into a single byte when the tag delta is small (which it usually is, since tags are sequential). This makes it almost as compact as Protobuf.

Interactive Binary Encoder

The simulation below encodes the same record in four formats. Watch how field names disappear in Protobuf and Thrift, replaced by tiny tag numbers. Compare the final byte counts.

Worked Example: Byte-by-Byte Protobuf Encoding

Let us encode {"userName": "Martin", "favoriteNumber": 1337} in Protobuf, byte by byte:

Eleven bytes. The JSON version was roughly 45 bytes. That is a 4x reduction for a single tiny record. At a million records per second, that is 34 MB/s of saved bandwidth. At a billion records in a database, that is 34 GB of saved storage.

python
# Varint encoder in Python — interview coding drill
def encode_varint(value):
    """Encode a non-negative integer as a Protobuf varint."""
    result = []
    while value > 0x7F:
        result.append((value & 0x7F) | 0x80)  # 7 data bits + MSB=1
        value >>= 7
    result.append(value & 0x7F)  # last byte: MSB=0
    return bytes(result)

def zigzag_encode(value):
    """ZigZag: map signed int to unsigned for efficient varint."""
    return (value << 1) ^ (value >> 63)

# Test
print(encode_varint(1).hex())       # "01"      (1 byte)
print(encode_varint(300).hex())     # "ac02"    (2 bytes)
print(encode_varint(1337).hex())    # "b90a"    (2 bytes)
print(zigzag_encode(-1))             # 1
print(zigzag_encode(1))              # 2

Chapter 4: Schema Evolution

We have established that Protobuf and Thrift are smaller and faster than JSON. But the real reason to use them is schema evolution. Because these formats use numeric field tags instead of string names, and because they have explicit rules for handling unknown fields, they make it possible to evolve your data format safely over time.

The Five Rules of Protobuf/Thrift Evolution

These rules determine what changes are safe and what changes will break your system. Memorize them for interviews.

Rule 1: You can add new fields with new tag numbers. If you add a field with tag 3 to a schema that only had tags 1 and 2, old code will simply skip tag 3 when it encounters it in the data. The field is unknown, and the decoder skips it using the wire type to determine how many bytes to consume. New code, when reading old data, will see that tag 3 is missing and use the default value.

Rule 2: You cannot rename fields — but you don't need to. Field names only exist in the schema definition, not in the wire format. The name user_name is purely for programmer convenience. You can rename it to display_name in your .proto file without changing a single byte of encoded data, as long as the tag number stays the same.

Rule 3: You cannot change a field's tag number. The tag IS the field's identity. Changing tag 2 from favorite_number to tag 5 means every existing encoded record still says "tag 2 = 1337," but your new code looks for tag 5 and finds nothing. The data is lost.

Rule 4: Never add a required field after initial deployment. If you add a required field, old code will produce records without it. When new code reads those records, it will reject them because the required field is missing. This breaks backward compatibility. In Protobuf 3, all fields are optional by default — Google learned this lesson the hard way.

Rule 5: You can change types if the wire format is compatible. Changing int32 to int64 is safe — both use varint encoding, and the bits are compatible. But reading a 64-bit value with 32-bit code may truncate (the high bits are silently discarded). Changing int32 to string is not safe — the wire types are different and the decoder will misinterpret the bytes.

Schema Evolution Visualizer

The simulation below shows two schema versions side by side. Try different evolution operations and see which ones are safe (green) and which ones break compatibility (red). This is exactly the kind of reasoning you must do in a system design interview when discussing API versioning.

Evolution Rules Summary

Chapter 5: Apache Avro

Protobuf and Thrift solve schema evolution with field tags. Apache Avro takes a radically different approach: no tags at all. Fields are matched by name, and the encoded data contains no field identifiers whatsoever — not tags, not names, nothing. Instead, the writer's schema and reader's schema are resolved at read time.

How Avro Encoding Works

An Avro schema looks like this:

json
{
  "type": "record",
  "name": "Person",
  "fields": [
    {"name": "userName", "type": "string"},
    {"name": "favoriteNumber", "type": ["null", "long"], "default": null}
  ]
}

When Avro encodes this record, it writes the fields in schema order with no field identifiers:

But wait — if there are no tags and no names, how does the reader know which bytes correspond to which field? The answer: the reader must have the writer's schema. Without it, the bytes are meaningless — just a stream of values with no field identity.

Schema Resolution: Writer vs Reader

This is Avro's key innovation. The writer and reader can have different schemas, and Avro resolves them at read time by matching fields by name:

And in reverse:

How Does the Reader Get the Writer's Schema?

Three strategies, each suited to a different dataflow pattern:

Avro Schema Resolution Visualizer

The simulation below shows writer and reader schemas side by side. Watch how Avro matches fields by name, fills in defaults for missing fields, and skips fields the reader does not know about.

Why Avro? Dynamic Schema Generation

Avro's killer feature is that schemas can be generated programmatically. Consider a database table:

sql
CREATE TABLE users (
  user_id  BIGINT,
  name     VARCHAR(255),
  email    VARCHAR(255)
);

You can automatically generate an Avro schema from this DDL. If someone runs ALTER TABLE users ADD COLUMN phone TEXT;, a new Avro schema is generated automatically. The old and new schemas coexist through Avro's resolution mechanism. No one has to manually assign tag numbers. No one can accidentally reuse a tag. The database schema is the serialization schema.

This is why Avro dominates in the Hadoop ecosystem: data pipelines that export database tables to HDFS can evolve their schemas automatically as the source database evolves.

Avro vs Protobuf: When to Use Which

python
# Writing and reading Avro with schema evolution
import avro.schema, avro.io, avro.datafile
import io

# Writer schema v1
writer_schema = avro.schema.parse("""
{
  "type": "record", "name": "User",
  "fields": [
    {"name": "userName", "type": "string"},
    {"name": "favoriteNumber", "type": ["null", "long"], "default": null}
  ]
}
""")

# Encode a record
buf = io.BytesIO()
encoder = avro.io.BinaryEncoder(buf)
writer = avro.io.DatumWriter(writer_schema)
writer.write({"userName": "Martin", "favoriteNumber": 1337}, encoder)
raw_bytes = buf.getvalue()
print(f"Encoded: {raw_bytes.hex()} ({len(raw_bytes)} bytes)")
# Encoded: 0c4d617274696e02f214 (10 bytes)

# Reader schema v2 — has an extra field with default
reader_schema = avro.schema.parse("""
{
  "type": "record", "name": "User",
  "fields": [
    {"name": "userName", "type": "string"},
    {"name": "favoriteNumber", "type": ["null", "long"], "default": null},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}
""")

# Decode v1 data with v2 reader schema — email gets default
buf = io.BytesIO(raw_bytes)
decoder = avro.io.BinaryDecoder(buf)
reader = avro.io.DatumReader(writer_schema, reader_schema)
record = reader.read(decoder)
print(record)
# {'userName': 'Martin', 'favoriteNumber': 1337, 'email': None}
# email filled with default — backward compatibility works!

Chapter 6: Schema Registries

We have seen that Avro needs the writer's schema at read time, and that Protobuf/Thrift need the .proto/.thrift files to decode messages. But how do you distribute these schemas across a distributed system with dozens of services? You cannot email .proto files around. You need a schema registry.

What a Schema Registry Does

A schema registry is a centralized service that stores versioned schemas and enforces compatibility rules. The most widely deployed is the Confluent Schema Registry, designed to work with Apache Kafka. Here is the workflow:

The 5-byte header is a brilliant design: it adds negligible overhead (5 bytes per message) but gives you full schema evolution support. Compare this to embedding the entire schema in every message (which Avro container files do for HDFS) — for a Kafka topic with millions of small messages, embedding the schema would be catastrophically wasteful.

Compatibility Modes

The registry does not just store schemas — it enforces rules about which schema changes are allowed. There are four compatibility modes:

Schema Registry Workflow

The simulation below animates the complete workflow: producer registers schema, registry checks compatibility, producer sends data with schema ID, consumer looks up schema, consumer decodes. Watch what happens when a compatibility violation is attempted.

Schema Registry in Practice

python
# Producing Avro messages with Confluent Schema Registry
from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

# Connect to registry
registry = SchemaRegistryClient({'url': 'http://schema-registry:8081'})

# Define schema (or load from file)
schema_str = """
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "userId", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}
"""

# Create serializer (auto-registers schema with registry)
avro_serializer = AvroSerializer(registry, schema_str)

producer = SerializingProducer({
    'bootstrap.servers': 'kafka:9092',
    'value.serializer': avro_serializer,
})

# Produce a message — schema ID is prepended automatically
producer.produce(
    topic='users',
    value={'userId': 42, 'name': 'Alice', 'email': 'alice@example.com'}
)
producer.flush()

Chapter 7: Modes of Dataflow

Schema evolution does not exist in a vacuum. The way data flows between processes determines which compatibility properties matter and which encoding formats are practical. There are three fundamental modes of dataflow, each with different implications for encoding and evolution.

Mode 1: Via Databases

When you write to a database, you are encoding data for your future self — or for a colleague who will read it months or years from now. The writer encodes, and a potentially very different reader decodes. This means you need both forward compatibility (old application reading newly-written data) and backward compatibility (new application reading old data that was written years ago).

The most dangerous scenario: a v1 application reads a record written by v2 (which has extra fields), modifies an unrelated field, and writes it back. If the application does not preserve unknown fields, the v2 fields are silently erased. This is called the read-modify-write hazard, and it is the single most common source of silent data corruption in production systems.

Mode 2: Via Service Calls (REST/RPC)

In service-to-service communication, one process (the client) makes a request and another (the server) responds. The client and server may be running different versions of the code.

gRPC (Protobuf over HTTP/2) has become the standard for internal service-to-service communication. It is smaller and faster than REST+JSON, has built-in schema evolution via Protobuf, supports streaming (server-side, client-side, and bidirectional), and generates client/server code automatically from .proto files.

Mode 3: Via Message Queues

In asynchronous messaging (Kafka, RabbitMQ, SQS), the producer and consumer are fully decoupled. The producer may have been deployed weeks before the consumer. Messages may sit in the queue for hours or days. The consumer may be running code that was written before the new fields even existed.

This is where schema registries shine. The producer registers its schema (with compatibility checks), embeds the schema ID in each message, and the consumer resolves schemas at read time. Without a registry, you are flying blind — every consumer must somehow know which schema the producer was using, and version mismatches cause silent data corruption.

The Big Simulation: All Three Modes

The simulation below shows data flowing through all three modes simultaneously. You can modify the schema (add a field) and watch how the change propagates. Notice where compatibility breaks happen and where the system handles evolution gracefully.

The Read-Modify-Write Hazard (Detailed)

This is so important that it deserves a concrete, step-by-step walkthrough:

The fix depends on the encoding format. Protobuf and Avro decoders can be configured to preserve unknown fields: when v1 code reads a record with unknown tags, it stores the raw bytes in a side buffer and re-emits them on write. This "round-trip safety" is not automatic in all implementations — you must verify that your serialization library supports it.

Chapter 8: Dataflow in Practice

Theory is clean. Practice is messy. Let us look at how real systems combine encoding formats, handle versioning, and debug the inevitable breakages.

The Modern Stack

Design Challenge: Schema Evolution Strategy for 50 Services

Imagine you are the staff engineer responsible for data contracts across a platform of 50 microservices. Here is the system you would design:

Debug Scenarios

These are the failure modes you will encounter in production (and in interviews):

Scenario 1: "Deserialization failures after deploying a new service version."

Root cause: a required field was added. Old messages in the Kafka topic (or old records in the database) do not have this field. The new code's deserializer throws because the required field is missing. Fix: make the field optional with a default. Redeploy. The error rate drops to zero because the deserializer now fills in the default for old records.

Scenario 2: "Silent data corruption — users report missing phone numbers."

Root cause: a v1 service reads records written by v2, modifies an unrelated field, and writes back without preserving unknown fields. phone_number (tag 3) is silently dropped on every round-trip through v1. Fix: upgrade all services to a Protobuf library version that preserves unknown fields. Add a monitoring check that detects field disappearance (compare field counts before and after writes).

Scenario 3: "Kafka consumer lag spike after producer schema change."

Root cause: the producer registered a new schema that is technically compatible, but the consumer's deserialization code has a bug: it tries to cast the new field to an incompatible type in application code (e.g., the schema says the field is a string, but the consumer's application code tries to parse it as an integer). The schema registry did not catch this because it only checks wire-format compatibility, not application-level semantics. Fix: add integration tests that deserialize sample messages with every consumer. Run these tests in the producer's CI pipeline before the schema change is merged.

Anti-Patterns

gRPC vs REST: A Concrete Comparison

Let us compare the same API call in both formats to make the trade-offs tangible.

protobuf
// gRPC: Define the service in a .proto file
service UserService {
  rpc GetUser(GetUserRequest) returns (User);
  rpc ListUsers(ListUsersRequest) returns (stream User);  // server streaming!
}

message GetUserRequest {
  int64 user_id = 1;
}

message User {
  int64 user_id = 1;
  string name = 2;
  string email = 3;
  optional string phone = 4;
}

python
# gRPC client (auto-generated from .proto)
import grpc
import user_pb2, user_pb2_grpc

channel = grpc.insecure_channel('user-service:50051')
stub = user_pb2_grpc.UserServiceStub(channel)

# Type-safe, IDE-autocompleted, binary-encoded
response = stub.GetUser(user_pb2.GetUserRequest(user_id=42))
print(response.name)   # "Alice"
print(response.phone)  # "" (default, field 4 missing in old data)

# Server streaming: get 10K users without buffering all in memory
for user in stub.ListUsers(user_pb2.ListUsersRequest(limit=10000)):
    process(user)  # each user arrives as it's ready

python
# REST equivalent — compare the developer experience
import requests

# No type safety, no auto-completion, text-encoded
response = requests.get('http://user-service:8080/api/v1/users/42')
data = response.json()  # dict, not a typed object
print(data['name'])    # KeyError if field missing
print(data.get('phone', ''))  # manual default handling

# No streaming: must buffer all 10K users in one JSON array
response = requests.get('http://user-service:8080/api/v1/users?limit=10000')
users = response.json()  # entire 10K-user array in memory at once

Format Comparison

The simulation below shows the same 1000-message workload processed by different format combinations. Compare throughput, payload size, and schema safety.

Chapter 9: Interview Arsenal

This chapter distills everything into the cheat sheets, drills, and patterns you need for system design and coding interviews.

Format Cheat Sheet

System Design Questions

Q: "Design a schema evolution strategy for a microservices platform."

Answer framework: (1) Schema repo with CI-enforced compatibility checks (buf breaking for Protobuf, schema registry with FULL mode for Avro/Kafka). (2) Code generation pipeline publishing versioned stubs. (3) Runtime validation middleware with dead-letter queues. (4) Monitoring: deserialization error rate per service per topic. (5) Policy: all new fields optional with defaults, never reuse tag numbers, never change types.

Q: "Compare REST vs gRPC for internal services."

Answer: gRPC wins on all internal metrics: (1) Payload size: Protobuf binary is 4-10x smaller than JSON. (2) Parse speed: binary parsing is ~10x faster. (3) Schema enforcement: .proto files catch type errors at compile time, not runtime. (4) Code generation: stubs in any language for free. (5) Streaming: gRPC supports server/client/bidirectional streaming natively. (6) HTTP/2: multiplexed connections, header compression. REST wins for external APIs: human readable, universally supported, curl-testable, no build step for consumers.

Q: "A Kafka consumer is deserializing messages 10x slower than expected. Diagnose."

Answer framework: (1) Check if the consumer is hitting the schema registry on every message instead of caching. (2) Check if the consumer is using reflection-based deserialization instead of generated code. (3) Profile the deserialization: is it CPU-bound (parsing) or memory-bound (allocations)? (4) Check if the messages contain deeply nested Avro unions that require extensive resolution. (5) Compare against a baseline: deserialize 1M messages of the same schema and measure throughput.

Coding Drills

Drill 1: Write a Protobuf schema with evolution.

protobuf
// v1: initial schema
syntax = "proto3";

message UserEvent {
  int64 user_id = 1;
  string action = 2;     // "login", "purchase", etc.
  int64 timestamp = 3;    // Unix millis
}

// v2: add optional metadata field
message UserEvent {
  int64 user_id = 1;
  string action = 2;
  int64 timestamp = 3;
  string ip_address = 4;  // NEW: optional (proto3 default)
  map<string, string> metadata = 5;  // NEW: flexible key-value
}

// v3: deprecate a field, add another
message UserEvent {
  int64 user_id = 1;
  string action = 2;
  int64 timestamp = 3;
  reserved 4;             // ip_address removed, tag 4 reserved forever
  reserved "ip_address";  // name also reserved
  map<string, string> metadata = 5;
  string session_id = 6; // NEW field gets NEW tag number
}

Drill 2: Implement varint encoder/decoder.

python
def encode_varint(n):
    """Encode unsigned integer as Protobuf varint bytes."""
    buf = []
    while n > 0x7F:
        buf.append((n & 0x7F) | 0x80)
        n >>= 7
    buf.append(n)
    return bytes(buf)

def decode_varint(data):
    """Decode varint from bytes. Returns (value, bytes_consumed)."""
    result = 0
    shift = 0
    for i, byte in enumerate(data):
        result |= (byte & 0x7F) << shift
        if not (byte & 0x80):
            return result, i + 1
        shift += 7
    raise ValueError("Varint not terminated")

def zigzag_encode(n):
    return (n << 1) ^ (n >> 63)

def zigzag_decode(n):
    return (n >> 1) ^ -(n & 1)

# Test round-trips
for v in [0, 1, 127, 128, 300, 1337, 1000000]:
    encoded = encode_varint(v)
    decoded, length = decode_varint(encoded)
    assert decoded == v, f"Failed for {v}"
    print(f"{v:>10} -> {encoded.hex():12} ({length} bytes)")

# Output:
#          0 -> 00           (1 bytes)
#          1 -> 01           (1 bytes)
#        127 -> 7f           (1 bytes)
#        128 -> 8001         (2 bytes)
#        300 -> ac02         (2 bytes)
#       1337 -> b90a         (2 bytes)
#    1000000 -> c0843d       (3 bytes)

Drill 3: Decode a Protobuf Message by Hand

This is the kind of question a senior interviewer at Google or Meta might ask. You are given raw Protobuf bytes and a schema. Decode the message.

The Five Interview Dimensions Applied

Chapter 10: Connections

Encoding and evolution do not exist in isolation. They interact deeply with every other layer of a data-intensive system. Here is where this chapter connects to the rest of the DDIA landscape.

Where Encoding Fits

The Encoding Decision Tree

When choosing a format for a new system, walk this tree:

What We Did Not Cover

This chapter focused on the fundamental concepts. More advanced topics include:

• Cap'n Proto — Zero-copy serialization. The on-wire format IS the in-memory format. No encoding/decoding step at all. Trades portability for speed.
• FlatBuffers (Google) — Random access to individual fields without parsing the entire message. Used heavily in games and mobile apps.
• Apache Arrow — Columnar in-memory format for analytics. Not a serialization format per se, but a shared memory layout that eliminates serialization between analytics tools.
• Protocol Buffer Editions — Google's successor to proto2/proto3 syntax, designed to unify the divergent feature sets.

"The key idea is that old and new code, old and new data, must coexist. Encoding formats are the contracts that make coexistence possible." — Martin Kleppmann, DDIA