Consistency & Consensus

Chapter 1: Linearizability

Before we can talk about how nodes agree, we need to define what "agreement" looks like from the outside. When a client reads a value from a distributed system, what should they see? The answer depends on the consistency model the system provides — a contract between the system and its clients about what behaviors are allowed.

The strongest consistency model is linearizability (also called atomic consistency or strong consistency). Here is the promise: the system behaves as if there is exactly one copy of the data, and every operation takes effect atomically at some single point in time between when the client sent the request and when the client received the response.

Let us make that precise. Every read or write operation has two timestamps: the invocation (when the client sends the request) and the response (when the client receives the reply). In a real system, there is some nonzero duration between these — the operation takes time. Linearizability says that you must be able to assign a single point in time within that duration where the operation "takes effect," and if you line up all operations by their effect points, the result must look like a sequential execution.

Is This History Linearizable?

The classic technique for checking linearizability: draw operations as horizontal bars on a timeline (one row per client), then try to place a "linearization point" (a dot) within each bar such that the dots form a valid sequential order.

Linearizability vs Serializability

These two terms are constantly confused — even in published papers. They are entirely different guarantees.

The combination of both — strict serializability — gives you transactions that are both serializable AND linearizable. This is what Spanner and CockroachDB provide, and it is the gold standard. But it comes at a cost: you need globally synchronized clocks (Spanner uses GPS and atomic clocks) or consensus on every transaction.

Implementing Linearizability

The Cost: CAP Theorem

The CAP theorem (Brewer, 2000; proved by Gilbert & Lynch, 2002) states: during a network partition, a distributed system must choose between Consistency (linearizability) and Availability (every non-crashed node responds). You cannot have both.

More precisely: if two nodes cannot communicate (partition), and a client writes to one node, the other node has two options: (1) refuse to answer reads until the partition heals (consistent but unavailable), or (2) answer reads with potentially stale data (available but inconsistent). There is no third option.

Worked Example: Is This Linearizable?

The CAP Spectrum in Real Systems

CAP is often presented as a binary choice, but real systems exist on a spectrum. Here is where common databases land:

When Do You Need Linearizability?

Not everything needs the strongest guarantee. Here is a decision framework:

Chapter 2: Ordering Guarantees

Linearizability is powerful but expensive. Can we get useful guarantees with something weaker? Yes — by focusing on causality. If event A caused event B, then A must be ordered before B. But if A and B are independent (neither caused the other), we don't care which comes first. This is causal consistency, and it is the strongest consistency model that does not sacrifice availability during partitions.

But how do you track "what caused what" in a distributed system? You cannot use wall clocks — we learned in the previous chapters that clocks drift, skew, and cannot be trusted. You need a logical clock.

Lamport Timestamps

Leslie Lamport (1978) invented a beautifully simple mechanism. Each node maintains a counter. The rules:

The result: if event A causally precedes event B, then A's timestamp is strictly less than B's. The contrapositive: if timestamp(A) ≥ timestamp(B), then A did NOT causally precede B.

Vector Clocks: Detecting Concurrency

Lamport timestamps are insufficient for detecting concurrent events. Vector clocks extend the idea: instead of a single counter, each node maintains a vector of counters — one for every node in the system.

Vector clocks are used in systems like Riak and Amazon's Dynamo (internally) to detect write conflicts. When two writes are concurrent, the system can either: (a) keep both versions (siblings) and let the application resolve, or (b) use a deterministic rule (last-write-wins by wall clock — but this loses data).

Total Order Broadcast

Total order broadcast (also called atomic broadcast) is a protocol that delivers messages to all nodes such that: (1) every non-crashed node delivers the same set of messages, and (2) every node delivers them in the same order. This is strictly stronger than Lamport timestamps because it gives you a total order that all nodes agree on, not just one that's consistent per node.

Here is the remarkable equivalence: total order broadcast is equivalent to consensus. If you can solve total order broadcast, you can solve consensus (decide on the first message delivered). If you can solve consensus, you can solve total order broadcast (use consensus to decide each next message). This equivalence is why we care so deeply about consensus algorithms — they are the universal primitive.

Formally, total order broadcast has two properties:

python
class LamportClock:
    def __init__(self, node_id):
        self.time = 0
        self.node_id = node_id

    def local_event(self):
        self.time += 1
        return self.time

    def send(self):
        self.time += 1
        return (self.time, self.node_id)  # attach to message

    def receive(self, msg_time):
        self.time = max(self.time, msg_time) + 1
        return self.time

# Usage
a = LamportClock("A")
b = LamportClock("B")

a.local_event()          # a.time = 1
ts = a.send()            # a.time = 2, msg carries (2, "A")
b.receive(ts[0])        # b.time = max(0, 2) + 1 = 3
b.local_event()          # b.time = 4

Worked Example: Tracing Lamport Timestamps

Worked Example: Vector Clocks Detect Conflict

Chapter 3: Two-Phase Commit

Before diving into real consensus algorithms, let's examine a simpler approach that almost works: two-phase commit (2PC). It guarantees atomicity across distributed nodes — all nodes commit or all nodes abort — but it is NOT consensus. Understanding why reveals exactly what consensus needs that 2PC lacks.

The Protocol

2PC uses a designated coordinator node that orchestrates a two-phase dance with the participants (the nodes that hold the data):

Notice the critical rule: once a participant votes "yes," it has made an irrevocable promise. It has written all data to its log and it CANNOT unilaterally abort — it must wait for the coordinator's decision, no matter how long that takes.

The Fatal Flaw

What happens if the coordinator crashes after sending "prepare" but before sending the decision? Every participant that voted "yes" is stuck. They have locks held on their data. They cannot commit (they don't know the decision). They cannot abort (they promised they could commit, and maybe the coordinator will come back and tell them to commit). They are blocked — holding locks, unable to proceed, potentially forever if the coordinator never recovers.

Worked Example: 2PC Timeline

2PC in the Real World

2PC is used in practice despite its limitations, because the alternatives are worse for many use cases:

From 2PC to Consensus

The fix for 2PC's blocking problem is to replicate the coordinator using a consensus algorithm. Instead of one coordinator, you have a group of nodes that agree on the decision using Raft or Paxos. If one node crashes, the others still know the decision. This is called three-phase commit (3PC) in theory, but in practice everyone just uses consensus directly.

The insight: 2PC's problem is that the coordinator is a single point of failure with irrecoverable state (the commit decision). Consensus eliminates single points of failure by replicating the decision across a majority of nodes. This is the bridge from "simple coordination" to "fault-tolerant agreement."

Chapter 4: Raft Consensus

In 2014, Diego Ongaro and John Ousterhout published "In Search of an Understandable Consensus Algorithm." They were frustrated that Paxos — the dominant consensus algorithm — was so notoriously difficult to understand that most implementations got it wrong. Their goal was to create an algorithm that was equivalent in correctness and performance but dramatically easier to understand. The result was Raft.

Raft decomposes consensus into three independent sub-problems:

Three Roles

Every Raft node is in exactly one of three states at any time:

Follower: passive. Responds to requests from leaders and candidates. If it hears nothing from a leader for a random timeout period (the election timeout, typically 150-300ms), it becomes a candidate.

Candidate: actively seeking votes. Increments its term number (a logical clock for the election cycle), votes for itself, and sends RequestVote RPCs to all other nodes. If it gets votes from a majority, it becomes the leader. If it hears from a new leader, it steps down to follower. If the election times out with no winner, it starts a new election with a higher term.

Leader: handles all client requests. Appends entries to its log, sends AppendEntries RPCs (also used as heartbeats) to all followers, and tracks which entries have been committed. There is at most one leader per term.

Leader Election in Detail

The randomized timeout is key to preventing split votes. If two candidates start elections simultaneously, they might split the vote and neither gets a majority. In that case, both time out and start new elections — but with freshly randomized timeouts, so one will almost certainly start before the other and win. In practice, split votes are rare (Ongaro reports them happening less than once per thousand elections in testing).

State Transition Diagram

Every Raft node moves between exactly three states. The transitions are triggered by specific events:

The Heartbeat Mechanism

A leader sends heartbeats (empty AppendEntries RPCs) to all followers at a regular interval — typically every 50-100ms. This serves two purposes: (1) it prevents followers from timing out and starting elections, and (2) it carries the leader's commitIndex, telling followers which entries are committed.

The election timeout must be significantly longer than the heartbeat interval. The Raft paper recommends:

Where broadcastTime is the average time for an RPC round trip (~1-20ms), electionTimeout is the random timeout (150-300ms), and MTBF is the mean time between failures for a single node (months to years). This ensures elections are rare, fast, and don't overlap with heartbeats.

Log Replication in Detail

Once a leader is elected, it handles all client requests. Each request becomes an entry in the leader's log: an ordered sequence of (term, command) pairs. The leader then replicates each entry to followers:

The Performance of Raft in Practice

How fast is Raft? The critical metric is commit latency: the time from when a client sends a request to when it receives a committed response. This depends on:

The key insight for system design: consensus is fast enough for metadata (who is the leader? what's the partition map?) but too slow for data-plane operations (every user write, every analytics event). This is why the pattern is always: consensus for coordination, something faster for data.

python
from enum import Enum
import random

class State(Enum):
    FOLLOWER = "follower"
    CANDIDATE = "candidate"
    LEADER = "leader"

class RaftNode:
    def __init__(self, node_id, peers):
        self.id = node_id
        self.peers = peers
        self.state = State.FOLLOWER
        self.current_term = 0
        self.voted_for = None
        self.log = []  # list of (term, command)
        self.commit_index = -1
        self.election_timeout = random.uniform(0.15, 0.3)

    def start_election(self):
        self.state = State.CANDIDATE
        self.current_term += 1
        self.voted_for = self.id
        votes = 1  # vote for self
        for peer in self.peers:
            if self.request_vote(peer):
                votes += 1
        if votes > len(self.peers + [self]) // 2:
            self.state = State.LEADER
            self.send_heartbeats()

    def append_entry(self, command):
        """Leader receives client request."""
        assert self.state == State.LEADER
        self.log.append((self.current_term, command))
        acks = 1  # leader has it
        for peer in self.peers:
            if self.replicate_to(peer):
                acks += 1
        if acks > len(self.peers + [self]) // 2:
            self.commit_index = len(self.log) - 1
            return True  # committed
        return False

Raft RPCs in Detail

Raft uses only two RPCs for the core protocol. This simplicity is intentional — fewer message types means fewer edge cases to reason about.

Per-Follower State Tracking

The leader maintains two indices for each follower:

The commit rule: an entry at index N is committed if matchIndex[i] >= N for a majority of nodes, AND the entry at N has the current term. This last condition — the entry must be from the current term — prevents a subtle bug where a leader commits entries from a previous term based on replication count alone. (See the Raft paper Section 5.4.2 for the counterexample that motivates this rule.)

Chapter 5: Raft Deep Dive

The elegance of Raft lies in two safety properties that guarantee consistency even through crashes and elections. Let us examine each in detail.

Log Matching Property

If two nodes have a log entry with the same index and the same term, then: (1) that entry stores the same command, and (2) ALL preceding entries are identical in both logs. This property is maintained by the consistency check in AppendEntries: the leader sends (prevLogIndex, prevLogTerm) and the follower rejects the append if its log doesn't match at that position. The leader then decrements the index and retries until it finds the point where the logs agree, then replays entries from there.

Leader Completeness Property

If a log entry is committed in a given term, then that entry will be present in the logs of all leaders in all higher terms. This is enforced by the election restriction: a candidate includes its (lastLogIndex, lastLogTerm) in RequestVote, and a voter rejects candidates whose logs are less up-to-date. Since committed entries are present on a majority, and a candidate needs a majority to win, at least one voter in any winning majority must have the committed entry — and will not vote for a candidate that lacks it.

Worked Example: 5-Node Cluster Through Failures

Let us trace a complete scenario. Our cluster has nodes {1, 2, 3, 4, 5}. We will go through leader election, writes, a crash, re-election, and log reconciliation.

How Raft Handles Edge Cases

Raft Cluster Membership Changes

One of the trickiest aspects of running Raft in production: what happens when you need to add or remove a node? You cannot just change the configuration on all nodes simultaneously — there will be a moment where some nodes think the cluster has 5 members and others think it has 6. During that overlap, you could get two different majorities, leading to split-brain.

Raft solves this with joint consensus: a two-phase approach where the cluster temporarily operates under both the old and new configurations simultaneously.

Read Consistency in Raft

There is a subtle problem with reads in Raft. A client reads from the leader, but how does the leader know it is still the leader? It might have been deposed by a network partition and not know it yet. Two solutions:

etcd uses the Read Index approach by default. CockroachDB uses lease-based reads with careful clock synchronization. The choice depends on whether you trust clocks (most engineers do not, and they are right).

Chapter 6: Paxos

Before Raft, there was Paxos. Leslie Lamport published it in 1989 (in a paper so whimsical that reviewers rejected it; it was finally published in 1998). Paxos solves the same problem as Raft — getting a group of nodes to agree on a value — but it takes a fundamentally different approach. Where Raft is structured around a strong leader, Paxos is more symmetric: any node can be a proposer.

Paxos has three roles:

In practice, the same physical node often plays all three roles. But separating them conceptually makes the algorithm easier to reason about.

Single-Decree Paxos

The simplest form of Paxos agrees on a single value. It runs in two phases:

The Livelock Problem

Paxos has a subtle liveness issue. Imagine two proposers, P1 and P2, competing:

This is called livelock: no value is ever decided because proposers keep interrupting each other. The solution is Multi-Paxos: elect a stable leader (a distinguished proposer) that skips Phase 1 for subsequent proposals. As long as the leader is stable, Paxos is fast. If the leader fails, a new one is elected — which temporarily risks livelock, but the randomized backoff makes it resolve quickly.

Why Is Paxos Hard?

Lamport's original Paxos paper is famous for being impenetrable. Even the "simplified" version confuses most readers. The difficulty comes from several sources:

Worked Example: A Complete Paxos Round

Paxos vs Raft: The Key Differences

Chapter 7: ZAB & Other Algorithms

Raft and Paxos are the two most well-known consensus algorithms, but they are not the only ones in production. Let us survey the landscape.

ZAB: ZooKeeper Atomic Broadcast

ZAB (ZooKeeper Atomic Broadcast) is the consensus protocol used by Apache ZooKeeper. It was designed specifically for total order broadcast: delivering messages to all nodes in the same order. ZAB is similar to Raft but was developed independently (published 2008, predating Raft by 6 years).

Key differences from Raft:

ZAB in Detail

ZAB has three phases that each new leader goes through:

The zxid (ZooKeeper transaction ID) is a 64-bit number: the high 32 bits are the epoch (like Raft's term), and the low 32 bits are a counter within the epoch. This makes it trivial to compare zxids and determine which transactions are from which leader's reign.

EPaxos: Leaderless Consensus

Egalitarian Paxos (EPaxos), published by Moraru, Andersen, and Kaminsky in 2013, takes a radical approach: no designated leader at all. Any node can process any command. Commands that don't conflict (they touch different keys) can be committed in one round trip with no coordination. Only conflicting commands require an extra round.

The appeal: in a geo-distributed system, a client in Tokyo can send a command to the Tokyo replica and get a response in one round trip to a majority, without routing through a leader in Virginia. For workloads with few conflicts (which describes most real workloads), this dramatically reduces latency.

The catch: EPaxos is significantly more complex than Raft. The dependency tracking, conflict detection, and execution ordering are all harder to implement correctly. As of 2024, no major production system uses EPaxos. But the ideas are influential and show up in research on geo-distributed databases.

Viewstamped Replication

Viewstamped Replication (VR), published by Oki and Liskov in 1988, is contemporaneous with Paxos and shares many ideas. Each "view" has a designated primary (leader). When the primary fails, a view change protocol selects a new one. The key insight: during a view change, the new primary collects logs from a majority and picks the one with the most entries. This is strikingly similar to Raft's election restriction.

The FLP Impossibility

In 1985, Fischer, Lynch, and Paterson proved the FLP impossibility result: in a purely asynchronous system (no timeouts, no clocks), where even a single process can crash, there is NO algorithm that guarantees consensus. This result shocked the distributed systems community.

Comparison Table

Performance Characteristics

In practice, Raft and ZAB dominate because the leader bottleneck is rarely the limiting factor for coordination workloads. The simplicity advantage of Raft and the battle-tested nature of ZAB far outweigh EPaxos's theoretical latency advantage for most teams.

Consensus and Blockchain

You may wonder: how does blockchain consensus relate to Raft and Paxos? The key distinction is the trust model:

For infrastructure engineering (which is what DDIA and this lesson focus on), you almost always use crash-fault-tolerant (CFT) consensus. You trust your own data center's servers not to lie — they just might crash. BFT consensus is for environments where participants are adversarial, which is a fundamentally different (and much harder) problem.

What Consensus Gives You

All of these algorithms solve the same abstract problem. In practice, consensus enables:

Chapter 8: Consistency in Practice

You will rarely implement Raft or Paxos yourself. Instead, you will use coordination services — systems like ZooKeeper, etcd, and Consul that implement consensus internally so your application does not have to. They expose simple APIs (get, set, watch, lock) while hiding the complexity of leader election, log replication, and failure recovery.

The Big Three Coordination Services

Design Challenge: Distributed Task Scheduler

You are building a system where thousands of tasks arrive every second, and a pool of 50 worker nodes must claim and execute them. Each task must be executed exactly once. How do you ensure no two workers claim the same task?

Debug Scenario: ZooKeeper Quorum Lost

You are on-call. At 3 AM, you get paged: "Kafka brokers are failing to produce/consume." You check and discover that 2 out of 3 ZooKeeper nodes are down. What happens?

Fencing Tokens: The Lock Safety Net

Even with a distributed lock service, there is a danger. Process A acquires a lock, then pauses (GC pause, swap, etc.). The lock expires (TTL runs out). Process B acquires the same lock. Now A resumes and thinks it still holds the lock. Both processes believe they have exclusive access. Data corruption ensues.

The solution is fencing tokens: every time a lock is acquired, the coordination service issues a monotonically increasing token. The resource (database, file, etc.) checks the token on every write and rejects writes with tokens older than the most recent one it has seen.

Service Discovery Pattern

A common use case for coordination services: service discovery. In a microservices architecture, services need to find each other. Which instances of the "payment service" are currently healthy? What are their IP addresses?

Consul adds a built-in health checking layer on top of this: it not only registers services but actively pings them (HTTP checks, TCP checks, gRPC checks) and removes unhealthy instances from the service catalog. This is why Consul is often preferred for service mesh architectures.

Anti-Pattern: Putting Everything in the Consensus Store

A common mistake in distributed system design: using etcd or ZooKeeper for too much. Here are the warning signs:

Chapter 9: Interview Arsenal

Consistency Models: Strongest to Weakest

Quick Decision Matrix for System Design Interviews

When the interviewer asks "what consistency do you need for this?", use this framework:

Raft Summary Card

System Design Talking Points

"Design a distributed lock service."

"Design a configuration management system."

"Why did Kafka use ZooKeeper, and why is it removing it?"

Coding Drills

python
# Drill 1: Implement a vector clock (extends Lamport timestamps)

class VectorClock:
    def __init__(self, node_id, num_nodes):
        self.id = node_id
        self.clock = [0] * num_nodes  # one counter per node

    def local_event(self):
        self.clock[self.id] += 1

    def send(self):
        self.clock[self.id] += 1
        return list(self.clock)  # copy

    def receive(self, other_clock):
        for i in range(len(self.clock)):
            self.clock[i] = max(self.clock[i], other_clock[i])
        self.clock[self.id] += 1

    def happens_before(self, other_clock):
        """Does self happen-before other? (All entries ≤, at least one <)"""
        return (all(self.clock[i] <= other_clock[i]
                   for i in range(len(self.clock)))
            and any(self.clock[i] < other_clock[i]
                    for i in range(len(self.clock))))

    def is_concurrent(self, other_clock):
        """Neither happens-before the other."""
        return (not self.happens_before(other_clock)
            and not VectorClock._static_hb(other_clock, self.clock))

# Usage:
a = VectorClock(0, 3)  # Node 0 in a 3-node system
b = VectorClock(1, 3)
a.local_event()              # a.clock = [1, 0, 0]
msg = a.send()               # a.clock = [2, 0, 0], msg = [2, 0, 0]
b.receive(msg)               # b.clock = [2, 1, 0] (merged + incremented)

python
# Drill 2: Raft leader election state machine

import random, time

class RaftElection:
    def __init__(self, node_id, cluster_size):
        self.id = node_id
        self.cluster_size = cluster_size
        self.state = "follower"
        self.term = 0
        self.voted_for = None
        self.votes_received = set()
        self.last_heartbeat = time.time()
        self.timeout = random.uniform(0.15, 0.3)
        self.log = []  # [(term, cmd), ...]

    def tick(self):
        """Called periodically. Triggers election if timeout expired."""
        if self.state == "leader":
            return None
        if time.time() - self.last_heartbeat > self.timeout:
            return self.start_election()

    def start_election(self):
        self.state = "candidate"
        self.term += 1
        self.voted_for = self.id
        self.votes_received = {self.id}
        self.timeout = random.uniform(0.15, 0.3)
        self.last_heartbeat = time.time()
        last_idx = len(self.log) - 1
        last_term = self.log[last_idx][0] if self.log else 0
        return ("RequestVote", self.term, last_idx, last_term)

    def handle_vote_request(self, candidate_term, cand_last_idx, cand_last_term):
        if candidate_term < self.term:
            return False
        if candidate_term > self.term:
            self.term = candidate_term
            self.state = "follower"
            self.voted_for = None
        # Check log freshness
        my_last_term = self.log[-1][0] if self.log else 0
        my_last_idx = len(self.log) - 1
        log_ok = (cand_last_term > my_last_term or
                  (cand_last_term == my_last_term and cand_last_idx >= my_last_idx))
        if self.voted_for in (None, candidate_term) and log_ok:
            self.voted_for = candidate_term
            return True
        return False

    def receive_vote(self, voter_id, granted):
        if granted:
            self.votes_received.add(voter_id)
        if len(self.votes_received) > self.cluster_size // 2:
            self.state = "leader"
            return True  # became leader!
        return False

python
# Drill 3: Distributed lock with fencing token

import time

class DistributedLock:
    """Simplified lock with fencing token support."""

    def __init__(self):
        self.holder = None
        self.token_counter = 0
        self.expiry = 0

    def acquire(self, client_id, ttl_seconds=10):
        """Try to acquire lock. Returns fencing token or None."""
        now = time.time()
        # If lock is held and not expired, reject
        if self.holder and now < self.expiry:
            return None
        # Grant lock with monotonic fencing token
        self.holder = client_id
        self.token_counter += 1
        self.expiry = now + ttl_seconds
        return self.token_counter

    def release(self, client_id, token):
        """Release lock. Only succeeds if token matches."""
        if self.holder == client_id:
            self.holder = None
            return True
        return False


class FencedResource:
    """A resource that rejects stale fencing tokens."""

    def __init__(self):
        self.data = None
        self.last_token = 0

    def write(self, value, fencing_token):
        """Write only if token is fresh (≥ last seen)."""
        if fencing_token < self.last_token:
            raise PermissionError(
                f"Stale token {fencing_token} < {self.last_token}")
        self.last_token = fencing_token
        self.data = value
        return True

# Demo: fencing prevents stale writes
lock = DistributedLock()
db = FencedResource()

token_a = lock.acquire("process_A", ttl_seconds=5)  # token=1
# Process A pauses (GC)... TTL expires...
token_b = lock.acquire("process_B", ttl_seconds=5)  # token=2
db.write("B's data", token_b)  # succeeds, last_token=2
# Process A resumes, tries to write with stale token:
try:
    db.write("A's data", token_a)  # REJECTED: 1 < 2
except PermissionError as e:
    print(e)  # "Stale token 1 < 2"

python
# Drill 4: Total order broadcast (simplified)

from collections import deque
from threading import Lock

class TotalOrderBroadcast:
    """Simplified TOB using a single sequencer (not fault-tolerant).
    In production, the sequencer would be a Raft/Paxos group."""

    def __init__(self, num_nodes):
        self.sequence_num = 0
        self.lock = Lock()
        self.queues = [deque() for _ in range(num_nodes)]

    def broadcast(self, message):
        """Assign a sequence number and deliver to all nodes."""
        with self.lock:
            self.sequence_num += 1
            ordered_msg = (self.sequence_num, message)
            for q in self.queues:
                q.append(ordered_msg)
        return self.sequence_num

    def deliver(self, node_id):
        """Node consumes next message in total order."""
        if self.queues[node_id]:
            return self.queues[node_id].popleft()
        return None

# All nodes see messages in the same order
tob = TotalOrderBroadcast(3)
tob.broadcast("set x=1")   # seq 1
tob.broadcast("set y=2")   # seq 2
tob.broadcast("set x=3")   # seq 3

# Every node delivers in order: (1,"set x=1"), (2,"set y=2"), (3,"set x=3")
# This is the foundation of replicated state machines.

Debug Scenarios

Common Interview Traps

Interviewers love these subtle distinctions. Get them wrong and you look like you memorized definitions without understanding.

Whiteboard Patterns

In a system design interview, these are the "consensus patterns" that recur:

The One-Slide Summary

If you had to put the entire chapter on one slide for a senior engineer audience, it would be this:

Chapter 10: Connections

Consistency and consensus sit at the top of the distributed systems pyramid. Everything we have studied builds toward this chapter.

How This Connects

The Hierarchy of Agreement

The Evolution: What's Next After Raft?

Raft was published in 2014 and is now the dominant consensus algorithm in industry. But research continues:

The trend is clear: Raft as the baseline, with optimizations for specific workloads. Multi-Raft is particularly important — it's how CockroachDB and TiKV scale to millions of ranges across hundreds of nodes, each range with its own independent 3- or 5-node Raft group.

Recommended Reading

The Big Picture: A Decision Tree

When designing a distributed system, work through this decision tree for each piece of state:

"We can solve any problem by introducing an extra level of indirection... except for the problem of too many levels of indirection." — David Wheeler. In distributed systems: we can solve any consistency problem by adding more consensus... except for the problem of consensus being too slow. The engineer's job is finding the minimum coordination that achieves the required guarantees.