Failure Modes & Isolation

Chapter 1: Hardware Faults

Hard drives spin at 7,200 RPM, 24 hours a day, for years on end. Eventually, the bearings wear out. The magnetic surface degrades. A cosmic ray flips a bit in RAM. A power supply capacitor dries out. Hardware is physical, and physical things break.

The good news about hardware faults is that they are usually independent. When one disk dies, that does not cause the disk in the next server to die. Each failure is a random, uncorrelated event. This means that simple redundancy — having a spare — is highly effective.

Mean Time Between Failures (MTBF)

Mean Time Between Failures is the average time a component runs before it fails. For a modern enterprise hard drive, MTBF is roughly 10-50 years (depending on the model and workload). But "10 years per drive" does not mean your first failure is in 10 years if you have many drives.

Google, with millions of servers, sees thousands of hardware failures per day. Amazon, Microsoft, and Meta see similar numbers. At this scale, hardware failure is not an exception — it is the steady state.

Types of Hardware Failures

The simulation below shows a datacenter with 100 servers. Each server has a random MTBF. Over time, servers fail and are replaced. Watch how the failure rate becomes predictable at scale.

The key insight: hardware faults are mostly independent. One server's disk dying does not cause another server's disk to die. This means that if you have data on two different servers, losing one disk does not lose the data. Simple replication defeats simple hardware faults.

But "mostly independent" has exceptions. A batch of drives from the same manufacturer, shipped at the same time, installed at the same time, may all reach end-of-life at roughly the same time. A power outage takes out every server on the same power circuit. These are correlated hardware faults, and we will address them with fault domains in Chapter 5.

A Worked Example: Disk Failure Probability at Scale

Let's do the math for a real datacenter. Suppose you operate 10,000 servers, each with 4 hard drives. That is 40,000 drives. Each drive has an annual failure rate (AFR) of 2% (a realistic figure for enterprise drives).

This is why large-scale systems are designed with the assumption that hardware will fail continuously. The system must handle failures automatically, without human intervention, as part of its normal operation.

ECC Memory: Silent Corruption

A particularly insidious hardware fault is a silent bit flip in RAM. Cosmic rays and natural radioactive decay can flip individual bits in memory. Without ECC (Error-Correcting Code) memory, these flips are silent — the data is corrupted but no error is reported. The program continues with wrong data.

A study at Google found approximately 1 single-bit error per gigabyte of RAM per year. With 64 GB per server and 10,000 servers, that is 640,000 bit flips per year across the fleet. ECC memory detects and corrects single-bit errors automatically. This is why every production server uses ECC memory.

Chapter 2: Software & Config Errors

Hardware faults are well-understood and well-mitigated. The bigger threat to modern systems is software faults and configuration errors. Unlike hardware, these are almost always perfectly correlated: the same bug exists on every replica, the same bad config is pushed to every server.

Software Bugs

A software bug is dormant until triggered. It might lurk for months, waiting for a specific combination of inputs, load level, or timing. When the trigger arrives, every replica running that software hits the bug simultaneously.

This pattern — a "poison pill" request that crashes every server it touches — is one of the most common causes of total outages in production systems.

Configuration Errors

Configuration changes are the number one cause of outages at large internet companies. Not hardware failures. Not software bugs. Config changes.

Why? Because configuration changes are typically pushed globally and immediately. A developer changes a DNS entry, a firewall rule, a feature flag, or a database connection string, and that change propagates to every server in seconds. If the change is wrong, every server is broken in seconds.

Resource Leaks

A resource leak is a slow-motion failure. Memory is allocated but never freed. File handles are opened but never closed. Database connections are checked out but never returned. The process works fine for hours or days, slowly consuming resources, until it hits a limit and crashes.

Resource leaks are insidious because they are time-correlated. If all your replicas started at the same time (which they usually did, because you deployed them at the same time), they all leak at the same rate, and they all crash at roughly the same time.

The simulation below demonstrates how resource leaks and config pushes create correlated failures. Each bar represents a server's health over time.

The Load Pressure Problem

There is one more failure type that deserves special attention: load pressure. A traffic spike — from a viral post, a marketing campaign, a DDoS attack, or simply normal daily peak — can overwhelm your system. Unlike hardware or software faults, load pressure affects every server simultaneously because they all see the increased traffic.

Load pressure is correlated by definition: if one server sees the spike, they all do (because the load balancer distributes it). And load pressure interacts badly with other failure types. A server that is running at 95% capacity has zero headroom for a GC pause, a slow query, or a background job. Load pressure turns normal events into failures.

The Interaction Between Failure Types

The most dangerous scenarios are not single failure types — they are combinations. A traffic spike (load pressure) hits while a server is restarting (reducing capacity by 1/N), during a GC pause on another server (reducing capacity further), which triggers retries (amplifying load). Each individual event is manageable. Together, they cascade.

This is why defense-in-depth matters. No single isolation strategy handles all failure combinations. You need multiple layers.

Chapter 3: Cascading Failures

One server fails. Its traffic shifts to the remaining servers. Those servers were already at 70% capacity. Now they are at 85%. Response times increase. Clients start timing out and retrying. Each retry doubles the load. The next server collapses. Its traffic shifts to the remaining servers. They are now at 100%. The entire cluster goes down in minutes.

This is a cascading failure: a chain reaction where one failure causes additional failures, which cause more failures, until the entire system is down. The initial trigger might be trivial — a single server restart, a small traffic spike, a GC pause. But the cascade amplifies it into a total outage.

The Anatomy of a Cascade

The Feedback Loops

Cascading failures are driven by three positive feedback loops (positive here means self-reinforcing, not "good"):

Let's do the math for a concrete example. You have 5 servers, each handling 1000 requests per second (RPS) at 80% capacity (max capacity = 1250 RPS each).

The simulation below lets you watch a cascade in real time. Five servers process incoming traffic. Click "Kill Server" to remove one. Watch the load redistribute and the cascade begin.

Breaking the Cascade

Every defense against cascading failures targets one of the feedback loops:

We will cover these patterns in detail in the Resiliency Patterns lesson. For now, the key insight is: cascading failures are caused by positive feedback loops, and the defense is to break those loops.

Chapter 4: Redundancy

The simplest defense against failure is having a spare. If one disk can fail, have two. If one server can crash, have three. If one datacenter can lose power, have two datacenters. This is redundancy: duplicating components so that when one fails, another takes over.

But not all redundancy is created equal. The two major patterns are active-active and active-passive, and choosing the wrong one can be worse than having no redundancy at all.

Active-Passive

In active-passive (also called primary-standby), one component handles all traffic. The other sits idle, doing nothing, waiting for the primary to fail. When the primary fails, the passive component takes over through a process called failover.

The dangerous thing about active-passive is the cold standby problem. If the standby has never handled real traffic, how do you know it works? Its cache is cold. Its connection pools are empty. Its code paths for handling real load have never been exercised. The first time it handles traffic might be during an incident — the worst possible time to discover it doesn't work.

Active-Active

In active-active, all components handle traffic simultaneously. There is no idle standby. When one fails, the remaining components absorb its traffic.

The Cold Standby Trap: A Worked Example

Let's trace through a typical active-passive failover that goes wrong. This scenario plays out in production systems all the time.

The fix: the standby should handle read traffic to keep its cache warm. This is "warm standby" vs. "cold standby." A warm standby has a hot buffer pool cache and warm connection pools. Its failover latency is seconds, not minutes.

The N+1 and N+2 Models

How many spare replicas do you need? This depends on what you are protecting against:

The simulation below compares active-passive and active-active redundancy. Watch how failover works in each mode and how quickly the system recovers.

Chapter 5: Fault Domains

You have three replicas of your database. Smart. But you put all three on the same physical rack in the same datacenter. The rack's top-of-rack switch fails. All three replicas are unreachable. Your redundancy is useless because all your replicas share the same fault domain.

A fault domain is a group of components that share a single point of failure. Everything on the same server shares the server's power supply. Everything on the same rack shares the rack's network switch and power distribution unit. Everything in the same datacenter shares the datacenter's power feed and internet uplink.

The Hierarchy of Fault Domains

Fault domains form a hierarchy, from small (a single process) to large (an entire region):

The rule for fault domains is simple: replicas should be in different fault domains. If you have 3 replicas, put them on 3 different racks (to survive rack failure), in 3 different AZs (to survive AZ failure), or in 3 different regions (to survive regional failure).

The Trade-off

Spreading replicas across fault domains increases reliability but also increases latency. The further apart your replicas, the longer it takes to synchronize them.

The simulation below shows how spreading replicas across fault domains protects against different failure scopes. Click on a fault domain level to simulate a failure at that scope.

Chapter 6: Shuffle Sharding

You run a multi-tenant service. Customer A and Customer B both send traffic to your system. You have 8 backend servers. In a simple setup, every customer's traffic can reach every server. If Customer A sends a traffic spike that overloads server 3, Customer B's traffic on server 3 also suffers.

The blast radius of Customer A's spike includes Customer B and every other customer on server 3. How do we shrink that blast radius?

Simple Sharding: The Starting Point

One approach: assign each customer to a fixed shard of servers. Customer A gets servers 1 and 2. Customer B gets servers 3 and 4. Customer C gets servers 5 and 6. Now if Customer A overloads their shard, only servers 1 and 2 are affected. Customer B on servers 3 and 4 is untouched.

This is better, but the blast radius of a shard failure is still large: every customer on that shard is affected. If you have 100 customers on shard 1, a problem on shard 1 hits all 100 customers.

Shuffle Sharding: Random Assignment

Shuffle sharding assigns each customer to a random subset of servers. Instead of fixed shards, each customer gets a randomly chosen pair (or trio) of servers. The magic is in the math of combinatorics.

The key insight: with fixed sharding, a shard failure has a 100% blast radius for all customers on that shard. With shuffle sharding, the blast radius is probabilistically limited. Most customers share zero or one servers with the affected customer.

Scaling the Numbers

As you increase the number of servers and decrease the shard size (relative to total), shuffle sharding becomes exponentially more powerful:

With 100 servers and shards of size 5, there are over 75 million possible combinations. The probability that two random customers share all 5 servers is vanishingly small. Even a catastrophic "noisy neighbor" affecting all 5 of their servers impacts almost no one else.

Shuffle Sharding: A Concrete Worked Example

Let's trace through a specific scenario. You have 8 workers (W1 through W8) and 6 customers (A through F). Each customer is assigned to 2 random workers:

The beauty of shuffle sharding is probabilistic: as you add more workers, the probability that any two customers share their entire set of workers drops exponentially. With enough workers, each customer effectively has a private shard — without the operational cost of actually running private shards.

The simulation below lets you see shuffle sharding in action. Each customer is assigned to a random pair of servers. Click on a customer to "overload" their servers and see how the blast radius is contained.

Chapter 7: Cellular Architecture

Shuffle sharding limits the blast radius within a single deployment. But what if the failure is not a noisy neighbor but a systemic issue — a bad deploy, a control plane bug, a shared dependency that goes down? In these cases, shuffle sharding does not help because the failure affects the entire fleet.

Cellular architecture (also called cell-based architecture) takes isolation to the extreme: instead of one large system, you build many small independent copies of the entire system, called cells. Each cell is a complete, self-contained instance of your service with its own compute, storage, configuration, and deployment pipeline.

What Makes a Cell

A cell is not just a shard. A shard shares infrastructure (deployment pipeline, config system, monitoring) with other shards. A cell is fully independent:

How Routing Works

A thin cell router sits in front of all cells. It maps each request to a specific cell based on a customer ID, tenant ID, or similar stable identifier. The router itself is the only shared component, so it must be extremely simple and reliable — it does nothing but route.

Real-World Cellular Architectures

The simulation below shows a cellular architecture with 4 cells. Each cell serves a subset of customers. Deploy a bad update to one cell and see how the damage is contained.

Designing a Cell

The hardest part of cellular architecture is deciding what constitutes a cell. The boundary must be complete: everything inside the cell is self-sufficient, with no shared mutable state across cells.

Cell Sizing

How large should a cell be? There is a tension between isolation (more, smaller cells) and efficiency (fewer, larger cells). A common heuristic:

Most organizations start with 3-5 cells and increase as they grow. The goal is that a single cell failure affects no more than 10% of traffic.

Cell Router Design

The cell router is the single point of failure in a cellular architecture. If it goes down, all cells are unreachable. Therefore, the router must be:

1. Stateless. No mutable state. Routing table is read from a static config that is cached locally. Even if the config system is down, the router uses its last known good config.

2. Deterministic. Given the same customer ID, always route to the same cell. This prevents split-brain where a customer's requests go to different cells and see inconsistent data.

3. Extremely simple. The router does one thing: hash the customer ID, look up the cell, forward the request. No business logic, no data transformation, no authentication. Every line of code in the router is a potential failure point.

python
# Minimal cell router
import hashlib

CELL_ASSIGNMENTS = {
    0: "cell-1.internal",
    1: "cell-2.internal",
    2: "cell-3.internal",
    3: "cell-4.internal",
}
NUM_CELLS = len(CELL_ASSIGNMENTS)

def route(customer_id):
    # Deterministic hash: same customer always goes to same cell
    h = int(hashlib.md5(customer_id.encode()).hexdigest(), 16)
    cell_idx = h % NUM_CELLS
    return CELL_ASSIGNMENTS[cell_idx]

Chapter 8: Blast Radius Simulator

This is the payoff chapter. Everything we have learned — fault domains, shuffle sharding, cellular architecture — exists to shrink the blast radius of a failure. Blast radius is the fraction of your system or your customers that are affected by a single failure event.

The simulator below lets you configure a distributed system with different isolation strategies, then inject failures and watch the blast radius in real time.

Expected Blast Radius by Strategy

Notice the pattern: no single strategy handles all failure types. Fault domains protect against infrastructure failures but not software failures. Shuffle sharding protects against noisy neighbors but not config pushes. Cells protect against everything — at the cost of operational complexity.

The best systems layer these strategies: cells for deployment isolation, shuffle sharding within each cell for tenant isolation, and fault domain awareness for infrastructure resilience.

Real-World Blast Radius Incidents

Let's look at real incidents through the lens of blast radius and isolation:

Notice the pattern: every major outage was a correlated failure caused by a change (config, deploy, maintenance) that propagated globally without isolation. Hardware failures almost never cause these outages — the isolation strategies (RAID, replication) handle those. It is the human-initiated changes that bypass isolation and cause global damage.

Designing for Blast Radius

When architecting a new system, ask yourself these questions for every component:

Chapter 9: Connections

Failure modes and isolation are the foundation. You now understand what fails and how to contain the damage. But containing damage is only half the story — you also need to recover gracefully when failures happen despite your isolation.

This Lesson vs. Related Topics

Key Takeaways

1. Hardware faults are independent. Redundancy works because one disk dying does not cause another to die. Simple replication defeats simple hardware faults.

2. Software and config faults are correlated. The same bug exists on every replica. The same bad config hits every server. Redundancy alone does not help — you need isolation.

3. Cascading failures are driven by feedback loops. Retry storms, load redistribution, and resource exhaustion amplify a small failure into a total outage. Break the loops.

4. Fault domains bound the blast radius of infrastructure failures. Put replicas in different racks, AZs, or regions.

5. Shuffle sharding bounds the blast radius of noisy neighbors. Random assignment means one customer's problem rarely affects another.

6. Cellular architecture bounds the blast radius of everything. Independent cells mean a bad deploy affects only one cell. The price is operational complexity.