Scaling Reads

Chapter 1: Read Replicas

The cheapest fix for “one box cannot answer 5,000 questions a second” is to build more boxes that know the same answers. That is the entire idea behind a read replica: a second Postgres server that continuously copies everything the first one writes, and that your application is allowed to read from. One leader (sometimes called a primary) accepts writes; one or more followers accept only reads and stay a fraction of a second behind. This chapter is about exactly how far behind, and what that gap costs you.

How a follower stays in sync

Postgres does not resend your SQL to the follower. It streams the write-ahead log (WAL) — the same append-only journal of raw byte-level changes the leader uses to recover from a crash. The follower receives that stream and replays it against its own copy of the data. Three steps happen between a write committing on the leader and that same write becoming visible on a follower:

Add the two steps that stand between the write and its visibility on a follower — ship and replay — and you get the number every replicated system lives with:

This is not a bug. It is the physical cost of copying bytes over a wire and re-executing them somewhere else. A follower that is 50ms behind the leader is a completely healthy, correctly functioning replica — the number to worry about is not whether lag exists, but whether your application knows it exists.

Asynchronous versus synchronous: a choice about who waits

The flow above describes the common case, asynchronous replication: the leader acknowledges a write to the client the moment it is durable on the leader itself, without waiting for any follower to catch up. That is what makes the leader’s write latency independent of follower count and network distance — and it is exactly what creates the staleness window this chapter is built around. The alternative, synchronous replication, has the leader wait for at least one follower to confirm it has applied the write before acknowledging the client at all:

Synchronous replication trades write latency for a staleness guarantee, and it is a legitimate choice for data where a lost or stale write is unacceptable — financial ledgers, for instance. A feed app’s read path, built around the assumption that most reads can tolerate some staleness in exchange for horizontal read scaling, is a textbook case for asynchronous replication, which is why the rest of this chapter assumes it and spends its effort on managing the staleness window rather than eliminating it.

Deriving the staleness window from first principles

Suppose the leader is accepting 2,000 writes per second — likes, comments, new posts, follows, all together. At any instant, how many of those writes have committed on the leader but have not yet been replayed on a given follower? That count is exactly the write rate times the apply delay:

One hundred writes, at any given moment, exist on the leader and do not yet exist on the follower. That is the staleness window in concrete terms: a follower that a reader hits right now is missing, on average, the most recent hundred writes system-wide, and for the specific row a given user just touched, is missing it for up to fifty milliseconds.

Replication has its own read wall

Here is a fact that rarely makes it into the introductory explanation of replicas, and it is a direct callback to Chapter 0: a follower’s replay process is itself a queue with a ceiling. WAL replay on a follower has historically run largely on a single thread (parallel apply exists in newer Postgres versions but has its own limits), and that thread can only replay so many bytes of change per second — call it a replay capacity of roughly 3,000 writes/second for a representative workload on a mid-sized follower.

Under normal traffic of 2,000 writes/second, the follower keeps up comfortably: replay capacity exceeds arrival rate, so the 50ms figure above is a genuine steady-state lag, not a growing one. Now run a marketing campaign that pushes writes to 8,000/second for ten minutes:

That is Chapter 0’s overload condition, transplanted onto the replication stream instead of the query stream. Lag does not settle at some larger constant — it grows for as long as the burst continues, and the “50ms average” figure quietly becomes meaningless. This is exactly why the session-pinning window in the next section is padded so far past the average: it is insurance against precisely this scenario, where a burst turns a comfortable lag into a widening one.

The failure mode this causes: read-your-writes

A user updates their profile photo. The write goes to the leader, commits, returns success. The app immediately re-renders the profile page — but that render fires a read, and your load balancer sends it to a follower that has not replayed the photo change yet. The user sees their old photo, on a page that just told them the save worked. Nothing is broken in the database. Everything is working exactly as specified. The specification was just missing a guarantee the user actually needed: read-your-writes — a user should always be able to see the effects of their own write, immediately, even while everyone else might still be looking at slightly stale data.

Not every stale read is this urgent, and it is worth distinguishing the case above from a second, much more common one: a user likes a post, the like count briefly still reads the old number, then a moment later updates. That is annoying but survivable — nobody files a support ticket over a like count that took fifty milliseconds to catch up, because the count is understood to be a live, slightly-lagging aggregate rather than a fact the user just personally asserted. The photo-upload case is different specifically because the user just performed an action and the interface told them it succeeded; the like-count case is a passive display the user is merely observing. This distinction — did the user themselves just cause this change — is exactly what determines whether a given read needs read-your-writes protection at all, and skipping that judgment call is how systems end up over-engineering staleness protection for data nobody was ever going to notice was stale.

Fix one: session pinning

The simple fix: after a user writes, route that user’s reads to the leader — or to a follower you have separately confirmed is caught up — for a short window afterward. Something like:

python
def route_read(user_id, redis):
    pin_key = f"pin:{user_id}"
    if redis.exists(pin_key):
        return LEADER            # this user wrote recently, do not risk it
    return pick_follower()

def after_write(user_id, redis):
    redis.setex(f"pin:{user_id}", 2, 1)   # pin for 2 seconds — margin over the 50ms average

Two seconds looks like wild overkill against a 50ms average lag, and it is deliberate: the average is not the tail. Under a burst of writes — a viral post, a marketing campaign — replication lag can spike to several hundred milliseconds while followers catch up. Two seconds of margin costs almost nothing (that user reads from the leader for a couple of extra requests) and protects against the case that actually causes support tickets. The cost of pinning is all extra load lands back on the leader, so a system with heavy write-then-read traffic from the same users needs the second fix as well.

Fix two: LSN tokens

The more precise fix asks the follower directly: “have you replayed my write yet?” Postgres exposes exactly this via its Log Sequence Number (LSN) — a monotonically increasing byte offset into the WAL that identifies precisely how far a follower has replayed. When a write commits, the leader hands back the LSN of that commit. The client stores it and attaches it to its next read:

python
lsn = leader.execute("UPDATE profiles SET photo=%s WHERE id=%s RETURNING pg_current_wal_lsn()", …)
session["min_lsn"] = lsn

# on the next read, only route to a follower that has caught up
def route_read(min_lsn):
    for follower in followers_by_freshness():
        if follower.applied_lsn >= min_lsn:
            return follower
    return LEADER   # nobody caught up yet — fall back rather than serve stale data

This is strictly more precise than pinning: it does not guess a safe window, it checks the actual replay position. The trade-off is a small extra query (or a cached, periodically refreshed table of each follower’s applied LSN) on every read, and a fallback path to the leader that still needs to exist for the moment right after a write when no follower has caught up. Production systems often run both: LSN tokens for correctness, session pinning as the coarse, cheap-to-implement fallback everywhere the LSN check is not worth wiring up.

Naming what this actually is: a consistency model

It is worth attaching the formal vocabulary here, because it transfers directly to other systems. Reading only from the leader gives strong consistency: every read sees every prior write, always, at the cost of the leader handling every single read. Reading from an unprotected follower gives eventual consistency: a write becomes visible everywhere eventually, with no promised bound on how long that takes under load, as Chapter 1’s burst example showed. Session pinning and LSN tokens both aim for something in between, sometimes called read-your-writes consistency or, more generally, bounded staleness: not as strict as routing every read to the leader, but strict enough that a specific, named guarantee (yourself, seeing your own write) always holds.

Almost nothing at real scale runs purely strong or purely eventual for its whole read path — production systems pick a consistency model per use case, not per database. A password change might warrant a strong read straight after; a feed refresh a minute later is happy with eventual. Naming which guarantee a given read actually needs is a prerequisite for choosing the cheapest mechanism that provides it.

This per-use-case decision is worth writing down explicitly for a system, not just holding as an intuition, precisely because it is easy to default to strong consistency everywhere out of caution and quietly rebuild Chapter 0’s wall by routing every read back to the leader. The discipline this chapter argues for is: name the guarantee each read path actually needs, in writing, then pick the cheapest mechanism — leader read, eventual follower read, or a bounded-staleness read — that honestly satisfies it.

How many replicas do you actually need?

Replicas are the direct fix for Chapter 0’s wall: each one is another 4,000-QPS box, and the load balancer spreads reads across all of them. Size the fleet from today’s peak plus a margin for losing one:

Two replicas give 8,000 QPS of capacity against 5,000 QPS of demand — comfortable, but with a problem: if either replica goes down, the survivor alone (4,000 QPS) is back under the 5,000 QPS load, right back at the wall. Add one more for headroom against a single failure:

This “N+1” (or, more conservatively, “N+2”) pattern — provision for peak load with at least one full unit of failure tolerance built in — is the sizing rule Chapter 8’s full assembly reuses once caching has cut the load the database tier actually has to survive down to a much smaller number.

Watching lag instead of guessing at it

Postgres exposes replication lag directly, and any serious deployment graphs it continuously rather than relying on the fixed constants used for the arithmetic above:

sql — run on the leader
SELECT client_addr, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
       extract(epoch from (now() - reply_time)) AS lag_seconds
FROM pg_stat_replication;

example output during the 8,000 write/sec burst
 client_addr  | state     | lag_bytes | lag_seconds
--------------+-----------+-----------+-------------
 10.0.4.12    | streaming |   4213880 |        1.84

lag_bytes answers “how far behind, in raw WAL”; lag_seconds answers the question a reader actually cares about. A dashboard alert on lag_seconds crossing, say, 500ms catches exactly the burst-overload scenario above well before session pinning or LSN fallback are asked to paper over several seconds of staleness.

Chapter 2: Connection Pooling

You added replicas. Now there is a second, more mundane problem underneath all of them: your application runs on a fleet of app servers, and every one of those servers wants to talk to the database. At any real scale, the raw number of clients trying to hold a connection open is far larger than the number of queries actually running at once — and Postgres was not built to hold that many connections open cheaply.

Note the word “mundane.” This chapter has no dramatic incident like Chapter 0’s 2:14 p.m. wall — connection exhaustion tends to arrive as a slow, quiet degradation rather than a sudden cliff, which is exactly why it is easy to under-provision for until the arithmetic below makes the gap concrete.

Why 10,000 clients does not mean 10,000 active queries

Say your app fleet maintains 10,000 concurrent client connections to the database — a believable number once you count every app server process across every host. Each client is not running a query continuously. It runs one, gets an answer, does some application work (rendering, calling another service, waiting on the user), and comes back roughly every 500 milliseconds for another. Each query itself still takes the familiar 2 milliseconds.

Little’s Law from Chapter 0 answers the real question directly: how many of those 10,000 clients are actually inside a query at any given instant? Each client’s personal arrival rate is one query every 500ms, so:

Forty. Out of ten thousand connections, at any instant, only about forty are doing real work. The other 9,960 are sitting open, idle, waiting for the app code to need them again — and every one of them, held open the naive way, costs Postgres a full backend process.

Why Postgres uses a process per connection at all

It is worth understanding why this cost exists rather than treating it as an arbitrary design flaw. Postgres, unlike some databases, forks a fresh OS process for every new connection rather than handling connections with lightweight threads inside one shared process. That choice buys real robustness — a crash in one connection’s query processing cannot corrupt another connection’s memory, because they are genuinely separate address spaces enforced by the operating system — at the direct cost of every connection being comparatively heavy to create and to hold open. A connection pooler exists precisely to buy back the efficiency of a thread-per-request model without touching Postgres’s process-per-connection internals: the pooler itself can be lightweight (an event loop, not a process per client), and it presents a small, stable number of real, heavy Postgres connections behind that lightweight front door.

Why an idle connection is not free

Postgres spawns one OS process per connection. Each process carries its own memory overhead — roughly 10 MB once you count its share of shared buffers, working memory allocations, and process bookkeeping. Connect all 10,000 clients directly and do the arithmetic:

Most database boxes do not have 100 GB to spend on idle sockets before a single query runs. And even where the RAM exists, Postgres’s own scheduler starts to choke well before that: process creation, context switching between thousands of mostly-idle backends, and per-connection catalog caches all add real overhead that has nothing to do with the 40 queries actually running.

Why not just cap max_connections and reject the rest?

Postgres does let you set a hard ceiling — max_connections — and refuse anything past it. That stops the memory bleed, but trades it for a worse failure: once the fleet legitimately needs more than that ceiling’s worth of connections (a deploy that briefly doubles app server count, for instance), new connection attempts are flatly rejected with a “too many clients already” error, which every affected request sees as a hard failure rather than the graceful queueing a pool provides. A raw connection limit protects the database at the cost of the application’s error rate; a pool protects the database while turning the same overload into a brief, bounded wait instead of an outright rejection.

The two are not actually competing choices — a well-run deployment sets max_connections generously above the pool size as a hard backstop (so a misconfigured pooler can never single-handedly exhaust the database), while relying on the pool itself, sized to real concurrency, to be the layer that normally does the work of keeping connection count sane.

One more number worth carrying forward: whatever max_connections is set to directly caps how many replicas and pooler instances can simultaneously hold real backend connections to a single Postgres box, which is a hard resource ceiling this lesson’s later chapters take as a given whenever they reference “a database box’s capacity.”

The fix: a pool sized to the work, not the clients

A connection pooler — PgBouncer is the standard one in front of Postgres — sits between the 10,000 clients and the database and holds open a much smaller, fixed number of real server connections. Clients connect to the pooler, which is cheap (a pooler connection is a lightweight event-loop socket, not a Postgres backend process); the pooler multiplexes those onto a small pool of real backends, handing a client a server connection only for the duration of one query or one transaction, then returning it to the pool.

Size the pool from the actual concurrency, with headroom for bursts — a factor of roughly 2.5× over the measured average is standard practice, to absorb the moments when concurrency spikes above its typical value without turning every spike into a queue:

Ten thousand clients, one hundred real backend connections. Recompute the memory bill:

A hundred-times reduction in connection overhead, for the same 40 queries actually running at any instant, because the pool is sized to the work rather than to the client count.

Sizing the pool for a burst, not just the steady state

The 40-query steady-state concurrency assumed think time stays a constant 500ms. Real traffic is lumpier: a marketing push, a viral post, or simply the ordinary daytime peak from Chapter 0 can compress that think time as more requests arrive back to back. Model a burst as think time dropping to 150ms while client count and query time stay fixed:

That is already above the 100-connection pool sized for the steady state. This is exactly why the 2.5× headroom factor was chosen rather than a tighter 1.1× or 1.2× — it is not padding for its own sake, it is capacity reserved for the burst case, computed the same way the steady-state number was. A pool sized with zero headroom looks perfectly adequate in every steady-state graph and then queues, exactly as Chapter 0 predicts, the first time real traffic gets lumpy.

What happens if the pool is undersized

The pool itself is now a queue, and it obeys the exact same M/M/1 arithmetic from Chapter 0 — only the “server” is a pool slot instead of a CPU core. Undersize the pool at, say, 30 connections against the 40-concurrency workload just derived:

This is worth pausing on, because it means Chapter 0’s M/M/1 formula generalizes far beyond the specific case of CPU cores serving queries. Any resource that is shared across more requesters than it can simultaneously serve — CPU cores, pool connections, worker threads, file descriptors — obeys the same latency-versus-utilization curve, and the same knee near ρ = 1. Once this pattern is recognizable, spotting it in a new part of a system is mostly a matter of asking “what is the shared, limited resource here, and what is ρ against it, right now?”

Requests do not fail here — they queue for a free pool slot, adding wait time on top of the 2ms the query itself takes, and that wait grows without bound for as long as concurrency exceeds the pool size. This is the same failure shape as the read wall, one layer higher in the stack, and it is why the 2.5× headroom factor exists: not as an arbitrary safety margin, but specifically to keep ρ_pool comfortably under 1 through ordinary variance in traffic.

Where to put the pool: L4 versus L7

There is a second, orthogonal decision: how does a query get routed to the right Postgres box in the first place — leader for writes, one of several followers for reads? That routing can happen at two different layers of the network stack.

Most systems end up doing both, at different layers: an L4 balancer spreads TCP connections evenly across a pool of pooler instances for availability, and an L7-aware layer above that (often just application code that knows “this call site is a read, send it to the follower pool”) does the actual read/write split. Pushing the split into the database wire protocol itself buys you correctness even when application code forgets, at the cost of that extra per-statement parse.

Where the pooler sits, physically

There are two common deployment shapes, and the choice affects how much the pool actually helps:

The centralized shape is what makes the arithmetic in this chapter actually apply: a global pool of 100 connections only means something if it is genuinely shared across the whole fleet, not subdivided into 100 fragments of 1 connection each that individually reintroduce the very per-client waste pooling was meant to remove.

Concept → realization: what the pooler is actually doing

pgbouncer.ini (transaction pooling mode)
[databases]
feed_reads = host=replica-pool.internal port=5432 dbname=feed

[pgbouncer]
pool_mode = transaction     # hand back the server conn after each transaction, not each client disconnect
default_pool_size = 100     # the number we just derived
max_client_conn = 20000     # cheap event-loop sockets, not backend processes

pool_mode = transaction is the load-bearing setting. In the naive mode (session), a client keeps its server connection for as long as it stays connected — which defeats the entire point, since an idle client would still pin a real backend. Transaction mode returns the server connection to the pool the instant a transaction commits, which is exactly what makes 100 real connections able to serve 10,000 idle clients: each client only occupies a server connection for the 2ms it is actually running a query.

Transaction mode versus statement mode, and why the difference matters

Transaction mode is the workhorse choice for exactly this reason: it gets nearly all of statement mode’s multiplexing efficiency while still allowing ordinary multi-statement transactions, provided the application avoids session-level state that would silently break when a connection hops between transactions.

Concretely, this rules out a common but easy-to-miss pattern: application code that runs SET search_path = tenant_42 on one query and expects it to still be in effect on the next, unrelated query from the same client. Under transaction pooling, those two queries may well land on two entirely different backend connections, and the second one never saw the first statement at all. The fix is to make every query self-contained — pass the tenant explicitly as a parameter rather than relying on connection-scoped state — which is, not coincidentally, also the discipline that makes an application horizontally scalable in the first place.

Watching the pool instead of guessing at its size

PgBouncer reports its own saturation directly, which is the number to alert on rather than recomputing the theoretical concurrency by hand every time traffic shifts:

SHOW POOLS; — run against the PgBouncer admin console

 database   | cl_active | cl_waiting | sv_active | sv_idle | maxwait
-------------+-----------+------------+-----------+---------+---------
 feed_reads |      9,940|         62 |        38 |      62 |   0.014

cl_waiting is the number of client requests sitting in the pool’s own queue right now, and maxwait is the longest any of them has been waiting, in seconds — the direct, measured version of the ρ_pool > 1 condition derived above. A healthy pool runs with cl_waiting at or near zero; a sustained nonzero value is the pool telling you, in real time, that its size has fallen behind actual concurrency.

The retry storm: a pool exhaustion failure worth naming

There is a specific cascading failure that undersized pools are prone to, and it is worth working through once so it is recognizable. A slow query — say, a missing index causes one query type to take 200ms instead of 2ms — holds its pool slot ten times longer than normal. If enough of those slow queries arrive together, they can occupy the entire pool, leaving zero slots for the fast, normal 2ms queries that would otherwise sail through:

Push the outlier count slightly higher and ordinary, previously-fast requests start queueing for a slot behind the slow ones — and if those requests have client-side timeouts and retry logic, each timed-out request’s retry becomes a second request competing for the same scarce pool, doubling the effective load on an already-saturated pool. This is a retry storm: the system’s own recovery mechanism amplifies the very overload it was trying to escape. The fix is not a bigger pool (that just delays the same collapse to a higher outlier count) but capping how long any single query may hold a pool slot — a statement timeout — so one slow query class can never starve out the fast one.

Chapter 3: Edge Cache

Replicas and pooling both make the database cheaper to talk to. This chapter is about not talking to it at all — for the slice of traffic that is the same answer for everyone, served from a server that sits close to the person asking, rather than close to the database.

It is also the first layer in this lesson that operates entirely outside your own infrastructure — a CDN’s edge nodes are machines you do not run, in locations you do not choose, governed by rules (cache keys, TTLs, headers) rather than code you deploy. That shift, from “infrastructure you operate” to “a contract you configure,” is worth noticing, because it is the same shift Chapter 8 revisits when discussing what an origin shield can and cannot control.

The problem with hundreds of independent caches

A CDN with 200 PoPs, naively, means 200 independent caches — each one, on a cold start or after its own TTL expiry, individually misses and goes to origin. A single popular item, requested from every region at once right after a deploy, can trigger 200 near-simultaneous origin fetches for the exact same content, one per PoP: a geographically distributed version of Chapter 4’s thundering herd, arriving before that chapter has even introduced the concept. The standard fix is an origin shield: one additional caching layer, sitting logically between all 200 edge PoPs and the actual origin, so a miss at any edge PoP first checks the shield — which is far more likely to already have the answer, since it serves as the single consolidation point for all 200 PoPs’ misses combined. Only a genuine miss at the shield itself reaches the origin server. This turns “up to 200 origin requests for one popular item” into “at most 1,” the same reduction in spirit as the single-flight lock two chapters from now, applied one layer earlier in the request path.

It is worth being precise about what an origin shield does and does not do for the incident described. It caps origin fetches at roughly 1 instead of 200 — not because it coalesces concurrent requests the way a single-flight lock does (Chapter 4 covers that mechanism in full), but because it collapses 200 independent caches into a two-tier hierarchy where only the top tier ever needs to ask the origin at all.

What is actually cacheable

Not every read qualifies. A CDN (content delivery network) caches a response and serves it to many different requesters without checking back with your origin server, so it can only safely hold responses that are the same for everyone who asks — or at least the same for everyone who shares the dimensions you cache on. In a feed app that includes:

The gray zone: mostly-shared content with a personalized edge

Not everything sorts cleanly into the two tables above. A public profile page usually shows a “Follow” button whose label depends on whether you already follow that person — the page is 95% identical across all visitors and 5% personalized. Caching the whole response defeats the point (back to 12,000,000 copies); refusing to cache any of it throws away the 95% that genuinely is shared. The standard resolution is edge-side includes or a client-side patch: cache the shared shell at the edge, and let a small, separate, uncached call (or a tiny client-side script) fill in the personalized fragment after the cached shell loads. The expensive, shared 95% gets full edge-cache economics; the cheap, personalized 5% pays for exactly the freshness it needs and nothing more.

This split-and-recombine pattern is worth recognizing because it recurs throughout system design under different names — the same idea shows up as request-level composition in API gateways, as fragment caching in server-rendered web frameworks, and, in a more elaborate form, as the hybrid fan-out strategy Chapter 7 builds for feed queries. In every case the underlying move is identical: separate a response into its shared, cacheable majority and its small, personalized remainder, and let each half be served by the mechanism suited to it rather than forcing one mechanism to handle both.

The cache key: what makes two requests “the same”

A CDN decides whether it has already answered a request by hashing a cache key — a chosen subset of the request that must match exactly for the cached response to apply. Get the key wrong in either direction and you get a real bug:

too narrow — key = path only
GET /profile/alice.jpg              # English-language client and Japanese-language client
GET /profile/alice.jpg              # both get served the SAME cached response —
                                     # if the response varies by Accept-Language, this is wrong

too wide — key includes a header that varies per user
GET /trending  Cookie: session=abc123
GET /trending  Cookie: session=xyz789    # same trending list, different session cookie —
                                          # keying on Cookie makes every user a cache miss

right — path + query + the Vary headers that actually change the response
key = path + query_string + Vary(Accept-Language, device-type)

The rule: the cache key must include every dimension the response actually changes on, and nothing else. Too narrow leaks the wrong content to the wrong audience. Too wide turns every request into a miss, because a key that includes something unique per user (a session cookie, an auth token) can never repeat.

The failure modes of these two mistakes are asymmetric in a way worth internalizing. A too-narrow key is a correctness bug — it leaks the wrong content, silently, to real users, and is often only discovered when someone notices a page in the wrong language or a stale value that should have varied. A too-wide key is a performance bug — every request misses, hit rate quietly collapses toward zero, and Chapter 5’s arithmetic shows exactly how much origin load that silently adds back. Both are invisible in a quick manual test (one request, one response, looks correct) and only show up once real, varied traffic exercises every dimension the key should or should not be sensitive to — which is why cache-key correctness deserves its own deliberate test coverage rather than a spot check.

In practice, this is expressed as HTTP headers your origin sends alongside the response, which the CDN reads and obeys:

stale-while-revalidate is worth dwelling on because it looks almost too good: a request that arrives just after expiry still gets served instantly, from the (briefly) stale copy, while a background fetch quietly refreshes it for the next request. It converts what would otherwise be one slow request per expiry into zero slow requests, at the cost of one extra window of staleness — a trade nearly every read-heavy endpoint is happy to make.

TTL versus purge

Once a response is cached, two mechanisms govern when it stops being served: a TTL (time to live) that expires it automatically after a fixed duration, and an explicit purge call your deploy pipeline or content-management system fires the instant something actually changes. A trending-topics list might carry a 60-second TTL — short enough that staleness is invisible, long enough to absorb nearly all the read traffic. A profile photo might carry a 1-hour TTL and an explicit purge on upload, so a user who changes their photo sees it update immediately instead of waiting up to an hour. Chapter 6 goes deep on this trade-off; for now, the rule of thumb is: TTL alone for content where staleness is cheap, TTL plus purge for anything a user expects to see change the moment they act.

Chapter 6 revisits this exact choice with a full worked incident and a formal comparison table across three invalidation strategies, because the same TTL-versus-purge trade-off reappears, unchanged in its logic, at the application-cache layer in Chapter 4. Learning it once here, in the simpler context of a CDN, pays off there.

What a purge actually costs

A purge is not instantaneous, and treating it as if it were is a common source of “I purged it, why is it still stale” confusion. Purging a key means telling every edge PoP holding a copy to drop it, and that message has to propagate:

Half a second is fast compared to a TTL measured in minutes, and it is the number that belongs in Chapter 6’s comparison of invalidation strategies — a CDN purge behaves like the event-driven purge described there, not like the near-instant write-through of an application cache, because it has to fan out across a genuinely distributed set of machines rather than update one process’s memory.

Negative caching: the miss you did not expect to cache

One more case worth naming explicitly: what happens when the origin returns a 404, because a post was deleted between when a link was shared and when someone clicked it? Cache that response too — a short TTL, typically far shorter than a successful response’s — and you protect the origin from being hammered by repeated requests for something that will never exist. Skip it, and a single widely shared link to deleted content becomes an uncached, unbounded stream of misses hitting the origin on every click, exactly the load pattern the whole cache tier exists to prevent.

The general principle behind negative caching extends past 404s: cache any deterministic response, success or failure, that is safe to reuse for the duration of its TTL. What must never be cached is a response whose failure was transient rather than a true, stable fact about the resource — a 500 from an origin that was merely overloaded for a second should not be remembered as “this resource errors,” or every subsequent request would be told the same false thing until the TTL expired.

Quantifying negative caching, by hand

Put a number on what negative caching actually buys, using the same shape of arithmetic this lesson has used everywhere else. A post gets deleted, but a link to it was shared in a group chat with 200,000 members, and roughly 4% of them click it inside the first ten minutes after the deletion — before anyone has had a chance to notice the link is dead and stop sharing it:

Without negative caching, every one of those 8,000 requests reaches the origin, each one paying a full database round trip just to confirm, again, that the row is gone — a small but genuinely wasted slice of Chapter 0’s capacity, spent entirely on re-deriving a fact that has not changed since the first request asked it. With a 60-second negative-cache TTL on the 404 response, only the first request per PoP per minute actually needs to reach the origin:

A four-times reduction from this one setting alone, and in practice far better than that worst case: the 200-PoP figure assumes every single PoP independently gets a click inside every 60-second window, which is pessimistic for anything short of a genuinely global, viral share. The real value of negative caching is not the specific ratio in any one incident — it is that a resource which will never again return anything but a 404 stops costing the origin anything at all, permanently, the moment its TTL is set correctly.

Why the distance to the edge matters: the speed-of-light floor

Even a perfect cache hit still has to travel from the CDN’s point of presence (PoP) to the requester and back, and that trip has a floor no engineering effort can beat: light in fiber optic cable travels at roughly 200,000 km/s — about two-thirds the speed of light in a vacuum, slowed by the glass’s refractive index. A round trip covers the distance twice:

Run that for two cases. First, a request to a single origin data center in Virginia from a user in Tokyo — roughly 10,850 km:

Real measured round trips run higher than this floor — routing hops, TLS handshakes, and paths that are never perfectly straight lines add roughly another 25–60ms on top, so a realistic figure for that Tokyo request lands around 165–170 ms. Now the same request against an edge PoP 50 km away, inside the same metro area:

The origin request and the edge request are answering the exact same question with the exact same bytes. The only difference is where the answer lives, and that difference is worth well over 100ms — larger than the entire database query budget from Chapter 0.

Anycast: how a request finds its nearest PoP without being told where it is

None of this works if the requester has to somehow know which of a CDN’s hundreds of PoPs is nearest and address it directly. The mechanism that makes it automatic is anycast: every PoP in the CDN announces the exact same IP address to the internet’s routing tables, and standard internet routing (BGP) naturally forwards each request toward whichever announcement of that address is topologically closest to the requester — not necessarily the geographically closest, but usually close enough that the two track together. The requester’s device does nothing special; it just connects to “the CDN’s IP address” and the network itself routes the packets to a nearby PoP as a side effect of how routing already works. This is also why CDN outages are usually regional rather than global — if one PoP goes down, its anycast announcement withdraws and traffic reroutes to the next-nearest PoP automatically, with no DNS change or client-side logic involved.

Anycast is also why a CDN cannot simply be reasoned about as “the origin, but closer.” Different requesters, on different networks, at different times of day as routing tables shift, can genuinely be routed to different PoPs for what looks like an identical request — which is precisely why cache behavior needs to be correct at every PoP independently, rather than assumed consistent because “it worked when I tested it,” since that one test only exercised whichever PoP anycast happened to route to at that moment.

The practical upshot for the arithmetic earlier in this chapter: the distance slider is not something an operator sets by hand for each requester. It is a natural consequence of how many PoPs a CDN operates and how well distributed they are — more PoPs, spread more evenly across population centers, pushes the typical requester-to-PoP distance down and the realized latency benefit up, without any per-request routing decision beyond what BGP already does automatically.

Concept → realization: what the browser actually does with these headers

It is worth tracing where a Cache-Control header actually gets read, because there are two separate caches in play for any real request, and conflating them causes confusion. The browser’s own local cache reads the same headers and may skip the network entirely on a repeat visit — the fastest possible case, since there is no request at all, not even to a nearby edge PoP. The CDN’s shared cache, one network hop away, serves everyone else asking for the same resource. The s-maxage directive from earlier exists specifically to let these two disagree on purpose: a shared CDN cache can hold a response for an hour (s-maxage=3600) for the benefit of thousands of different users, while each individual browser is told to treat it as fresh for only a minute (max-age=60) so any one user does not go a full hour without seeing a genuinely new value if they happen to reload.

This split also explains a common piece of confusion: a purge fired against the CDN clears the shared cache immediately, but a browser that cached the old response locally, under its own max-age, has no way of knowing the purge happened at all and will keep serving its local copy until that window expires. For anything a purge needs to guarantee freshness on immediately, the max-age given to browsers has to be kept short regardless of how long the shared s-maxage is allowed to run — the purge only reaches as far as the CDN, never past it into millions of individual browsers.

Chapter 4: The Cache-Aside Race

The edge cache in Chapter 3 handles public, universal content. Most of what a feed app serves is neither — it is expensive to compute but shared across many requests within a short window: a trending list recomputed every minute, a popular post’s comment count, a feed template shared by everyone following the same account. That is the job of an application cache, usually Redis, sitting between your app servers and the database. And it introduces a genuinely new kind of bug that neither replicas nor edge caching produce: a race.

“Race” is worth defining precisely before diving in, since the word gets used loosely. A race condition is a bug whose existence depends on the relative timing of independent operations, not on any single operation being individually wrong — which is exactly what makes it invisible to a test that runs one request at a time, and exactly what makes it appear only once real concurrent load exposes the timing window it depends on.

Why this belongs in the application tier, not the edge

It is worth being explicit about why Chapter 3’s CDN cannot simply absorb this workload too. A trending list changes every minute and is computed from live, server-side aggregation over recent activity — it is not a static asset a CDN fetches once from an origin URL. More importantly, an application cache like Redis sits inside your own infrastructure, reachable by your application code with sub-millisecond latency and full programmatic control (locks, coalescing, custom eviction), none of which a CDN’s edge nodes expose. The edge is for content the internet at large can treat as a static resource; the application cache is for values your own backend computes and needs to reuse across requests within its own process boundary.

The two caches also fail differently under load, which reinforces the same point. A CDN miss falls through to origin as effectively one request, thanks to the origin shield from Chapter 3. The application cache’s cache-aside pattern, as the rest of this chapter shows in detail, can turn one expiry into hundreds of simultaneous database queries if left unprotected — a failure mode with no equivalent at the edge, and the specific reason this chapter needs its own, dedicated defense rather than inheriting Chapter 3’s.

Concretely, this difference traces back to control. A CDN operator built and tuned the coalescing behavior of every one of its edge nodes; you inherit that protection as a platform feature the moment you use the CDN, without writing a line of code for it. Your own application cache has no such built-in protection, because Redis itself is just a key-value store — it has no opinion about what your application should do when a key is missing, and the single-flight lock this chapter builds is exactly the piece of behavior a CDN vendor already wrote for you, that you now have to write for yourself.

The cache-aside pattern

The standard shape is called cache-aside (also “lazy loading”): the application checks the cache first; on a miss, it queries the database, then writes the result into the cache for next time.

python
def get_trending(redis, db):
    cached = redis.get("trending:global")
    if cached is not None:
        return cached                      # hit — no database touched
    result = db.compute_trending()          # miss — ~40 ms of real work
    redis.setex("trending:global", 60, result)
    return result

On its own this is correct and simple. The trouble starts at scale, at the exact moment the cached value expires.

Cache-aside is one of three shapes, and it is worth knowing why this lesson picked it

Cache-aside is the right default for a feed app’s read-heavy, miss-tolerant workloads precisely because it never risks losing data: the database is always the source of truth, and the cache is only ever a shortcut to it. The remainder of this chapter is about making that shortcut safe under real concurrency.

“Safe under real concurrency” deserves a moment of precision before diving into the race itself. The code above is correct for a single request in isolation — check, miss, query, populate, return. It says nothing about what happens when two, or two hundred, requests run that exact same sequence at the exact same moment, because a single-request trace can never reveal a race condition. The next section is where that gap becomes concrete and expensive.

Why the database, not Redis, is usually the actual bottleneck here

It is worth being precise about where the danger in this chapter actually lives, because it is easy to assume the cache itself is the fragile part. Redis, single-threaded per shard, can typically serve on the order of 100,000–200,000 simple key lookups a second from a single instance — nowhere close to being the bottleneck at the traffic levels this lesson discusses. The database sitting behind it, at 4,000 QPS per box from Chapter 0, is the fragile part by a factor of 25–50×. A thundering herd is dangerous specifically because it takes traffic that Redis was effortlessly absorbing and redirects a slice of it, all at once, at the one component in the whole stack with the least headroom.

Concept → realization: what the lock actually protects

It helps to be precise about the boundary of the lock’s job. It does not make the regeneration faster — the winning request still pays the full 40ms. It does not prevent the eventual 8,000 requests per second from all reading the trending list — every one of them still gets an answer. The single thing it changes is how many independent database queries that population of requests triggers for one expiry event: from a number that scales with request rate and regeneration time (up to 320, in this example) down to a constant, exactly 1, regardless of how many requests happen to be waiting.

Thundering herd, by the numbers

Suppose this trending key is read at 8,000 requests a second during peak hours — every feed load checks it. Suppose computing it fresh from the database takes 40 milliseconds. The key expires. The very next request misses, and starts a 40ms database query. But it is not the only request in flight — every request that arrives during those same 40 milliseconds, before the first miss has finished repopulating the cache, also finds nothing there:

Three hundred twenty requests, each independently deciding “cache is empty, I will compute it myself,” each firing the same expensive query at the same moment. This is a thundering herd (also called a cache stampede): a single expiry event turns into a 320× spike of identical, redundant load hitting the database all at once — precisely the kind of burst that pushes utilization past the wall from Chapter 0, even on a system that was comfortably under capacity a moment before.

The fix: a single-flight lock

The fix is to let exactly one of those 320 requests do the work, and have the other 319 wait for it to finish and then read the result it produced — a technique called request coalescing, implemented with a short-lived lock:

python
def get_trending(redis, db):
    cached = redis.get("trending:global")
    if cached is not None:
        return cached

    got_lock = redis.set("lock:trending", 1, nx=True, ex=2)  # only one caller wins this
    if got_lock:
        try:
            result = db.compute_trending()      # only ONE of the 320 gets here
            redis.setex("trending:global", 60, result)
            return result
        finally:
            redis.delete("lock:trending")
    else:
        for _ in range(20):                    # the other 319 poll briefly for the winner’s result
            sleep(0.005)
            cached = redis.get("trending:global")
            if cached: return cached
        return db.compute_trending()          # timed out waiting — fall back, rare in practice

Count the database hits again with the lock in place: exactly 1, down from 320.

The other 319 requests each spend a few milliseconds polling Redis — far cheaper than a database query — and get the same answer the winner computed, typically within 10–30ms.

An alternative to polling: pub/sub instead of a busy wait

The polling loop in the code above — sleep 5ms, check again, up to 20 times — is simple and works, but it is not the only shape this fix can take. A pub/sub variant has the losing requests subscribe to a channel keyed on the lock, and the winner publish the result to that channel the instant it finishes, waking every waiter immediately instead of on the next poll interval:

python — pub/sub instead of polling
else:
    with redis.pubsub() as sub:
        sub.subscribe("trending:ready")
        for msg in sub.listen(timeout=2):
            if msg: return msg["data"]        # woken the instant the winner publishes
    return db.compute_trending()               # fallback, same as before

Polling wastes a few milliseconds of average latency (waiting for the next poll tick) in exchange for far simpler code and no persistent subscription to manage; pub/sub trims that latency to whatever the winner’s publish takes, at the cost of holding an open subscription per waiting request. At the scale this lesson operates at — hundreds of waiters, not hundreds of thousands — both are reasonable, and the polling version is the one most teams reach for first because it needs no additional infrastructure beyond the lock itself.

Either implementation is invisible to the caller. A request that hits get_trending during a herd either reads a cached value directly, wins the lock and computes it, or waits briefly and gets the same computed value — the function’s return type and contract never change. That invisibility is exactly the property that makes this fix safe to retrofit onto an existing cache-aside call site without touching anything that calls it.

Sizing the lock timeout, by hand

The callout above is easy to nod along with in the abstract; it is worth actually computing both failure directions so the right number stops being a guess. Suppose the query this lock protects normally takes 40ms but, under database load, its p99 stretches to 180ms. Set the lock’s expiry too tight, at ex=0.1 (100ms):

Now two queries are racing to populate the same key, and if the database is already the reason the query is slow, adding a second concurrent copy of it makes the slowness worse, not better — a mild version of Chapter 0’s overload condition, self-inflicted by an overly cautious timeout. Set it far too loose instead, say ex=30 (30 seconds), and imagine the winning request’s process crashes mid-query, before it ever reaches the finally block that releases the lock:

That fallback line in the code above is exactly what keeps a crashed lock-holder from taking down the whole key for the full 30 seconds — every waiter still gets an answer, just at full cost, once its own poll budget runs out. The right timeout sits comfortably above the slowest realistic query (the 180ms p99, not the 40ms average) so a slow-but-alive winner is never second-guessed, while the poll-and-fallback path stays short enough that a genuinely dead winner does not strand every waiter for the lock’s entire duration. A reasonable choice here is an expiry of roughly 3–5× the p99 query time — about 1 second for this workload — with the polling fallback, not the lock timeout itself, doing the actual work of bounding how long a genuine crash can stall waiters.

A subtler fix: never let the whole cache expire at once

The single-flight lock handles a herd once it starts. A complementary technique tries to prevent the herd from starting in the first place, by making expiry probabilistic instead of a hard cliff. The idea, sometimes called early recomputation: as a cached value approaches its TTL, give each request a small, growing chance of recomputing it early — on its own, ahead of the crowd — so that by the time the real expiry arrives, the value has usually already been refreshed by one lucky early request, quietly, with no herd at all.

python — probabilistic early expiration
def should_recompute_early(created_at, ttl, regen_time):
    age = now() - created_at
    remaining = ttl - age
    # the chance of an early recompute grows sharply as remaining time shrinks
    p_recompute = math.exp(-remaining / regen_time)
    return random() < p_recompute

With a 60-second TTL and a 40ms regeneration time, this probability is essentially zero until the last few hundred milliseconds before expiry, then rises sharply — so out of 8,000 requests a second landing in that final window, only a handful independently decide to recompute, and whichever one finishes first refreshes the key before it ever actually goes empty. Combined with the single-flight lock as a backstop for the rare case where the probabilistic trigger fires too late, this two-layer defense is what production caching libraries actually ship.

The other trap: caching a miss

One more race worth naming, distinct from the herd: what does get_trending do if db.compute_trending() itself fails — a timeout, a bad connection? A careless implementation catches the exception, returns an empty result to the caller, and never writes to the cache — which sounds safe, but under sustained load means every concurrent request hits the failing database at once, on every single request, for as long as the outage lasts, because nothing was ever cached to short-circuit the next attempt. The fix mirrors Chapter 6’s “never cache a refusal” rule from a different angle: cache a short-lived sentinel on failure (a few seconds), not the missing data itself, so a struggling database gets a brief moment of relief instead of the full, uncoalesced weight of live traffic.

Concept → realization: why this specific race did not exist before Chapter 4

Neither replicas nor edge caching produce this bug. A replica serves whatever data it has — there is no moment where a replica is “empty.” An edge cache miss is a single request falling through to origin, not 320 identical ones, because a CDN’s edge nodes typically already coalesce concurrent misses for the same key internally. An application cache you write to by hand, on a key with a hard expiry, under real concurrent load, is where this specific failure mode is born — and it is worth understanding exactly why: it is the first layer in this lesson where your own cache, not the database, is briefly the thing that is down.

It is a useful chapter to return to mentally whenever a new caching layer gets added anywhere in a system, at any company: the question worth asking is always “what happens when this specific key expires while many concurrent requests want it,” because that question has the same answer — a herd, unless something explicitly prevents it — regardless of which technology is doing the caching.

Chapter 5: Hit-Rate Arithmetic

Every layer so far has been about correctness under load. This chapter is a single piece of arithmetic, worth its own chapter because almost nobody does it by hand before shipping a caching layer — and it is the number that determines whether the database tier you are building needs to survive 500,000 QPS or 5,000.

Read it as the bridge between the two halves of this lesson. Chapters 1 through 4 each build a mechanism; this chapter is the ledger that converts what those mechanisms are worth into the actual currency Chapter 8 spends — database boxes.

It is a short chapter by design. The formula itself is one line; the value of this chapter is entirely in taking that one line seriously enough to plug real numbers into it before a design review, rather than after an incident forces the question.

Keep the formula visible next to whatever hit-rate number a dashboard reports, because the two together are what turn a monitoring metric into a capacity forecast rather than just a health indicator.

The formula, derived from nothing but the definition of a hit

Call the total read traffic hitting your system T, and the fraction of those reads the cache answers without touching the database the hit rate, h (a number between 0 and 1). By definition, the reads the cache does not answer — the ones that reach the database, which system-design conversations call the origin — are the complement:

That is the entire formula. Everything interesting in this chapter comes from plugging real numbers into it.

What happens when the formula runs twice: compounding hit rates

Chapter 8 stacks two caching layers — the CDN from Chapter 3 and the app cache from Chapter 4 — one after the other, so it is worth previewing what the formula does when applied twice in sequence rather than once. If the first layer catches h₁ of traffic and the second layer catches h₂ of what is left, the traffic that survives both is:

The two miss rates multiply, not add. Two mediocre 80% layers stacked in sequence still remove

of traffic combined — noticeably better than either layer alone, and a useful intuition for why multiple imperfect caching layers, stacked in the right order, comfortably outperform investing everything into perfecting a single one.

Where hit rate itself comes from: the shape of the traffic

Before trusting a hit-rate number, it helps to see where one actually comes from, because “99% hit rate” is not a knob you turn — it falls out of how concentrated your traffic is and how many distinct keys your cache holds. Feed traffic, like most human-generated traffic, is Zipf-shaped: a small number of keys account for a large share of requests, and a long tail of keys are each requested rarely.

This is why hit rate climbs so slowly past a certain point: caching the first thousand keys buys 62 points of hit rate almost for free, because that is where the traffic is concentrated. Caching the next 49,000 buys another 35 points — still a good trade, but each additional key is doing far less work than the ones before it. And no amount of additional cache capacity reaches 100%, because the tail is, by construction, requested too rarely to ever repeat inside any reasonable cache lifetime.

This same shape is why Chapter 3’s CDN and Chapter 4’s Redis both reach genuinely high hit rates without needing to store every distinct key a feed app has ever produced — the Zipf concentration means a comparatively small, bounded set of keys accounts for nearly all repeat traffic, and both layers are sized against that concentrated head, not against the unbounded tail.

It also explains a pattern operators notice but rarely name: hit rate on a brand-new cache climbs extremely fast in its first minutes (the head fills in almost immediately, since it is requested constantly) and then rises with visibly diminishing speed afterward, as each additional point of hit rate requires waiting for progressively rarer keys to be requested a second time before they can register as a hit at all.

The number that should end every “let’s add a cache” conversation

Recall the design target from Chapter 0: this system needs to survive T = 500,000 QPS at peak. Compute origin load at two hit rates that sound, in casual conversation, roughly similar — “90% cached” and “99% cached”:

A nine-point difference in hit rate — 90% versus 99%, which sound like “pretty good” and “excellent” — is a ten-times difference in how much database capacity you need to build and pay for. This is the arithmetic every cache-sizing conversation is actually about, whether or not anyone writes it down.

Push one digit further. At h = 0.999 (one miss in a thousand):

From 90% to 99.9% — two additional nines — origin load falls from 50,000 QPS to 500, a hundred-fold reduction, and 500 QPS is comfortably inside a single box’s 4,000-QPS ceiling from Chapter 0. This is why production caching systems obsess over the third and fourth nine of hit rate long after “it is mostly caching” would satisfy a casual read of a dashboard.

Do the arithmetic backwards: what hit rate does a target require?

The formula runs just as usefully in reverse. Suppose the database tier, per Chapter 1, has been provisioned for exactly 3 replicas × 4,000 QPS = 12,000 QPS of capacity, and you want to know the minimum hit rate that keeps origin traffic under that ceiling at the full 500,000 QPS target. Rearrange the formula:

This is the number a capacity-planning conversation actually needs: not “a high hit rate,” but a specific figure, 97.6%, below which the provisioned database tier is mathematically guaranteed to be overloaded, no matter how well everything else in the system behaves. Any cache design review for this system should be checked against this number directly, not against a vague sense of “caching helps.”

Run the same inversion for the target this lesson is actually building toward — the full 500,000 QPS from Chapter 0, against the eventual 3-replica, 12,000-QPS database tier Chapter 8 assembles — and the required hit rate is the same 97.6%. That single number, computed once here, is the load-bearing target every later chapter’s design has to clear.

Notice, too, that this inversion is symmetric with the forward calculation from a few paragraphs above: forward, a hit rate predicts an origin QPS; backward, a database tier’s provisioned capacity predicts the minimum hit rate it demands. A capacity-planning conversation can enter the formula from either direction depending on which side is fixed — a hard infrastructure budget, or a hard traffic target — and the same one-line formula answers both questions.

The mistake this arithmetic prevents

It is tempting to size a database tier off the top-line traffic number and treat the cache as a “nice to have” that reduces average load. That gets the direction of the design backwards. The database tier only needs to be sized for the traffic the cache does not catch — and Chapter 8’s full assembly is built entirely around this: figure out the realistic hit rate first, size the database for the resulting origin QPS, and treat every point of hit rate lost (a cache flush, a cold cache after a deploy, a spike in unique keys) as a capacity event, not a performance nuisance.

The crossover point: when caching pays for its own infrastructure

Every point of hit rate above 62% (the Zipf table above) requires holding more distinct keys in Redis — more RAM, more cost. It is worth pricing that cost against what it buys, using Chapter 0’s database box as the unit of comparison. Suppose a Redis box holding enough keys for 90% hit rate costs $400/month, and pushing to 99% requires a bigger box at $1,200/month. Compare against database boxes saved:

At any realistic database box cost (even a modest few hundred dollars a month), the 90%-hit-rate cache pays for itself many times over just from the first 112 boxes it removes — the crossover happens almost immediately, which is the actual reason caching is close to mandatory at this scale rather than an optimization to consider later. The interesting decision is not “cache or no cache,” it is how far past that first crossover — 90%, 99%, 99.9% — to keep paying for, and that answer depends on how much the remaining origin boxes cost relative to the next slice of cache memory.

A measurement trap: the average hides the emergency

A single global hit-rate number, reported hourly, can look perfectly healthy while a specific, important slice of traffic is on fire. Suppose overall hit rate sits at a comfortable 97% — but that number blends two very different populations:

The blended 96.98% looks fine on a dashboard with a 95% alert threshold. But nearly all of that origin load — 10,000 of the 10,950 QPS — is coming from 5% of traffic with a genuinely bad hit rate, and a single global metric buries that entirely. The fix is not a different formula; it is measuring hit rate per endpoint or per feature, not only in aggregate, so a newly-launched, poorly-cached surface shows up as its own alarm instead of being diluted into a comfortable global average.

A regional outage, worked through

The blended-average trap above hides a slow degradation. A sharper version of the same lesson shows up during an outright partial outage. Suppose this system runs three geographic regions, each normally holding its own warm cache and handling a third of total traffic — roughly 166,667 QPS apiece out of the 500,000 QPS target. One region’s Redis cluster goes down entirely. Traffic for that region does not vanish; it fails over to the other two regions’ application tier, but arrives at a cache that has never seen those users’ keys before — functionally a cold cache for that slice of traffic, at 0% hit rate, even while the surviving regions’ own traffic keeps its usual 99%:

One failed region turns a comfortable ~5,000 QPS of total origin traffic into 170,000 — a 34× spike, and it is dominated almost entirely by the one region running at 0%, not by any change to the two regions that are otherwise perfectly healthy. This is why a global, blended dashboard is dangerous in exactly the way the previous section warned: a per-region hit-rate view would show one region pinned at 0% the instant the outage starts, while the blended figure might still read a deceptively survivable 66% (two-thirds of traffic at 99%, one-third at 0%) — a number that sounds rough but not catastrophic, until it is checked against actual provisioned capacity.

Compare 170,000 QPS of origin load against the 3-replica, 12,000-QPS database tier Chapter 8 provisions for the steady state, and the gap is stark: this single-region cache failure alone demands roughly 14× more database capacity than exists. The honest response here is not a bigger database — it is routing the failed-over region’s reads to a neighboring region’s warm cache where the same keys are likely already resident, or, failing that, invoking Chapter 0’s load-shedding fallback specifically for the affected region rather than trying to serve it in full from a cold start.

Concept → realization: hit rate as a live gauge, not a monthly report

python — the two counters that produce every number in this chapter
def get_trending(redis, db, metrics):
    cached = redis.get("trending:global")
    if cached is not None:
        metrics.incr("cache.hit")
        return cached
    metrics.incr("cache.miss")               # this counter IS origin QPS
    return db.compute_trending()

# hit_rate = hit_count / (hit_count + miss_count), over any rolling window
# origin_qps = miss_count / window_seconds — measured directly, no formula needed

Two increments, placed at the two exit points of the same function that already existed in Chapter 4, are the entire instrumentation this chapter’s arithmetic depends on in production. Hit rate is not a number you estimate from a spreadsheet — it is a ratio of two counters that were sitting right there in the code the whole time, and origin QPS is simply the miss counter’s rate, measured directly rather than derived from the formula at all. The formula’s value is predictive — it tells you what to expect before you have the counters, when you are still deciding how much cache capacity to buy.

Why hit rate has diminishing returns even with unlimited cache memory

It is tempting to think a big enough Redis box could push hit rate arbitrarily close to 100%. Two things stop that in practice. First, the Zipf tail from earlier in this chapter never fully repeats — a key requested exactly once in a month cannot be a cache hit no matter how much memory exists, because there is no second request to serve from cache. Second, real caches use LRU (least recently used) eviction under memory pressure: even generously sized, a working set that shifts over time (yesterday’s trending topics are not today’s) means some fraction of memory is always holding entries on their way out, not their way to being reused. Both effects mean the honest ceiling on hit rate for any real, unbounded traffic distribution is comfortably below 100% — which is exactly why this chapter’s target was 99%, not 100%, and why Chapter 8’s assembly still needs a database tier at all.

This is worth internalizing before Chapter 8, because it means the database tier is never fully optional at this scale, only ever smaller. Every layer of caching in this lesson reduces origin QPS; none of them, even in the limit of unbounded memory, drives it to exactly zero. The lesson’s target has been, from Chapter 0 onward, a database tier that survives the traffic it actually gets — not a database tier eliminated altogether.

Chapter 6: Invalidation

“There are only two hard things in computer science: cache invalidation and naming things.” It is a joke because it is true, and this chapter is about earning that truth rather than just repeating it — with a specific incident, a specific dollar figure, and three concrete strategies for keeping a cached value from lying about the world after the world has changed.

Every prior chapter in this lesson made a system faster or cheaper to run. This one is different in kind: it is about correctness, the same axis Chapter 4’s race condition lives on, and it is where a caching decision made carelessly stops costing milliseconds and starts costing real money.

It is also, not coincidentally, the chapter most likely to be skipped under deadline pressure — a TTL is one line of code, and shipping it without a second thought feels like finishing the work rather than deferring a decision. The rest of this chapter is the case for treating that one line with the same scrutiny given to the threshold and the lock in earlier chapters.

Why this problem does not go away once you have a cache-aside race fix

It is tempting to think Chapter 4 already solved the correctness problems an application cache can have — the single-flight lock guarantees the value in the cache is consistent, in the sense that everyone reads the same thing. It says nothing about whether that value is current. A perfectly coalesced cache, with zero thundering herd, can still confidently and consistently serve last week’s price to every single reader, because coalescing only controls who computes a value, not when that value should be thrown away and recomputed. This chapter is about that second, separate axis.

An incident, worked through in full

At 9:00:00 a pricing team pushes a correction: a promotional item’s price, mistakenly listed five dollars too low, gets fixed in the database. The product page for that item is cached with a 300-second (5-minute) TTL. The catch: that cache entry had last been populated two minutes before the fix, at 8:58:00, and TTLs do not know or care when the underlying data changed — they only count down from when the entry was written.

So the cache keeps serving the old (wrong) price from the moment of the fix, 9:00:00, until it naturally expires three minutes later, at 9:03:00:

Checkout traffic for this item runs at roughly 3 orders per second. Every order placed inside that window is billed at the wrong price:

Nothing crashed. No error appeared in any log. The system did exactly what a TTL-only cache is specified to do: serve a value for up to its TTL, regardless of what happens to the source of truth in the meantime. The bug is in the specification, not the code.

That last sentence is the entire reason this incident is worth a full chapter rather than a footnote. A code review looking for bugs finds none, because there are none — the TTL logic, the cache-aside pattern, the database write, all execute exactly as written. The gap lives one level up, in the unstated assumption that a 300-second TTL was an acceptable staleness bound for a field where staleness has a direct, measurable dollar cost. Fixing it means revisiting that assumption, not debugging a function.

This is the pattern behind the joke this chapter opened with. Naming things is hard because the right name depends on a shared, often unstated understanding of what a thing actually is. Cache invalidation is hard for the identical reason: it depends on a shared, often unstated understanding of how long a value is allowed to be wrong, and that understanding lives in people’s heads far more often than it lives in a config file where a code reviewer could catch it.

A second incident, in the other direction: the coupon that would not expire

The price incident above is a cache serving a value that is wrong because it is old. The same mechanism produces an equally real, opposite-flavored bug: a cache confidently serving a value that is wrong because it has expired in the real world but not yet in the cache. A promotional coupon, SUMMER20, is configured to stop working at exactly 23:00:00. The checkout service caches “is this coupon code currently valid” with a 600-second (10-minute) TTL, and the last cache population before the cutoff happened at 22:54:00.

So the cache keeps answering “valid” for four minutes after the coupon has actually expired, from 23:00:00 until it naturally falls out at 23:04:00:

This coupon is being applied at checkout roughly 2 orders per second during this promotion’s closing rush, at a 20% discount on an average $60 cart:

The direction of the error is the mirror image of the price incident — there, a correction that should have raised the effective price got delayed; here, an expiry that should have removed a discount got delayed — but the root cause and the fix are identical: a TTL that does not know when the underlying fact actually changed. Write-through on the coupon’s validity flag, updated in the same transaction that flips it inactive, closes this window down to the same single-digit milliseconds the price fix achieves, for the identical reason.

Choosing a TTL by what the data actually is

Before reaching for write-through everywhere, it helps to sort the things a feed app caches into buckets by how much a stale read actually costs, because the right default TTL is completely different across them:

The instinct to pick one TTL policy and apply it everywhere is exactly the mistake this table is built to prevent. A financial fact and a like count are not the same kind of data, and treating them identically either over-invests in invalidation machinery for data that does not need it, or under-invests in it for data where the cost, as this chapter’s incident shows, is real money.

Concept → realization: what event-driven purge actually looks like

python — the write path publishes, every cache node subscribes
def update_price(item_id, new_price, db, pubsub):
    db.execute("UPDATE items SET price=%s WHERE id=%s", new_price, item_id)  # 1. write, durable
    pubsub.publish("cache-invalidate", f"item:{item_id}")                    # 2. tell everyone

# every cache node, subscribed once at startup:
def on_invalidate_message(key, local_cache):
    local_cache.delete(key)         # local Redis instance drops its copy — propagation delay is this fan-out

The 800ms propagation-delay figure used earlier is the time for that publish call to reach every subscribed cache node across every region — a real number, not a rounding convenience, and one that grows with the number of regions and the reliability of the transport carrying the message. A missed message (a cache node that was briefly disconnected from the pub/sub channel) silently reverts that one node to TTL-only behavior for that key, which is why systems that lean on event-driven purge for correctness-critical data usually pair it with a much shorter backstop TTL — not the 5 minutes from this chapter’s incident, but something like 30 seconds, so a missed invalidation message self-heals quickly rather than persisting indefinitely.

That backstop TTL is not a contradiction of the invalidation strategy — it is the safety net underneath it. Event-driven purge handles the common case in under a second; the short TTL handles the rare case where the message itself was lost, at a cost far lower than the incident this chapter opened with, because a 30-second bound instead of a 300-second one shrinks the same arithmetic by a factor of ten before it ever compounds into real money.

The pattern generalizes past caching, too: whenever a fast primary mechanism has a rare failure mode, pairing it with a slower but simpler backstop is usually cheaper than trying to make the primary mechanism itself perfectly reliable. Chapter 8’s failure toggles later in this lesson lean on the same idea — redundancy for the common failure, graceful degradation for the rest.

Three strategies, and what each one actually bounds

Rerun the incident’s arithmetic under each strategy, holding the 3 orders/second checkout rate fixed:

Write-through and event-driven purge are not exotic techniques — they are just a way of making the cache’s staleness window track the actual write, instead of an arbitrary timer that started whenever the entry happened to be populated. The trade-off runs the other way: TTL is the cheapest to implement (nothing to build besides a number) and the most forgiving of a missed message, while write-through and event-driven purge both require the write path to reliably notify every cache node, and a missed invalidation message quietly reverts to TTL-only behavior.

Why nobody runs write-through everywhere

If write-through bounds staleness to single-digit milliseconds, why does any system still use a plain TTL? Two honest reasons. First, write-through couples the cache into the write path’s critical section — if the cache is slow or unavailable, does the write fail too, or does it silently skip the cache update and risk exactly the staleness it was meant to prevent? Second, and more common: most cached values are not tied to one clean write event at all. A trending list is a continuous aggregate over the last hour of activity — there is no single “the write that changed it,” only a TTL that accepts staleness because the value itself is inherently approximate. Reach for write-through or event-driven purge specifically for values with a clear, identifiable source write — a price, a profile field, an inventory count — and accept a TTL everywhere the underlying value is a moving aggregate rather than a fact.

There is also an operational cost to write-through that is easy to underweight in a design discussion and easy to feel in an incident: it adds a dependency. A cache-aside system, as built in Chapter 4, can lose its entire cache and keep serving correct (if slower) answers straight from the database. A write-through system that fails open — skipping the cache update on error rather than blocking the write — degrades gracefully in the exact same way. One that fails closed, blocking every write until the cache update succeeds, has just made cache availability a prerequisite for the ability to write data at all, which is a much larger promise than most systems intend to make when they first reach for write-through as “the more correct option.”

Deciding which failure mode is acceptable — block the write, or risk a brief staleness window — is itself a decision about the data, mirroring the earlier table sorting financial facts from moving aggregates. A payment system may reasonably choose to block; a feed app, almost never.

The second race: getting the write order wrong

Write-through and event-driven purge both introduce a new question that a plain TTL never has to answer: in what order do you update the database and touch the cache? Get it backwards and you create a second race condition, distinct from Chapter 4’s thundering herd, that has nothing to do with concurrency volume and everything to do with sequencing.

wrong order — invalidate, then write
1. delete cache key
2.                                    <-- a concurrent READER misses, queries the OLD price,
                                          and repopulates the cache with it
3. write new price to database        <-- too late: the cache now holds the stale value again,
                                          and nothing will fix it until the next explicit write

right order — write, then invalidate
1. write new price to database        -- durable, committed
2. delete (or update) cache key       -- any reader after this point either misses and
                                          reads the new value, or was served the old value
                                          from before step 1, which is a normal, bounded race

The window in the wrong-order case is not bounded by any TTL at all — it persists until the next write happens to touch the same key, which could be minutes or hours later. The right order narrows the dangerous window down to the single instant between the database commit and the cache update, which is exactly the commit-latency figure — a few milliseconds — used in the table above. Sequencing, not just mechanism, is what makes write-through actually deliver the bound it promises.

Who is responsible for invalidating a key nobody wrote to directly?

The price example is clean because one write touches one key. Real systems have cached values that depend on data spread across several tables — a product page’s cached response might depend on the item’s price, its stock count, and its category’s current promotion, each owned by a different part of the write path. The discipline that keeps this tractable is provenance: record, alongside the cache entry, every underlying source it was built from, so any write to any of those sources knows exactly which cache keys to invalidate — rather than requiring every future author of a new write path to remember, by convention alone, that touching the promotions table also means purging every product page in that category. Missing this is a common, quiet source of exactly the kind of incident this chapter opened with: not because invalidation was never implemented, but because it was implemented for the write path someone thought of, and not for the one added six months later by someone else.

Testing invalidation before an incident finds the gap for you

Because invalidation bugs are specification gaps rather than obvious code defects, they rarely surface in ordinary testing — a unit test that writes a value and immediately reads it back will pass whether or not the cache was properly invalidated, since a fresh write almost always populates the cache correctly on the very next read regardless of the invalidation path. The test that actually exercises this chapter’s failure mode has to simulate the race directly: write a value, assert the old cached value is still being served (proving the cache was populated before the write), then assert it stops being served within the strategy’s promised bound — 5ms for write-through, 800ms for event-driven purge, the full TTL for TTL-only. Writing that test once, for the highest-stakes cached field in a system, is usually enough to catch the ordering bug from the previous section before it ever reaches production.

It is worth running this same test periodically against production, not only once at build time. An invalidation path that worked correctly at launch can silently regress months later — a refactor that reorders a write and a cache purge, a new write path added by someone unaware the purge was load-bearing. A recurring, automated check of the staleness bound is the only reliable way to know the guarantee this chapter is built around still holds.

Chapter 7: Denormalized Reads

Everything so far assumes the query itself — the 2ms lookup from Chapter 0 — is cheap, and the problem is purely how many times you run it. That assumption breaks the moment a read actually needs to combine data from many places, which is exactly what a feed does: show me the newest posts from the 300 accounts I follow, merged and sorted. This chapter is about what happens when replicas and caching are not enough, because the query underneath them was never cheap to begin with.

Every earlier chapter treated the query itself as a fixed unit of work and asked how to run it less often or on more boxes. This chapter asks the more fundamental question those chapters quietly assumed away: what if the unit of work itself is the thing worth shrinking?

The two words in this chapter’s title, defined precisely

“Denormalized” and “materialized” get used almost interchangeably in casual conversation, and it is worth pinning down what each actually means before using them. Normalization is the database-design discipline of storing each fact exactly once — a post lives in one posts row, a follow relationship in one follows row, and a feed is derived by joining them at read time. That is what Chapter 0’s original query does, and it is why it is expensive: deriving is work. This is the standard vocabulary a system-design conversation reaches for, so it is worth having it land correctly the first time rather than picking it up piecemeal from context. Denormalizing means deliberately storing redundant, precomputed copies of derived data — the user_feed table below duplicates information that could, in principle, be recomputed from posts and follows. A materialized view is the general database term for exactly this: a precomputed, stored result of a query, refreshed on some schedule or trigger rather than recomputed on every read. This chapter builds a hand-rolled materialized view via explicit fan-out-on-write, which is the same idea Postgres’s own MATERIALIZED VIEW feature provides more generically, at the cost of controlling the refresh trigger less precisely.

Postgres’s built-in REFRESH MATERIALIZED VIEW recomputes the entire view from scratch on a schedule, which is simple but coarse — a full recompute of every user’s feed, on a timer, regardless of whether that particular user’s followed accounts posted anything since the last refresh. The hand-rolled fan-out-on-write approach this chapter builds instead updates only the specific rows a specific new post actually affects, which is more code to maintain but avoids paying the cost of recomputing 12,000,000 users’ feeds to reflect the handful that actually changed. This targeted-versus-blanket distinction mirrors the same choice Chapter 6 makes between event-driven purge and a scheduled full refresh — a targeted update touches only what actually needs to change, while a blanket refresh recomputes everything on a fixed schedule regardless. The right choice in both chapters depends on whether the underlying data changes in small, identifiable increments or requires genuinely global recomputation to stay correct.

Costing the live join by hand

Recall the query from Chapter 0’s concept-to-realization: fetch a user’s 300 followed authors, then find and merge their recent posts, sorted by time. Unlike a single indexed lookup, this one has to touch data proportional to how many accounts the user follows. Break the cost into its two real pieces:

Fifteen milliseconds instead of the 2ms baseline — not because Postgres is doing anything wrong, but because there is genuinely more work: fan out to 300 authors’ worth of rows, then merge and sort them. This is called fan-out on read: the expensive work happens every single time someone opens their feed.

Notice something the earlier chapters could not fix, even in principle. A replica in Chapter 1 runs this exact same 15ms query — replicating the box does not touch the cost of the query itself, only how many of them can run in parallel. An application cache in Chapter 4 could store the result of this query, but the result is different for every one of 12,000,000 users, so a cache keyed on user ID would need 12,000,000 entries and would still have to pay the full 15ms on every miss, refresh, and cold start. Both prior tools multiply or hide the cost; neither one shrinks it. That is precisely the gap this chapter exists to close.

What that does to box capacity

Redo Chapter 0’s capacity math with the real 15ms query time instead of the simplified 2ms:

Five hundred thirty-three queries a second, for the exact same box that handled 4,000 simple lookups a second. Now ask how many of these boxes you would need just to survive the 5,000 QPS of read traffic from Chapter 0 — ignoring caching entirely, to isolate this one effect:

Ten boxes, running the live-join query, just to hold the line at today’s traffic — and this is before applying any of the growth target from Chapter 0.

The alternative: fan out on write instead

Flip the work from read time to write time. Instead of computing a user’s feed live every time they open the app, maintain a precomputed materialized read model: a table that already holds “this user’s current feed, ready to read” — refreshed the instant someone they follow posts, not the instant someone asks to see it.

sql — the fan-out, on a new post
-- author posted; push this post_id into every follower’s precomputed feed table
INSERT INTO user_feed (user_id, post_id, created_at)
SELECT follower_id, $new_post_id, now()
FROM follows WHERE followed_id = $author_id;

sql — the read, now trivial
SELECT post_id, created_at FROM user_feed
WHERE user_id = $1 ORDER BY created_at DESC LIMIT 50;

The read is back to a single indexed lookup on user_id — the 2ms query from Chapter 0, not the 15ms join. Recompute:

Two boxes instead of ten — a five-times reduction, purely from moving the expensive join off the read path and onto the write path, where it happens once per post instead of once per feed view.

It is worth stating precisely what changed and what did not. The total amount of work the system does per post has not shrunk — if anything, fan-out on write does strictly more total work than fan-out on read, since it precomputes the join for every follower whether or not they ever open their feed that day. What shrank is when that work happens: once, at write time, amortized across however many times the post is subsequently read, instead of freshly recomputed on every single read. This trade only pays off because reads vastly outnumber writes in a feed app — each post is written once and, if popular, read thousands of times.

The materialized view has its own staleness window

Notice what this trade actually did: it turned a read problem into a write-propagation problem that looks structurally identical to Chapter 1’s replication lag. The fan-out insert in the code above is not instantaneous — it has to run, and under load it queues, exactly like every other piece of work in this lesson. If fan-out inserts are processed by a worker pool handling 50,000 inserts/second and a single popular post triggers 300,000 follower inserts:

For up to six seconds after a popular account posts, some followers’ materialized feed tables have the new post and others do not — a staleness window with exactly the same shape as Chapter 1’s replication lag, just measured in fan-out completions instead of WAL bytes. A materialized read model does not eliminate staleness; it relocates it from the query itself (Chapter 7’s original problem) to the write pipeline that keeps the model current.

The bill this moves, not erases: write amplification

Fan-out on write does not make the work disappear — it moves it, and moves it in a way that can get very lopsided. A typical account with a few hundred followers costs a few hundred row inserts per post: cheap. A celebrity account with 4,000,000 followers costs 4,000,000 row inserts for a single post:

That is a write-scaling problem, not a read-scaling problem — and it is the exact subject of this lesson’s companion, Scaling Writes, where partition keys, write buffering, and the question of what to do about a single account whose fan-out dwarfs everyone else’s get the full chapter they deserve. For this lesson, the takeaway is narrower: a materialized read model is how you buy back read latency and read capacity, and the price is a write-side cost that scales with followers instead of with readers — a trade worth making for read-heavy traffic, and one that needs its own design once any single account’s fan-out gets large enough to dominate the write path.

What the materialized table costs to store, by hand

Write amplification is a rate problem — extra inserts per second. There is a second, separate cost worth pricing: the user_feed table itself has to live somewhere, continuously, whether or not anyone is actively posting. Model a realistic bound across all 12,000,000 users: keep only the last 50 feed entries materialized per user, matching the LIMIT 50 already used in the read query above. Each row — a user ID, a post ID, and a timestamp — costs roughly 32 bytes once index overhead is included:

Nineteen gigabytes is trivially cheap on any real database box — well under what a single replica’s working set already budgets for — and that cheapness is not an accident, it is the direct payoff of the LIMIT 50 cap. If the table instead kept every post a user had ever been shown, rather than capping at the most recent 50, storage would grow linearly and unboundedly with total lifetime post volume instead of staying a flat, predictable multiple of the user count:

A 40× difference between the bounded and unbounded versions of the identical table, from one design decision — capping how far back the materialized view reaches — which is exactly why the read query above always carries a LIMIT 50 and why a background job periodically trims rows older than that window, rather than letting the table grow with total historical post volume the way the naive version above does.

A hybrid: fan out on write for almost everyone, fan out on read for outliers

Production feed systems rarely pick one strategy for every account. Instead, they set a follower threshold — say, 10,000 followers — and split the population:

The read for a user who follows a mix of both now does two things instead of one: fetch the precomputed rows from user_feed as before, and separately fetch recent posts from the small number of high-follower accounts that user follows (typically a handful, since few people follow more than a few celebrities), then merge and re-sort the two result sets in the application. Concretely:

sql — the hybrid read, two cheap queries instead of one expensive one
-- 1. the precomputed part: everyone this user follows who is NOT high-follower
SELECT post_id, created_at FROM user_feed
WHERE user_id = $1 ORDER BY created_at DESC LIMIT 50;    -- ~2ms

-- 2. the live part: the handful of high-follower accounts this user follows
SELECT post_id, author_id, created_at FROM posts
WHERE author_id = ANY($2)          -- typically 0-5 accounts, not 300
ORDER BY created_at DESC LIMIT 50;                 -- ~3ms — small fan-out, cheap join

-- application: merge both result sets by created_at, take top 50 — sub-millisecond

That extra merge step costs a few milliseconds — far less than the full 15ms live join across all 300 followed accounts, because it only touches the small high-follower subset instead of everyone. This hybrid is exactly the kind of design captured by this lesson’s taxonomy from Chapter 0: most of the traffic gets “answer without querying” treatment, and the genuinely expensive outlier case gets a smaller, targeted version of the original live join.

Picking the threshold itself is a genuine design decision, not an arbitrary round number. Set it too low and ordinary accounts fall into the expensive live-join path unnecessarily; set it too high and a small number of celebrity posts still trigger enormous fan-out writes. In practice, the threshold is chosen empirically — measure the actual follower-count distribution, find the point past which write amplification starts to dominate total system cost, and set the cutoff there, revisiting it as the user base and its distribution of follower counts grow.

It is a threshold worth revisiting on a schedule, not just once at launch, because a platform’s own growth changes the input to this calculation: a follower-count distribution that justified a 10,000-follower cutoff at one scale of user base may look completely different once that same platform has a hundred times as many users, some of whom now have followings the original threshold never anticipated.

The part fan-out on write does not automatically handle: deletes and unfollows

The fan-out code above pushes new posts into every follower’s materialized table on write. The reverse case needs its own handling, and it is easy to overlook: if a post is deleted, or a user unfollows an author, every materialized user_feed row that came from that author-follower pair is now wrong, sitting in potentially thousands of separate followers’ precomputed tables. Two honest options, with the same trade-off Chapter 6 already named:

Most production systems land on a hybrid here too: lazy filtering for the common case (unfollows are rare relative to posts), with a periodic background sweep that eventually cleans up accumulated stale rows so the lazy-filter join does not grow unbounded over time. It is the same lesson as Chapter 6’s invalidation strategies, wearing a different hat: correctness has a cost, and the right amount to pay for it depends on how often the underlying data actually changes.

That, in one sentence, is the entire lesson of this chapter, and it is the piece Chapter 8 needs in hand before it can size a real database tier: caching and replication multiply the value of cheap queries; they do nothing to make an expensive query cheap. Shrinking the query itself was always a separate problem, solved by a separate tool.

Chapter 8: Assembling the Read Path

Every layer up to this point has been studied in isolation. Now put them in the order a real request actually travels through them, at the actual target from Chapter 0 — 500,000 QPS at peak — and see what each layer removes before the next one even has to work.

Read this chapter as the payoff for the previous seven. Nothing here is new mechanism — every formula and every number below was already derived, in isolation, in an earlier chapter. What is new is seeing them compound against each other, at the real target, for the first time.

Treat the numbers below as a worked example, not a universal blueprint. A different application with a different traffic shape, different hot-key concentration, or different query complexity will land on different hit rates and a different replica count — the method (fold each layer in order, size the last one from what actually reaches it) is what transfers, not the specific 60%/99%/3-replica figures this particular feed app happens to produce.

That method is worth restating plainly, since it is the actual takeaway of this chapter: total traffic, folded through every real cache layer in the order requests actually encounter them, determines the load the last, most expensive layer has to survive — and that final number, not the original traffic figure, is what a database tier is sized against.

Why the order of the stages is not arbitrary

It matters that the CDN sits before everything else, and that the database sits last. Each stage is ordered from cheapest-to-answer-from to most-expensive-to-answer-from: an edge cache hit costs a network round trip inside one metro area (Chapter 3’s low single-digit milliseconds); an app cache hit costs one Redis lookup (Chapter 4’s sub-millisecond figure); a database hit costs a real query against real disk-backed storage (Chapter 0’s 2ms, or Chapter 7’s 15ms without the materialized model). Putting the database first and the CDN last would mean paying the most expensive cost for every single request before ever getting the chance to answer cheaply — the funnel only works because each layer is asked the question before the next, more expensive layer ever sees it.

The full path, stage by stage

Sizing every stage by hand, at 500,000 QPS

Start at the top of the funnel. Suppose the edge cache, per Chapter 3, catches 60% of total traffic — the public, cacheable slice:

Of that 200,000, the application cache, per Chapters 4 and 5, is running at a 99% hit rate:

Two thousand queries per second, out of an original 500,000 — a 250× reduction, purely from two layers of caching, before a single replica has to answer anything. This is Chapter 5’s compounding-hit-rate formula from earlier in this lesson, applied to the real two-layer stack rather than a toy example: two imperfect filters, chained, remove far more than either alone. Now size the database tier for that remaining 2,000 QPS, using the materialized-read-model capacity of 4,000 QPS per box from Chapter 7:

Reading the two failure toggles

Kill a replica. With 3 replicas provisioned against a 2,000 QPS load, losing one leaves 2 replicas × 4,000 QPS = 8,000 QPS of capacity against 2,000 QPS of demand — still comfortable. The redundancy margin from the “+2” in the sizing above exists for exactly this: a single lost box should never be the thing that turns a Tuesday into an incident.

Notice this is Chapter 1’s N+2 sizing rule, applied for real once the traffic it actually has to survive was known. Sizing replica count off the raw 500,000 QPS target would have meant provisioning for over 125 boxes; sizing it off the 2,000 QPS that actually reaches the database tier, after two layers of caching, means 3. The order of operations in this lesson — understand the caching layers and their arithmetic first, size the database tier last — is not incidental. It is the entire reason the database tier stays this small.

This is also the honest answer to why this lesson spent seven chapters getting here instead of opening straight with “add a CDN and a Redis cache.” Every one of those chapters supplied a number this final sizing step depends on: the box ceiling from Chapter 0, the staleness budget from Chapter 1, the pool headroom from Chapter 2, the edge hit rate from Chapter 3, the protected hit rate from Chapter 4, the compounding formula from Chapter 5, the invalidation bound from Chapter 6, and the per-query cost from Chapter 7. Remove any one of them and this chapter’s sizing arithmetic stops being trustworthy.

This is the honest reason an eight-chapter build-up preceded a one-page summary: the summary would have been unearned. “Add a CDN, add Redis, add replicas” is advice; a specific hit rate, a specific replica count, a specific dollar figure, each traced back to a derivation you could reproduce by hand, is engineering.

Flush the app cache. This is the one that actually hurts. Set the app-cache hit rate to near zero and the traffic reaching the database tier jumps from 2,000 QPS back to nearly the full 200,000 QPS that made it past the CDN — a hundred-fold spike, arriving all at once, against 3 replicas that can absorb 12,000 QPS between them. That gap — 200,000 arriving against 12,000 available — is Chapter 0’s wall again, just reached by a different road: not organic growth this time, but an operational event (a bad deploy that cleared the cache, a Redis failover that lost its data) recreating the exact same overload.

This is worth sitting with for a moment: the database tier in this design was never made bigger or faster than the one in Chapter 0. It is the identical 4,000-QPS-per-box hardware, replicated to three boxes for redundancy. Every bit of the 500,000 QPS this architecture survives is survived by the caching layers absorbing traffic before it arrives, not by the database itself having gotten stronger. That is precisely why a cache flush is uniquely dangerous here: it does not degrade the database’s performance, it removes the only thing standing between the database and the traffic it was never sized to handle directly.

The same logic runs in reverse for capacity planning: growing this system further does not mean buying a bigger database. It means pushing the CDN and app-cache hit rates even higher, or adding a third caching layer, and only reaching for more database boxes once the arithmetic shows the existing caching layers are genuinely saturated rather than merely misconfigured.

A third toggle worth reasoning through: both at once

Real incidents rarely arrive as one clean failure. Consider losing a replica during a cache flush — a deploy that both restarts the app fleet (clearing local state) and happens to land while a database box is being patched. Database capacity drops to 2 replicas × 4,000 QPS = 8,000 QPS at the exact moment origin load is spiking toward 200,000 QPS. Neither failure alone was survivable at that combined moment, and the arithmetic makes clear why runbooks for this kind of system treat “cache flush” and “replica maintenance” as events that should never be scheduled to overlap deliberately, and why the load-shedding fallback from Chapter 0 — reject or degrade the least important slice of traffic — is the honest last line of defense when two independent failures do happen to coincide.

Put the timeline in numbers instead of prose. Origin load during the flush is 200,000 QPS, as derived above; surviving database capacity is 2 replicas × 4,000 QPS = 8,000 QPS. Using Chapter 0’s M/M/1 relationship one more time, ρ = 200,000 ÷ 8,000 = 25 — wildly over 1 — so the queue does not stabilize at some larger latency, it grows without bound for as long as both conditions hold. Applying Little’s Law at a representative moment, half a second into the overlap:

Ninety-six thousand requests, backed up behind a database tier that can only ever finish 8,000 of them a second, in the time it takes a person to glance away from a dashboard. That number is precisely why the runbook rule above is not bureaucratic caution: the two failures do not add, they multiply, and the load-shedding fallback from Chapter 0 has to trigger in well under that half-second window to keep the queue from spiraling into the tens of thousands before a human alert has even fired.

How long recovery actually takes

Once the immediate crisis is contained by load shedding, the cache still has to rewarm before normal service resumes, and that takes real time — it is not instantaneous just because the Redis process itself came back up. If the caching layer rebuilds hit rate the way it built up the first time — the Zipf table from Chapter 5, where the top 1,000 keys absorb 62% of traffic almost immediately and the long tail arrives far more slowly — expect the head of the distribution to rewarm within the first few minutes, with the tail taking substantially longer, on the order of the cache’s full TTL window before it has genuinely seen every key it will eventually hold again. The practical upshot for an on-call engineer: do not lift load shedding the instant the cache process is reachable again. Lift it once the hit-rate dashboard shows the head of the distribution has actually rewarmed — a later, measurably different moment than “the process is running.”

Concept → realization: the whole system, one request at a time

Trace one concrete request end to end, with real numbers at every hop, for a user opening their feed after a friend just posted:

one feed-open request, worst case (edge miss, app-cache miss)
1. request arrives at nearby edge PoP           ~1 ms   (Chapter 3, but this one is a MISS)
2. edge forwards to app tier over backbone       ~15 ms
3. app checks Redis — miss                       ~1 ms   (Chapter 4/5, coalescing engaged)
4. one request wins the lock, queries DB          15 ms   (Chapter 7 — materialized user_feed row IS a hit; a cold materialized view would cost 15ms of live join instead)
5. app writes result back to Redis                ~1 ms
6. response returns to edge, then to user        ~15 ms
                                                  ------
                                                  ~48 ms, worst case, once

And the same request, moments later, from a different user reading the same freshly-populated cache entry:

the next 999 requests for the same content
1. edge PoP — HIT                                 ~1 ms
                                                  ------
                                                  ~1 ms, every time after the first

Forty-eight milliseconds once, one millisecond nine hundred ninety-nine times after — and every layer in this lesson exists to make that ratio as lopsided as possible, for as much of your traffic as it honestly can be.

What this costs, against the alternative from Chapter 0

Chapter 0 raised, and set aside, the option of simply buying bigger boxes — vertical scaling. It is worth closing the loop on that with real numbers, now that the full assembled architecture has a shape to price. Ballpark monthly figures for a system at this traffic level:

Compare against trying to reach 500,000 QPS on Chapter 0’s vertical-scaling path alone: even granting an unrealistically generous 64-core box at 32,000 QPS of capacity, holding the full target would need roughly sixteen of them running the simple 2ms query — and that estimate ignores that a query touching real, larger data (Chapter 7’s 15ms join) would need the fan-out architecture regardless. The caching layers are not a nice-to-have layered on top of an otherwise-sufficient database tier; they are the reason the database tier can stay small enough to be affordable at all.

What we built, and what it is worth

None of these layers is optional past a certain scale, and none of them is free below it — every one adds an operational surface (a replica to monitor, a lock to tune, a cache to keep warm) in exchange for QPS the single box in Chapter 0 could never have served. The judgment call in real systems is not whether to add these layers, but when — and the arithmetic in this lesson is exactly what tells you when a layer is worth its own operational cost and when it is solving a problem you do not have yet.

A small, single-Postgres-box feed app at a few hundred users does not need any of this — the 8-core box from Chapter 0 comfortably outpaces its traffic, and every layer this lesson built would be pure operational overhead with no traffic to justify it. The value of having derived every number by hand, rather than having simply been told “add a CDN, add Redis, add replicas,” is that the arithmetic tells you exactly when that stops being true for a specific system, instead of leaving it as a guess.

→ Scaling Writes — the write-side twin of this lesson: sharding, partition keys, and the fan-out-on-write cost this chapter deliberately deferred
→ Replication — the deeper mechanics of leader-follower replication this lesson took as given in Chapter 1
→ Semantic Caching — a very different kind of cache, where a hit can be wrong even when the key matches
→ Vector Databases — another read-path specialization, for similarity search instead of exact lookup