Async Work & Long-Running Tasks

Chapter 1: Task Queue Anatomy

Chapter 0 ended on a hand-off: instead of transcoding inline, the handler calls enqueue_job and returns. That one function call is doing a lot of quiet work, and this chapter opens it up. A task queue has exactly three moving parts, and nearly every bug in an async system traces back to a misunderstanding of how one of them hands off to the next.

The producer’s job ends the instant the message is durably stored — that is the 5ms Chapter 0 measured. The broker’s entire reason to exist is to hold that message safely between “the producer wrote it” and “a worker finished it,” a gap that can be milliseconds or, if every worker is busy, minutes. The worker pool is where this lesson’s remaining chapters spend almost all their time, because it is where every interesting failure mode lives.

Two promises a broker can make, and why they are different

When a worker claims a message and then crashes before finishing, what happens to the job? The answer is a design decision the broker makes, not a law of nature, and it has a name:

For a user-uploaded video, losing the job silently is unacceptable — the customer paid for a transcode they will never get, and nothing in the system will ever tell anyone it happened. That rules out at-most-once for this use case outright. The rest of this lesson assumes at-least-once delivery, which trades “might lose work” for “might do work twice” — a trade this chapter now shows you exactly how to get wrong, and how to get right.

How at-least-once actually works: the visibility timeout

A broker cannot know a worker crashed — there is no shared memory, no synchronous call that would fail. What it can do is a much cruder but effective trick: when a worker claims a message, the broker hides it from everyone else for a fixed window called the visibility timeout (VT). If the worker finishes and explicitly deletes the message before the VT expires, the job is done and gone. If the VT expires first — because the worker crashed, hung, or is simply still working — the broker assumes the worst and makes the message visible again for another worker to claim.

Read that condition again: the broker cannot distinguish “the worker crashed” from “the worker is still legitimately working and just hasn’t finished yet.” Both look identical from the broker’s side — the message is still unacknowledged when the clock runs out. That ambiguity is not a bug in the design; it is the entire mechanism, and it is exactly where the next section’s bug comes from.

Deriving the redelivery bug by hand

Suppose the visibility timeout is configured the way it often is in practice: copied from a different, faster job type, without checking it against the job actually running. Set VT = 300 seconds (five minutes) and recall Chapter 0’s transcode job takes 600 seconds (ten minutes) on average, with a p99 up to 900 seconds for large 4K files. Walk the timeline of a single job by hand:

Nothing crashed. Nothing errored. Every individual step behaved exactly as specified. The bug is purely a timing mismatch: visibility timeout (300s) < actual work duration (600–900s), and the broker’s only defense against a truly crashed worker — redelivering the message — fires against a worker that was never actually in trouble.

What this actually costs, in dollars

This is not a theoretical inefficiency. Say the bug fires on roughly 1% of a platform doing 500,000 transcodes/day — the large, slow jobs are disproportionately the ones that exceed a too-short VT, so 1% is a conservative estimate for a platform that has not yet fixed this:

Each collision wastes roughly the overlap window — 300 seconds of genuinely duplicated compute, from the timeline above:

At a modest $0.04/worker-hour for CPU-bound transcode compute:

Five hundred dollars a month is not the number that should alarm you here — it is the shape of the cost: silent, proportional to traffic, invisible on any dashboard that only tracks “jobs completed,” and it grows linearly with platform scale forever until someone reads a message-redelivery metric and asks why it is nonzero.

Two fixes, and why one is strictly better

The obvious fix is to set the visibility timeout above the worst case: VT = 1,200s, roughly 2× the 900s p99. This works — the collision in the timeline above cannot happen anymore, because the message stays hidden for longer than any real job takes. But it trades one problem for a subtler one: if a worker genuinely crashes at t=10s into a job, the broker has no way to know that until the full 1,200 seconds elapse. A real crash now sits undetected for up to twenty minutes before anyone retries it.

The better fix decouples “how long can this job legitimately run” from “how fast should we detect a real crash” — a heartbeat. The worker periodically tells the broker “I am still alive, extend my claim,” and the broker resets a much shorter timer each time:

python
def process_transcode_job(msg, broker):
    heartbeat_interval = 60   # seconds
    last_beat = time.time()
    for chunk in encode_chunks(msg.video_id):
        process_chunk(chunk)
        if time.time() - last_beat > heartbeat_interval:
            broker.extend_visibility(msg, 180)   # reset to 180s from NOW
            last_beat = time.time()
    broker.delete(msg)   # job succeeded — release the message for good

With a 180s reset window and a heartbeat every 60s, a genuinely crashed worker is discovered within roughly 180 seconds — three missed heartbeats — regardless of whether the job’s total duration is 10 seconds or 10,000. A legitimately slow job never trips the timer at all, because it keeps proving it is still alive. The static-timeout fix is a single number gambled against the worst case; the heartbeat fix makes the timeout track reality.

What is actually inside the message

It is worth seeing the payload the producer writes, because it previews two ideas the rest of this lesson depends on: a durable job identity, and a forward reference to Chapter 5’s idempotency key.

json — the message body
{
  "job_id": "job_8f2a1c",
  "job_type": "transcode",
  "video_id": "vid_5591",
  "resolutions": ["1080p", "720p", "480p"],
  "idempotency_key": "upload_9931_attempt_1",
  "created_at": "2026-08-18T14:02:11Z",
  "attempt_count": 0
}

Every field above earns its place in a later chapter: job_id is what the state machine in Chapter 3 keys its compare-and-swap on, idempotency_key is what Chapter 5’s dedup window checks, and attempt_count is what Chapter 4’s retry cap reads before deciding whether to try again or move the job to a dead-letter queue. None of it is decoration — a task queue message is really a small, durable record of intent that has to survive being read more than once.

Brokers differ in how literally they implement this contract

“The broker” has been treated as one abstract box so far, but real brokers make different tradeoffs in how they honor at-least-once delivery, and picking one is picking a set of defaults worth knowing before you inherit them:

Kafka is worth a specific note: it does not have a “visibility timeout” concept at all in the SQS sense, because it is fundamentally a log, not a queue — a consumer simply resumes from the last offset it committed. The same collision from this chapter can still happen (a consumer crashes before committing its offset, another consumer resumes from the earlier, uncommitted offset and reprocesses the same records), it just wears a different name: consumer offset lag instead of visibility timeout expiry. The underlying lesson — a gap between “claimed” and “durably acknowledged” is where duplicates live — transfers directly.

Ack-early vs. ack-late, in code

Section one of this chapter named the two delivery guarantees; here is the one-line difference between them, because it really is one line, and it is worth being able to spot at a glance in a code review.

python — at-most-once (ack early — do not use for this job)
def worker_loop_unsafe(broker):
    msg = broker.receive()
    broker.delete(msg)              # acked BEFORE the work — if the process dies right here, the job is gone forever
    transcode(msg.video_id)

python — at-least-once (ack late — what this lesson builds on)
def worker_loop_safe(broker):
    msg = broker.receive()
    transcode(msg.video_id)         # if the process dies here, the message is still claimed —
                                     # VT eventually expires and another worker retries it
    broker.delete(msg)              # acked only AFTER success

Price the silent-loss risk of the unsafe version with a number: if worker processes crash mid-job at a rate of 0.01% of jobs (a realistic figure for infrastructure failures, OOM kills, and deploys landing mid-request), and the platform runs 500,000 transcodes/day:

Fifty customers a day whose upload simply vanishes, with no error, no retry, and no log entry anyone would think to look for — because from the broker’s point of view, that message was deleted successfully. Ack-late converts every one of those 50 into a redelivered, eventually-completed job instead, at the cost of the collision risk this chapter spends the rest of its time managing.

The broker itself is not infallible either

Everything above assumes the broker is a reliable, durable store once a message is written to it — but that durability is itself an engineering choice, not a given. A broker that acknowledges a producer’s write before the message is replicated to more than one node can lose that message if the single node holding it crashes before replication completes. Production brokers guard against this with a replication factor — SQS replicates across multiple availability zones transparently; a self-hosted RabbitMQ or Kafka cluster needs this configured explicitly (Kafka’s min.insync.replicas, for instance). The visibility-timeout mechanics this chapter derives assume the message survives to be redelivered at all — a broker configured for single-node durability quietly removes that assumption, and no amount of correct worker-side logic recovers a message the broker itself lost.

Why not just disable redelivery entirely?

A tempting shortcut: if redelivery is the source of every collision in this chapter, why not simply turn it off — set an effectively infinite visibility timeout and let a crashed job stay claimed forever? The reasoning is worth walking through explicitly, because it is exactly backwards from what it looks like at first glance.

A worker that crashes at t=10 seconds into a 600-second job, with visibility timeout disabled, leaves that message permanently claimed and permanently unprocessed. Nothing will ever redeliver it, because redelivery is the mechanism being disabled. The customer’s video never transcodes, with no error, no retry, and no signal anywhere that anything went wrong — which is precisely the at-most-once failure mode Chapter 1 opened by ruling out. Disabling redelivery does not remove the tradeoff between losing work and duplicating work; it just silently picks “lose work” and hides that choice inside an infrastructure setting nobody thought of as a design decision.

There is no version of this tradeoff where both risks disappear at once, for the same reason a broker cannot distinguish a crashed worker from a slow one: the only information it has is silence, and silence is genuinely ambiguous between the two. Every remaining chapter in this lesson is really about narrowing that ambiguity’s cost — a correctly-sized timeout narrows how often it misfires, an idempotent handler narrows how much a misfire costs, and a state machine narrows how far a misfire can propagate before something notices and stops it.

Chapter 2: Sizing the Worker Pool

Chapter 0 sized an API worker pool for fast requests. Now do the same exercise for the background worker pool that actually runs the jobs — and this time, zoom out from the single 10-minute transcode example to the shape of a real production queue, which is never just one job type.

A real queue is a blend, not a single number

A video platform’s task queue does not only carry 10-minute full transcodes. It also carries 5-second thumbnail extractions, 10-second audio waveform generations, and 15-second preview-clip generations — and in volume, those small, fast jobs vastly outnumber the rare full transcode. Sizing a worker pool means sizing it for the blended average across the whole mix, not for Chapter 1’s single worst-case job type. Take the platform’s measured numbers:

Little’s Law, the same formula from Chapter 0, converts that directly into how many jobs must be in flight — queued or actively running — at any given instant:

From “jobs in flight” to “workers needed”

If every worker processes exactly one job at a time serially, the pool needs at least as many workers as there are jobs in flight simultaneously — 3,000 — just to keep up with arrivals on average. But “exactly at the average” means utilization ρ = 1, and Chapter 0 of the companion Scaling Reads lesson already showed what happens at ρ = 1: latency stops being a number and becomes an ever-growing queue. A worker pool needs headroom for the same reason a database replica fleet does.

Three thousand seven hundred fifty container instances, each capable of running one job at a time. At roughly $0.02/worker-hour for a lightweight, mostly-I/O-bound instance:

That number is the real cost of undersizing or oversizing this decision by even a small margin — it is worth deriving by hand rather than picking round numbers, because the next section shows exactly how much latency you buy or lose per worker added.

The utilization curve, transplanted from database boxes to worker pools

A worker pool behaves like the M/M/1 queue from the Scaling Reads lesson, just with c parallel servers instead of one. As an honest approximation — the exact result for c parallel servers (the Erlang C formula) is more involved and outside this lesson’s scope, but the shape is the same — utilization is offered load divided by total capacity:

and expected wait grows the same way it did for a single database box, worse as ρ climbs toward 1:

Work the table by hand at several pool sizes, holding L = 3,000 fixed:

The curve is exactly the shape Chapter 0 of the Scaling Reads lesson derived: flat, then bending, then vertical as ρ approaches 1. The 3,750-worker recommendation from the previous section sits at ρ = 0.8, giving a 150-second expected queue wait on top of whatever the job itself takes — a reasonable number to show a user as “processing, usually ready within a few minutes,” and a genuine, quantified design choice rather than a guess.

Why not just massively overprovision?

If more workers always means less waiting, why not run 20,000 workers and make ρ vanishingly small? Two honest reasons, both concrete. First, the table above already shows diminishing returns — from 3,750 to 6,000 workers, wait drops from 150s to 60s (a 90-second improvement), but the next equivalent jump, from 6,000 to roughly 10,000, buys back only about another 30–40 seconds, for 60% more infrastructure spend. Second, most of those idle workers sit polling an empty queue most of the time, and idle compute is not free — it is simply money spent on latency you may not need, at a rate that only gets worse as the pool grows past the point where the queue is already comfortably drained.

The honest answer is: size for the wait you are willing to show a user, verify it against real traffic percentiles (not just the average λ and W used here), and revisit the number as traffic grows — exactly the same discipline the read-scaling lesson applies to replica counts.

Sizing for the average quietly under-provisions for the peak

The 100 jobs/second figure is a daily average, and the same lesson the Scaling Reads companion lesson teaches about read traffic applies here without modification: nobody experiences the average. Daytime usage — more uploads, more thumbnail regenerations as more users browse — commonly runs at something like 1.8× the daily average for a consumer platform. Recompute L and the recommended pool at that peak, holding the blended W = 30s fixed:

A pool sized only from the average — 3,750 workers — sitting under peak traffic of 5,400 jobs in flight is a pool at ρ = 5,400/3,750 = 1.44: past Chapter 0’s wall, during the exact hours the platform is busiest and most visible. Provisioning has to target the peak, with the average serving only as the baseline the multiplier is applied to.

Autoscaling: closing the gap without paying for peak capacity all day

Running 6,750 workers around the clock to cover a peak that lasts a few hours a day wastes money the other twenty hours. The standard fix is a reactive autoscaler: watch queue depth, and add workers when it crosses a threshold.

python — a minimal reactive scale-out rule
def maybe_scale_out(queue_depth, current_workers, target_wait_s=150, avg_duration_s=30):
    # if the current backlog alone would take longer than the target wait to drain,
    # add enough workers to bring drain time back under target
    drain_time = queue_depth * avg_duration_s / current_workers
    if drain_time > target_wait_s:
        needed = math.ceil(queue_depth * avg_duration_s / target_wait_s)
        scale_to(needed)

Autoscaling has its own lag worth costing out: if a new worker container takes 90 seconds to boot (pull the image, start the process, pass a health check), the queue keeps growing at the full arrival rate for that entire window before relief arrives. At the peak arrival rate of 180/s:

That number argues for two things at once: scale-out triggers should fire on a leading indicator (queue depth trending up) rather than a lagging one (wait time already breached), and a platform with a hard latency SLA should keep a standing floor of workers sized well above the trough traffic, autoscaling only the margin between the floor and the peak — never scaling from zero.

Not every worker is a full machine

The 3,750-worker figure has been treated as 3,750 physical or virtual machines, one job each. For I/O-bound job types — ones that spend most of their time waiting on a network call rather than burning CPU — a single host can run many jobs concurrently using asynchronous I/O, without needing one host per job.

This does not change Chapter 2’s core sizing arithmetic at all — the pool still needs to sustain 3,750 units of concurrent processing capacity to hold ρ at 0.8 — it only changes how many physical machines deliver that capacity, which changes the dollar figure computed earlier. CPU-bound transcode work (the actual video encoding) does not get this discount; only the surrounding I/O — downloads, uploads, status writes — does. Real pipelines often split job types onto separate pools for exactly this reason: a CPU-bound transcode pool sized one host per worker, and a lighter I/O-bound pool (thumbnail fetches, status callbacks) sized with concurrency.

Cost per job, not just cost per month

The $54,750/month figure from earlier becomes more actionable divided down to a per-job unit economics number, which is what a pricing or margin conversation actually needs:

A fraction of a cent per job in pure compute — small in isolation, but the number that makes Chapter 1’s $500/month redelivery waste and Chapter 4’s poison-pill leak worth catching: at this scale, inefficiencies that look small in dollars are still real percentages of a real infrastructure line item, and they compound as the platform grows.

Scale-in matters as much as scale-out

Autoscaling discussion tends to focus entirely on adding capacity; removing it correctly matters just as much, and gets it wrong in a distinctly dangerous way if done naively. A worker that is mid-job when the autoscaler decides to terminate it for being “idle capacity” is not idle — and killing it recreates Chapter 1’s exact collision, just triggered by infrastructure automation instead of a network blip.

python — scale-in that respects in-flight work
def maybe_scale_in(current_workers, target_workers, workers):
    excess = current_workers - target_workers
    if excess <= 0:
        return
    idle = [w for w in workers if not w.busy]
    for w in idle[:excess]:
        terminate(w)                      # only ever terminate workers with no claimed message
    # if fewer idle workers exist than the excess demands, mark the rest
    # "draining" — accept no new claims, terminate once their current job completes

The “draining” state in the comment is the detail that actually matters: a worker told to scale in should stop claiming new messages immediately but be allowed to finish whatever it already claimed, so its in-flight job’s natural completion — not a forced termination — is what releases the message. This is the same principle Chapter 1 built the heartbeat pattern around, applied to a different trigger for the same underlying risk: any process that might disappear mid-job needs the system around it to assume that is possible and plan for the message to survive it.

A note on measuring W in production, not assuming it

Every number in this chapter treated W — the 30-second blended average duration — as a given. In a real system it is a measured quantity, and it is worth being specific about how: instrument the moment a worker claims a message and the moment it deletes it (success) or moves it to the DLQ (final failure), and record that interval per job type. The blended average feeding Little’s Law is the traffic-weighted mean across every job type’s own measured distribution, not a single number anyone should type into a spreadsheet once and forget:

Recomputed automatically as job-type mix shifts — more full transcodes relative to thumbnails as video length trends up, say — this keeps the worker-pool sizing in Chapter 2 honest against reality instead of quietly drifting stale against a number someone measured once, a year ago, before the product changed.

What the pool size does NOT protect against

It is worth being precise about the boundary of what Little’s Law sizing actually buys. A correctly-sized pool keeps ρ comfortably under 1 for the traffic pattern it was measured against — it says nothing about an individual job that misbehaves once it is running. A worker stuck in an infinite loop on a malformed input, or one silently leaking memory across jobs until the host is killed by the OS, consumes its slot in the pool the same as a healthy job would, and no amount of correct capacity planning catches it. That is a different problem — a job-level correctness and resource-limit problem — and it is why real worker processes are typically wrapped in their own hard per-job execution timeout, independent of and usually tighter than the broker’s own visibility timeout, specifically to bound the damage a single misbehaving job can do to the rest of the pool’s capacity.

That per-job timeout and the broker’s visibility timeout are deliberately not the same setting, even though both bound how long a single job can run: the per-job timeout is a correctness backstop enforced by the worker process itself (kill the job, report a failure, free the slot), while the visibility timeout is the broker’s independent assumption about when to consider a claim abandoned. Keeping them distinct, with the per-job timeout set comfortably tighter, means a genuinely stuck job gets caught and freed by its own process before the broker ever has to guess about it at all.

Put the three timeouts this lesson eventually derives — per-job execution timeout, visibility timeout, and lock TTL — on one mental timeline and the ordering explains why each exists separately: the per-job timeout should trip first, catching a stuck job at its own worst case; the visibility timeout should trip second, comfortably after, catching a worker that vanished entirely without even the chance to hit its own timeout; and nothing should ever need to wait for the third.

Ordering them this way also determines which alert actually pages someone. A per-job timeout firing is an application-level signal — log it, count it, but a single occurrence rarely needs a human at 2 a.m. A visibility-timeout-triggered redelivery firing at any real rate, by contrast, means workers are disappearing without even the grace of their own timeout catching it first — a stronger signal, worth a real alert, because it points at infrastructure instability rather than one job’s own bad input.

Chapter 3: Status Delivery

The upload handler now returns in 55 milliseconds with a job ID and a 202 Accepted. The user is still staring at a progress bar. Somehow, the client has to find out when the job actually finishes — and the naive way to find out, just like the naive way to transcode, is the one that looks obvious and quietly costs the most.

Polling: the obvious answer, priced by hand

The client app calls GET /jobs/{id}/status every couple of seconds until it sees done. Simple to implement on both ends. Price it against Chapter 2’s numbers: roughly 3,000 jobs are in flight at any instant (Chapter 2’s L), and each one has a client polling for its status.

Fifteen hundred requests a second, and the overwhelming majority of them return the exact same answer as the poll before: “still running.” Compare that to the rate at which jobs actually finish — which, at steady state, equals the arrival rate λ from Chapter 2:

Fifteen hundred polls a second to learn about one hundred completions a second — a 15× overhead, almost all of it wasted work answering “nothing changed.” Widen the poll interval and the ratio improves, but never to parity, and every widening trades responsiveness for it:

Webhooks: push instead of pull

The alternative flips who initiates contact. The client registers a callback URL; when a worker finishes a job, it makes one outbound HTTP call to that URL, once, with the result. No polling infrastructure on the status-check path at all — the request volume on that path drops to the completion rate itself, 100/s, a flat 15× reduction versus 2-second polling, with zero added latency (the client learns the instant the job is actually done, not up to one poll-interval late).

The cost moves rather than vanishes, and it is worth naming honestly: the worker now needs outbound network access to arbitrary client-supplied URLs (a real security surface — validate and sandbox those callbacks), and it needs its own retry logic if the callback fails (the webhook delivery is itself a small instance of the at-least-once problem from Chapter 1). Webhooks trade infrastructure simplicity for a new, smaller failure mode to manage.

WebSockets: when the client cannot receive a callback

A browser tab has no public URL a worker can call. For that case, a persistent connection — a WebSocket, held open between the client and a connection-serving tier — lets the server push the same completion event the instant it happens, without either polling or an inbound callback URL. Memory cost, at Chapter 2’s watched-job count and a typical ~10KB per open connection:

The real cost of this option is not memory, it is the stateful connection tier it requires — sticky routing, reconnection handling, a consistent-hash ring of socket servers if you scale past one box. That machinery is the entire subject of the companion Realtime Updates lesson; this lesson borrows only the conclusion: sockets are the right choice when the client cannot be reached any other way, and the wrong choice to reach for by default when a webhook or a modest poll interval would do.

The job-state machine underneath all three

Whichever delivery mechanism the client uses, it is reading the same underlying object: a job whose state moves through a small, fixed set of values.

Chapter 1’s redelivery bug puts two workers into the running transition for the same job at once. If both are allowed to write running unconditionally, the state machine cannot tell that happened. The fix is a database pattern called compare-and-swap: the transition only succeeds if the row is still in the state the worker expects it to be in.

sql
-- only the worker that finds the row still 'queued' wins the transition
UPDATE jobs
SET status = 'running', worker_id = $1, started_at = now()
WHERE id = $2 AND status = 'queued';
-- returns 0 rows affected if someone else already claimed it — the loser checks
-- the row count and backs off instead of doing 300 seconds of wasted work

This is a second, independent line of defense against Chapter 1’s exact collision: even if the visibility timeout is misconfigured and both Worker A and Worker B claim the same message, only one of them wins the UPDATE and is allowed to proceed as the authoritative processor. The loser sees zero rows affected and can abort immediately — turning a 300-second wasted transcode into a wasted database round trip instead.

Long polling: a middle option worth naming

Between plain polling and a persistent socket sits a technique that gets less attention than either: long polling. The client still issues an ordinary HTTP GET, but the server does not answer immediately — it holds the connection open, waiting up to some maximum duration (say 25 seconds) for the job to change state, and answers the instant it does, or with a “still running” timeout response if the window elapses first.

python — a long-poll status endpoint
def get_status_long_poll(job_id, db, max_wait_s=25):
    deadline = time.time() + max_wait_s
    while time.time() < deadline:
        job = db.get_job(job_id)
        if job.status in ("done", "failed"):
            return job, 200
        time.sleep(0.5)                 # check locally every 500ms, well under the connection's own cost
    return {"status": "running"}, 200   # window elapsed — client immediately reconnects

The client-visible effect is close to a webhook’s responsiveness — it learns about completion within roughly 500ms rather than a full poll interval — while requiring no outbound callback infrastructure on the server and no persistent connection tier. The real cost moves to the server, which now holds open connections for up to 25 seconds each instead of answering in milliseconds; sizing that connection-holding capacity is a smaller version of Chapter 2’s own worker-pool arithmetic, just applied to HTTP handler threads instead of job workers.

Caching headers cut even ordinary polling’s cost

Plain 2-second polling can shed real bandwidth without changing its interval at all, using HTTP’s own conditional-request machinery. The server returns an ETag identifying the current status; the client sends it back on the next poll via If-None-Match, and if nothing changed, the server answers 304 Not Modified with an empty body instead of resending the full status object.

At the 1,500 polls/second this chapter derived, and the overwhelming majority returning “nothing changed”:

This does not touch the request-count overhead — the server still has to process 1,500 requests/second either way — but it materially cuts payload bandwidth for free, and it is a one-line addition (ETag + a conditional check) to an endpoint that already exists.

The full state machine, with a retrying state

Chapter 3’s three-state sketch (queued → running → done/failed) hides one detail Chapter 4 needs: a failed attempt that is about to be retried is not the same as a final failure, and conflating them would report a job as “failed” to the client one attempt before it actually succeeds.

That last invalid row is worth dwelling on: it is exactly Chapter 1’s collision, caught by the state machine instead of prevented by timing. Even if a visibility-timeout misconfiguration lets a redelivered message reach a worker after the original already finished, the CAS-guarded transition from done back to running simply fails — zero rows updated — and the redundant worker can detect that and exit immediately rather than doing any wasted work at all.

A complete status endpoint

python
def get_job_status(job_id, db):
    job = db.get_job(job_id)
    if job is None:
        return {"error": "not found"}, 404
    body = {
        "job_id": job.id,
        "status": job.status,               # queued | running | retrying | done | failed
        "attempt_count": job.attempt_count,
        "result_url": job.result_url if job.status == "done" else None,
    }
    etag = hashlib.sha1(json.dumps(body, sort_keys=True).encode()).hexdigest()
    return body, 200, {"ETag": etag}

Webhook delivery is itself an at-least-once problem

It is worth closing the loop on a point raised earlier and left open: a webhook call can fail — the client’s server is down, a network blip, a 500 from their endpoint. The worker that fires it faces the exact same delivery-guarantee choice Chapter 1 named for the broker itself.

python
def deliver_webhook(job, client_url, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            resp = http_post(client_url, json=job.to_dict(), timeout=5)
            if resp.status_code < 300:
                return                          # delivered — done
        except NetworkError:
            pass
        time.sleep(random.uniform(0, 1 * 2**attempt))   # Chapter 4's jittered backoff, reused here
    mark_webhook_failed(job)                # give up — client falls back to polling the status endpoint

Note the fallback in the last line: a well-designed status-delivery system never relies on the webhook alone. The status endpoint from Chapter 3’s state machine stays available regardless, so a client whose webhook silently failed to deliver can always fall back to a single poll to find out what actually happened — push is an optimization on top of pull, not a replacement for it.

Choosing between the three, by client type

A hybrid most real products actually ship

Production systems rarely pick exactly one of these and stop. A typical upload flow uses all three, layered by how long the user has been waiting: WebSocket or long polling while the tab stays open and the job is expected to finish soon; a fallback to ordinary polling at a wide interval if the connection drops but the tab is still open; and a push notification or email if the job is still running once the user has closed the tab entirely, so the platform can still reach them without holding any connection open at all. Each layer covers the gap the one before it leaves, which is the same layered-defense shape Chapter 5 uses for idempotency — no single mechanism has to be perfect on its own.

Building all three layers at once is rarely the right order of operations for a real team, though, and it is worth saying so plainly: plain polling at a modest interval is enough to ship a correct, if not maximally efficient, first version of any status-delivery feature. Webhooks and sockets are optimizations layered on afterward, once the QPS or responsiveness cost of polling alone is actually measured and found wanting — not a mandatory starting point. The state machine underneath, by contrast, is not optional at any stage; it is the correctness layer every delivery mechanism reads from, and it is worth getting right before any of the three transport choices above it.

That ordering — correctness first, transport efficiency second — is worth stating as a general rule for this entire lesson, not just this chapter: every optimization from here on (jittered backoff, a dedup window, a weighted queue, a distributed lock) sits on top of the state machine and the delivery guarantee this lesson built in its first three chapters. None of the later chapters make sense, or are even safe to add, on top of a system that gets those first three wrong.

It is a useful test to apply to any proposed addition to a system like this one: ask whether the new piece assumes the state machine is already correct, or whether it is quietly trying to compensate for the state machine being wrong. A caching layer in front of the status endpoint is the first kind — a reasonable optimization. A client-side workaround that polls twice and takes whichever answer arrives first, because status reads are sometimes seen to be stale, is the second kind — a symptom that the underlying state transitions are not actually being written and read consistently, and the fix belongs in Chapter 3’s layer, not in every client that talks to it.

Chapter 4: Retries & Backoff

Workers fail. A transient network blip, a downstream storage API returning a 503, a brief outage in the object-storage service every worker writes its output to. At-least-once delivery (Chapter 1) already guarantees a failed job gets tried again — this chapter is about the difference between a retry policy that quietly heals the system and one that actively prevents it from recovering.

The setup: a shared dependency goes down

Every transcode job’s last step uploads its output to object storage. That storage API has a brief outage — ten seconds — and at the moment it goes down, 1,000 workers happen to be mid-call to it, because that is roughly how many are active at Chapter 2’s steady state. All 1,000 calls fail within the same short window, essentially simultaneously.

Naive retry: fixed exponential backoff, no jitter

The standard fix for a transient failure is exponential backoff — wait longer between each successive retry, so a brief outage does not get hammered by an immediate resubmission:

Work out the schedule by hand for the first few attempts:

Here is the problem: every one of the 1,000 workers failed at nearly the same instant and is running the exact same deterministic formula. All 1,000 retry at t=1s. All 1,000 retry again at t=3s. Again at t=7s. Each of those is a synchronized spike of 1,000 simultaneous requests slamming a storage service that may have just started to recover — a thundering herd, identical in shape to the cache-stampede problem in the Scaling Reads lesson, except this time the herd is retries instead of cache misses, and it can knock a recovering dependency back down, prolonging the very outage it was trying to survive.

Jitter: spread the spike instead of removing it

The fix does not change how many retries happen — it changes when, by adding randomness to the delay instead of using a fixed value:

This is called full jitter. For the first retry, instead of all 1,000 workers firing at exactly t=1s, each one independently picks a delay uniformly between 0 and 1 second. Slice that window into 100 buckets of 10 milliseconds each and count the expected workers per bucket:

Same total number of retries, same eventual recovery — but the peak simultaneous load on the recovering dependency drops by roughly 100×. Ten requests in any given 10-millisecond window looks like ordinary traffic to the storage API; a thousand at once looks like a second outage.

Poison pills and the dead-letter queue

Backoff assumes the failure is transient — the dependency will recover, so retrying later helps. Some jobs fail for reasons that will never resolve no matter how many times or how patiently you retry: a corrupted upload with an unparseable video container, a file that claims to be MP4 but is actually a zip bomb. Retrying one of these forever, with no upper bound, is called a poison pill, and it is a slow, quiet capacity leak.

Suppose poison pills are 0.1% of the arrival stream at Chapter 2’s λ = 100/s:

Without a retry cap, each one occupies a worker slot in an endless claim-fail-redeliver loop. After one day:

The fix is a bounded attempt count — say, max_attempts = 5, matching the backoff schedule derived above — after which the job is moved out of the live queue entirely and into a separate dead-letter queue (DLQ) for a human to inspect.

python
def handle_failure(msg, attempt_count, broker):
    if attempt_count >= 5:
        broker.move_to_dlq(msg, reason="max attempts exceeded")
        broker.delete(msg)          # stop it from ever being redelivered to the live queue
    else:
        delay = random.uniform(0, 1 * 2**(attempt_count-1))
        broker.retry_after(msg, delay)

Each poison pill now costs exactly five wasted attempts — roughly 31 seconds of backoff schedule plus five attempt durations — then stops consuming live-queue capacity forever, instead of consuming it forever, growing without bound.

Linear vs. exponential backoff, worked side by side

Exponential backoff is not the only schedule available — it is worth comparing against the simpler alternative, linear backoff (delay grows by a fixed increment each attempt rather than doubling), to see exactly what the exponent buys.

The two schedules land at nearly the same total wait by attempt 5 in this example, but they get there differently: exponential backs off far more aggressively on later attempts (16s vs 10s), which matters specifically when the failure is a sustained outage rather than a one-off blip — it stops hammering a dependency that has now been down for a while, where linear backoff keeps retrying at a nearly constant, still-meaningful rate. Exponential is the better default specifically because failure duration is unknown in advance; it adapts its own aggressiveness downward the longer the failure persists, without anyone having to guess how long an outage will last.

A retry budget, not just a retry count

Chapter 4’s cap so far is a count — 5 attempts. That is not quite the same guarantee as a bound on wall-clock time, and the difference matters when individual attempts themselves can be slow. Consider a job whose attempts occasionally hang for 60 seconds before timing out, rather than failing fast:

A user watching that job’s status waits over five minutes to be told it failed, even though the retry count policy was followed correctly the whole time. A retry budget — a hard wall-clock ceiling independent of attempt count — closes that gap:

python
def handle_failure(msg, attempt_count, first_attempt_at, broker, max_attempts=5, max_wall_s=120):
    elapsed = time.time() - first_attempt_at
    if attempt_count >= max_attempts or elapsed > max_wall_s:
        broker.move_to_dlq(msg, reason=f"attempts={attempt_count} elapsed={elapsed:.0f}s")
        broker.delete(msg)
    else:
        delay = random.uniform(0, 1 * 2**(attempt_count-1))
        broker.retry_after(msg, delay)

Whichever bound trips first — five attempts, or two minutes of real time — sends the job to the DLQ. This gives client-facing code a genuine upper bound on “how long before I know this failed for good,” which a pure attempt-count cap cannot promise on its own when individual attempt durations are unpredictable.

Circuit breakers: stop trying instead of trying more slowly

Backoff and jitter both assume it is worth retrying, just more carefully. A circuit breaker is the pattern for when it stops being worth trying at all, for a while: after enough consecutive failures against a dependency, stop sending it traffic entirely — fail fast, locally, without even attempting the call — for a cooldown period, then send one experimental “probe” request to check if it has recovered.

The number worth deriving here is what a circuit breaker saves during a real outage that a pure retry-with-jitter policy would not: without a breaker, all 1,000 workers from the retry-storm example keep attempting calls — jittered, so not synchronized, but still a real, continuous trickle of doomed requests — for as long as their individual backoff schedules allow. With a breaker tripped after the first handful of failures, the other several hundred workers fail their very next call attempt in microseconds (a local check, no network round trip) instead of waiting out a network timeout, freeing them to pick up other queued jobs instead of sitting blocked on a dependency everyone already knows is down.

Distinguishing retryable from non-retryable failures

Everything so far assumed the failure was transient — worth retrying. Not every error a job encounters is. A malformed request to a downstream API and a temporary network blip both raise an exception; only one of them will ever succeed on retry, and treating them identically wastes the retry budget on a failure class that backoff cannot fix.

python — routing failures by class, not treating them uniformly
def handle_failure(msg, error, attempt_count, broker):
    if isinstance(error, (NetworkError, RateLimitedError)):
        if attempt_count < 5:
            broker.retry_after(msg, random.uniform(0, 2**attempt_count))
            return
    # non-retryable, OR retries exhausted — either way, this job cannot succeed by trying again
    broker.move_to_dlq(msg, reason=str(error))
    broker.delete(msg)

Sorting failures this way changes the DLQ’s arithmetic from earlier in this chapter for the better: a malformed-input job now costs one attempt before reaching the DLQ instead of five, since there is no reason to spend four wasted backoff cycles on a failure class that will never resolve. The zombie-job accumulation math from earlier — 8,640/day without a cap — assumed every failure got the full retry treatment; classified failure handling shrinks that number further by skipping the retry schedule entirely for the failures it can already recognize as permanent.

Alerting on the DLQ, not just filling it

A dead-letter queue that nobody watches is only a slower version of the zombie-accumulation problem it was built to prevent — the jobs stop consuming live worker capacity, but they also stop being anyone’s problem, silently, forever. The operational half of this chapter’s pattern is an alert on DLQ depth crossing a threshold and, separately, a rate of new arrivals — a sudden spike in DLQ arrivals is frequently the first visible signal of an upstream problem (a corrupted batch upload, a downstream API that started rejecting a previously-valid request shape) well before any other dashboard would show it.

Replaying a DLQ after the root cause is fixed

The dead-letter queue is not a graveyard — it is a holding pen. Once whatever caused a batch of jobs to land there is fixed (a corrupted-upload bug patched, a downstream dependency back online), those jobs are frequently legitimately processable now, and replaying them back into the live queue recovers work that would otherwise be silently lost to a customer who never got their video. A replay tool is a small but important piece of DLQ tooling worth building deliberately rather than leaving as a manual database operation:

python
def replay_dlq(broker, reason_filter=None, limit=1000):
    messages = broker.peek_dlq(limit=limit)
    for msg in messages:
        if reason_filter and reason_filter not in msg.dlq_reason:
            continue
        msg.attempt_count = 0              # reset the retry budget for a fresh attempt
        broker.enqueue(msg)               # back into the live queue
        broker.remove_from_dlq(msg)

Note the reason_filter: replaying selectively, by the failure reason recorded when the job was dead-lettered, avoids re-injecting jobs that failed for a genuinely permanent reason (a truly corrupted file) alongside ones that failed only because of the now-fixed transient issue. Replaying everything indiscriminately just recreates the same DLQ a few minutes later, with the permanent failures bouncing right back.

Notice that replay is only safe because of a property the next chapter formalizes: the handler being replayed into has to be idempotent, since a replayed job is, by definition, being processed a second time relative to whatever partial work its original failed attempt may have already done. Retries, backoff, dead letters, and replay all, underneath, lean on the same single guarantee Chapter 5 is about to make explicit.

Put differently: every mechanism in this chapter controls WHEN and HOW OFTEN a failed job gets tried again. None of them, on their own, control what happens if two of those attempts somehow overlap or land out of order — that is a different question, belonging to the handler itself, and it is exactly where this lesson goes next.

Chapter 5: Idempotency

Every fix so far has been an honest trade, not a magic bullet: at-least-once delivery (Chapter 1) guarantees no job is silently lost, at the cost of possibly running a job more than once — from the redelivery collision itself, or from a client retrying an upload after a 504 (Chapter 0), or from a worker retrying after a transient failure (Chapter 4). This chapter answers the question all three of those chapters deferred: what actually stops a duplicate delivery from becoming duplicate, visible, harmful work?

The exactly-once myth

It is tempting to want a broker that guarantees exactly-once delivery — never lost, never duplicated. No distributed system can honestly promise that for an operation with a real side effect (writing a file, charging a card, sending an email), because the promise would require the sender to know, with certainty, whether the receiver acted on a message before a crash could occur between “receiver did the work” and “receiver confirmed it did the work” — and no amount of protocol cleverness closes that gap for an arbitrary network. What is achievable, and what every system in this lesson actually relies on, is at-least-once delivery of the attempt, combined with a handler written so that trying twice produces the same observable result as trying once. That combination is called effectively-once, and the property the handler needs is idempotency: applying the same operation multiple times has the same effect as applying it a single time.

Making the transcode handler idempotent

The fix is not exotic — it is a deterministic output path plus a status check before starting real work:

python
def process_transcode_job(msg, db, storage):
    job = db.get_job(msg.job_id)
    if job.status == "done":
        return                                    # already finished by another delivery — no-op

    output_path = f"transcoded/{msg.job_id}/output.mp4"   # deterministic — keyed by job_id, not by attempt
    result = transcode(msg.video_id)
    storage.write(output_path, result)               # a duplicate attempt overwrites the SAME path,
                                                        # not a new one — safe by construction
    db.mark_done(msg.job_id)

Two separate design choices are doing the work here. The status check at the top turns most duplicate deliveries into a fast no-op before any real transcoding happens. The deterministic output path means that even if two workers genuinely race past that check — exactly Chapter 1’s collision — the worst outcome is one file getting overwritten by an equivalent copy of itself, not two divergent files at two different paths.

Idempotency keys and sizing the dedup window

The status check above works because the job already has a durable ID. Client-initiated requests need the same protection at the API boundary: if a client’s upload retry (Chapter 0’s 504-then-retry scenario) is treated as a brand-new job, you get a second, fully duplicate transcode. The fix is an idempotency key — a client- or server-generated identifier tied to one logical operation, checked against a short-term store before any new work is created.

How far back does that store need to remember a key? Long enough to cover the worst-case gap between an original attempt and any legitimate duplicate of it. Chain the worst cases from earlier chapters: Chapter 1’s fixed visibility timeout (1,200s) plus the p99 processing time (900s) bounds how late a genuine redelivery-driven duplicate can arrive:

Round up with margin for clock skew and a slow client retry, to a clean 1-hour (3,600s) dedup window. Storage cost of remembering every key for that long, at Chapter 2’s platform-wide λ = 100/s:

Thirty-six megabytes to safely deduplicate an hour’s worth of every job on the platform — trivial for a key-value store like Redis, and cheap insurance against the exact duplicate-charge, duplicate-email class of bug idempotency keys exist to prevent.

A window is a tradeoff, not a free parameter

Widening the window catches more legitimate duplicates but costs more storage and, more subtly, risks a false positive: a genuinely new, unrelated request that happens to reuse an old key (rare, but possible with poorly generated keys) gets incorrectly treated as a duplicate and silently dropped. Narrowing it saves storage but lets slow, legitimate duplicates — a mobile client that retried after being offline for forty minutes — slip through uncaught. The 3,600s figure above is not a universal constant; it is this system’s specific worst-case delivery chain, rounded up. A different visibility timeout or a different p99 changes it, and the chapters that produced those numbers are exactly where you go to recompute it.

Two layers of idempotency, protecting two different boundaries

Everything so far protected the worker’s own processing — a job-id check inside the handler. There is a second boundary, further upstream: the client’s own retry of the original API call from Chapter 0’s 504-then-retry scenario, before a job even exists. That needs its own guard, checked at the API layer, before enqueue_job is ever called.

python — API-layer idempotency check
def handle_upload(request, redis):
    idem_key = request.headers.get("Idempotency-Key")   # client-supplied, one per logical upload attempt
    existing = redis.get(f"idem:{idem_key}")
    if existing:
        return json.loads(existing), 202          # return the SAME response as the original call

    file_bytes = request.files["video"].read()
    video_id = save_to_storage(file_bytes)
    job_id = enqueue_job("transcode", video_id=video_id, idempotency_key=idem_key)
    response = {"video_id": video_id, "job_id": job_id, "status": "queued"}
    redis.setex(f"idem:{idem_key}", 3600, json.dumps(response))   # 1hr window, Ch5's derived size
    return response, 202

This is the layer that catches Chapter 0’s exact scenario — a client that received a 504 and retried — before a second job is even created, which is strictly cheaper than letting a duplicate job reach the worker pool and relying on the job-id check there to catch it. The two layers are complementary, not redundant: the API-layer key catches duplicate-request retries; the job-id check inside the worker catches duplicate deliveries of the same already-created job (Chapter 1’s collision). A system needs both, because they guard different points where duplication can enter.

Not every operation is naturally idempotent

The transcode handler earlier in this chapter is idempotent almost by luck of its shape: writing a file to a deterministic path is naturally safe to repeat, because the second write simply produces the identical file again. Not every side effect a job performs has that property, and conflating the two is a common, expensive mistake.

The right column is the actual engineering task: an operation that is not naturally idempotent needs an explicit guard — a compare-and-swap flag checked before the side effect fires, exactly like Chapter 3’s state-machine pattern applied to one more field.

sql — guarding a non-idempotent side effect
-- only the caller that flips notified from false to true is allowed to send the email
UPDATE jobs
SET notified = true
WHERE id = $1 AND notified = false;
-- 1 row affected → send the email. 0 rows affected → someone already sent it, skip.

This is the same idea as Chapter 3’s job-state CAS and Chapter 1’s at-most-once comparison, generalized into a single rule: any effect that is not safe to repeat needs its own explicit, atomically-checked flag, distinct from the job’s overall status field, because a job can legitimately be retried as a whole while a specific side effect inside it must fire exactly once.

What if the dedup store itself is unavailable?

The idempotency check above depends on Redis being reachable. What should handle_upload do if that specific call times out? There is no answer that is free of risk, and the honest way to choose is to ask which failure is cheaper for this specific operation.

This lesson’s transcode pipeline can safely fail open, because every layer downstream — the deterministic output path, the CAS-guarded state transitions — already makes an uncaught duplicate merely wasteful rather than actually harmful. A payments system wrapping the same pattern around “charge this card” cannot make that same choice, because nothing downstream of a duplicate charge un-charges it.

A worked walk-through: the collision from Chapter 1, fully defended

It is worth tracing Chapter 1’s exact collision one more time, now with every layer this lesson has built standing between the redelivery and a real duplicate side effect, to see concretely how much of it survives.

Four independent layers, each closing a gap the one before it could theoretically miss. This is defense in depth in the same sense a database uses it for security: no single layer is trusted to be perfect, so each layer assumes the one before it might fail and protects the outcome anyway. The visible cost of Chapter 1’s bug, with all four layers in place, shrinks from “a customer gets two emails and a possibly-corrupted file” down to “a worker briefly does redundant compute that a CAS check catches within one database round trip.”

Idempotency keys are not the same as request IDs

A subtle distinction worth being precise about: a request ID (or trace ID) uniquely identifies one HTTP call, generated fresh by the client or a gateway on every attempt — including retries. An idempotency key identifies one logical operation, and a well-behaved client deliberately reuses the SAME key across retries of the same logical attempt. Confusing the two defeats the entire mechanism:

javascript — client-side, correct
const idempotencyKey = crypto.randomUUID();  // generated ONCE per upload attempt
async function uploadWithRetry(file, maxAttempts = 3) {
  for (let i = 0; i < maxAttempts; i++) {
    try {
      return await fetch("/upload", {
        method: "POST", body: file,
        headers: { "Idempotency-Key": idempotencyKey }   // SAME key on every retry
      });
    } catch (e) { await sleep(1000 * 2 ** i); }
  }
}

If the client instead generated a fresh key on every retry attempt (the same mistake as generating a fresh request ID and treating it as an idempotency key), the API-layer check from earlier in this chapter would never find a match — every retry would look like a brand-new logical operation, and the whole mechanism would silently do nothing while looking, on the surface, correctly wired.

The read side needs idempotency too

Everything in this chapter protected a write — a transcode, an email, a charge. Reads inside a job can have the same problem in a quieter form: a job that queries “has this video already been flagged for moderation” and then acts on the answer needs that read to be from a source that reflects its own prior writes, or a redelivered duplicate attempt can make a decision based on stale information from before the first attempt’s writes landed. This is the same read-your-writes concern the companion Scaling Reads lesson names for database replicas, applied here to a worker reading back data it may have itself just written moments earlier under a different attempt.

Testing idempotency deliberately, not hoping for it

Idempotency is exactly the kind of property that looks correct in every normal test run and fails silently the one time a redelivery actually happens in production, months later, on a job nobody thought to watch closely. The only reliable way to trust it is to test the duplicate path directly, on purpose, rather than waiting for a real collision to prove or disprove it:

python — a test that actually exercises the duplicate path
def test_transcode_handler_is_idempotent():
    job = make_test_job()
    process_transcode_job(job)                      # first, "original" attempt
    output_after_first = storage.read(job.output_path)

    process_transcode_job(job)                       # second, "duplicate" attempt — same job, unchanged
    output_after_second = storage.read(job.output_path)

    assert output_after_first == output_after_second   # same result, not a corrupted or doubled one
    assert count_emails_sent(job.user_id) == 1       # the notification guard fired exactly once

Running the handler twice in a row, in a test, and asserting the visible side effects happened exactly once is a small addition to a test suite that catches an entire class of bug this chapter spent its time deriving — before it ever reaches a real duplicate delivery in production.

A useful habit for any new job handler, independent of a formal test suite: run it twice on the same input, by hand, before shipping it. If the second run produces anything different from the first — a second file, a second email, an incremented counter that should have stayed put — that difference is exactly the gap Chapter 1’s redelivery bug will eventually find in production, on its own schedule rather than yours.

Chapter 6: Priorities & Fairness

Everything so far has treated the queue as one undifferentiated stream. Real platforms have tiers — a paying customer’s transcode should not wait behind a flood of free-tier jobs. This chapter takes Chapter 2’s exact 100 jobs/second and splits it by customer tier, then shows, by hand, what a single shared FIFO queue does to the tier that is supposed to be premium.

The split, and the baseline it inherits

Of the platform’s 100 jobs/second, 90 come from the free tier and 10 from paying pro customers. Chapter 2’s pool of 3,750 workers at ρ = 0.8 gave a blended expected wait of 150 seconds for everyone — and in a single shared FIFO queue, a pro job is not treated any differently from a free one, so it waits the same 150 seconds too, despite being only 10% of the volume. That is the problem: fairness by construction (first-in, first-out) produces an outcome nobody actually wants.

Fix one: hard-reserved capacity per tier

Carve the pool into two dedicated sub-pools. Give pro traffic a generously oversized reserved share — 1,000 workers, far more than its 10% traffic share, specifically to buy a strong latency guarantee — and leave the remaining 2,750 for free traffic.

Pro’s wait drops from 150s to 42.9s — a genuine, quantified improvement, purchased by deliberately overprovisioning relative to its traffic share. But look at what happened to free: its wait ballooned from 150s to 27.5 minutes, because it went from sharing the full 3,750-worker pool down to a smaller 2,750-worker pool while carrying nearly the same absolute load. This is the whole tension named plainly: a hard reservation does not create extra capacity out of nothing — it takes capacity from one tier and gives it to another.

Fix two: weighted polling instead of a hard wall

A softer alternative lets both tiers draw from the same shared pool, but biases which queue a newly-free worker checks first. Instead of a hard partition, each worker polls the pro queue with probability 0.7 and the free queue with probability 0.3 on every poll cycle. When one tier is quiet, idle capacity naturally flows to the other — unlike the hard split, which wastes pro-reserved capacity whenever pro traffic is light.

python
def next_job(pro_queue, free_queue, weight_pro=0.7):
    if random.random() < weight_pro and not pro_queue.empty():
        return pro_queue.pop()
    elif not free_queue.empty():
        return free_queue.pop()
    elif not pro_queue.empty():
        return pro_queue.pop()          # fall back if free is also empty
    return None

The weight, w_free = 0.3, is also a bound on how badly free tier can be starved, even in the worst case where pro traffic never lets up: free jobs are guaranteed roughly 30% of polling slots as a floor, not zero. Compare that to a strict priority policy — always serve pro fully before touching free at all — which has no such floor: if the pro queue never empties, the free queue can, in principle, starve indefinitely. Weighted polling trades a little pro-tier optimality for a guaranteed minimum service floor on the lower-priority tier.

Choosing between the two, honestly

Hard reservation gives a precise, provable latency ceiling for the protected tier — useful when a contract promises a specific number (“pro jobs complete within 60 seconds”). Weighted polling gives better overall utilization and a bounded-but-softer starvation guarantee — better when the goal is “pro should generally feel fast” without a hard SLA, and free-tier capacity should not go to waste when pro traffic is quiet. Production systems frequently run both at once: a small hard-reserved floor for pro (so it never fully starves even under a weighted-polling worker crash), plus weighted polling across the remaining shared pool.

Strict priority: the policy with an unbounded worst case

It is worth naming a third policy, if only because it is the one people reach for first and the one with the sharpest failure mode: strict priority, where a worker always serves the pro queue to complete exhaustion before glancing at the free queue at all.

python — strict priority (do not use without a floor)
def next_job_strict(pro_queue, free_queue):
    if not pro_queue.empty():
        return pro_queue.pop()
    return free_queue.pop() if not free_queue.empty() else None

The failure mode is a short, formal argument rather than a measured number: if pro-tier arrivals alone are enough to keep the pool at or above ρ_pro = 1 — a sustained pro traffic spike, or simply enough pro growth over time — the free queue never gets touched at all, for as long as that condition holds. Not “free tier waits a long time.” Free tier waits forever, formally, until pro traffic drops below saturation. Weighted polling’s w_free floor from earlier in this chapter is the direct fix for exactly this unbounded case — it guarantees free tier a nonzero minimum share of service slots no matter how saturated pro traffic becomes.

Aging: give a bounded guarantee without reserving anything up front

A third option side-steps the fixed-weight tradeoff entirely: instead of deciding a fairness policy in advance, let a job’s own wait time raise its priority the longer it sits in the queue — a technique called aging. A free-tier job that has waited past some threshold gets promoted to be served alongside pro jobs, even under a strict-priority-like base policy.

python
def next_job_aging(pro_queue, free_queue, age_threshold_s=300):
    aged_free = [j for j in free_queue if (time.time() - j.enqueued_at) > age_threshold_s]
    if aged_free:
        return aged_free[0]              # an aged-out free job jumps ahead of ordinary priority rules
    if pro_queue:
        return pro_queue.pop(0)
    return free_queue.pop(0) if free_queue else None

Set age_threshold_s = 300 and the guarantee becomes concrete and easy to state to a customer, even under the strict-priority base policy’s unbounded worst case from above: no free-tier job waits longer than 300 seconds before being forcibly promoted ahead of the pro queue’s ordinary priority. This converts an unbounded worst-case wait into a hard ceiling, without pre-committing any capacity the way the hard split does — pro traffic gets full priority treatment right up until a specific free job has waited long enough to demand its turn.

The fully-separate-infrastructure option, and when it earns its cost

There is a fourth answer that skips the shared-pool tradeoff entirely: give pro and free tiers completely separate queues and worker fleets, with zero shared infrastructure between them. Nothing free tier does — a traffic spike, a bug that floods the queue, a poison-pill storm — can touch pro tier’s capacity at all, because there is no shared resource left for it to contend over.

This is the strongest isolation guarantee available, and it is also the most expensive: it forfeits every efficiency gain from pooling, including the very thing weighted polling was designed to preserve — idle pro capacity flowing to free tier when pro traffic is light. Two pools sized independently for their own peaks will, in total, need more workers than one shared pool sized for the combined peak, for the same reason Chapter 2’s pooling argument favored one PgBouncer pool over one connection per client. Full separation is the right call when the isolation requirement is not just about latency but about blast radius — a regulatory or contractual requirement that one tier’s failure literally cannot touch another’s infrastructure — rather than simply wanting better latency.

Not every broker makes this easy

Worth knowing before reaching for a specific technology: not every broker natively supports priority. RabbitMQ has built-in priority queues. SQS has no native priority concept at all — weighted or strict priority has to be implemented at the application layer, exactly the way this chapter’s next_job functions do, by maintaining separate named queues and choosing between them in the polling logic. Kafka’s consumer-group model assigns whole partitions to consumers, which makes fine-grained per-message priority awkward; priority tiers on Kafka are usually implemented as entirely separate topics, closer to this chapter’s fully-separate-infrastructure option than to weighted polling.

Fairness inside a tier, not just between tiers

Everything so far treated “pro” and “free” as internally uniform, but a single free-tier customer submitting a burst of 500 jobs at once raises the same question one level down: should that one customer’s burst be allowed to delay every OTHER free-tier customer’s single job behind it? This is exactly Chapter 6’s tier-fairness problem, recursively applied to per-customer fairness within a tier.

The fix is structurally identical to weighted polling, just keyed by customer ID instead of tier: round-robin across distinct customers with pending jobs, rather than draining one customer’s entire backlog before touching the next customer’s single job.

python — round-robin by customer within a tier
def next_free_job(free_queue_by_customer, rr_cursor):
    customers = sorted(free_queue_by_customer.keys())
    if not customers:
        return None
    for i in range(len(customers)):
        cid = customers[(rr_cursor + i) % len(customers)]
        if free_queue_by_customer[cid]:
            rr_cursor = (rr_cursor + i + 1) % len(customers)
            return free_queue_by_customer[cid].pop(0), rr_cursor
    return None

Worked example: one customer submits 500 jobs at once; nine other customers each submit one. Under plain FIFO within the free queue, the nine single-job customers wait behind however many of the 500 happened to be enqueued ahead of them — up to the full 500-job backlog in the worst case. Under per-customer round robin, each of the nine gets served within their own first turn of the rotation — at most 9 other customers’ single jobs ahead of them, regardless of how large the burst customer’s backlog is. The burst customer’s own 500 jobs still all get processed, just interleaved one-at-a-time with everyone else’s, rather than as one contiguous block that starves every other customer for its full duration.

Deciding the split with real cost accounting

The 90/10 free/pro traffic split and the 1,000-worker pro reservation were given as inputs; in a real system they are a business decision with a real cost basis worth making explicit. If pro-tier customers pay, say, $0.02/job in subscription-amortized revenue and free-tier jobs generate no direct revenue, the 1,000 reserved workers can be priced against what they are worth:

At this ratio, the reserved capacity is a small fraction of the revenue it protects — an easy call. The arithmetic matters more as the ratio tightens: a platform reserving capacity for a tier that does not clearly cover that capacity’s cost is optimizing a latency number that free-tier growth, ironically, is subsidizing. Fairness policy and unit economics are the same spreadsheet, not two separate conversations.

What happens when both tiers spike together

Every table in this chapter held one tier’s traffic fixed while varying the other. Real incidents rarely cooperate that way — a platform-wide traffic spike (a viral moment, a marketing push) can hit both tiers proportionally at once. Scale the whole 100/s baseline by 2× and recompute the hard-split table:

Pro tier survives a 2× platform-wide spike comfortably, still under its reserved capacity. Free tier does not — its own utilization alone exceeds 1 even before accounting for anything else, which is the honest answer to “what does a hard reservation buy you during a real incident”: it buys the protected tier isolation from exactly this scenario, at the cost of the unprotected tier being the one that absorbs a platform-wide spike unbounded, alone.

Communicating tiered latency honestly

Every table in this chapter was computed as an expected wait — a mean, not a guarantee. A platform quoting a specific SLA number to pro customers (“jobs complete within 60 seconds”) is making a promise about a percentile, not an average, and the distinction is not cosmetic: an M/M/c-style queue’s wait time is heavily right-skewed, so a mean of 42.9s can still carry a p99 several multiples higher during ordinary variance, well before any incident. A defensible SLA is derived from the tail of a measured wait-time distribution under real traffic, using this chapter’s formulas as the sizing tool that gets you into the right neighborhood, not as the number printed on the contract itself.

It is also worth remembering what fairness in this chapter never claimed to solve: none of these policies make the platform faster overall. They redistribute a fixed total capacity across competing demands according to a chosen priority rule. The only way to raise every tier’s wait time at once is the one lever this chapter deliberately held fixed — adding more workers, priced out in Chapter 2 — and no amount of clever scheduling substitutes for capacity that genuinely is not there.

Treat every fairness policy in this chapter, then, as a way to spend a fixed budget of worker-seconds deliberately rather than by accident — plain FIFO spends it in arrival order, a hard split spends a portion up front regardless of demand, weighted polling spends it probabilistically, and aging spends it reactively as a job’s own wait grows. None of them print more worker-seconds than Chapter 2’s sizing already provisioned.

Choosing among them, then, is really choosing which stakeholder gets to know the answer to “how long will my job wait” in advance, and how precisely. A hard split gives pro customers a number a sales team can put in a contract. Weighted polling gives better average outcomes for everyone but a softer per-customer promise. Aging gives free-tier customers a promise too, just a much longer one. None is universally correct; each is a different answer to who this fixed capacity is being spent to reassure.

Chapter 7: Scheduled & Recurring Work

Not all background work is triggered by a user action. Some of it is triggered by the clock: “every five minutes, re-check any job stuck in running longer than expected” or “every night, reprocess every user’s recommendation feed.” Scheduled work looks simple — it is just a cron entry — until you remember the service running that cron entry is not one process. It is a fleet.

The bug: one cron entry, many replicas

Say the API fleet runs 10 replicas for redundancy, and each one independently runs the same in-process scheduler, firing the same “reprocess-stale-jobs” task every five minutes. Nobody coordinated them, because nobody thought they needed to — each replica is just running its own copy of the same code.

Ten times the compute for no reason is the mild version of this bug. The severe version is when the task has a real side effect — “email users whose transcode finished overnight” — and it fires ten times: every affected user gets ten identical emails.

The fix: a distributed lock makes the job a singleton

Before running the scheduled task, each replica first tries to acquire a lock — a single row or key that only one holder can own at a time, using a mechanism like Redis’s SET key value NX EX ttl (set only if not already set, with an expiry) or Postgres’s advisory locks. Only the replica that wins the lock runs the task; the other nine see the lock already held and skip this cycle entirely.

python
def run_scheduled_task(redis):
    got_lock = redis.set("lock:reprocess-stale-jobs", worker_id, nx=True, ex=120)
    if not got_lock:
        return                                # someone else already has it — skip this cycle
    try:
        reprocess_stale_jobs()
    finally:
        redis.delete("lock:reprocess-stale-jobs")   # release once done, so the next cycle can acquire fresh

Sizing the lock TTL: the exact same bug as Chapter 1, wearing a new name

The lock in the snippet above carries a 120-second expiry — a safety net, so that if the replica holding the lock crashes mid-task without reaching the finally block, the lock does not stay held forever. That expiry has to be sized correctly, and getting it wrong reproduces Chapter 1’s collision exactly.

Say the reprocessing task takes up to 45 seconds at its p99, and someone sets the lock TTL to a plausible-looking round number, 30 seconds — shorter than the task’s own worst case:

The fix is the same lesson learned twice: set the TTL comfortably above the worst-case duration with margin for clock skew across nodes — TTL = 120s, nearly 3× the 45s p99 — and, for tasks whose duration is unpredictable, extend the lock with a heartbeat exactly the way Chapter 1’s worker extends its visibility timeout.

Sharding a job that touches every row

A different scheduling problem shows up at scale: “reprocess every user’s recommendation feed, once a day.” With 10,000,000 users, firing that as one giant job at midnight either takes hours to run serially or needs a huge burst of workers all at once — the exact spiky-load pattern Chapter 0 warned against, just on a daily clock instead of a request-response one.

The fix is to shard the schedule itself. Hash each user’s ID into one of 1,440 buckets — one per minute of the day — and have each minute’s cron tick process only its own bucket:

Instead of a midnight spike of ten million simultaneous reprocessing jobs, the platform sees a flat, predictable 115.7/second all day long — a load profile that fits inside a worker pool sized the ordinary way, using Chapter 2’s Little’s Law arithmetic, rather than one sized for a once-a-day burst that would sit idle the other 23 hours and 59 minutes.

Clock skew: the margin inside the margin

The 120s TTL recommendation already includes a margin over the 45s p99 — but that margin was sized against one replica’s own clock. A distributed lock is checked by different machines, and different machines do not agree on the time perfectly. Network Time Protocol keeps well-run fleets tight, but a couple of seconds of drift between nodes is realistic, and the direction of that drift matters here: if the lock holder’s clock runs slightly behind the newly-checking replica’s clock, the checking replica perceives the lock as having expired slightly earlier than the holder itself believes.

Seventy-three seconds of real margin remains even after subtracting a pessimistic 2-second skew — comfortable, and the reason to compute it explicitly rather than trust the raw 120-vs-45 gap: on a fleet with looser time synchronization, or a lock TTL set closer to the task’s worst case to begin with, that skew term can eat the entire margin and silently reopen Chapter 1’s exact bug, with no code change and no alert — only a slowly drifting set of system clocks.

Leader election: one long-lived winner instead of a race every tick

The per-tick lock this chapter has built so far re-runs the acquire-race every five minutes, for every scheduled task independently. An alternative shifts the unit of coordination: instead of racing for a lock per task execution, the 10 replicas race once for a leader role, held via a longer-lived lease (say, 60 seconds, continuously renewed by the current leader via the same heartbeat pattern Chapter 1 used for visibility timeouts). Whichever replica holds the lease runs every scheduled task locally, in-process, with no further lock acquisition needed per task.

Leader election trades a larger blast radius during its own failover window for meaningfully less coordination overhead once a large number of distinct scheduled tasks exist — the choice is really a bet on which is more valuable at the fleet’s actual scale: minimizing per-task overhead, or minimizing how much stops working when a single leader briefly goes away.

Implementing the shard hash

Chapter 7’s minute-bucket sharding needs a concrete, stable hash — stable meaning the same user always lands in the same bucket, so the daily job schedule for any given user does not jitter from run to run.

python
def bucket_for_user(user_id, num_buckets=1440):
    h = hashlib.sha256(str(user_id).encode()).hexdigest()
    return int(h, 16) % num_buckets

# the minute-N cron tick processes only its own bucket
def run_minute_bucket(minute_of_day, all_user_ids):
    due = [u for u in all_user_ids if bucket_for_user(u) == minute_of_day]
    for user_id in due:
        enqueue_job("reprocess_feed", user_id=user_id)

A cryptographic hash is deliberately overkill for uniformity here — any well-distributed hash works — but it is cheap and removes any risk of a naive hash (like a raw modulo of a sequential user ID) clustering unevenly across buckets, which would silently recreate a smaller version of the very burst this sharding scheme exists to remove.

Choosing the bucket count

1,440 buckets (one per minute) is a specific choice, not a universal default, and the tradeoff it makes is worth stating: more buckets means smaller, more evenly-spread per-tick batches, at the cost of more total cron ticks to manage and coordinate a lock for. Fewer buckets means larger, lumpier per-tick batches that more closely resemble the original all-at-once burst.

The sustained long-run rate is identical at every granularity — 115.7 users/second is simply 10,000,000 divided by 86,400 seconds in a day, however you slice the buckets. What changes is how lumpy the arrivals are within each bucket’s own window, and how much scheduling overhead you accept to smooth that lumpiness further. Per-minute is a common sweet spot precisely because it is granular enough that no single tick meaningfully dents Chapter 2’s worker pool, while being coarse enough that a standard cron scheduler can manage 1,440 triggers a day without needing its own dedicated infrastructure.

How real schedulers handle exactly-once triggering

The common thread: every one of these either hands the coordination problem to a single external, durable resource (a cloud scheduler, a database row acting as a lock) or sidesteps it by convention (run exactly one process, and hope nobody scales it). This chapter’s own distributed-lock pattern is the general-purpose version of the same idea, usable regardless of which specific scheduler triggers the tick.

Idempotency, again, as the backstop

Even a correctly-sized lock is a probabilistic guarantee, not a mathematical one — clock skew estimates can be wrong, a network partition can produce behavior nobody modeled, an operator can manually run the task while a lock is held for testing. The same principle Chapter 5 built for job processing applies here without modification: a scheduled task should be written to be safe if it somehow runs twice, not merely relying on the lock to guarantee it never does.

python — a reprocessing task that is safe even if it double-runs
def reprocess_stale_jobs(db):
    stale = db.query("SELECT id FROM jobs WHERE status='running' AND started_at < now() - interval '20 minutes'")
    for job_id in stale:
        # idempotent by construction: this UPDATE only affects rows STILL in the stale state,
        # so a second concurrent run of this exact task finds nothing left to touch
        db.execute("UPDATE jobs SET status='queued' WHERE id=%s AND status='running' AND started_at < now() - interval '20 minutes'", job_id)

Written this way, two concurrent executions of the same “reprocess stale jobs” task are not merely tolerable — they are provably harmless, because the second one’s UPDATE clauses simply match zero rows for anything the first one already fixed. The lock is still worth having, because it avoids wasting compute on a redundant run entirely — but the task’s own correctness no longer depends on the lock being perfect.

A quick reference: sizing every timeout this lesson introduced

Three different chapters each derived a timeout-shaped number from the same underlying principle — protect against a worst case, not an average — and it is worth seeing them side by side, because the rule connecting them is identical every time.

Every row follows the same rule stated once, in Chapter 1, and reused twice more without needing to be rediscovered: size a protective timeout from the worst case the thing it protects can legitimately take, never from its average — and when that worst case is uncertain or highly variable, prefer a heartbeat that extends the timeout while the protected work is provably still alive, over a single static guess.

One more scheduling failure mode: the missed tick

This chapter has focused entirely on preventing a tick from firing too many times. The opposite failure — a tick that should have fired but did not, because every replica was mid-deploy, or the leader crashed and re-election took longer than expected — deserves a name too, since a naive schedule has no memory of what it missed. A robust scheduled task checks, on its next successful run, whether the previous expected run actually happened, and catches up if not:

python
def run_with_catchup(db, task_name, interval_s=300):
    last_run = db.get_last_run(task_name)
    if time.time() - last_run > interval_s * 1.5:   # missed at least one tick
        log.warning(f"{task_name} missed a tick — running catch-up now")
    reprocess_stale_jobs(db)
    db.set_last_run(task_name, time.time())

This closes the loop with the rest of the chapter: locks and TTLs prevent a tick from over-firing; catch-up logic like the above prevents a tick from silently under-firing. Scheduled work needs both halves of that guarantee, not just the one this chapter spent most of its time on.

Both halves reduce to the same discipline this whole lesson keeps returning to: name the failure explicitly — a duplicate execution, a missed one — and build a specific, checkable mechanism against it, rather than trusting a cron entry to simply behave the way a single-machine mental model assumes it will once it is running across a fleet.

Chapter 8: Assembling the Pipeline

Every chapter built one piece: a hand-off out of the request/response cycle, a broker with a visibility timeout, a worker pool sized by Little’s Law, a way for the client to find out what happened, jittered retries with a dead-letter ceiling, an idempotent handler, priority lanes, and a safely-singleton cron. This chapter assembles all of it into one running simulation, and injects the one failure every earlier chapter analyzed on paper: a worker dying mid-job.

The assembled pipeline

What “kill worker” actually simulates

The simulation below runs on a compressed demo timescale — roughly 20× real speed, so a full redelivery cycle that would take twenty real minutes in production completes in a few seconds on screen. The underlying mechanism is identical to Chapter 1’s timeline, just fast-forwarded: a demo visibility timeout and a demo job duration, in the same VT-shorter-or-longer-than-duration relationship the real system uses.

Click upload a video to enqueue a job and watch it move from the queue into an idle worker. While it is running, click kill worker to simulate a crash — the worker disappears mid-job, but the message it was holding is still marked invisible in the broker, exactly as Chapter 1 described. Nothing happens immediately. Only once the demo visibility timeout elapses does the broker notice the silence and make the message visible again — watch a fresh worker pick it up and finish the job the crashed worker never got to.

Reading the simulation like an incident review

Watch the readouts at the bottom of the widget as jobs move through the system. Queue depth rising with active workers already at their cap is Chapter 2’s ρ → 1 pattern, live. A completed count that climbs steadily is the healthy path. A redelivered count that climbs is not, by itself, a failure — it is the system doing exactly what Chapter 1 designed it to do: notice a silent worker and recover the job anyway, without losing it and without a human paging anyone at 2 a.m.

That is the actual measure of success for everything this lesson built: not that failures stop happening — they do not, in any real system — but that every failure mode this lesson named (a slow endpoint starving fast ones, a redelivered duplicate, an unbounded queue, a lost status update, a retry storm, a duplicate side effect, a starved tier, a double-run singleton) has a specific, hand-derivable fix, and none of those fixes require the system to be perfect. They require it to be honest about what can go wrong, and to have a number — a timeout, a worker count, a window, a weight — ready for exactly that failure before it happens.

What you would reach for in production, and what it buys you

Every mechanism in this lesson was built from first principles — a broker, a visibility timeout, a hand-sized worker pool, a hand-written state machine. Production teams rarely build every piece from scratch; they compose managed primitives, or reach for a framework that bundles several of these chapters into one dependency. It is worth knowing what each level of the stack actually buys, so the choice is deliberate rather than accidental.

The honest reason to know this stack is not that the framework or workflow-engine rows make this lesson’s chapters obsolete — a Temporal workflow still has a visibility- timeout-shaped setting (its activity timeout), still needs its retry policy configured with backoff and jitter, and still needs idempotent activity handlers for exactly the reasons Chapter 5 derived. Adopting a higher-level tool moves where these decisions get made, not whether they need to be made at all.

What this lesson did not cover

Three real topics sit just past this lesson’s edge, worth naming honestly rather than implying they were covered:

Where the numbers in this lesson came from, and where to change them

Every derived number in this lesson traces back to one of five inputs, and the whole point of deriving rather than memorizing them is that a real system can recompute every downstream number the moment any one input changes:

Change any input — traffic doubles, a new job type with a different p99 ships, a new pricing tier launches — and the fix is not a rewrite. It is walking back through this table and recomputing the specific downstream numbers that input actually touches.

A dashboard that watches every chapter at once

Nine chapters each named a metric worth watching for their own specific failure mode. Pulled together, they form the actual on-call dashboard for a system built this way — not nine separate tools, but nine rows on one screen, because a real incident frequently moves through several of them in sequence.

No single row on that table is sufficient on its own to diagnose a real incident — the 2:14 p.m. scenario that opened this lesson would show up first as row one, but a genuine root cause investigation often needs to walk down several rows to find where the actual fix belongs, the same way the misdiagnosis table in Chapter 0 showed symptoms pointing away from their real cause.

Reading a harder failure in the simulation: rapid successive kills

Click kill worker more than once in quick succession, before the first redelivery has even happened, and watch what the simulation does — it is a useful edge case to reason through by hand before trusting it on screen. Each kill marks a different worker’s current job as dead independently; each of those jobs sits invisible until its own visibility timeout elapses, and because the demo VT is fixed, several redeliveries can land in the same short window rather than being spread out. This is a scaled- down version of exactly the scenario Chapter 4 analyzed for retries: a burst of near-simultaneous failures produces a burst of near-simultaneous recoveries, which is why production systems watch redelivery rate as a rate over a window, not just a raw counter — a burst is a different signal than the same total spread evenly across an hour, even though the counter ends at the same place either way.

What “good” looks like in a system built this way

It is tempting to judge a system like this on whether failures happen — they will, regardless of how carefully every chapter’s numbers are chosen, because workers run on real hardware that really does crash, networks really do drop packets, and dependencies really do have outages. The actual measure of a well-built async pipeline is narrower and more achievable: every failure this lesson named resolves itself automatically, within a bounded and known amount of time, without losing a customer’s job and without a human being paged to intervene. A redelivered job recovers within the visibility timeout. A retry storm self-heals within a few backoff cycles, jittered. A poison pill costs exactly five attempts, never more. A duplicate cron execution costs at most one wasted run, never a corrupted result. None of that is the absence of failure. It is failure with a ceiling on its cost, derived by hand, chapter by chapter, before it ever happened for real.

A final worked example: pricing the whole pipeline

It is worth closing with one number that pulls every chapter’s cost together, because a system design decision is never free and the honest version of this lesson says so plainly. Sum the recurring monthly figures this lesson derived, at the platform’s measured scale:

The takeaway is not any single number in that table — it is the ratio between them. The Chapter 1 bug this lesson spent an entire chapter deriving costs, left unfixed, about 1% of the worker pool’s own monthly bill. That is simultaneously small enough that it is easy to overlook and large enough that it is real, recurring money, growing with traffic forever until someone reads a redelivery-rate metric and asks why it is nonzero — which is exactly the shape every silent inefficiency in this lesson takes: individually cheap to fix, collectively expensive to ignore, and invisible on a dashboard that only tracks whether jobs eventually complete.

The one question worth asking about any new background job

Nine chapters produced a lot of individual mechanisms — a useful way to compress them back down, for the next time a new job type gets added to this pipeline, is a short checklist derived directly from the questions each chapter answered:

Every row is a specific, answerable question with a specific chapter’s worked arithmetic behind it — which is the actual deliverable of this lesson: not nine isolated facts, but one repeatable checklist that turns “is this background job going to be reliable” from a hope into something you can verify, in advance, by hand.

Related lessons

This lesson leaned on two ideas that get their own full treatment elsewhere in the System Design track, worth reading if either felt underexplained here: Little’s Law and the queueing-latency curve first appear in Scaling Reads, applied to database replicas instead of task workers — the M/M/1 math is identical. The retry-storm pattern in Chapter 4 is a direct cousin of the cache thundering-herd problem covered in that same lesson. And Chapter 3’s WebSocket option for status delivery is a narrow slice of the full stateful-connection-tier design covered in Realtime Updates, including the consistent-hash ring and reconnection-storm handling this lesson only gestured at.

Scaling Writes is the fourth companion worth naming directly: this lesson treated the broker as an infinite, reliable buffer between producer and worker, but a broker under extreme sustained write pressure — millions of jobs enqueued per second, not this lesson’s hundreds — runs into exactly that lesson’s sharding and partition- key problems, since a task queue’s broker is, underneath, a write-heavy distributed system in its own right. Every hot-partition and skew argument that lesson makes about a database applies just as directly to a queue with a single overloaded partition key.

One closing distinction worth carrying forward

This lesson used one running example — video transcoding — deliberately, so every number stayed concrete and checkable by hand. The mechanisms themselves are not specific to video at all: a report-generation job, a bulk data export, an ML batch-inference request, a large file conversion — anything whose processing time meaningfully exceeds an HTTP request’s reasonable lifetime faces exactly the same nine questions this lesson answered, with different numbers plugged into the same formulas. The video was the vehicle; Little’s Law, the visibility timeout, the idempotent handler, and the rest are the actual portable content, applicable to any long-running task hiding behind an API that was never meant to wait for it.

The test to apply the next time a feature spec includes the word “processing” or “generating” is the same test Chapter 0 opened with: how long does the real work take, and does that number fit inside a request/response cycle. If it does not, everything from Chapter 1 onward is not extra engineering overhead layered on top of a simpler design — it is what “correct” actually means once the work has to outlive the request that started it.

Ten minutes of transcoding behind a 200-millisecond API was never really the problem this lesson solved. The problem was that request/response was never designed to describe work whose duration nobody can promise in advance — and once that gap is named honestly, the nine chapters above are not nine separate tricks. They are one consistent answer, applied to one gap, from nine different angles — hand off the work, protect the claim, size the pool, report back, survive failure, guarantee safety, share fairly, and coordinate the clock.

Carry that gap forward, not just the nine mechanisms that close it. The next slow endpoint will not look like a video upload. It will look like a report export, a bulk import, an AI-generated summary that takes forty seconds instead of four hundred. The recognition — this does not belong in the request/response cycle — is the one habit worth keeping long after the specific numbers in this lesson stop applying.