The Real-Time Mental Model
The tail is where robots break

Chapter 1: Hard / Firm / Soft

In Chapter 0 the robot tripped because one cycle came in at 4 ms against a 1 ms deadline. But we never asked the question underneath that: what actually happens when a deadline is missed? The answer is not one answer. It depends entirely on what kind of deadline it is — and getting that classification wrong is how engineers over-build cheap systems and under-build dangerous ones.

Everyone reaches for the word "real-time" to mean "fast." That is the single most expensive misconception in the field. A real-time system is one whose correctness depends on when a result is produced, not just whether it is correct. A loop that produces the right torque 2 ms too late has produced a wrong output, even though the arithmetic was flawless. Real-time means predictable, and a slow-but-bounded system is more real-time than a fast-but-unbounded one.

The value function is the whole taxonomy

The cleanest way to define the three flavours is with a single object: the value function V(t), which says how useful a result is if it arrives at time t. Let D be the deadline. Then:

• Hard. V(t) = 1 for t ≤ D, and V(t) = −∞ for t > D. A late result is not merely useless — it is catastrophic. The airbag that fires 50 ms late. The flight-control surface that responds after the aircraft has departed controlled flight. The collision-avoidance stop that computes the brake command after the impact. There is no partial credit; a single miss is a system failure.
• Firm. V(t) = 1 for t ≤ D, and V(t) = 0 for t > D. A late result is worthless but harmless — you throw it away and carry on. The video frame that missed its display refresh: skip it, show the next one. A stale sensor reading in a fusion filter: drop it, wait for a fresh one. Occasional misses degrade quality but do not destroy the system, as long as they stay rare.
• Soft. V(t) = 1 for t ≤ D, then V(t) decays for t > D — still positive for a while, sliding toward zero. A late result is worth less, but still worth something. The keystroke that echoes 100 ms late still echoes. The web page that loads a beat slow is still the page you wanted. Lateness costs utility gradually rather than falling off a cliff.

Read those as one picture: hard falls to negative infinity, firm falls to a flat zero, soft slides down a ramp. That is the entire taxonomy, and it is why "real-time" alone tells you almost nothing — the useful question is always which shape is this deadline's value function?

Reading the three curves by hand at three completion times

Before we classify real subsystems, let us prove to ourselves that we can evaluate each value function by hand at specific instants — the slider does it visually, but the arithmetic is what you will do on a whiteboard. Fix the deadline at D = 100 (arbitrary units) and the soft decay window at τ = 200. We will read all three flavours at three moments: one on time, one just barely late, and one far late.

At t = 80 (on time, t ≤ D). Every value function is defined as V(t) = 1 for t ≤ D, so there is nothing to compute: hard = 1, firm = 1, soft = 1. This is the flat shelf on the left of the widget where all three curves lie on top of each other. On-time work is on-time work; the flavour is invisible until you are late.

At t = 130 (30 units late, t > D). Now the branches diverge. Hard takes the catastrophic branch: V(130) = −∞. Firm takes the zero branch: V(130) = 0. Soft takes the ramp: substitute into 1 − (t − D)/τ = 1 − (130 − 100)/200 = 1 − 30/200 = 1 − 0.15 = 0.85. So a result that is 30 units late is a disaster for a hard task, garbage for a firm task, and still 85 % as good for a soft one. Same lateness, three completely different costs.

At t = 200 (100 units late, exactly one τ past D). Hard is still −∞, firm is still 0. Soft: 1 − (200 − 100)/200 = 1 − 100/200 = 1 − 0.5 = 0.50. Being late by a full decay window costs the soft task exactly half its value — that is what τ means: the number of units of lateness that burns half the value. Push to t = 300 and the ramp reaches 1 − 200/200 = 0; beyond that the max(0, ·) clamp holds it at zero, so a soft deadline eventually becomes indistinguishable from a firm one. Drag the completion-time slider to 0.80·D, 1.30·D, and 2.00·D and watch the three dots land on exactly these numbers.

Worked example: classify four subsystems by hand

The taxonomy is only useful if you can apply it. Take four real subsystems on the same robot arm and classify each, then read the value at a specific late arrival — with the arithmetic shown, not asserted.

(1) The safety torque monitor checks every cycle whether the commanded torque exceeds a limit and cuts power if so. Deadline D = 1 ms (one control period). If it is late, a dangerous torque is applied for an extra millisecond and the arm can injure someone. Evaluate the hard value function at a 1 ms overrun, t = 2 ms: since 2 > D = 1, we take the catastrophic branch, V(2 ms) = −∞. There is no arithmetic to soften — the branch itself is the answer. Hard. This subsystem justifies a dedicated core, a watchdog, and formal WCET analysis.

(2) The camera-based object detector feeds grasp targets to the planner at 30 Hz, deadline D = 33 ms. If a frame's detection is late, the planner reuses the previous target for one cycle and the new detection is discarded. The value of a late frame is the firm branch: V(50 ms) = 0. But the sharper question a designer asks is the expected value per frame given a miss rate p — and that is real arithmetic. A firm result is worth 1 when on time and 0 when missed, so E[V] = (1 − p)·1 + p·0 = 1 − p. At a measured miss rate of p = 0.02, E[V] = 1 − 0.02 = 0.98. Read as a rate: over a one-second window the detector runs 30 frames, of which 30·(1 − 0.02) = 29.4 update the planner and 30·0.02 = 0.6 are dropped — roughly one dropped frame every ~1.7 s. Harmless as long as p stays small. Firm. This justifies a "drop the frame" policy, not a dedicated core.

(3) The teach-pendant UI echoes the operator's jog commands to a screen. Deadline D = 100 ms (the threshold where lag feels sluggish). If the echo is 150 ms late, it still shows — just feels slightly laggy. Take a linear decay over a τ = 200 ms window: V(150 ms) = 1 − (150 − 100)/200 = 1 − 50/200 = 1 − 0.25 = 0.75. Still three-quarters as useful. Soft. This justifies nothing special; a best-effort OS thread is fine.

(4) The trajectory-preview overlay on the operator display predicts the arm's path ~500 ms ahead and redraws it at 20 Hz, deadline D = 50 ms. A late preview is still a readable path for a while, decaying over τ = 150 ms as the drawn path drifts from where the arm actually is. Read it at three arrival times, all with the same ramp 1 − (t − D)/τ: at t = 80 ms, V = 1 − (80 − 50)/150 = 1 − 30/150 = 1 − 0.20 = 0.80; at t = 125 ms, V = 1 − (125 − 50)/150 = 1 − 75/150 = 1 − 0.50 = 0.50; at t = 200 ms, V = 1 − (200 − 50)/150 = 1 − 150/150 = 1 − 1 = 0, and the max(0, ·) clamp keeps it there for any later arrival. Same shape as the pendant, faster decay because a stale prediction rots quicker than a stale echo. Soft (with the shorter τ that a preview demands).

The number that sizes the budget: expected value under a miss rate

Classification tells you the shape; the budget question asks how much a given miss rate costs you, and for firm and soft that is a genuine expected-value calculation you can run on the back of an envelope. Imagine two subsystems facing the same 2 % miss rate, p = 0.02 — one firm, one soft that lands at 0.75 when late (the pendant). We want the average value each delivers per invocation.

Firm. A firm task is worth 1 on time and exactly 0 when missed, so the expected value is a two-term average weighted by the miss rate:

Soft (lands at 0.75 when late). A soft task keeps partial value on a miss, so the missed branch contributes 0.75 rather than 0:

Now read the two losses relative to a perfect run (value 1). The firm task loses 1 − 0.98 = 0.020 per invocation; the soft task loses 1 − 0.995 = 0.005. The firm task's loss is 0.020 / 0.005 = 4× larger for the identical 2 % miss rate — entirely because a firm miss throws the whole result away while a soft miss keeps three-quarters of it. This is the quantitative reason a firm deadline deserves a tighter miss-rate budget than a soft one: the same misbehaviour is four times as expensive. Push the rate to p = 0.10 and the gap widens — firm E[V] = 0.90 (loss 0.10), soft E[V] = 0.975 (loss 0.025), still exactly the 4× ratio, because the ratio of losses is fixed at 1 / (1 − 0.75) = 4 regardless of p.

Inverting the question: the miss-rate budget each flavour buys you

The expected-value formula runs the other direction too, and that is the direction a real spec is written in. Instead of "given a miss rate, what value?" ask "given a required value floor, how many misses can I afford?" Write the expected value for any late-branch value v_late in one line:

Demand E[V] ≥ a floor of 0.99 (deliver at least 99 % of ideal value on average) and solve for the largest tolerable p:

For a firm task, v_late = 0, so p ≤ 0.01 / (1 − 0) = 0.01 — a 1 % miss-rate budget. For the soft pendant, v_late = 0.75, so p ≤ 0.01 / (1 − 0.75) = 0.01 / 0.25 = 0.04 — a 4 % budget, four times more forgiving for the identical value floor. And the hard task? Its late branch is −∞, so any p > 0 drives E[V] to −∞: the only tolerable budget is p = 0, which is why hard tasks are never specified by an average value at all. They are specified by a failure probability over a mission — a 1 kHz hard loop runs 1000×3600 = 3.6×10⁶ cycles an hour, and a safety standard demands the probability of a dangerous failure per hour sit in a defined band (for IEC 61508 / ISO 13849 SIL 3, that band is 10⁻⁸ to 10⁻⁷ per hour). Different mathematics entirely, because "on average" is meaningless when a single miss is catastrophic.

Read the table as a budget ladder: the same 0.99 value floor buys a hard task zero misses, a firm task 1 %, and a soft task 4 % — and the engineering column is what that budget costs to guarantee. Misclassify a firm subsystem as hard and you spend a dedicated core to buy a zero-miss guarantee nobody needed; misclassify it as soft and you allow 4× too many misses. The value function is not describing the task — it is writing the spec.

From scratch: the value functions in code

python
import numpy as np

def value_hard(t, D):
    return np.where(t <= D, 1.0, -np.inf)      # cliff to catastrophe

def value_firm(t, D):
    return np.where(t <= D, 1.0, 0.0)         # cliff to zero, then harmless

def value_soft(t, D, tau):
    late = np.clip((t - D) / tau, 0, 1)         # fraction of the decay window used
    return np.where(t <= D, 1.0, 1.0 - late)  # ramp down to zero over tau

# classify the pendant echo: D = 100 ms, tau = 200 ms, arrives at 150 ms
print(value_soft(np.array([150.0]), 100, 200))   # -> [0.75]
# the preview overlay: D = 50 ms, tau = 150 ms, arrives at 125 ms
print(value_soft(np.array([125.0]), 50, 150))    # -> [0.50]

def expected_value(v_late, p):
    return (1 - p) * 1.0 + p * v_late          # on-time worth 1, miss worth v_late

print(expected_value(0.0,  0.02))                # firm -> 0.98
print(expected_value(0.75, 0.02))                # soft -> 0.995  (4x smaller loss)

The only difference between the three functions is what fills the else branch after the deadline: negative infinity, zero, or a ramp. That single line is the taxonomy. There is no library for this because there is nothing to abstract — the value function is the specification, and you write it per subsystem. And expected_value is the one-liner that turns a flavour plus a measured miss rate into the number a budget meeting actually argues over.

The failure, and the metric that reveals it

Derivation: how a firm miss becomes a hard stall, with timestamps

"A firm miss propagates into a hard-loop stall" is easy to say and easy to disbelieve — a firm task's misses are supposed to be harmless. Let us trace it on a concrete timeline until the mechanism is undeniable. Two tasks:

• Control loop — hard, 1 kHz, so its period and deadline are both 1 ms. It ticks at t = 0, 1, 2, 3, … ms and each tick must finish inside its 1 ms slot.
• Object detector — firm, 30 Hz, deadline 33 ms. It normally computes a fresh grasp target in ~20 ms, but on a bad cycle (a cache-cold re-plan) it takes 45 ms. Start this bad cycle at t = 0.

Step 1 — the firm miss, in isolation. The detector's bad cycle finishes at t = 45 ms against its 33 ms deadline. It missed by 45 − 33 = 12 ms. By the firm definition that late detection is worth 0 — you should discard it and let the planner reuse the previous target. On its own, this costs one stale frame and nothing else. So far, harmless, exactly as "firm" promises.

Step 2 — the wiring mistake. Suppose the control loop was written to read the target synchronously: target = detector.get_target(), a blocking call that does not return until the detector's current cycle finishes. This one line silently converts the detector's firm deadline into a hard dependency of the control loop.

Step 3 — the stall, timestamped. The control tick at t = 40 ms issues that blocking read. The detector is still busy until t = 45 ms, so the call cannot return until t = 45 ms. That single control cycle therefore takes 45 − 40 = 5 ms against its 1 ms deadline — a stall of 5 / 1 = 5× the control period. The control ticks that should have fired at t = 41, 42, 43, 44 ms never ran: 4 skipped control updates, during which the actuator holds its last torque for the full 5 ms. That is precisely the 4 ms-versus-1 ms overrun from Chapter 0 — only now it is 5 ms, and its root cause is not a slow control loop but a firm task the control loop was told to wait for.

The inverse mistake: a soft task starving a hard one

The blocking-read bug hides a firm task inside a hard loop. The inverse mistake is just as common and just as timestampable: giving a soft task higher priority than a hard one on a shared core, so the soft task can preempt and starve the loop that actually matters. Trace it:

• Control loop — hard, 1 kHz, needs only 0.3 ms of CPU per 1 ms period (30 % utilisation — trivially schedulable if it can run when it wants to).
• Telemetry flush — soft, and mistakenly pinned at a priority above the control loop. Usually it does nothing, but on a bad cycle it flushes a full buffer in one 2.5 ms burst without yielding.

The stall, timestamped. The soft burst starts at t = 10.0 ms and holds the single core until t = 12.5 ms. Because it outranks the control loop, the hard ticks scheduled at t = 10, 11, and 12 ms cannot run until the burst releases the CPU at 12.5 ms. That is 3 consecutive missed control deadlines: the tick due at t = 10 runs 12.5 − 10 = 2.5 ms late, the one due at t = 11 runs 1.5 ms late, the one due at t = 12 runs 0.5 ms late — every one of them past its 1 ms deadline. A soft task with no business near the control loop just caused three hard misses, and the CPU was only 30 % loaded the whole time. Utilisation looked healthy; the priority order was the bug.

Frontier: mixed-criticality systems

Real robots and vehicles run hard, firm, and soft tasks on the same hardware — you cannot afford a separate computer per criticality level. The research field that studies this is mixed-criticality scheduling, launched by Vestal's 2007 paper "Preemptive Scheduling of Multi-Criticality Systems with Varying Degrees of Execution Time Assurance" (RTSS 2007). Its core idea: a high-criticality task is analysed with a pessimistic (large) worst-case execution time, a low-criticality one with an optimistic estimate, and the scheduler is allowed to shed low-criticality work if a high-criticality task overruns its optimistic budget. This is the formal version of "when the safety monitor needs the CPU, drop the video frame" — and it is why the DO-178C (avionics) and ISO 26262 (automotive) safety standards care so much about how you partition tasks by criticality. We list the standards in the Field Guide.

Chapter 2: Latency, Jitter, WCET

Chapter 1 gave us the deadline and what a miss costs. This chapter gives us the three numbers that tell us whether we will miss it. They are constantly confused — people say "latency" when they mean "jitter," or quote a mean when they need a worst case — and every one of those confusions has tripped a real robot. We will define each precisely, work an example by hand, then build the measurement in code.

Three numbers, one signal

Take a single repeating operation — one iteration of a control loop — and measure how long it takes on each of many repetitions. That stream of durations is your raw signal. Three summaries of it matter:

• Latency is the time from stimulus to response — how long one operation takes. It is a single measurement, or a distribution of them. "The loop took 1030 µs this cycle" is a latency. Latency has a whole distribution, and reducing it to one number is where mistakes begin.
• Jitter is the variation in latency — how much the timing wobbles cycle to cycle. Precisely: jitter is the spread of the latency distribution, quoted as either peak-to-peak (max − min) or as a standard deviation. A loop with 1 ms mean latency and 5 µs jitter is rock-solid; the same mean with 3 ms jitter is a time bomb. Jitter, not latency, is what breaks fixed-rate control (Chapter 3 proves it).
• WCET, the Worst-Case Execution Time, is the largest latency the operation can ever take — the upper bound. Not the observed max in your test run (which is a lower bound on the WCET), but the provable ceiling across all inputs, all cache states, all interrupt patterns. WCET is the number a hard deadline must be compared against, because a hard deadline can tolerate exactly zero excursions above it.

WCET vs observed max: why a short soak cannot see the ceiling

The WCET bullet said the observed max is only a lower bound on the true worst case. That is not a hedge — it is arithmetic, and the arithmetic tells you exactly how long you must run to observe a rare excursion at all. Suppose the event that violates your deadline happens once in a million cycles — a probability p = 10⁻⁶ per cycle. How many times do you expect to see it in a soak of length N? The expected count is simply N · p.

Short soak, N = 10,000 cycles. The expected number of occurrences of the once-in-a-million event is:

You expect to see the event about once every hundred such runs. Ninety-nine soaks out of a hundred finish with a clean bill of health and never touched the failure at all. The observed max is the max of the healthy cluster; the true WCET is nowhere in the sample.

Long soak, N = 10,000,000 cycles. Same product, three orders of magnitude more cycles:

Now the event shows up about ten times, so the observed max finally has a chance to reach into the tail. The rule of thumb falls straight out: to observe a 1-in-k event with any reliability you must run at least a few times k cycles — here a few times 10⁶, which is why 10⁷ is the honest floor and 10⁴ is theatre.

The same fact wrecks deep percentiles on short runs. A percentile is a sorted index (we prove this below), so p99.99 on N samples sits at fractional rank r = (99.99/100)·(N−1). On a 10,000-cycle run:

There is no such thing as a stable p99.99 in 10,000 samples: the index it would need does not correspond to a repeatable event, so the number it returns is whatever noise happened to land on top this run.

Turn that rule into a wall-clock soak time, because that is the number a test plan actually schedules. Our arm's torque loop runs at 1 kHz, so it produces exactly 1000 cycles every second. Suppose the requirement is to certify that a deadline-violating stall happens less than once in a million cycles. To observe such an event with any stability — say around ten occurrences — the reciprocal rule demands N ≈ 10×10⁶ = 10⁷ cycles. Convert to time by dividing by the rate:

If instead you follow the stricter "run for at least 100× the reciprocal of the fault rate" habit — so the estimate rests on ~1000 tail draws, not ten — the soak grows to N = 100×10⁶ = 10⁸ cycles:

Now put the short bench beside it. A 60-second smoke test at 1 kHz collects

An expected count of 0.06 means you see the failure in roughly 1 run in 17, and the other sixteen ship a green light over a real defect. The gulf between "0.06 expected events in a minute" and "10 expected events in three hours" is the entire reason field units miss deadlines that the CI bench swore were impossible: the bench was three orders of magnitude too short to have met the fault even once.

Why p99.9 and not the mean — derived

Here is the argument that makes the whole lesson click. Suppose your control loop runs at 1 kHz — 1000 cycles per second, 3.6 million cycles per hour. Suppose a rare stall makes one cycle in a thousand come in late. How often does the robot meet a late cycle?

Do the wall-clock arithmetic explicitly. One late cycle in a thousand, at a thousand cycles per second, is one late cycle every second:

Scale that up to an hour and a day:

So if a late cycle can trip the robot, "it only happens 0.1 % of the time" means it happens about 86,400 times a day. Rarity in percentage terms is not rarity in wall-clock terms when the loop runs fast — the fast loop turns a tiny per-cycle probability into a large daily count.

Now ask which statistic sees that 0.1 % event. The mean averages it in with 999 healthy cycles, so it barely moves. The median (p50) is the middle cycle — it is defined to ignore the top half entirely, so it never sees the tail. The p99 — the value exceeded by 1 % of cycles — sees events at or above the 1-in-100 rate, so a 1-in-1000 event is still in its tail. Only the p99.9 — the value exceeded by 0.1 % of cycles — lands exactly where the 1-in-1000 stall lives.

The rule is a one-line formula, not a memorised table. A fault that happens at rate f per cycle lives in the top f fraction of cycles, so the percentile that lands on it is the one below which the remaining 1−f of cycles fall:

Turn the crank for the fault rates a 1 kHz loop actually meets, and also read how often each fault strikes in wall-clock time (rate×1000 per second):

Two things jump out. First, the "9"s in the percentile name are literally counting the zeros in the fault rate — a 1-in-10³ fault needs a percentile with three nines (p99.9), a 1-in-10⁶ fault needs six (p99.9999). Second, even the rarest row here — a one-in-a-million stall — still strikes about 86 times a day on a loop that runs a thousand times a second. That is the whole reason we chase these deep percentiles at all: on fast loops, "astronomically rare per cycle" is still "several times an hour" on the wall clock.

At 0.1 % tail, only the p99.9 marker is out in the tail — every other statistic sits in the healthy cluster, blind. Push the tail rate to 1 % and the p99 marker jumps out too; at 5 % even the p99 is deep in the tail and the mean finally starts to drift. The lesson is mechanical: a percentile only sees a tail event once the event is more common than one minus that percentile.

Worked example: percentiles of ten samples, by hand

Take the ten sorted loop times from Chapter 0 again, in microseconds:

The standard "linear interpolation" percentile (what numpy.percentile uses by default) maps a percentile q to a fractional rank r = (q/100)·(n−1) into the sorted array, and interpolates between the two nearest samples. With n = 10, so n−1 = 9:

p50 (the median), step by step. First the fractional rank, then the bracketing indices, then the interpolation:

The median lands calmly in the healthy cluster, completely blind to the 4000 µs straggler.

p90, step by step. The rank now climbs to the very edge of the tail:

At 0.1 of the way into the last gap, p90 has already leaked 297 µs of the straggler's pull into its value — the tail has begun to bite.

The mean, for contrast. Sum all ten and divide:

Notice the mean (1304.5) is lower than the p90 (1327) here — the single 4000 outlier pulls the mean up above the bulk, but the p90 index sits right against that outlier and inherits almost all of it. The two "typical" numbers disagree by roughly 300 µs, which is your first sign the distribution is not the bell curve your intuition assumes.

Why ten samples cannot hold a p99. Try to push the rank deeper and the reason falls out. Both p99 and p99.9 land in the very last gap, between index 8 (1030) and index 9 (4000):

The two ranks differ by only 8.991 − 8.91 = 0.081, so p99 and p99.9 are almost the same number and both are just the lone 4000 µs straggler read at slightly different fractions. There is nothing to interpolate against in the tail because there is only one tail sample. This is exactly why the next section moves to twenty samples with two tail cycles — you need at least a couple of points in the tail before a deep percentile means anything.

That is the entire idea of a percentile: a sorted index, with interpolation between neighbours. Once you see that, it is obvious why the mean cannot recover a percentile — the mean throws away the ordering, and the ordering is the whole point.

Deep-tail percentiles by hand: p99 and p99.9 on twenty samples

Ten samples cannot even hold a p99 — the top 1 % of ten cycles is a tenth of a cycle, so p99 collapses onto the single largest value. To watch the deep-tail interpolation the Code Lab automates, we need a longer vector. Here are twenty sorted loop times in microseconds: eighteen healthy cycles and two preempted ones.

Now n = 20, so n−1 = 19, and the sorted indices run 0 (value 980) through 19 (value 4200). Index 18 is 3800 and index 19 is 4200 — the two tail cycles. Apply the same fractional-rank rule r = (q/100)·(n−1):

p50. The rank is right in the middle, far from either tail cycle:

Squarely in the healthy cluster, blind to the tail — exactly as a median should be.

p90. The rank has just crossed into the gap between the top healthy cycle (index 17) and the first spike (index 18):

p90 sits in no-man's-land — higher than any healthy cycle, lower than any real preemption — because its rank fell in the wide gap between cluster and tail.

p99. Now the rank pushes past index 18, into the interval spanning the two spike cycles:

With two spikes in twenty cycles (a 10 % tail), the 1-in-100 value has fully entered the preemption band.

p99.9. Deeper still into the same top interval — the rank barely moves, but it moves toward the maximum:

Almost the raw maximum, because at n = 20 the p99.9 index (18.981) is only 0.019 below the very last sample.

Read the four numbers in order — 1009, 1316, 4124, 4192 — and you can see the rank climb out of the cluster (p50), cross the gap (p90), and then walk up the two tail samples (p99, p99.9). That walk is the interpolation arithmetic; the Code Lab below does exactly this at n = 1000, where the deep-tail rank finally rests on many spike observations instead of two, which is why its p99.9 is a stable estimate and this one is not.

Jitter by hand: peak-to-peak and standard deviation

We defined jitter as the spread of the latency distribution, quoted two ways — peak-to-peak (max − min) or standard deviation. Prose is not a realization; let us put numbers on both for one small set. Take five consecutive loop times, in microseconds:

Peak-to-peak is the crudest jitter measure and the easiest by hand: the largest minus the smallest. Here max = 1010, min = 990, so peak-to-peak jitter = 1010 − 990 = 20 µs. It is a pure worst-case-of-this-sample number: one wild outlier sets it entirely, which is both its virtue (it catches the single bad cycle a stddev would average away) and its flaw (it is unstable and grows as you collect more samples).

Standard deviation is the smoother measure. Compute it in five steps — mean, deviations, squared deviations, variance, then the root:

Step 1, the mean (a round number here on purpose, so the deviations stay clean):

Step 2, subtract the mean from each sample to get the deviations:

Step 3, square each deviation and sum them:

Step 4, the variance is the average squared deviation (population form, dividing by n = 5):

Step 5, the standard deviation is the square root of the variance:

Notice the two jitter numbers disagree — 20 µs peak-to-peak versus 7.07 µs standard deviation — and that disagreement is the whole subtlety. The stddev has divided the energy of the two ±10 extremes across all five samples; peak-to-peak reports the extreme raw. For control stability (Chapter 3) the peak matters more than the average, because a single Δt excursion is what the controller was never designed for — which is exactly why a real-time report should carry both a spread number and a worst-case number, never just one.

Work that tight second set through the same recipe so the contrast is not just asserted. Its mean is (999 + 1000 + 1000 + 1000 + 1001)/5 = 5000/5 = 1000 µs — identical to the first set. But its deviations are only −1, 0, 0, 0, +1, so:

Set both results beside each other: same 1000 µs mean, but σ = 7.07 µs for the first loop versus σ = 0.63 µs for the second — an 11× difference in jitter that the mean is completely blind to. A report that quotes only the mean would call these two loops identical; a control engineer would call one a time bomb and the other rock-solid.

One more number falls straight out of the standard deviation, and it explains why even the honest statistics need a sample count. The standard error of the mean — how much the estimate of the mean itself wobbles across repeated runs — is the standard deviation divided by √N. With our σ ≈ 7.07 µs, watch it shrink as the soak lengthens:

Read the pattern: to pin the mean down to ±0.1 µs you need SEM ≤ 0.1, which means N ≥ (7.07 / 0.1)² = 70.7² ≈ 5000 samples. That is the reassuring case — the √N law makes the mean converge, and a few thousand samples nail it. The unsettling contrast is the tail: while the mean's error falls like 1/√N, the observed maximum only ever rises as you collect more samples, because each new draw can only push the max up, never down. That asymmetry — the mean converges, the max diverges upward — is the deep reason a benchmark that is long enough to trust its mean can still be far too short to trust its worst case.

Here is the whole jitter-and-confidence calculation as a small function, so you can reproduce every number above from scratch rather than trusting the prose:

python
import numpy as np

def jitter_stats(x):
    x = np.asarray(x, float)
    ptp   = x.max() - x.min()          # peak-to-peak: the raw worst-case spread
    sigma = x.std()                    # population std (ddof=0): the smoothed spread
    sem   = sigma / np.sqrt(len(x))   # standard error of the MEAN estimate
    return ptp, sigma, sem

loop = [1000, 1005, 995, 1010, 990]
ptp, sigma, sem = jitter_stats(loop)
print(ptp, sigma, sem)          # -> 20.0   7.071   3.162  (matches the hand derivation)

# how many samples to pin the mean to +-0.1 us?  SEM = sigma/sqrt(N) <= 0.1
n_needed = int(np.ceil((sigma / 0.1) ** 2))
print(n_needed)                 # -> 5000

# the WCET margin check: torque loop C=350,T=1000 preempted by a monitor C=120,T=500
acts = int(np.ceil(1000 / 500))       # activations of the monitor in one torque period
interference = acts * 120          # -> 2 * 120 = 240 us
print(350 + interference <= 1000)  # -> True, slack 410 us

The three-line jitter_stats is the entire realization of this chapter's spread numbers: ptp is the raw worst case, sigma is the smoothed spread, and sem is how much the mean itself is still guessing. There is no deeper library call — peak-to-peak is a subtraction, the standard deviation is the root of the average squared deviation, and the standard error just divides by √N. Every number the prose derived by hand comes back out of this function unchanged.

Design: the WCET margin, in numbers

A hard-deadline task's design rule is one inequality: WCET ≤ deadline − interference. Put numbers on it for the arm's torque loop. The compute WCET is 350 µs (measured max, plus a safety factor to bound the true worst case). The deadline is the 1000 µs period. The interference — the worst-case time the loop can be preempted by higher-priority work — is what the design must bound. If the logger can preempt for 800 µs, then 350 + 800 = 1150 > 1000: the inequality fails, and you have a design defect that will show up as exactly the once-a-second trip from Chapter 0. Fix it by lowering the interference (move the logger to a lower priority, cap its burst, run it on another core) until 350 + interference ≤ 1000, i.e. interference ≤ 650 µs.

That was the loop against a single interferer. Real cores host several tasks, and "interference" is not a number you guess — it is a number you compute from who can preempt whom. Take three periodic tasks on one core, each with a worst-case compute time C and a period T:

Step 1 — per-task utilisation, then the total. Each task's utilisation is its compute over its period — the fraction of the CPU it consumes on average:

Step 2 — the rate-monotonic bound. For n tasks scheduled by rate-monotonic priorities (shorter period = higher priority), Liu & Layland's sufficient bound is n·(2^1/n − 1). For n = 3:

Since U = 0.69 ≤ 0.78, the set is guaranteed schedulable by this test — every task will meet its deadline. But that guarantee is average utilisation; it says nothing about the torque loop's worst single period, which is what a hard deadline actually cares about. For that we go back to the margin inequality and compute the torque loop's interference explicitly.

Step 3 — interference from higher-priority work, by activation count. Only tasks of higher priority than the torque loop can preempt it: here, just the safety monitor. In one torque-loop period of 1000 µs, how many times can a 500 µs-period monitor fire? The ceiling of the period ratio:

Each activation costs its own worst-case compute of 120 µs, so the worst-case interference the monitor can inject into one torque period is:

Step 4 — the margin check. Substitute into WCET + interference ≤ deadline:

The loop survives with 410 µs to spare — as long as only the safety monitor preempts it. Now expose the defect from the single-interferer version: if a priority inversion lets the low-priority logger's 800 µs burst run inside the torque period (a held mutex, a non-preemptible section), the sum becomes

and the loop misses. The utilisation test passed (0.69 ≤ 0.78) yet the loop still misses — because a mean-based test cannot see a worst-case preemption stack-up. The fix is a cap: the logger's burst inside any torque period must satisfy 350 + 240 + X ≤ 1000, i.e.

The same slack number answers a second design question: how much can the torque loop's own WCET grow — from added filtering, a new safety check — before it breaks, given only the legitimate 240 µs of monitor interference? Solve WCET + 240 ≤ 1000:

So the torque compute has headroom from its current 350 µs up to 760 µs — a 410 µs budget you can spend on features, and the exact same 410 µs the logger was about to steal. That single slack figure is the whole margin: spend it on the logger and the loop misses; spend it on the loop's own work and you are fine. The inequality just tells you it is one budget, not two.

The failure, and the metric that reveals it

Make the "confidence interval shrinks" claim concrete — you can reproduce this yourself. A p99.9 estimate is really a count of how many samples land in the top 0.1 %, and the noise on that count follows the binomial standard deviation √(N·p·(1−p)) with p = 0.001. Compare a short run to a long one:

Before — N = 1000 samples. First the rank the p99.9 interpolates against, then the count of tail samples and its noise:

Whether you saw the fault zero, one, or two times is pure luck; the p99.9 you print swings wildly run to run.

After — N = 1,000,000 samples. The same three quantities, now with a thousand-fold longer soak:

The estimate has stopped rattling because a thousand independent tail draws, not one, now define it. The relative spread fell from ~100 % to ~3.2 % purely from running 1000× longer — a factor of √1000 ≈ 31.6, exactly the √N law. That is the metric to reproduce and to report: not just the percentile, but the number of tail observations behind it and how much the estimate wobbles across repeated soaks.

Frontier: static WCET analysis

Measuring the worst case can never prove the worst case — you only ever observe a lower bound. For hard-real-time avionics, the field instead computes a provable upper bound by static analysis of the binary, modelling the pipeline and caches. The canonical tool is aiT from AbsInt, built on the abstract-interpretation framework of Wilhelm et al.'s survey "The Worst-Case Execution-Time Problem — Overview of Methods and Survey of Tools" (ACM TECS, 2008). The hard part is not the arithmetic; it is that modern CPUs have caches, branch predictors, and out-of-order execution that make the worst path depend on history in ways that are exponential to enumerate — which is exactly why safety-critical robotics often chooses simpler, more predictable processors (or a bare-metal microcontroller) over a fast but timing-opaque application core. We name the tools in the Field Guide.

Chapter 3: Fixed-Rate Loops

We now have the deadline taxonomy and the statistics. This chapter answers the question a fresh engineer always asks: why not just run the control loop as fast as the CPU allows, best-effort, instead of pinning it to a fixed rate? It sounds efficient. It is, in fact, the direct cause of the Chapter 0 trip. A control loop's rate is a fixed physical constant of the closed-loop system, not a scheduling convenience — and running it best-effort silently changes the math the controller was designed around.

A controller is designed at one specific rate

Here is the piece most software engineers have never been told. When a control engineer designs a discrete controller — a PID, a state-feedback law, a filter — they design it for a specific sample interval Δt. Every gain in the controller is a number that depends on Δt. Change Δt and you have silently changed the controller into a different one, which may no longer be stable.

Take the simplest case: a derivative term. Velocity is estimated as

where θ is the joint angle. The code divides by the Δt it was told to assume — the nominal period — not the Δt that actually elapsed. If the loop was late and the true interval was 1.3 ms but the code divides by 1.0 ms, the velocity estimate is off by the ratio 1.3/1.0 = 1.3: a 30 % overestimate. Work it: a genuine angle change of 0.5 rad over a true 1.3 ms is a velocity of 0.5/0.0013 = 385 rad/s, but the code computes 0.5/0.001 = 500 rad/s. The derivative gain then commands a braking torque 30 % too large, and the arm jerks. A 3 ms straggler like Chapter 0's produces a 4× error — more than enough to trip the safety envelope.

The deeper reason: stability is not linear in Δt

It gets worse than a scaled gain. For many discrete controllers, stability itself depends on Δt in a convex way — small Δt is stable, large Δt is unstable, and the boundary is a hard edge. When the loop jitters, some cycles run with a small Δt (comfortably stable) and some with a large Δt (unstable). The stable cycles pull the state toward zero; the unstable ones push it away. Whether the loop survives is decided by the average growth rate, and because the growth rate is convex in Δt, jittering Δt around a fixed mean makes the average growth worse than running at that mean — even though the mean Δt is unchanged.

This is the same "the mean lies" theme as Chapter 2, now for stability. A best-effort loop whose mean rate matches the design rate can still diverge, because the tail of large-Δt cycles injects energy the small-Δt cycles cannot remove. Formally: if the per-step growth factor ρ(Δt) is convex, then by Jensen's inequality the average of ρ over a jittered Δt exceeds ρ at the mean Δt. Sit the controller right at its stability edge, where ρ(mean) = 1, and any jitter tips the average above 1 and the loop diverges. We will watch exactly this happen in code.

Set the jitter to zero and the two curves are identical — both are the fixed-rate loop, both bounded. Raise it and the red curve, whose mean interval is still exactly 20 ms, climbs away from zero and diverges. That is the entire argument against best-effort control in one picture: identical average rate, opposite fate, decided by the tail.

Design: will three tasks fit? The utilisation test

Fixing the rate is necessary but not sufficient — you also have to prove the tasks fit on the CPU. The tool is CPU utilisation. For a task with worst-case execution time C_i and period T_i, its utilisation is U_i = C_i/T_i — the fraction of the CPU it needs. Sum them for total utilisation U = ∑_i C_i/T_i.

For rate-monotonic scheduling (fixed priorities, shortest period highest priority), Liu and Layland's 1973 bound says n tasks are guaranteed schedulable if U ≤ n(2^1/n − 1). Work a concrete case — three tasks on one core:

Total U = 0.200 + 0.150 + 0.100 = 0.450. The RMS bound for n = 3 is 3·(2^1/3 − 1) = 3·(1.2599 − 1) = 3·0.2599 = 0.780. Since 0.450 ≤ 0.780, the three tasks are guaranteed to meet their deadlines under rate-monotonic scheduling. Notice the CPU is only 45 % busy on average — the utilisation test is not about average business, it is about whether the worst-case arrivals can still be served in time.

Ten misses per thousand cycles — and the mean compute load is 300 µs, a comfortable 30 % of the period. The average says the CPU is nearly idle. The deadline misses come entirely from the coincidence of the preemption, which the average cannot see. This is Chapter 0's once-a-second trip, reduced to eight lines you can count.

Read the two outputs. Both schedules average to a 20 ms interval — the fixed one exactly, the jittered one to four decimal places. The fixed loop stays bounded; the jittered loop, with identical average rate, diverges by a factor of a hundred. If you ran this controller best-effort and looked only at the mean loop rate, every dashboard would say "on target" while the arm shook itself apart. That is why a control loop's rate is fixed, not best-effort.

The failure, and the metric that reveals it

Frontier: analysing jitter's effect on control

The formal study of how timing jitter degrades control performance is a whole subfield. The landmark is Cervin, Åström, and colleagues' Jitterbug and TrueTime tools (Lund University, early 2000s; see Cervin et al., "How Does Control Timing Affect Performance? Analysis and Simulation of Timing Using Jitterbug and TrueTime," IEEE Control Systems Magazine, 2003). They let you compute the exact performance loss from a given latency-and-jitter distribution, turning "jitter is bad" into a number you can trade against CPU cost. The modern descendant is the control–scheduling co-design literature, which designs the controller and the schedule together so the controller is robust to the jitter the schedule actually produces — rather than pretending the schedule is perfect. We point to the tools in the Field Guide.

Chapter 4: Timeline Showcase

This is the payoff. Everything in the lesson — the deadline, the tail, the jitter, the miss — is one live instrument now. A fixed grid of deadline ticks marches left to right. Each control cycle is a dot that should land on its tick. Inject jitter and the dots start to wobble off their ticks; inject preemption and some dots slip clean past the deadline line and turn red. Beside the timeline, a histogram of loop times updates in real time with the p50, p99, and p99.9 markers you now know how to read.

Start with jitter at 15 % and preemption at 0.8 %. The dots wobble gently, and every few seconds one slips red. Read the histogram: the mean and p50 sit calmly at the nominal period while the p99.9 marker lives out in the tail with the preempted cycles. Now crank the preemption rate: the red misses multiply, the tail of the histogram fattens, the p99.9 marches right — and the mean still barely moves until the misses are frequent. You are watching, in real time, the exact reason a mean-based dashboard is blind to the fault that trips the robot.

Trace one cycle through the instrument, by hand

The instrument is not magic; it is the arithmetic of Chapters 0–2 running sixty times a second. Every constant it uses is named in the code: the nominal loop is SHOW_NOMINAL = 700 µs, the deadline is SHOW_PERIOD = 1000 µs, jitter is symmetric with amplitude jitAmp = jit × 700 × 0.4, and a preemption adds a one-sided stall of +1800…3300 µs. Let us push one nominal cycle and one preempted cycle through it and predict, on paper, exactly what the readout will say.

The healthy cycle. Take the default jitter of 15 %. The jitter amplitude is

so this cycle draws a loop time somewhere in 700 ± 42 = 658…742 µs. Say the random draw lands at 720 µs. Compare to the deadline: 720 < 1000, so miss = false.

On the timeline this dot renders teal, sitting comfortably inside the on-time band; in the histogram it drops into a bin left of the green deadline line. The widest this cycle could ever be at 15 % jitter is 742 µs — still 258 µs of clearance.

This is the whole reason jitter alone rarely misses: symmetric jitter at any sane amplitude stays inside the 300 µs headroom the nominal loop leaves. It shifts a dot left or right of its tick; it does not launch it over the deadline.

The preempted cycle. Now suppose the preemption coin comes up (probability 0.8 %) and the stall draws 2500 µs out of its 1800–3300 range. The loop time is

Against the 1000 µs deadline that is

— a red dot flung far above the on-time band, and a lonely bar way out in the right tail of the histogram. That factor of 3.20× is a computed number, not a guess: 3200 µs of work against a 1000 µs window.

Notice how badly a single such cycle blows the deadline — over-run by 2200 µs, more than double the whole period. And yet, as the next worked example shows, it barely disturbs the mean. That asymmetry — catastrophic for the cycle, invisible to the average — is the entire lesson in one dot.

One honest subtlety the instrument teaches. If you shrink the window so that only a single cycle in 1000 is preempted, the p99.9 index (0.999 × 999 ≈ 998) lands just below that lone spike and interpolates to about 702 µs, not 3200 — you need the miss rate to exceed 0.1 % before the p99.9 marker climbs onto the tail at all.

This is the "quote the percentile that matches your fault rate" rule from Chapter 2 made concrete: a 1-in-1000 fault needs p99.9; a 1-in-10 000 fault hides from p99.9 and needs p99.99. The percentile you watch has to be as rare as the fault you are hunting.

The instrument's default 0.8 % — 8 faults per 1000 — is deliberately chosen to sit just above the p99.9 threshold, so the marker jumps out onto the tail where you can see it. Nudge preemption below 0.1 % and the p99.9 slides back into the healthy cluster even though real misses are still happening; that is not a bug, it is the percentile telling you to look further out.

That is the honest teeth behind the “look further out” advice above. As the fault gets rarer, the percentile you must quote marches from p99 to p99.9 to p99.99 to p99.999, one decimal place per order of magnitude in the fault rate. A dashboard frozen on p99.9 is exactly one order of magnitude blind: it will show green through a fault that trips the robot once every ten thousand cycles — roughly once every ten seconds at 1 kHz.

Design with numbers: what the constants encode

The instrument's constants are not arbitrary; each one is a real-time design decision you can read off in numbers. Every pixel you see on cv-show is one of these seven values doing its job — here is the whole mapping from on-screen element to the exact variable in showStep/showDraw and the number it carries:

Utilisation. The nominal loop takes 700 µs inside a 1000 µs period, so the healthy utilisation is SHOW_NOMINAL / SHOW_PERIOD = 700/1000 = 0.70. That leaves 300 µs of slack — 30 % of the period.

This slack is exactly why symmetric jitter of ±42 µs (or even the ±280 µs worst case) never crosses the line: jitter eats into slack, and there is a lot of slack to eat. The deadline sits at 1.0× the period because in a periodic control loop the cycle must finish before its own next release; overrun past 1.0 means the next cycle starts late, which is the definition of a miss.

Set the nominal to 950 µs instead of 700 (utilisation 0.95) and even modest jitter would start clipping the deadline. The 30 % headroom is a design choice, not luck — it is the margin the loop rate is deliberately given so that ordinary jitter cannot turn into a miss.

Dots to wall-clock. The timeline shows SHOW_MAXDOTS = 60 cycles at once. If this loop runs at 1 kHz (the rate the whole lesson has assumed — 1000 µs per cycle), then 60 dots span 60 × 1 ms = 0.06 s = 60 ms of real time. The full 2000-sample statistics window is 2000 × 1 ms = 2.0 s of history. So the timeline is a 60 ms magnifying glass and the histogram is a 2-second memory — which is why the dots refresh in a blink while the percentile markers settle more slowly.

Preemption slider to misses per second. A preemption rate p at loop rate R produces, in expectation, p × R preemptions per second. At the default p = 0.8 % = 0.008 and R = 1 kHz that is 0.008 × 1000 = 8 misses per second — roughly one red dot every 125 ms.

On the 60-dot timeline you therefore expect 60 × 0.008 = 0.48 red dots visible at any instant. About one frame in two shows a miss, which matches "every few seconds one slips red" once you remember the timeline is only a 60 ms window into a stream running far faster than your eye.

Double the slider to 1.6 % and you double to 16 misses per second; halve it and you halve them. That linear relationship — misses per second grows in direct proportion to the rate — is the whole point of putting a rate control on a Poisson-like fault: the slider is a dial straight onto the fault frequency.

Now watch the statistics react to that same sweep. Below is the full 2000-sample window computed by hand at four slider settings, each stall taken as a representative 3200 µs = 3.20× cycle. Read it top to bottom and the lesson is a single column: the mean creeps, the p99.9 jumps to the tail the instant even one stall lands past its rank, and the miss count grows in lockstep with the slider. Every number is arithmetic you can redo on paper — the mean is ((2000−k)×700 + k×3200)/2000 for k stalls, and each percentile is the rank test from Chapter 0.

Two boundaries in that table earn their place. The p99 is nominal at 0.8 % but 3200 at 1.6 %: the p99 rank is index 0.99 × 1999 ≈ 1979, so it only climbs onto the tail once the stalls number more than 2000 − 1979 ≈ 21 — which happens between 0.8 % (16 stalls, below 21) and 1.6 % (32 stalls, above 21). That is why doubling the fault rate flips the p99 from green to red while the p99.9 was already red at half that rate: each percentile has its own fault-rate threshold, exactly its own tail fraction. And the mean× column is the alibi — even at a punishing 8 % preemption, one cycle in twelve blown, the mean is still 0.90× the deadline. A mean-only alarm would call this healthy while the robot is missing 160 deadlines a second.

The instrument's kernel, from scratch

The showcase is doing streaming statistics: it keeps a bounded ring of recent loop times, re-sorts it, and reads off the same percentile function you wrote in Chapter 0. Here is that kernel in the same four-line spirit — a ring buffer, a re-sort, and the percentile read — exactly what showStep and showDraw perform every frame:

python
from collections import deque

WINDOW = 2000                       # SHOW_HIST_N: keep the last 2000 loop times
times  = deque(maxlen=WINDOW)         # a ring buffer: push evicts the oldest for free

def on_cycle(t_us):                 # called once per control cycle, like showStep()
    times.append(t_us)              # O(1) push; maxlen drops the stalest sample

def stats():                        # called to refresh the readout, like showDraw()
    s = sorted(times)              # re-sort the whole window; tail stats need order
    n = len(s)
    def pct(q):                     # same linear-interp percentile as Chapter 0
        idx = q / 100 * (n - 1)
        lo  = int(idx)
        return s[lo] + (idx - lo) * (s[min(lo + 1, n - 1)] - s[lo])
    mean = sum(times) / n
    return mean, pct(50), pct(99), pct(99.9), sum(1 for x in times if x > 1000)  # mean, p50, p99, p99.9, misses

The library one-liner for the whole percentile trio is numpy.percentile(times, [50, 99, 99.9]), and the miss count is numpy.count_nonzero(numpy.asarray(times) > 1000) — same answer, same interpolation convention as the from-scratch version. The re-sort every frame is O(n log n) over 2000 samples, which is trivial at 60 Hz; a production profiler would keep an order statistic tree or a t-digest instead, but the toy re-sort is exactly right for making the mechanism visible. This is the same showPct the instrument calls at cv-show's draw — you have now written its engine.

python
# the percentile markers the showcase draws, as a numpy one-liner
import numpy as np
p50, p99, p999 = np.percentile(times, [50, 99, 99.9])   # the teal / purple / red lines on the histogram
mean            = np.mean(times)                        # the warm line — the one that lies
misses          = np.count_nonzero(np.asarray(times) > 1000)  # dots that crossed the deadline

The bars the histogram draws are the other half of the kernel, and they are just as simple: a fixed value range split into equal-width bins, one increment per sample. This is verbatim the arithmetic showDraw runs at the cv-show canvas — a bin width binw = (xhi−xlo)/nb and a bin index b = floor((t−xlo)/binw) — only written here as plain Python so you can see there is no library magic under the bars:

python
def histogram(times, xlo=600, xhi=3800, nb=48):   # same range/bin count as showDraw()
    binw = (xhi - xlo) / nb                     # bin WIDTH in us: (3800-600)/48 = 66.67 us/bin
    hist = [0] * nb                            # one counter per bin, all starting at zero
    for t in times:
        b = int((t - xlo) / binw)             # which bin: floor of (value - lo) / width
        if 0 <= b < nb:                       # clip anything outside [xlo, xhi]
            hist[b] += 1                       # drop this sample into its bin
    return hist                                # bar heights, left (fast) to right (tail)

Trace one sample to see the bins are not arbitrary. A nominal 700 µs cycle lands in bin floor((700−600)/66.67) = floor(1.5) = 1 — the second bar from the left, right where the healthy cluster piles up. A 3200 µs stall lands in bin floor((3200−600)/66.67) = floor(39.0) = 39 — far out in the right tail, past the green deadline line at bin floor((1000−600)/66.67) = 6. So the deadline sits at bar 6, the healthy mass at bar 1, and every miss at bar 39-and-beyond: the visual gap between the cluster and the tail is 33 empty bins wide, which is exactly why a lonely stall reads as a bar stranded out in space rather than a fatter version of the main hump. The library one-liner for the same counts is numpy.histogram(times, bins=48, range=(600, 3800))[0] — identical convention, same left-closed bins.

One last piece of the kernel is worth doing on paper end to end: the pct() interpolation, run against a concrete sorted window so you can see every arithmetic step the red marker takes. Take the default 0.8 % window — 1984 nominal cycles at 700 µs followed by 16 stalls at 3200 µs, sorted so the stalls occupy indices 1984…1999. Here is pct(99.9) line by line:

Now run the same four steps for pct(50) on the same window, so you can lay the two side by side and see the machine is identical:

Identical machine, four identical steps, and the answer diverges by a factor of 4.6 — p50 = 700 versus p99.9 = 3200 — purely because the rank of p99.9 falls inside the 16-sample stall block while the rank of p50 falls in the 1984-sample healthy block. That divergence — not any special “tail formula” — is the entire mechanism behind the two markers you watch pull apart on the screen. Slide preemption up and you are literally pushing more stalls above the p99.9 rank, then above the p99 rank, then eventually above the p50 rank; each marker jumps the instant the stall count crosses its own tail fraction.

What to notice

• Jitter alone rarely misses. At high jitter but zero preemption, the dots wobble a lot but most stay under the deadline — symmetric jitter widens the histogram both ways. The tail that misses comes from preemption, which is one-sided (a stall only makes a cycle longer, never shorter).
• The mean is the last number to react. Drive the preemption up and the p99.9 has crossed the deadline long before the mean has moved a hair. If your alarm is on the mean, you find out last.
• Misses cluster. Because preemption is periodic, the red dots arrive on a rhythm — exactly the once-a-second signature from Chapter 0. A random-looking fault that is actually periodic is a task, not luck.
• The deadline line is the only thing that matters for hard real-time. The instrument's smallest possible stall is 1800 µs, so the gentlest preempted cycle is 700 + 1800 = 2500 µs = 2.50× the period, and the harshest is 700 + 3300 = 4000 µs = 4.00×. Every red dot is a computed 2.5–4.0× over the line — a miss whether the mean is beautiful or not. Hard real-time is a pass/fail on every single dot.
• Each percentile has its own fault-rate threshold. A percentile only climbs onto the tail once the stalls outnumber its own tail fraction of the window. p99.9 needs just 2000 × 0.001 = 2 stalls; p99 needs 2000 × 0.01 = 20…21; p50 would need a full 1000. So as you raise the slider the markers light up in order — p99.9 first, then p99, and the mean last of all — and which ones are red tells you the fault rate without reading a single count.
• The bars use a square-root height so the tail stays visible. showDraw scales each bar as sqrt(count/max), not count/max. At 0.8 % the tallest bar holds ~1984 samples and the lonely stall bar holds 16; a linear height would draw that stall bar at 16/1984 = 0.81 % of the panel — a single invisible pixel. The square root lifts it to √(16/1984) = 8.98 %, an 11× boost, so the rare tail you most need to see does not disappear under the healthy hump. It is a deliberate readability choice, not a distortion of the counts — the numbers in the readout are still exact.

Frontier: what a real profiler measures that this toy abstracts

This instrument draws loop times you can already reason about; a production tail-latency profiler measures the same quantity on real silicon and formalises why it is the number that governs the system.

The reference is Dean and Barroso, "The Tail at Scale" (Communications of the ACM, 2013). Their claim: in a pipeline of stages, the loop's latency is set by the slowest stage on each pass. So if a control cycle chains, say, ten stages that each go slow 1 % of the time independently, the loop hits a tail on 1 − 0.99¹⁰ ≈ 9.6 % of cycles — the tail amplifies as you add stages, and shrinking each stage's average does almost nothing to it.

That is precisely why the showcase plots a distribution and a p99.9, not a bar of averages: it is a picture of the exact statistic Dean and Barroso argue governs the system.

The modern robotics counterpart is what tools like cyclictest and ftrace on a PREEMPT_RT kernel (mainlined into Linux in 2024) actually record: the true preemption latency the OS imposes — the real-world source of the 1800–3300 µs stall this sim injects by hand.

The toy abstracts scheduler internals, cache misses, and IRQ storms into one +1800…3300 µs draw. A real profiler measures each of those separately and attributes the tail to its true cause — which is the difference between seeing that the loop has a tail (this sim) and knowing which line of code produced it (the profiler).

Now confirm it on the instrument itself. The screen is the second half of the test: drag preemption to zero and jitter to 100 %, and you will watch the histogram widen symmetrically toward — but never past — the green line, the mean and p99.9 markers staying within 280 µs of the nominal, and the miss count frozen at zero. Then flick preemption back up and watch the p99.9 leap to the tail while the mean barely stirs. If you can predict which marker moves and whether a dot goes red before you touch a slider, you hold the real-time mental model — that is the whole point of the instrument.

Chapter 5: Field Guide

This is the reference chapter — the page you keep open next to the terminal. It condenses the whole lesson into the cheat sheet, the design patterns, the drills, and the debugging table you reach for when a real robot is misbehaving at 2 a.m. Nothing new is introduced; everything here was derived above.

The one-page cheat sheet

Derivation you should be able to rebuild: the utilisation bound

The cheat sheet quotes the rate-monotonic guarantee — a task set is schedulable if the total utilisation U stays under n(2^1/n−1) — as a fact to reach for. A field guide is not a place to memorise a formula you cannot rebuild, so here it is derived from nothing, then run on a concrete task set. If someone erases the whiteboard and asks you to reproduce the bound, this is the path.

Set the stage. Rate-monotonic (RM) priority assignment means: the task with the shortest period gets the highest priority, always, and never changes. A task i is described by two numbers — its worst-case compute time C_i (how long one job takes if nothing goes right) and its period T_i (how often a new job arrives). Its individual utilisation is the fraction of the CPU it demands: u_i = C_i/T_i. The total is U = ∑ C_i/T_i.

The question RM answers is: for what values of U is every deadline met, no matter how the periods line up? Liu & Layland's 1973 answer is the least-upper-bound — the largest U below which schedulability is guaranteed. Above it, some task sets still work and some do not; below it, all of them work. We derive that bound.

Build the worst case by hand for two tasks, then read off the pattern. Order them by period, T₁ ≤ T₂, so task 1 is higher priority. The stressful arrangement is when task 2's period is less than twice task 1's, so during one job of task 2 the higher-priority task 1 fires twice and steals CPU twice. Push each task's compute up until the schedule is exactly full — any more and a deadline slips. Working through that "just barely fits" algebra and then minimising the resulting utilisation over the ratio T₂/T₁ gives, for two tasks, the tightest packing at

Generalise the same "fill until exactly full, then minimise over the period ratios" argument to n tasks and the packing worsens smoothly: the more tasks you interleave, the more ways their firings can gang up, so the guaranteed bound drops. The closed form is

Sanity-check its shape before trusting it. For one task, U₁ = 1·(2¹−1) = 1 — a single periodic task can use the whole CPU, which is obviously right. For two, 0.8284 as above. For three, 0.7798. As n → ∞ the bound descends monotonically to ln 2 ≈ 0.6931 — the famous "69 %" rule of thumb. So the bound is 100 % for one task and sinks toward 69 % for many; if your total utilisation is under 69 % you are safe for any number of RM tasks without even counting them.

Now run it on a real three-task loop — the exact drill from the coding list, worked here with every intermediate value so nothing is asserted:

Sum the demands, then compute the bound for n = 3, then compare — substitute the numbers, then simplify, one step at a time:

Derivation you should be able to rebuild: the Δt bug in drill 6

Drill 6 asserts a "30 % velocity overestimate." A field guide that only states the answer is the thing this lesson warns against. Here is the arithmetic in full, so you can reproduce it at the whiteboard and, more importantly, see why a timing slip becomes a control error.

The setup, from Chapter 0's mechanism. A joint controller runs nominally every T = 1 ms. Each cycle it estimates angular velocity by finite difference — the change in angle over the interval — and the code is written v = (angle_now − angle_last) / T, hard-coding the nominal interval T. Now one cycle runs 30 % late: the CPU was stolen, so the real elapsed interval is

Take a concrete rotation over that late cycle: the joint actually moved Δθ = 0.13 rad during the 1.30 ms. Compute both the truth and what the buggy code reports — substitute, then simplify:

The ratio is not a coincidence of the numbers — it is structural. Both estimates share the same numerator Δθ, so the ratio is just the interval ratio flipped: dividing by a T that is 1.30× too small inflates the result by exactly 1.30×. The velocity error equals the timing error, one for one. (Cross-check against Chapter 0's 4 ms straggler: dt_real = 4T, so the overestimate there is 4×, matching the "velocity overestimated by 4×" claim in the hook.)

Now carry it into torque, which is where the robot lurches. Near the target the position term is small, so the braking command is dominated by the derivative term τ_d = k_d · v. Take a gain k_d = 0.5 N·m·s/rad:

Derivation you should be able to rebuild: which percentile to quote

The cheat sheet's tail row says "match the percentile to the fault rate: 1-in-1000 → p99.9." That is the single most useful sentence in the guide, and it is worth deriving rather than trusting, because getting it wrong is how a healthy-looking dashboard hides the fault that trips the robot.

Start from the definition. The p_q percentile of a sample is the value that q % of the sample falls below; equivalently, (100−q) % falls at or above it. So a percentile is a knob that selects a fraction of the worst cycles: the higher q, the thinner the slice of stragglers you are looking at.

Now match the knob to the fault. A fault that happens once every N cycles occupies a fraction 1/N of all cycles — it lives in the top 1/N of the sorted loop times. To point the percentile exactly at that slice, choose q so that the fraction above it, (100−q)/100, equals 1/N:

Substitute the rates and simplify, one line each — this is the whole cheat-sheet row, derived:

This is why the lesson's once-a-second trip needs p99.9 and nothing softer. At the 1 kHz control loop, one thousand cycles pass every second, so a once-a-second fault is exactly a 1-in-1000 event: N = 1000, and the formula lands on p99.9. Quote p99 instead and you are averaging over the top 1 % — ten times wider than the fault — so the straggler is diluted by nine healthy-ish cycles and the number looks fine. The median and mean are hopeless: they describe the middle and the bulk, and the fault is neither.

Put the same 1-in-1000 rate into human units so the stakes are concrete. One fault per second is 3,600 per hour and 86,400 per day — every one a chance for the arm to overshoot. A statistic that hides 86,400 daily faults is not conservative; it is blind.

Design pattern, instantiated with numbers: the detector → planner seam

The patterns table lists "non-blocking read + last-known-good" as an abstraction. Here it is made concrete with a latency budget and a bandwidth budget in actual numbers — the two figures a design reviewer will ask for by name.

The seam: a 30 Hz object detector (firm) feeds a 1 kHz motion planner (hard). Their rates do not divide evenly — the planner fires 1000/30 ≈ 33 times per detection — so most planner cycles have no fresh detection, and any single detection may be late. Wire this with a blocking read and the hard 1 kHz loop stalls whenever the firm 30 Hz producer is late: a firm miss becomes a hard miss, and the whole robot faults on a dropped video frame. Wire it with a non-blocking read of a double-buffered slot and a late detection is simply skipped — the planner reuses the previous target for one more of its 33 cycles, a ≤33 ms staleness the controller already tolerates.

Latency budget (numbers). The planner's own deadline is its period, 1 ms. The non-blocking read must cost effectively zero against that budget: reading one pointer-swap slot is ~tens of nanoseconds, so it consumes <0.01 % of the 1 ms and the hard loop's worst case is unchanged. A blocking read, by contrast, could add up to a full detector period — 33 ms, or 33× the planner's entire deadline — the instant the producer stalls. That single number, 33 ms of possible blocking against a 1 ms deadline, is the whole argument for the pattern.

Bandwidth budget (numbers). Say each detection is a list of up to 20 boxes, each 6 floats (x, y, w, h, class-id, score) at 4 bytes = 24 bytes, so 480 bytes per detection. At 30 Hz that is 480×30 = 14.4 KB/s — utterly negligible, which confirms the seam is latency-bound, not bandwidth-bound: the design problem is when the data arrives, never how much. Double-buffering costs one extra 480-byte slot to decouple producer and consumer — a trade of under a kilobyte of RAM to turn a hard fault into a harmless skip.

From-scratch: the schedulability checker

The two derivations above become fifteen lines you can carry in your head. This is the whole utilisation test with no library — sum the demands, compute the least-upper-bound, compare, and report where the margin sits. Run it on the three-task set above and it prints the same 0.70 ≤ 0.7798 verdict you worked by hand.

python
def rms_schedulable(tasks):
    # tasks: list of (compute_C, period_T) in the same time unit
    U = sum(C / T for C, T in tasks)   # total demanded fraction of the CPU
    n = len(tasks)
    bound = n * (2 ** (1 / n) - 1)         # Liu & Layland least-upper-bound
    return {
        "U": round(U, 4),
        "bound": round(bound, 4),
        "margin": round(bound - U, 4),      # positive = headroom to the guarantee
        "guaranteed": U <= bound,             # True = every deadline provably met
    }

# the torque / estimator / telemetry loop, times in milliseconds
loop = [(0.30, 1.00), (0.10, 0.50), (0.05, 0.25)]
print(rms_schedulable(loop))
# -> {'U': 0.7, 'bound': 0.7798, 'margin': 0.0798, 'guaranteed': True}

Field-guide explorer

The cheat sheet packs three deadline classes and one schedulability rule into a table. Here they are made live: drag the two sliders and the panel redraws the value-function shape and the utilisation-vs-bound bar together, so you can feel how a class and a load interact before you ever quote a number.

System-design patterns

Coding drills

Work these by hand or in the Code Labs above until they are reflex:

1. Percentiles from a vector. Given loop times, compute mean, p50, p99, p99.9 and the tail-to-mean ratio. Explain why the mean cannot recover the p99.9. (Code Lab, Chapter 2.)
2. Count the misses. Given per-cycle compute and a periodic preemption, compute finish times and count deadline misses at a given budget. Show that raising the budget past the preemption spike drives misses to zero. (Code Lab, Chapter 3.)
3. Break the loop with jitter. Run a controller at its stability edge with fixed vs jittered Δt of the same mean; show the jittered one diverges. State the Jensen's-inequality reason. (Code Lab, Chapter 3.)
4. Utilisation check. Given three tasks (C, T each), compute U and compare to the RMS bound; state whether the schedule is guaranteed. (Worked in Chapter 3: U = 0.45, bound 0.78 → schedulable.)
5. Classify a deadline. For a given subsystem, write V(t) and label it hard/firm/soft; read the value at a specified late arrival. (Worked in Chapter 1.)
6. Derivative-gain error. A cycle runs 30 % late but the code divides by the nominal Δt. Compute the resulting velocity overestimate (30 %) and the torque error it produces.

Debugging scenarios: symptom → cause → the metric that reveals it

Classical vs modern

Where this sits in the track

This lesson is the mental model; the mechanisms that deliver real-time behaviour live in the sibling lessons. Follow these to go deeper:

Recommended reading

• Liu & Layland, "Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment," JACM, 1973 — the rate-monotonic bound.
• Dean & Barroso, "The Tail at Scale," Communications of the ACM, 2013 — why the tail, not the average, governs a multi-component system.
• Vestal, "Preemptive Scheduling of Multi-Criticality Systems…," RTSS 2007 — the founding mixed-criticality paper.
• Cervin et al., "How Does Control Timing Affect Performance?… Jitterbug and TrueTime," IEEE CSM, 2003 — quantifying jitter's cost to control.
• Wilhelm et al., "The Worst-Case Execution-Time Problem…," ACM TECS, 2008 — the survey of WCET methods and tools (aiT and others).
• Buttazzo, Hard Real-Time Computing Systems (Springer, 3rd ed.) — the standard textbook for scheduling theory.

"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman. A healthy mean is exactly the fool's comfort; the tail is the truth.