Fusion Architectures & the JDL Model

Chapter 1: The JDL Levels — Five Rungs of "What Am I Even Reasoning About?"

Before two engineers can argue about where to fuse, they need to agree on what they're fusing toward. "Sensor fusion" is a maddeningly broad phrase — it covers everything from "average two voltage readings" to "decide whether that approaching object intends to harm us." Those are not the same problem, and confusing them wastes meetings. In 1991 the U.S. Joint Directors of Laboratories (JDL) Data Fusion Group cut the chaos into a ladder of levels, and Steinberg, Bowman, and White revised it in 1999 into the version everyone now cites. The JDL levels are not an algorithm — they are a vocabulary, a shared map so people know which rung of the problem they're standing on.

The key idea: as you climb the ladder, the thing you reason about gets more abstract and more meaningful. At the bottom you fuse raw signals; near the top you fuse judgments about intent and threat; and there is a special level off to the side that watches the fusion itself. Here are the five rungs, from the ground up.

Level 0 — Sub-object / signal assessment

Fuse the raw signals before anything is identified as an object. This is the basement: pixels, radar returns, raw waveforms. A camera's photons and a radar's reflected energy are combined or cleaned before the system decides "that's a car." Classic Level-0 work is signal conditioning, denoising, and pixel-level or waveform-level alignment — for example, summing two microphones' raw waveforms to boost a faint sound, or stacking two camera frames into a multi-channel tensor. Nothing here knows what an "object" is yet; it is fusion of measurements, not of things. (The 1999 revision added Level 0 explicitly — the original 1991 model started at Level 1.)

Level 1 — Object assessment (the workhorse)

Combine data to estimate the state of individual objects: their position, velocity, identity, and a track over time. This is where most of robotics and most of this entire 22-lesson series lives. "There is a car at (18 m, 2 m), moving 12 m/s, ID = vehicle, and here is its trajectory." Level 1 takes detections from multiple sensors and answers what is out there and where. The Kalman filter (Lesson 5), multi-target tracking, and data association are all Level-1 machinery. When someone says "sensor fusion" with no qualifier, they almost always mean Level 1.

Level 2 — Situation assessment

Reason about the relationships among objects — the whole scene, not one thing. Level 1 gives you a list of objects; Level 2 asks what they mean together. "The car ahead is braking, the pedestrian is stepping off the curb toward the crosswalk, and they are on a collision course." It's the difference between a list of facts and a story. Situation assessment groups objects, infers their interactions, and builds a coherent picture of the tactical environment. In a self-driving stack this is "scene understanding."

Level 3 — Impact / threat assessment

Project the situation forward and assess consequences — how bad could this get, and what's the threat? Level 2 says "they're on a collision course"; Level 3 says "if nobody acts, impact in 1.4 seconds, severity high — this is a threat that demands braking now." It's predictive and evaluative: it estimates intent, projects futures, and ranks dangers. In its defense origins this was literally "is that aircraft hostile, and what can it do to us?" In a car it's "which of the twelve tracked objects is actually about to hurt someone?"

Level 4 — Process refinement (the meta-level)

Fuse information about the fusion process itself, and act on it — this is the level that closes the loop. Level 4 doesn't reason about the world; it reasons about how well we're sensing the world. "We have a blind spot to the left; turn the steerable radar that way." "Camera 2's confidence has dropped — trust it less and request a cleaning cycle." "We're uncertain about that pedestrian; allocate more compute to tracking it." Level 4 is sensor management, resource allocation, and self-monitoring — the brain deciding where to point its eyes. It sits beside the ladder because it feeds back into every other level.

One pedestrian, all five levels

To feel how distinct the rungs are, run a single event — a pedestrian stepping off a curb — up the whole ladder:

Same pedestrian, five completely different products. A team building Level-1 object tracking and a team building Level-3 threat assessment are doing fundamentally different jobs, even though both say "we work on sensor fusion." That is precisely the ambiguity the JDL levels exist to kill.

A short history — why the defense world built this ladder

It helps to know where the JDL model came from, because the origin explains its shape. In the 1980s, military command-and-control systems were drowning in sensor feeds — radar, sonar, infrared, electronic-intelligence intercepts — and there was no shared language for what "fusing" them even meant. One contractor's "fusion" was cleaning radar noise; another's was deciding whether an incoming aircraft was hostile. They were both right, and they could not talk to each other. The U.S. Department of Defense's Joint Directors of Laboratories convened a Data Fusion Group around 1985 to fix the Tower of Babel, and by 1991 they had the original four-level model (Levels 1–4). The 1999 Steinberg/Bowman/White revision added Level 0 (because people kept doing signal-level fusion that the model couldn't name) and sharpened the definitions into the version we use today.

The lasting lesson is that the JDL model is a communication tool first and a technical one second. Its highest value is letting two engineers in a meeting say "I'm working Level 1 tracking, you're working Level 3 threat" and instantly know they are not competing — they're building adjacent rungs of the same ladder. A taxonomy that prevents an argument is worth more than one that wins it.

Why Level 4 is the secret sauce — a worked sensor-management decision

Level 4 deserves a closer look because it's where a system stops being a passive observer and starts being an active investigator. Say a car is tracking twelve objects, and one — track #7, the pedestrian — has high positional uncertainty (its variance is 9 m² while the others are around 1 m²). A Level-1 system would just report all twelve and move on. A Level-4 system reasons about the uncertainty itself and acts:

• Allocate the steerable radar beam to #7 for the next few cycles — trading attention away from well-known tracks toward the uncertain one, which drives #7's variance down fast.
• Raise the compute budget for tracking #7 (more particles, a finer model), because it's the one that matters.
• Request a sensor health action — if #7's uncertainty traces to camera glare, trigger a wash/wipe or down-weight that camera.

This is fusion about the fusion: the system fuses information about where its own knowledge is weakest and reallocates sensing to fix it. It's the difference between a person who stares straight ahead and one who turns their head toward the rustle in the bushes. Level 4 closes the perception loop, and it's why the original 1991 JDL model put "process refinement" at the top — it governs all the others.

A second walk-through — an airliner's air-data, up the ladder

To prove the ladder isn't car-specific, run a completely different system — an airliner's air-data — up the same five rungs:

Note Level 1 here is the median-voting from Lesson 1 — competitive fusion is a Level-1 object-assessment activity. And Level 4 is doing something striking: it noticed a sensing problem (a pitot disagreement) and took a physical action on the sensors (turn on the heaters). That feedback from Level 4 down to the hardware is what separates a smart system from a passive pipeline, and it's exactly the loop the original 1991 model put at the top.

Why a model at all? Because it makes architecture decisions concrete

Here's the payoff that connects this chapter to the rest of the lesson. The level you're operating at strongly suggests an architecture. Level 0 (raw signals) is bandwidth-hungry — you're shipping pixels — so it pushes you toward fusing close to the sensor or toward a fat centralized link. Level 3 (threat) is compact and judgment-heavy — you're shipping a short threat list — so it lives naturally in a central decision-maker. Level 1 (objects/tracks) is the sweet spot for distributed architectures: each sensor can produce a track locally and ship the small track inward. The JDL ladder and the architecture choice are deeply intertwined, which is why we learned the ladder first.

JDL is one of several lenses — how it relates to the others

The JDL levels aren't the only way the field slices fusion, and knowing how they relate keeps you from confusing them. Three lenses, three questions:

These are four independent axes, not competing answers. A single fusion block might be: Level 1 (object), DEI–DEO (decisions in/out), competitive (voting), distributed (nodes ship verdicts to a fuser) — all true at once, each describing a different facet. When someone hands you a fusion system, asking all four questions pins it down completely. Confusing the axes — "is it Level 1 or is it distributed?" — is a category error; it's both, because they're orthogonal.

Chapter 2: Dasarathy's DFD Model — Classifying Fusion by What Goes In and What Comes Out

The JDL levels told you what abstraction you're reasoning about. But there's a second, sharper way to slice fusion, proposed by Belur Dasarathy in his influential 1997 Proceedings of the IEEE survey, and it's the one engineers reach for when designing actual modules. Dasarathy's insight: every fusion block has an input and an output, and each can be one of three things — Data (raw measurements), Features (extracted descriptors), or Decisions (final verdicts). Classify a fusion block by its input-output pair, and you've pinned down exactly what it does.

In Lesson 1 we met the three classic levels — raw, feature, decision — as the single stage where you splice two pipelines. Dasarathy's contribution is to realize the input and the output need not be the same kind. A fusion block can take in features and emit a decision. It can take in data and emit cleaner data. There are five input-output combinations that actually make sense (you never get more information out than you put in), and naming them gives a precise label to every fusion module you'll ever build.

The three building blocks: Data, Features, Decisions

Think of every sensor feeding a little pipeline: raw data → extracted features → a decision. Dasarathy names the three substances that can flow between fusion blocks:

• Data (D) — the raw, unprocessed measurement. Pixels, radar returns, raw waveforms, point clouds. Maximum information, maximum volume.
• Features (F) — descriptors extracted from the data. "An edge here, a corner there, a bounding box, a cluster of points 12 m ahead." Compressed but still structured.
• Decisions (De) — a final verdict. "Pedestrian, 90% confident." "Airspeed = 249 knots." A label or a number, the smallest of all.

The five named modes

Now combine an input kind with an output kind. Only five combinations are useful (you can't manufacture detail you discarded), and each has a memorable acronym — here they are with the canonical example:

Read the acronyms aloud once and they stick: DAI = "data in," FEO = "feature out," DEI = "decision in," and so on. The five form a staircase: you can stay on a step (DAI–DAO, FEI–FEO, DEI–DEO) or step down the abstraction ladder (DAI–FEO, FEI–DEO), but you can never step up — there is no "DEI–DAO," because you cannot reconstruct the original pixels from the word "pedestrian." Information only flows downhill from data to decision.

One scenario through three Dasarathy modes

Take the camera + radar pedestrian question once more and route it through three different DFD modes, watching how the choice changes what crosses the wire:

Watch the data shrink by orders of magnitude top to bottom, and the modularity loosen: DAI–DAO welds the sensors into one network; DEI–DEO lets each run a self-contained pipeline and just reconciles two opinions. This is the same bandwidth-versus-information trade you met in Lesson 1, now with crisp names for every input-output pair. And it connects straight to architecture: DEI–DEO (decision fusion) is what makes distributed and decentralized architectures cheap — each node decides locally, then ships only its tiny verdict.

Putting numbers on the compression — the three modes for one frame

To feel why the mode choice is an architecture decision and not a cosmetic one, count the bytes for a single camera+radar pedestrian frame:

Six megabytes versus sixteen bytes for the same conceptual question, "is there a pedestrian?" The mode you pick is not a detail — it changes the data on your bus by five orders of magnitude, which is precisely what decides whether you can centralize (DAI–DAO, fat bus required) or must distribute (DEI–DEO, fits anywhere). Dasarathy's contribution is giving you a precise label for each rung of that compression staircase.

Why not always pick the cheapest mode (decision fusion)?

If DEI–DEO is 375,000× cheaper on the wire, why isn't everything decision fusion? Because each step down the ladder discards information the fuser might have needed. Concretely: a camera sees a faint vertical shape and reports "pedestrian, 0.55" — barely above chance. A radar sees a slow-moving return and reports "pedestrian, 0.55". At the decision level, the fuser has two weak 0.55 verdicts and can only weakly conclude "maybe." But at the feature level, the camera's "vertical edge at bearing 12°" and the radar's "moving return at bearing 12°, 1.2 m/s" line up at the same bearing and consistent motion — a coincidence the decision-level fuser couldn't see because each sensor had already collapsed its rich feature into a single number. Feature fusion would conclude "yes, pedestrian, high confidence" from evidence that decision fusion threw away. The cheaper the mode, the more pre-fusion information you discard, and the more confident-but-wrong sensors can slip through. You pay for bandwidth with the fuser's ability to notice agreement that only shows up in the details.

How to choose the mode — three quick questions

Faced with a real fusion block, you can pin its mode with three questions:

• Are the sensors the same type and perfectly aligned? If yes and you have bandwidth, you can do DAI–DAO (raw-in, raw-out) — e.g. two identical LiDARs into one cloud. If they're different types, raw fusion is impossible (you can't stack a waveform on an image), so move up the modes.
• Can the sensors output a meaningful intermediate descriptor? If yes, FEI–FEO or FEI–DEO lets you mix dissimilar sensors at modest bandwidth — the dominant choice for camera+radar+LiDAR.
• Do you want fully independent, swappable modules and minimal bandwidth? Then DEI–DEO — each sensor decides, you fuse verdicts. This is the home of competitive voting and the enabler of distributed/decentralized architectures.

Composing the two models — a full system labeled

JDL (altitude) and DFD (plumbing) compose into a complete description. Trace one camera+radar pedestrian-detection system through both at once:

One system, walked simultaneously up the JDL ladder and across the Dasarathy modes. Notice the modes descend the abstraction (DAI–DAO → DAI–FEO → FEI–DEO → DEI–DEO) as the JDL level climbs (L0 → L1 → L2 → L3) — you compress the data form precisely as you elevate the meaning. That coupling — less data, more meaning, as you go — is the deep unity behind both models, and it's why engineers cite them together.

Chapter 3: Centralized Fusion — Optimal On Paper, Brutal In Hardware

We've named the levels and the modes. Now the central question of the lesson: where does the fusion live? The simplest, most powerful answer is centralized fusion — ship every sensor's raw measurements to one fuser, and combine them all in one place. It is the architecture a statistician would draw, because it is the one that throws away nothing. And it is the architecture that breaks first when it meets a real vehicle's wiring harness.

The defining feature: each sensor is a dumb pipe. The forward radar doesn't track anything; it sends its raw detections inward. The cameras don't classify; they send pixels (or near-raw detections) inward. All the intelligence sits in one central fuser that sees everything at once and produces the single fused estimate. Nothing is decided locally.

Why it's optimal — nothing is thrown away

Here's the deep reason engineers are forever tempted by centralization: a central fuser with access to all the raw measurements can, in principle, compute the statistically optimal estimate. Why? Because no information has been discarded before fusion. Recall the inverse-variance rule from Lesson 1 — the precisions (1/variance) of independent measurements simply add. With all raw data in one place, the fuser sums every available scrap of information:

Let's make it concrete with three radars all measuring the same target's range, with variances 4, 9, and 16 m². A centralized fuser with all three raw readings computes:

The fused variance, 2.36 m², is smaller than the best single sensor's 4 m² — the centralized fuser squeezed every drop of precision out of all three. No distributed scheme can beat this, because the moment a node pre-processes its raw data into a track and throws the raw data away, some information is lost (and, worse, correlations get introduced — the topic of Chapter 5). Centralized is the gold standard against which every other architecture is measured.

And the fused estimate itself is the precision-weighted mean — each reading weighted by its vote strength. With raw readings 100.2, 99.5, and 101.0 m and precisions 0.25, 0.111, 0.0625:

The fused estimate, 100.1 m, leans toward the most precise sensor (the variance-4 one, reading 100.2) and barely heeds the sloppy variance-16 sensor (reading 101.0) — exactly the inverse-variance weighting from Lesson 1, now over three raw inputs in one place. Every sensor contributed in exact proportion to how sure it was, and nothing was wasted. That is what "optimal" buys you: not just a tight variance, but the best point estimate too.

The three brutal costs

If centralized is optimal, why doesn't everyone use it? Because "optimal estimate" and "system that survives contact with reality" are different things. Centralization buys statistical perfection at three steep prices:

Cost 1 — Bandwidth. Raw streams are enormous. One forward automotive camera at 1080p/30 is ~1.5 Gbit/s of raw pixels; a LiDAR spins out millions of points per second. Shipping eight such raw streams to one fuser can exceed the entire data bus of the vehicle. You'd need a fatter, more expensive, heavier wiring harness — or you simply can't ship the raw data at all. Bandwidth is usually the first wall centralized fusion hits.

Let's put a number on it, because the gap is staggering. Take a modest car: three 1080p/30 cameras and five radars. Raw, each camera is 1920×1080×3 bytes × 30 frames/s ≈ 1.5 Gbit/s; the three together are ~4.5 Gbit/s. The five radars, raw, might add another ~0.5 Gbit/s. Centralized raw fusion would demand ~5 Gbit/s into one box. Now compare the distributed alternative (Chapter 4): if each sensor instead ships a track — say "object at (18, 2) m, velocity (12, 0) m/s, covariance 4 numbers" — that's roughly a dozen floats, ~50 bytes, at maybe 50 Hz, or ~20 kbit/s per sensor. Eight sensors: ~160 kbit/s total.

That's a factor of roughly 30,000× less data on the wire. The centralized number needs automotive Ethernet or coax and a serious harness; the distributed number fits comfortably on an ordinary CAN-FD bus. This single ratio is why the entire automotive industry moved away from centralizing raw sensor data and toward shipping tracks — the bandwidth simply isn't there for raw. The information you give up (Chapter 5 will show it's not free) is the price of fitting on the bus.

Cost 2 — Single point of failure. All the intelligence sits in one box. If that central fuser crashes, overheats, or its power rail glitches, the entire perception system goes dark at once — every sensor is now a dumb pipe with no one listening. This is the architectural sin that safety standards (automotive ASIL-D, avionics DO-178) punish hardest: a single failure must not take down a safety function. Centralized fusion with one fuser is, by construction, exactly that single point of failure.

Put it in the reliability arithmetic from Lesson 1. Suppose each sensor is 99% reliable and the central fuser is also 99% reliable. The whole centralized system works only if the fuser works and at least one sensor works — but because the fuser is required, it sits in series with everything: R_system ≤ R_fuser = 0.99. No matter how many redundant sensors you add, the system can never be more reliable than its single fuser — the fuser caps it. Add a competitive ring of five sensors (reliability 0.99999) and it's wasted, because the lone fuser drags the product back to 0.99. A single mandatory fuser is a series element that caps your whole system's reliability at the fuser's own reliability. That's why safety-critical designs either duplicate the fuser (two in parallel: 1−0.01² = 0.9999) or go decentralized so no fuser exists to cap them.

Cost 3 — Compute concentration & latency. One box now has to process all the raw data from all sensors in real time. That demands a very powerful (hot, expensive) central computer, and it creates a latency bottleneck — everything queues behind that one fuser. There's no parallelism across nodes; the fuser is doing every sensor's work serially.

The latency story, with a number

Latency in a centralized system is the round trip: sensor → (ship raw data over the bus) → central fuser → (process all sensors) → decision. Each leg adds delay, and crucially the processing leg is serial across all sensors. If the fuser needs 2 ms to process each sensor's raw stream and there are eight sensors, that's 16 ms of pure fusion time before any answer emerges — on top of the transmission time for all that raw data. In a distributed system, each node processes its own data in parallel (8 nodes × 2 ms, but happening simultaneously = 2 ms wall-clock), and the central fuser only combines eight tiny tracks (sub-millisecond). The centralized architecture pays a latency penalty precisely because it refuses to let the work happen in parallel at the edge — another reason "optimal estimate" and "fast enough to brake in time" can be at odds.

When centralized actually wins

It's not always wrong — it's the right call in specific regimes:

• Few sensors, modest data rates. A drone with one IMU, one GPS, one barometer? The raw streams are tiny; ship them all to one flight controller and fuse optimally. No bandwidth problem.
• Bandwidth is cheap and short. Sensors a few centimeters from the fuser on a fast internal bus — e.g. a phone's IMU + magnetometer + GPS feeding one fusion chip. Centralized is simplest and optimal.
• You need the absolute best accuracy and can pay for it. A laboratory motion-capture rig, or a high-value satellite, where one beefy central processor and a fat bus are affordable, and the single-point-of-failure risk is mitigated by redundant fusers.

A worked "centralized wins" example — the small drone

Quantify why a drone is centralized's natural home. Suppose its three position-relevant sources, in the same instant, have variances: GPS 9 m², visual odometry 4 m², and the IMU-propagated estimate 16 m². A centralized fuser holding all three computes:

That fused σ of 1.54 m is tight enough to aim for a 1-meter landing pad with margin — and it's the best achievable, because nothing was discarded. Now ask the three cost questions: bandwidth? Tiny — three sources are a handful of numbers each, easily carried on the drone's internal bus. Single point of failure? The flight controller is already the only thing flying the drone, so adding the estimator to it introduces no new single point of failure. Compute? Trivial — fusing three scalars is microseconds. All three costs evaporate, leaving only the upside (optimal accuracy). This is the precise situation where centralized fusion is unambiguously correct, and it's why nearly every small drone, phone, and embedded IMU+GPS device fuses centrally.

Centralized fusion in code

The optimal centralized fuser is the inverse-variance sum from Lesson 1, generalized to n sensors. Notice every sensor's raw reading and variance must be present in one place:

python
def centralized_fuse(readings, variances):
    # all raw data lives here, in ONE place
    total_precision = 0.0
    weighted_sum    = 0.0
    for z, var in zip(readings, variances):
        p = 1.0 / var          # this sensor's "vote strength"
        total_precision += p     # precisions ADD -> optimal
        weighted_sum    += p * z
    x_fused   = weighted_sum / total_precision
    var_fused = 1.0 / total_precision
    return x_fused, var_fused

# three radars, same target's range
centralized_fuse([100.2, 99.5, 101.0], [4, 9, 16])
# -> (100.18, 2.36)   fused var 2.36 < best single sensor's 4  (OPTIMAL)

This is unbeatable for accuracy — but read the requirement again: all raw readings must arrive at this one function. That zip(readings, variances) assumes every sensor's raw stream made it to one box. The whole bandwidth-and-failure story is hiding in that innocent assumption.

How much accuracy does distribution actually cost?

A fair question: if centralized is "optimal," how much worse is distributed? The honest answer is "usually only a little, if you avoid double-counting." When the local nodes' estimates are genuinely independent and you fuse them correctly, distributed fusion can match the centralized variance — that's the magic of the information form (Chapter 4): adding independent information gives the same total whether you do it all at once (centralized) or piecewise at the nodes (distributed). The loss creeps in two ways: (1) when nodes throw away raw detail their tracker couldn't use but the central fuser could (e.g. a raw radar return that's ambiguous alone but disambiguated by camera context); and (2) when correlations sneak in and you must fuse conservatively (covariance intersection deliberately gives up some sharpness for safety). So the trade is: distributed is near-optimal in accuracy and 30,000× cheaper in bandwidth, as long as you do the consistency bookkeeping. Centralized's optimality is the ceiling; good distributed design gets close to it for a fraction of the bus.

Below, build a centralized network and watch the fused precision climb as you add sensors — then kill the central fuser and watch everything die at once.

Chapter 4: Distributed & Decentralized — Spreading the Intelligence Out

Centralized fusion hit three walls: bandwidth, single point of failure, and compute concentration. Every one of them comes from the same root cause — the sensors are dumb and one box does all the thinking. The fix is to push intelligence out toward the sensors, so that what travels on the wire is small and no one box is irreplaceable. There are two ways to do this, and they differ in one crucial detail: whether there's still anyone in charge.

Distributed fusion — smart nodes, one fuser at the top

In distributed fusion, each sensor node does its own local tracking and ships only the small estimate (a track) to a central fuser. The forward radar runs its own little Kalman filter and concludes "target at 18 m, moving 12 m/s, my uncertainty is ±1.5 m." It sends that — a handful of numbers — not its raw return cube. The central fuser then combines the tracks from all nodes into one global track. This is Dasarathy's decision/estimate fusion (Chapter 2) applied across the whole system.

The win is dramatic. Instead of 1.5 Gbit/s of raw pixels, a track is maybe a few dozen bytes per update. Bandwidth collapses by orders of magnitude. Each node's local processing happens in parallel, so the central fuser does far less work. And the system is modular: you can swap a radar for a better one without touching the central fuser, because the interface is just "send me a track." This is why distributed fusion is the default in modern automotive radar networks — each radar is a self-contained tracking unit, and a lightweight domain controller fuses the tracks.

Decentralized fusion — no one in charge at all

In decentralized fusion there is no central fuser. Every node is a peer; nodes exchange their estimates directly with their neighbors, and each maintains its own fused belief. There's no special "top" node to crash. If you have four sensor nodes in a ring, each one talks to its neighbors, folds in what they say, and over time the whole network converges toward a consensus — with no coordinator orchestrating it.

The headline benefit: no single point of failure. Kill any node and the rest keep talking and keep fusing. This is exactly why decentralized fusion dominates multi-robot teams (a swarm where any robot might be destroyed), sensor networks scattered across a field, and resilient military systems — there's no command node to take out. Durrant-Whyte and Rao's 1990s work showed that, remarkably, you can do this with purely local communication and still reach the right global answer, using the information filter form of the Kalman filter (Lesson 9), because information from independent sources simply adds — and addition doesn't care what order or by whom it's done.

The cost: decentralized fusion is the hardest to keep consistent. With no coordinator, how do you make sure node A and node C — who both heard from node B — don't both count B's information and end up overconfident? Managing that shared information without a central referee is genuinely hard, and it is the entire reason covariance intersection (Lesson 11) was invented. Decentralized buys you ultimate robustness and pays for it in consistency headaches.

Why "no referee" makes consistency hard — a concrete loop

Picture three peers A, B, C in a triangle, each talking to the other two. A measures the target and tells B. B folds A's info into its estimate, then — doing its job — tells C. C folds in what B said (which secretly contains A's measurement). Later C talks to A. Now A receives, from C, an estimate that contains A's own original measurement, bounced through B and C. If A naively adds it, A counts its own measurement twice. In a centralized system the one fuser would simply remember "I already have A's measurement" and not re-add it — but in a peer mesh, no node has the global bookkeeping to know what everyone else already knows. Each node sees only its neighbors' summaries, not the provenance inside them. That missing global memory is exactly why decentralized consistency is hard, and why a fusion rule that's safe without knowing the provenance (covariance intersection, Lesson 11) is so valuable.

Hierarchical fusion — the pragmatic tree

Hierarchical fusion is the middle ground that most real, large systems actually ship: a tree of fusers. Sensors feed local fusers; local fusers feed group fusers; group fusers feed a top-level fuser. It's neither flat-centralized nor flat-decentralized — it's layered. The forward-left and forward-right radars fuse at a "front" node; the rear radars fuse at a "rear" node; those two feed the vehicle-level fuser. Each layer compresses, so bandwidth stays bounded; each layer has local autonomy, so a branch failure doesn't kill everything; and the structure mirrors how the problem naturally decomposes (front vs rear, vision vs radar).

Hierarchical fusion inherits a bit of every other architecture's character: it's partly centralized (there's a top), partly distributed (nodes pre-process), and you can even make sub-trees decentralized. It's the reason "what architecture is this car?" rarely has a one-word answer — real systems are trees that blend all three.

A worked hierarchical example — the front/rear/top tree

Make the tree concrete. A car has four corner radars. Instead of all four feeding one vehicle-level fuser (flat distributed), build a two-level tree:

• Leaf level: each radar tracks locally and emits a track (Dasarathy DEI–DEO / FEI–DEO).
• Group level: a "front" fuser combines the two front radars' tracks; a "rear" fuser combines the two rear radars' tracks. Each group fuser emits one consolidated front (or rear) track.
• Top level: the vehicle fuser combines just the front track and the rear track into the global scene.

Three benefits fall out. Bandwidth stays bounded — the top fuser only ever sees two tracks, not four. Fault isolation — if the front group fuser dies, the rear half still works; you lose a quadrant pair, not the car. The structure mirrors the problem — front and rear are natural groupings. The cost is the same one that haunts all estimate-passing: each level must guard against double-counting (if a target is seen by both a front and a rear radar, the front and rear tracks share that target's information, and the top fuser must not credit it twice). The tree gives you modularity and graceful degradation; the price is consistency bookkeeping at every junction.

The bandwidth math makes the benefit concrete. Say each track is ~50 bytes at 50 Hz = 20 kbit/s. In flat distributed, the central fuser receives all four radars directly: 4 × 20 = 80 kbit/s into one box. In the hierarchical tree, the front fuser receives 2 × 20 = 40 kbit/s, the rear fuser likewise 40 kbit/s, and the top fuser receives just 2 consolidated tracks = 40 kbit/s. No single link or box ever carries more than 40 kbit/s — the load is spread, and the top fuser's burden halved versus flat distributed. As you scale to dozens of sensors, this matters enormously: a flat fuser receiving 30 tracks is a bottleneck, while a balanced tree keeps every node's fan-in small. That's why genuinely large multi-sensor systems are almost always trees.

Track-to-track fusion — what the distributed fuser actually does

Let's see the distributed fuser's job concretely, because "combine the tracks" sounds vaguer than it is. Two radars each track the same car and ship their tracks. Radar A says "x = 18.2 m, variance 4 m²"; radar B says "x = 17.6 m, variance 9 m²". The fuser does inverse-variance weighting on the tracks (not raw data):

Out comes 18.0 m with variance 2.77 m² — tighter than either track. This is the exact same arithmetic as the centralized fuser, just applied to tracks instead of raw measurements. And that's the catch that haunts the whole architecture: this calculation is only valid if radar A's and radar B's tracks are independent. If both radars were initialized from a shared map, or both heard a prior broadcast, their tracks share information and this clean inverse-variance fusion over-counts — the trap of Chapter 5. The distributed fuser's job looks identical to the centralized one; the danger is entirely in whether the independence assumption holds.

The federated filter — a famous hierarchical pattern

One named hierarchical design is worth knowing because it shows the double-counting fix in action: the federated Kalman filter (Carlson, 1990). Several local filters each fuse a subset of sensors and ship their estimates to a master filter; the master combines them. To avoid double-counting the common information (the shared process model and any shared reference), the federated filter deliberately divides the common information among the local filters by a set of fractions that sum to one — the "information-sharing" principle. Each local filter is handed only its share of the common information, so when the master adds them back up, the common part sums to exactly one copy, not several. It's a clean, engineered answer to the trap of Chapter 5, built right into the architecture — a preview of the conservative-fusion thinking that Lesson 11 generalizes.

The four architectures, side by side

Why information form makes decentralized possible — the key trick

There's one beautiful fact that makes peer-to-peer fusion work, and it's worth seeing now (we prove it in Lesson 9). In the ordinary Kalman filter you track a state and a covariance; combining two estimates is a fiddly weighted average. But there's an equivalent form — the information filter — that tracks the information matrix (the inverse of covariance) and an information vector. In that form, fusing two independent estimates is dead simple: you just add their information.

where Y is the information matrix (1/σ² in scalar form — exactly the "vote strength" from Lesson 1) and y is the information vector. Addition is associative and commutative — it doesn't matter what order you add things, or who does the adding. That's the secret of decentralized fusion: each node can add its neighbors' information in any order and still reach the same global answer the centralized fuser would, provided the information being added is genuinely independent. The instant it isn't — the instant two nodes share a common source — the simple addition double-counts, and you get the trap of Chapter 5.

Watch a decentralized network reach the centralized answer — by hand

Let's prove that peer-to-peer addition reaches the optimum. Three nodes each measure the same target's range, independently. Their measurements (and precisions, i.e. information) are:

A centralized fuser would compute total information 0.25 + 0.111 + 0.0625 = 0.4236 and a weighted-mean estimate. Now let the decentralized network reach the same answer with only neighbor-to-neighbor talk, in whatever order. Say A and B talk first — each adds the other's information:

Then that pooled result reaches C (via B, who now knows A's info too), and C adds its own:

Identical to the centralized total — 0.4236, no matter the order the nodes happened to talk in. Had C talked to A first, or all three gossiped in a ring, the sum is the same, because addition doesn't care. This is the precise mathematical reason decentralized fusion can be optimal despite having no coordinator: information is additive, and addition is order-independent. The one fatal assumption hiding in "A and B each add the other's info" is that A's and B's measurements were independent — which here they were (three different sensors). When they secretly share a source, this exact addition over-counts. Hold that thought; it is the whole of Chapter 5.

The same scenario, all four architectures — what you'd actually ship

Take one concrete problem — four corner radars tracking a car ahead — and design it four ways, so the trade-offs sit side by side:

Same four sensors, same target, four very different systems. Centralized is the only one with the fattest bus and a hard single point of failure; decentralized is the only one with no irreplaceable node; hierarchical bounds bandwidth and isolates faults to a quadrant. There's no "right" row — you'd pick centralized for a lab rig, distributed for a cost-sensitive car, decentralized for a swarm, hierarchical for a big complex platform. The constraint that bites hardest picks the row.

Below, watch the same four-node network run under distributed versus decentralized routing, and break links and nodes to feel the robustness difference.

Chapter 5: The Double-Counting Trap — When Fusing Estimates Makes You Lie To Yourself

Here is the most dangerous bug in all of distributed and decentralized fusion, and the reason an entire later lesson exists to fix it. It is silent — nothing crashes, no error is thrown — and it makes your system confidently wrong, which is far worse than being obviously broken. The trap is called double-counting common information, and once you see the arithmetic you'll never trust a naive distributed fuser again.

The setup. In Chapter 3 we learned the beautiful rule: independent precisions add. Fuse two independent estimates and the fused precision is the sum, so the fused estimate is sharper than either. That rule powered both centralized fusion and the information-form addition that makes decentralized fusion possible. But it has a fatal fine-print clause, italicized every time we stated it: provided the estimates are independent. When two estimates secretly share information — a common prior, a common source — they are not independent, and adding their precisions counts the shared part twice.

Why should you care so much about one bad number? Because in a fused estimate, the covariance is a promise. When the fuser reports "position 100 m, variance 2.5 m²," it is promising every downstream consumer: "I'm confident the truth is within about ±1.6 m." The path planner uses that promise to decide how much clearance to leave; the threat assessor uses it to decide whether to brake. If the promise is a lie — if the truth is really ±1.8 m — the planner leaves too little clearance and the car clips the obstacle, confidently. A loud crash you can debug; a quietly-too-small covariance ships to production and bites a customer. That's why double-counting is the trap an entire later lesson exists to fix.

How two estimates come to share information

In a centralized architecture this can't happen: there's only raw data, fused once. But the instant you go distributed or decentralized, nodes pass estimates around, and estimates carry history. Three common ways the sharing sneaks in:

• A common prior. Two nodes both started from the same initial estimate (say, the last fused track broadcast to everyone). Their "independent" updates both contain that shared starting point.
• A relayed measurement. Node A tells node B about a target; B incorporates it and later tells node C. Now B's estimate and A's estimate both contain A's original measurement — C hears it twice if it fuses both.
• A feedback loop. The central fuser sends its fused result back to the nodes (to help them); the nodes then send updated tracks back to the fuser, which now re-fuses information it already had. The "rumor from three friends who all heard it from you" problem.

The worked example — watch the covariance lie

Let's make it brutally concrete with the simplest possible case: two nodes that share a common prior. We'll track a one-dimensional quantity — say a target's range — and we'll work entirely in precision (1/variance), because that's the currency that's supposed to add.

Step 1 — the common prior. Both nodes start from the same broadcast prior estimate: range = 100 m, with variance σ²_prior = 10 m², i.e. precision 1/10 = 0.10. This prior is one piece of information, shared by both nodes.

Step 2 — each node adds one genuinely independent measurement. Node A takes its own radar reading with variance 10 m² (precision 0.10); node B takes its own, also variance 10 m² (precision 0.10). These two measurements are independent of each other — different sensors, different noise.

Step 3 — each node forms its local estimate. Each node fuses (prior + its own measurement), so each node's local precision is:

Each node honestly believes it has precision 0.20, variance 5 m². So far, no bug — each node did the right thing locally.

Step 4 — the central fuser naively adds the two local precisions. The distributed fuser receives P_A = 0.20 and P_B = 0.20 and applies the "independent precisions add" rule:

The fuser proudly reports variance 2.5 m² — a tight, confident estimate. This number is a lie.

Step 5 — what the honest answer is. Count the distinct pieces of information that actually went in. There were exactly three: the shared prior (precision 0.10, counted once), node A's measurement (0.10), and node B's measurement (0.10). The correct total precision is:

The honest variance is 3.33 m², not 2.5 m². The naive fuser counted the prior twice — once inside P_A and again inside P_B — so it added 0.40 of precision when only 0.30 was real. It over-credited itself by 0.10, exactly the duplicated prior.

The geometric picture — the uncertainty ellipse drawn too small

In two dimensions the lie becomes visual, which is how engineers usually catch it. An estimate's uncertainty is an ellipse: the region where the true value probably lives. A big ellipse means "I'm unsure"; a tight ellipse means "I'm confident." When you fuse two estimates, the honest fused ellipse should be the intersection-ish region — smaller than either, but only as small as the genuinely independent information justifies. The double-counting fuser, treating shared information as independent, draws the fused ellipse too small — tighter than the truth warrants. Picture it: the true value sits comfortably inside the honest (larger) ellipse, but pokes outside the naive (shrunken) one. The fuser is announcing "the target is almost certainly in this tiny region" when it's actually often outside it. Every downstream planner that trusts that tiny ellipse — routing around obstacles, deciding it's safe to proceed — is being lied to. The visual tell of double-counting is an uncertainty ellipse that's suspiciously small and that the truth keeps escaping; Chapter 9's consistency test makes that "keeps escaping" precise.

It gets worse as the shared part grows

The 33% error above came from a prior that was as strong as each measurement. When the shared information dominates, the lie balloons. Suppose the common prior is very strong — precision 1.0 (variance 1 m²) — and each node adds a weak measurement of precision 0.10:

Now the reported variance (0.45 m²) is nearly half the true variance (0.83 m²) — the system thinks it's almost twice as certain as it is. And in a feedback loop where the same information cycles many times, the naive estimate's reported variance can collapse toward zero — the system becomes infinitely confident in a number that's merely echoing itself. This runaway is exactly why uncontrolled feedback in a distributed fuser is forbidden.

The rumor analogy, made numerical

The intuition is the gossip everyone has lived. You hear "the meeting moved to 3pm" from Alice, then from Bob, then from Carol — and you feel very sure. But all three heard it from the same email Dave sent. You have one piece of evidence (Dave's email), not three, yet your confidence acted as if you had three independent confirmations. In precision terms: Dave's email is information 0.30; each of Alice, Bob, Carol relays it (adding a little of their own, say 0.02 each from "I think I remember that too"). If you naively add all three relayed reports as independent, you compute 3 × (0.30 + 0.02) = 0.96 of information — but the honest total is 0.30 (Dave, once) + 0.06 (the three small independent contributions) = 0.36. You felt nearly three times as certain as the evidence warrants. That is double-counting, and a distributed sensor network does it for exactly the same reason a rumor mill does: the same original information arrives by multiple paths and gets credited each time.

The feedback runaway — confidence collapsing to a lie

The single worst form is a feedback loop, and it's worth tracing because the failure is so dramatic. Suppose the central fuser sends its fused result back to a node "to help," the node folds it into its local estimate, then sends an updated estimate back to the fuser, which re-fuses it. Each round, the fuser counts its own previous output again. Track the reported precision over a few rounds, starting from an honest 0.30:

The reported variance halves every round and races toward zero — the system becomes infinitely certain of a number that is merely its own output reverberating. There was never any new information after round 0; the fuser was applauding its own echo. This is precisely why uncontrolled feedback in a distributed fuser is forbidden: a fuser must never consume information it produced, or it will talk itself into total, false certainty. (In the showcase, Chapter 6, you'll flip this feedback loop on and watch "honesty" flip to OVERCONFIDENT in real time.)

Why centralized fusion is immune (and distributed is not)

Notice what just happened: the bug appeared only because nodes shipped estimates (which carry shared history) instead of raw measurements (which don't). A centralized fuser holding the raw prior and both raw measurements would have simply summed 0.10 + 0.10 + 0.10 = 0.30 — the honest answer — because it sees each distinct measurement exactly once. Double-counting is the price of distribution. The moment you compress raw data into estimates and pass them around, you lose track of who-already-knows-what, and the simple "precisions add" rule becomes a liability.

This is the hidden virtue of centralized fusion we kept hinting at: it is the only architecture that cannot double-count, because there is exactly one fusion event and it operates on distinct raw measurements. Every other architecture — distributed, decentralized, hierarchical — must actively defend against double-counting, because they all pass estimates around. When you choose to leave centralized fusion (usually for bandwidth, Chapter 3), you take on the double-counting problem as part of the bargain, whether you realize it or not.

The trap in code

The bug is a single line — the line that adds two local precisions as if they were independent:

python
def naive_distributed_fuse(P_A, P_B):
    # DANGER: assumes A and B share NO information
    return P_A + P_B                 # double-counts any common prior

def honest_fuse(P_prior, P_measA, P_measB):
    # count each distinct piece of information exactly once
    return P_prior + P_measA + P_measB

# shared prior 0.10, two independent measurements 0.10 each
P_A = 0.10 + 0.10     # node A local precision = 0.20
P_B = 0.10 + 0.10     # node B local precision = 0.20
naive_distributed_fuse(P_A, P_B)            # -> 0.40  (var 2.5)  OVERCONFIDENT
honest_fuse(0.10, 0.10, 0.10)            # -> 0.30  (var 3.33) CORRECT

The naive function looks identical to the perfectly-correct centralized sum from Chapter 3 — that's what makes it so insidious. The same + that is exactly right for raw measurements is exactly wrong for estimates that share a prior. The honest version needs to know the provenance of every piece of precision, which is precisely the bookkeeping a distributed system finds hard.

The two honest fixes — and why one is hard

There are exactly two principled ways out, and seeing both sets up Lesson 11:

python
# FIX 1 — track and subtract the common information (correct, but needs bookkeeping)
def decorrelated_fuse(P_A, P_B, P_common):
    # remove the shared part that both A and B already counted
    return P_A + P_B - P_common      # subtract the duplicate

decorrelated_fuse(0.20, 0.20, 0.10)   # -> 0.30  CORRECT — but you must KNOW P_common

# FIX 2 — covariance intersection: assume the WORST correlation, stay safe (Lesson 11)
def covariance_intersection(P_A, P_B, w=0.5):
    # conservative blend — never overconfident, no knowledge of correlation needed
    return w * P_A + (1 - w) * P_B   # <= the naive sum, guaranteed safe

covariance_intersection(0.20, 0.20)        # -> 0.20  conservative (a touch under-confident, but NEVER a lie)

Fix 1 is exact but demands you know the common precision — the very provenance a peer mesh can't track. Fix 2, covariance intersection, sidesteps that entirely: it makes no assumption about how correlated A and B are, so it's guaranteed never to be overconfident, at the price of being slightly under-confident (it reports 0.20 here, below even the honest 0.30). For a safety system, a little under-confidence is fine; overconfidence is a menace. That trade — give up a little sharpness to guarantee you never lie — is the entire idea of Lesson 11.

Below, slide the strength of the shared prior up and watch the gap between the naive (overconfident) variance and the true variance widen — the system's self-deception, made visible.

Chapter 6: Showcase — The Architecture Explorer

We've met the architectures one at a time. Now put all of them into a single live multi-sensor network you can break on purpose, because the deepest understanding comes from watching the same failure — a dead node, a cut link — produce three completely different outcomes depending only on the architecture. This is the payoff chapter: no new theory, just every idea from chapters 0–5 made visible and breakable at once.

Below is a network of five sensors observing one target. You choose the architecture — centralized, distributed, or decentralized — and then you can fail any node or any link and read off, in real time, three numbers that summarize the whole lesson: the bandwidth the architecture demands, whether the network still survives, and whether the fused estimate is honest about its uncertainty or secretly double-counting.

This sim is the answer to a question you should ask of every fusion design: "if I poke it, what breaks, and what does it cost me while it's healthy?" The three readout numbers are the three constraints that fought each other back in Chapter 0 — bandwidth, fault-tolerance, honesty — and the explorer lets you watch them trade off as you change architecture and break things. By the end of a minute with it, you'll have felt the impossibility of having all three maxed at once.

Play with it for a minute and the architectures announce themselves through their failure behavior and their costs, not their definitions:

• Centralized. The links are drawn fat (raw data) and all point at the fuser. Kill a sensor — the estimate degrades but survives. Kill the fuser — the entire network goes dark instantly, every healthy sensor now shouting into the void. Bandwidth reads HIGH. But honesty always reads CALIBRATED: a single raw fuse can't double-count.
• Distributed. The links are thin (tracks) into one fuser. Kill a sensor — survives, just less precise. Kill the fuser — still goes dark (it's still a single point of failure), but bandwidth reads LOW. Now flip the feedback loop ON: honesty flips to OVERCONFIDENT, because the fuser's output cycles back and gets re-counted — the Chapter 5 trap, live.
• Decentralized. No fuser exists — it's a mesh. Kill any node and the rest keep talking and keep fusing; survival stays GREEN no matter which single node you remove. Bandwidth reads MEDIUM (peer chatter). Honesty reads HARD — without a referee, keeping the shared information straight is the standing challenge.

A guided experiment — reproduce every claim in six clicks

Don't just watch; falsify. Run this sequence and confirm each prediction with your own eyes:

1. Centralized, then click the fuser. Survival flips to LOST. One node death (the fuser) blinded the whole system — the single-point-of-failure of Chapter 3, made literal.
2. Restore, switch to Distributed, kill one sensor. Survival stays SURVIVING (degraded); bandwidth dropped to LOW versus centralized's HIGH. Smart nodes shipping thin tracks — Chapter 4.
3. Still distributed, turn the feedback loop ON. Honesty flips from CALIBRATED to OVERCONFIDENT, with no node failing at all. The fuser is now eating its own output — the double-counting trap of Chapter 5, born purely from the architecture.
4. Kill the distributed fuser. Survival flips to LOST again — distributed still has a central fuser, so it inherits a single point of failure even though bandwidth was cheap.
5. Restore, switch to Decentralized, kill any one node. Survival stays SURVIVING — there's no special node to kill. This is the robustness centralized and distributed can't match.
6. Kill a second, then a third node (decentralized). It keeps surviving as long as the mesh stays connected — only when the survivors get isolated does it finally fail. No single point of failure anywhere.

The crucial contrast is steps 1, 4, and 5 against each other. The same action — "kill one node" — is catastrophic for centralized (if it's the fuser), catastrophic for distributed (if it's the fuser), and survivable for decentralized (any node). One identical event, three outcomes, decided purely by where the intelligence lives. The architecture is the system's destiny under failure — exactly as the relationship was in Lesson 1, now one layer down in the stack.

What the three readout numbers actually compute

The readout isn't decoration — each number is a small calculation you could do by hand, tying the sim back to the chapters:

• Bandwidth follows the Dasarathy mode (Chapter 2): centralized ships raw data (HIGH), distributed and decentralized ship estimates (LOW/MEDIUM). It's literally "how many bytes per link" — megabytes for raw, bytes for tracks.
• Survival is the architecture-ablation test (Chapter 9): does the function survive killing this node? Centralized/distributed fail if you kill the fuser (single point of failure); decentralized survives any single node because the mesh reroutes. The sim is computing "is there still a path from the survivors to a fused answer?"
• Honesty is the double-counting check (Chapter 5): centralized fuses raw data once, so it's always CALIBRATED; distributed-with-feedback re-counts its own output, so it flips to OVERCONFIDENT; decentralized is HARD because keeping peers' shared information straight has no referee. The sim is asking "could the same information reach a fuser by two paths?"

So the explorer is a live evaluator of the exact arithmetic you worked by hand: bandwidth = Dasarathy byte-count, survival = ablation test, honesty = cycle/consistency check. Three chapters of theory, condensed into three numbers that move when you break things.

The deepest contrast — centralized's honesty vs decentralized's robustness

Run two specific configurations back to back and you'll feel the lesson's central tension. (1) Centralized, all sensors alive: honesty CALIBRATED, survival fine, but bandwidth HIGH and one click on the fuser ends everything. (2) Decentralized, kill two nodes: survival still green, bandwidth MEDIUM, but honesty reads HARD — you bought bulletproof robustness and paid in consistency difficulty. Neither is "better" — centralized gives you an honest optimal estimate on a fat, fragile bus; decentralized gives you unkillable robustness with a consistency headache. The explorer makes that trade physical: you literally cannot find a configuration that's HIGH-robust, LOW-bandwidth, and CALIBRATED-honest all at once, because the constraints genuinely fight (Chapter 0).

Chapter 7: Use Cases & Real Products — Architectures In the Wild

Every Engineermaxxing lesson in this series ends with the same four chapters — Use Cases, Practical Application, Debugging, Connections — because an architecture you can't point at in a shipped product is one you don't really own yet. So before any more theory, let's ground centralized, distributed, and decentralized in hardware you can buy or ride in today.

The payoff of having learned the architectures is this: once you can name a product's fusion architecture, you can predict exactly how it fails and what it costs to wire. A spec sheet becomes a failure-mode forecast. Let's tour real, shipped systems and label each.

And here's the diagnostic superpower in one move: when you see a multi-sensor product, ask "where does the fusion happen, and what's on the wire?" If raw data goes to one box, it's centralized — so look for a redundant computer or a fail-degraded mode (or worry). If smart sensors ship tracks, it's distributed — so look for how they avoid double-counting in any feedback. If there's no central box at all, it's decentralized — so look for how they keep peers consistent. The architecture tells you where the bodies are buried before you ever read the failure-analysis report.

The pattern across products

Notice the recurring logic in how engineers chose each architecture:

The deep case: why the same car uses different architectures for different jobs

The richest example is a self-driving stack, because a single vehicle deliberately mixes architectures by sub-function:

• Centralized for the highest-fidelity perception where bandwidth is affordable and the optimal estimate matters: e.g. fusing two forward cameras' near-raw streams in one vision computer for stereo + classification.
• Distributed for the wide sensor ring where bandwidth is scarce: each corner radar tracks locally and ships tracks, because routing five raw radar cubes to the center would swamp the bus.
• Decentralized / redundant for the safety core: two independent compute paths cross-check each other so no single computer failure can disable braking — the same robustness logic as a robot swarm, applied to fault-tolerant compute.

One vehicle, three architectures, each chosen for its job — and the choices follow the cost arithmetic exactly: centralize where bandwidth is cheap and accuracy is king; distribute where bandwidth is the bottleneck; decentralize/redundify where a single failure would be catastrophic. A naive design that centralized everything would need an impossible harness and would be one crash from blindness; one that decentralized everything would struggle to keep its world model consistent. The good designs match architecture to the constraint that dominates each sub-problem.

A fifth case: avionics navigation — FEDERATED (hierarchical)

Aircraft navigation is the textbook federated/hierarchical system. An inertial navigation system (INS) drifts but is fast; GPS, an air-data computer, and sometimes terrain-referenced navigation each correct it. Rather than one giant filter ingesting everything (centralized, and a single point of failure for the whole nav function — unacceptable in flight), avionics use a federated structure: local filters fuse INS+GPS, INS+air-data, etc., and a master filter combines their outputs, using the information-sharing trick (Chapter 4) so the common INS information isn't counted multiple times. The win is fault tolerance with honest covariance: if the GPS local filter fails, the others carry on, and because the common information was divided, no branch was ever falsely overconfident. This is hierarchical fusion engineered to certification standards — robustness and consistency both, at the cost of design complexity.

The cost trade, summarized across the products

Lay the products on the bandwidth–robustness–consistency axes and the pattern is unmistakable:

Read the "dominant constraint" column top to bottom and the architecture writes itself. Nobody picks an architecture for its own sake; they pick the one that survives the constraint that bites hardest, and accept that architecture's characteristic weakness as the cost of doing business.

How to read the architecture off a product spec sheet

You can now reverse-engineer any multi-sensor product's architecture from its marketing copy. Three diagnostic phrases:

• "Central compute" / "domain fusion ECU" / "one brain" → centralized (watch for: how do they handle the single point of failure? Redundant computer? Fail-degraded mode?).
• "Smart sensors" / "on-sensor tracking" / "object-list output" / "sensor sends tracks" → distributed (cheap bus, swappable — ask how they prevent double-counting in any feedback path).
• "Mesh" / "peer-to-peer" / "no single point of failure" / "swarm" → decentralized (robust — ask how they keep estimates consistent across peers, i.e. do they use covariance intersection?).

The vocabulary changes by industry — avionics says "federated," automotive says "domain controller," robotics says "decentralized estimation" — but underneath, it is always a point on the centralized↔decentralized spectrum you now know by heart.

Chapter 8: Practical Application — A Decision Guide for Choosing Your Architecture

You now know the four architectures, the JDL levels, the Dasarathy modes, and the double-counting trap. But knowing the parts is not the same as knowing which to reach for when a real design lands on your desk. This recurring "now build it" chapter turns the theory into a procedure you can run on any project — a small flowchart of questions that converts "I have these sensors and these constraints" into "use this architecture, here's why."

The choice comes down to four constraints, and the trick is figuring out which one dominates. We'll state the procedure, then run it end-to-end on two completely different systems — a delivery drone and a building-wide sensor net — to prove the same questions work regardless of hardware.

The four constraints that pick an architecture

• Bandwidth. How much data can your bus carry? Raw streams are huge; tracks are tiny. If the bus can't carry raw data from all sensors, centralized is off the table → lean distributed/decentralized (ship estimates).
• Latency. How fast must the answer arrive? Centralized adds a round-trip to the fuser and a serial bottleneck; local processing at the node is faster for that node's own decisions. Hard real-time at the edge → push intelligence outward.
• Fault-tolerance. What happens if one box dies? If a single failure must not kill the function, you cannot have a lone central fuser → decentralize, or duplicate the fuser, or go hierarchical with redundant branches.
• Compute. Where can you afford to put processing? A powerful central computer is hot and expensive; many cheap smart sensors spread the load. Cheap edge compute available → distribute.

The decision procedure

Ask these in order; the first hard constraint that bites usually picks your architecture:

And one universal safety check, regardless of the choice: if any node ships estimates (not raw data) and there's any feedback path, you must guard against double-counting — track correlations or use covariance intersection (Chapter 5, Lesson 11). This applies to distributed, decentralized, and hierarchical alike.

Picking the Dasarathy mode (the level) to go with the architecture

Architecture answers where; the Dasarathy mode (Chapter 2) answers what flows on the wire, and the two interlock. A quick mapping:

• Centralized usually pairs with DAI–DAO / DAI–FEO — raw or near-raw data to the center, because the whole point is to preserve information. (If you're shipping decisions to a "central" fuser, you're really doing distributed fusion with one fuser.)
• Distributed pairs with FEI–DEO / DEI–DEO — nodes ship features or verdicts (tracks), which is exactly the bandwidth win.
• Decentralized pairs with DEI–DEO over the information form — peers exchange estimates and add information.
• Hierarchical mixes them by tier: raw-ish near the sensors, decisions near the top.

The rule of thumb: the tighter your bandwidth, the further down the Dasarathy ladder (toward decisions) you must push, and the more distributed/decentralized your architecture becomes. Bandwidth and architecture and Dasarathy mode are three views of the same constraint.

Worked design example: a small outdoor delivery drone

The task: fuse a GPS, an IMU, a barometer, and a downward camera to estimate the drone's pose, reliably enough to land on a 1-meter pad. Walk the procedure.

Q1 — bandwidth. How much raw data? GPS, IMU, and barometer output a handful of numbers each at modest rates; the downward camera is the only fat stream, and on a small drone it sits a few centimeters from the flight controller on a short, fast bus. So the bus can carry it — bandwidth is not the binding constraint here. Centralized stays on the table.

Q2 — fault-tolerance. Must it survive losing the one flight controller? On a small drone, if the flight controller dies the drone falls regardless of architecture — there's only one thing flying it. So there's no benefit to decentralizing the estimator; a single fuser is acceptable because the platform already has a single point of control. (A large multi-rotor cargo drone with redundant flight computers would answer differently — then you'd want redundant/hierarchical fusion.)

Q3 — few sensors, short bus, want optimal? Yes on all counts: four sensors, centimeters apart, and we want the tightest possible pose for a precision landing. Verdict: centralized fusion — route all four sensors' measurements to one estimator (a Kalman filter, Lesson 5) and fuse optimally. This is the right call because bandwidth and fault-tolerance didn't bind; the drone is the textbook home of centralized fusion.

The double-counting check. Since it's centralized with raw measurements and no estimate-relaying between nodes, there's no shared-information path — no double-counting risk. Centralized's hidden virtue strikes again: it's immune to the Chapter 5 trap.

A second example on a totally different sensor set — a building-wide occupancy sensor net

The procedure is sensor-agnostic; run it again where the constraints flip. The task: a smart building must estimate which rooms are occupied, fusing motion sensors, door sensors, and CO₂ sensors scattered across 200 rooms over five floors.

Q1 — bandwidth. Could you route raw streams from hundreds of sensors across a whole building to one central server? The aggregate raw data over building-scale wiring (or wireless mesh) is large and the links are long and lossy. Centralizing raw data is impractical → sensors must ship small estimates ("room 3F-12: occupied, 0.9 confidence"). Distributed or decentralized.

Q2 — fault-tolerance. Must the system survive losing one box? A building automation system shouldn't go fully blind because one hub failed — and there's no single "flight controller" equivalent that everything depends on anyway. The sensors are physically scattered and independently powered. This pushes toward decentralized / mesh: each floor (or zone) maintains its own occupancy estimate and gossips with neighbors, so a dead hub on floor 3 doesn't blind floors 1, 2, 4, 5.

Verdict: hierarchical-leaning-decentralized. Per-zone fusers (distributed within a zone: sensors ship estimates to a zone controller) that gossip peer-to-peer across zones (decentralized between zones), with no single building-wide brain to lose. Bandwidth stays bounded (only small estimates travel), and a single hub failure degrades one zone, not the building.

The double-counting check — and it bites here. Because zones gossip estimates peer-to-peer, the shared-information problem is real: if zone A's "hallway occupied" estimate propagates to B and C, and B and C both report back, the system could become falsely certain the hallway is occupied. This design needs covariance intersection (Lesson 11) or correlation tracking — exactly the safety check the procedure flagged. The drone needed no such guard; the sensor net does. Same procedure, opposite answers, because the dominant constraints differ.

A third example — a warehouse robot fleet

One more, to cement that the procedure is universal. A fleet of 30 warehouse robots must each know where it is and where the others are (to avoid collisions). Each robot has wheel odometry, an IMU, a LiDAR, and a radio to talk to other robots and to overhead ceiling beacons.

Q1 — bandwidth. Could every robot stream raw LiDAR to one central server over Wi-Fi? Thirty LiDAR streams over shared wireless would saturate the channel instantly. So robots must ship small estimates (their pose + a short list of detected neighbors), not raw scans. Centralized-raw is out.

Q2 — fault tolerance. Must the fleet survive losing the warehouse server? Yes — a server outage shouldn't make 30 robots collide. And robots roam in and out of radio range, so there can't be a single mandatory coordinator. This pushes hard toward decentralized: each robot maintains its own world model and exchanges pose estimates peer-to-peer with robots in range.

Verdict: decentralized for inter-robot pose sharing, with each robot internally running a centralized fuser over its own four onboard sensors (they're all on the same robot, on a short bus — centralized is optimal there). So the fleet is decentralized between robots and centralized within each robot — a hierarchy emerging naturally from the constraints.

Double-counting check — severe here. If robot A tells B its position, B tells C, and C tells A, a robot's own position estimate can loop back to it amplified — the rumor mill on wheels. This fleet requires covariance intersection (Lesson 11) for the inter-robot sharing, exactly as the decision procedure's safety check warns. Notice the pattern across all three examples: the moment estimates flow peer-to-peer with any cycle, the double-counting guard becomes mandatory; the drone (no estimate-passing) needed none, the building and the fleet (heavy estimate-passing) cannot ship without one.

Computing reliability/honesty on paper before you build

Two paper checks turn the procedure into a guarantee. First, the single-point-of-failure check: list every box and ask "if this dies, does the whole function die?" For the warehouse fleet, no single box is fatal (decentralized) — pass. For the drone, the flight controller is fatal, but it's fatal regardless of fusion, so it's acceptable — pass. Second, the cycle check: trace whether any single measurement can reach a fuser by two paths. Drone: no (raw, one fuser) — no double-counting possible. Building & fleet: yes (gossip cycles) — must add covariance intersection. Both checks are a few minutes with a pencil and a box-and-arrow diagram, and they catch the two deadliest architecture bugs (SPOF and double-counting) before a single line of code ships.

Make the SPOF check quantitative when safety demands it. Suppose a building's fire-detection fusion must hit reliability 0.9999. A centralized design with one fuser of reliability 0.99 caps at 0.99 — fails the requirement no matter how good the sensors are (the fuser is a series element). Two options: duplicate the fuser in parallel (1 − 0.01² = 0.9999 — passes), or go decentralized so there's no fuser to cap it. The arithmetic tells you, on paper, that the single-fuser design was never going to meet spec — saving you from discovering it in a field failure. Always compute the reliability of the required elements (multiply their R's, since required = series) and compare to spec before building; if a single box caps you below requirement, change the architecture, not the sensors.

A sanity check before you build

Before committing, trace one piece of information through your design and ask the Chapter 5 question: could this same measurement reach the fuser by two different paths? Draw the data-flow graph; if there's any cycle (an estimate that can return to where it came from), you have a double-counting hazard and you must either break the cycle, track the correlation, or fuse conservatively. Doing this on paper, before building, catches the overconfidence trap before it ships — the architectural twin of computing reliability on paper in Lesson 1.

Chapter 9: Debugging & Failure Modes — The Four Architecture Traps

Knowing the architectures is half the battle; the other half is recognizing when a real system has quietly violated its own design. Architecture bugs are rarely loud crashes — they are subtle gaps between the topology you drew and the one the data actually flows through, and they hide until exactly the wrong moment (a node dies, a clock drifts, a measurement arrives late). This recurring chapter catalogs the classic traps, the symptom each presents, and the specific test that exposes it before it ships.

Trap 1 — Overconfidence from double-counting

The deadliest and most insidious, straight out of Chapter 5. Your distributed or decentralized fuser reports a beautifully tight covariance, but it's a lie — it's counting shared information multiple times. The estimate looks better than the truth, which is worse than looking broken, because every downstream consumer trusts a number that's shakier than advertised.

A concrete instance: a fuser feeds its global track back to the sensor nodes "to help them initialize," and the nodes then send updated tracks back. The global track now contains its own past output. Each cycle, the reported variance shrinks a little more, until the system is absurdly confident in a number that's merely echoing itself. The cure is to never let a fuser consume information it produced — or to rigorously subtract the common part.

Trap 2 — Out-of-sequence measurements (OOSM)

A measurement taken at time t arrives at the fuser after a measurement taken at a later time t+Δ — the data showed up out of order. This is endemic in distributed and networked fusion: different sensors have different processing delays and different network latencies, so a slow-but-old measurement can land after a fast-but-newer one. If the fuser naively folds the old measurement into its current (newer) state, it corrupts the estimate — it's applying stale information as if it were fresh.

A worked feel for the magnitude: a target moving 20 m/s, and a camera measurement that's 100 ms late, describes a position that's 2 meters stale by the time it arrives. Fold that into a state that has already moved on, and you've injected a 2-meter error in the wrong direction. OOSM is why distributed automotive fusion lives or dies on accurate, synchronized timestamps — which leads directly to the next trap.

A worked hidden-SPOF discovery

Here's how the hidden single point of failure usually hides. A team proudly ships a "fully distributed" radar network: four smart radars, each tracking locally, feeding a domain controller. They reason "it's distributed, so it's robust." But run the architecture ablation on paper: kill radar 1 — survives (3 radars + controller). Kill radar 2 — survives. Kill the domain controller — total blackout, because all four radars were feeding only it. The "distributed" system has the same single point of failure as a centralized one; distribution moved the data processing to the edge but left the fusion in one box. The word "distributed" lulled them. The fix is to recognize that distributed fusion still has a central fuser (Chapter 4), and if single-fault tolerance is truly required, you must either duplicate the controller or let the radars also gossip peer-to-peer (a decentralized fallback) so they can fuse among themselves when the controller dies.

Trap 3 — Single point of failure (hidden in a "distributed" design)

You believe you built a robust distributed system, but in practice one node is irreplaceable — the central fuser, a shared clock source, a single gateway — and when it dies the whole "distributed" system goes dark. The word "distributed" lulls you into assuming fault-tolerance you didn't actually build, because distributed fusion still has a central fuser (Chapter 4).

Trap 4 — Clock skew across nodes

Every node has its own clock, and clocks drift. If node A thinks it's 12:00:00.000 and node B thinks it's 12:00:00.030, then a measurement A timestamps and a measurement B timestamps are mis-aligned by 30 ms before fusion even starts — the fuser believes two measurements were simultaneous when they were 30 ms apart, or vice versa. For a fast-moving target this silently smears the fused estimate, and it compounds with the OOSM problem (you can't re-order correctly if you can't trust the timestamps).

A worked consistency check — catching double-counting in the field

How do you actually detect overconfidence, since the estimate looks beautiful? You compare the errors you really make against the errors the covariance claims you should make. The rule of thumb: if your reported standard deviation is σ, then roughly 68% of your errors should fall within ±σ and ~95% within ±2σ (for a well-calibrated Gaussian). Run a hundred trials of a fuser that reports σ = 1.6 m (variance 2.5 m² — the overconfident number from Chapter 5) but whose true σ is 1.8 m (variance 3.33 m²):

When errors routinely land outside the claimed bands more often than they should, the covariance is too small — you're overconfident, almost always from double-counting. A formal version of this is the Normalized Estimation Error Squared (NEES) test, which checks whether the average squared error, scaled by the inverse covariance, sits inside a statistical bound; if it consistently exceeds the bound, the filter is inconsistent (overconfident). The takeaway: you cannot see double-counting by staring at one estimate — you catch it only by comparing many errors to the covariance over time.

A worked clock-skew calculation

Clock skew's signature — error scaling with target speed — comes straight from arithmetic. If two nodes' clocks differ by skew Δt and the target moves at speed v, the fuser misplaces the target by:

With a 30 ms skew (Δt = 0.030 s): a pedestrian at 1.5 m/s is misplaced by 1.5 × 0.030 = 4.5 cm (annoying but tolerable); a car at 30 m/s is misplaced by 30 × 0.030 = 0.9 m (now you're in the wrong lane); a highway closing speed of 60 m/s gives 60 × 0.030 = 1.8 m of pure timing error. The same 30 ms skew produces wildly different errors depending only on speed — which is exactly why "the error gets worse the faster things move" is the fingerprint of a clock problem, not a sensor-accuracy problem. A noisy sensor's error doesn't care how fast the target moves; a timing error scales with it linearly. That diagnostic alone often points straight at the bug.

Trap 5 (bonus) — network partition & stale data in decentralized meshes

Specific to decentralized/peer architectures: the network splits (a "partition") so two groups of peers can no longer talk, and each group drifts to its own answer; or a peer goes silent and its neighbors keep using its last (now stale) estimate as if it were fresh. The mesh's great strength — no coordinator — is exactly why no one notices the split: there's no referee to declare "we've lost contact with the east wing."

The five tests as a checklist you can run on any fusion system

A worked OOSM detection, step by step

Out-of-sequence detection is mechanical once you timestamp at the source. Suppose the fuser's state is currently valid as of t_state = 1.000 s, and three measurements arrive in this order with these sensing timestamps:

The second measurement was sensed 60 ms before the state's current time — the camera was just slow to deliver it. Folding it in as if it were current would drag the state backward. The fix: detect it by the timestamp comparison (older than state → OOSM), then either roll the state back to 0.950, incorporate, and roll forward, or hold a short re-ordering buffer. The detection is one comparison; the corruption it prevents is enormous.

How big is the OOSM corruption? A worked number

Quantify the damage of naively applying that 60 ms-old measurement. The target moves at, say, 25 m/s. In 60 ms it travels 25 × 0.060 = 1.5 m. The stale measurement describes where the target was 1.5 m ago. If the fuser treats it as current, it yanks the state estimate 1.5 m backward — against the direction of travel — injecting a 1.5 m error in exactly the wrong direction. Worse, because the fuser thinks this is a fresh, valid measurement, it tightens its covariance around the wrong position, so it becomes confidently wrong (echoing the overconfidence theme). Retrodiction fixes it cleanly: roll the state back 60 ms (subtract the predicted motion), fuse the measurement at its time, then roll forward 60 ms again — now the old measurement contributes correctly, at the position it actually describes. The cost is keeping a short history of states; the benefit is not poisoning your track with stale data. In a networked car where camera and radar have different latencies, OOSM handling is not optional — it's the difference between a smooth track and one that jitters backward every time the slow sensor reports.

Chapter 10: Connections, References & Cheat-Sheet

You now hold the architectural skeleton that the rest of the series hangs its math on. Lesson 1 told you what relationship the sensors have; this lesson told you where the fusing lives and who decides. Before we name what comes next, here is everything in one place — the cheat-sheet to screenshot, the motto, the real sources, and the cross-links.

The fusion-architecture cheat-sheet

The five questions that pin down any fusion system

Combine this lesson with Lesson 1 and you have a five-question interrogation that fully characterizes any fusion design you encounter:

1. Relationship (Lesson 1): complementary, competitive, or cooperative? → tells you if combining helps reliability.
2. JDL level (Ch 1): signal, object, situation, threat, or process? → tells you what abstraction it reasons about.
3. Dasarathy mode (Ch 2): what kind of thing flows in and out? → tells you the bandwidth and modularity.
4. Architecture (Ch 3–4): centralized, distributed, decentralized, or hierarchical? → tells you bandwidth bill and fault-tolerance.
5. Consistency (Ch 5): does it ship estimates with any feedback cycle? → tells you if it's at risk of double-counting / overconfidence.

Five questions, five independent axes. Ask them of any radar, any robot, any phone, any aircraft, and you will have characterized its fusion completely — what it reasons about, how it combines, where the combining lives, what it costs, and whether you can trust its confidence. That is the whole of classical fusion's conceptual layer, and you now hold it.

The motto

The architectures in one quick-reference table

Screenshot this. It is the entire lesson collapsed into a single lookup:

Where this lesson sits in the series

This was Lesson 2 of 22 — the architectural foundation. The recurring chapter structure you just walked (Why → concepts → showcase → Use Cases, Practical Application, Debugging, Connections) repeats in every lesson, so you always know where you are. Here is the road ahead, so you can see how the architectures grow into a full toolbox:

Next up, Lesson 3: Coordinate Frames & Time Synchronization — the unglamorous but essential plumbing under every architecture. Before two sensors' data can be fused at all, they must agree on where (a common coordinate frame — the registration problem behind raw fusion) and when (a common clock — the skew and out-of-sequence problems we met in Chapter 9). Lesson 3 turns those two debugging traps into a discipline.

Related lessons on Engineermaxxing

These lessons go deeper on machinery we touched:

• Lesson 1: Why Fuse At All? — the three relationships (complementary, competitive, cooperative) that this lesson's architectures are built to implement. Architecture is where; relationship is what.
• Lesson 9: The Information Filter — the additive (Y = Y₁ + Y₂) form of the Kalman filter that makes decentralized peer-to-peer fusion possible, because information from independent sources simply sums.
• Lesson 11: Covariance Intersection — the conservative fusion rule that cures the double-counting trap from Chapter 5 by making no assumption about how correlated the estimates are. The direct sequel to this lesson's central worked example.
• The Kalman Filter — the optimal estimator that, in a centralized architecture, computes the gold-standard fused estimate this lesson measures everything against.

References

1. Steinberg, A. N., Bowman, C. L., and White, F. E. "Revisions to the JDL Data Fusion Model." Proceedings of SPIE, Sensor Fusion: Architectures, Algorithms, and Applications III, vol. 3719, pp. 430–441, 1999. The canonical revision that added Level 0 and standardized the five-level JDL data-fusion model. doi:10.1117/12.341367
2. Dasarathy, B. V. "Sensor Fusion Potential Exploitation—Innovative Architectures and Illustrative Applications." Proceedings of the IEEE, vol. 85, no. 1, pp. 24–38, 1997. Introduces the data/feature/decision (DFD) input–output classification used in Chapter 2. doi:10.1109/5.554206
3. Castanedo, F. "A Review of Data Fusion Techniques." The Scientific World Journal, vol. 2013, Article 704504, 2013. A modern survey tying together fusion architectures (centralized / distributed / decentralized), the JDL model, and Dasarathy's classification. doi:10.1155/2013/704504
4. Durrant-Whyte, H. F., Rao, B. Y. S., and Hu, H. "Toward a Fully Decentralized Architecture for Multi-Sensor Data Fusion." Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 1331–1336, 1990. The foundational work on fully decentralized, peer-to-peer sensor fusion using the information filter. doi:10.1109/ROBOT.1990.126184