Reasoning I: Bayesian Inference over LLM Beliefs

Chapter 1: Induction, Abduction, Deduction

Chapter 0 diagnosed the disease: a single forward pass that outputs “probability: 70%” is not a calculation, it's a completion. The cure requires naming exactly which step in the reasoning process is missing, and that requires three words philosophers have used since Aristotle and Peirce — because it turns out LLM prompting research reinvented all three, and knowing the names makes BIRD's architecture almost obvious.

Three ways to get from evidence to a belief

Given a context C (a scenario plus whatever partial information you have) and a decision Y (which of two outcomes is more likely), there are three distinct logical moves you could make:

A vanilla LLM prompted for a probability performs pure induction: C → Y in one shot, using whatever statistical pattern its training data associated with sentences that look like this one. Chain-of-thought prompting looks like it adds structure — the model writes out reasoning steps — but the final number is still produced by the same induction machine; the intermediate text is not constrained to be a reusable, checkable structure. It's induction wearing a lab coat.

The formal setup

BIRD's problem statement, stripped to its essentials: you have a scenario S (a general situation, e.g. “you want to charge your phone while using it”) and an additional condition U (extra information or a preference, e.g. “you'll be pacing around the room”). Together they form the context C = (S, U). There are two complementary outcomes, O₁ and O₂ (“a shorter cord is better” vs. “a longer cord is better”). The task is to compute P(O_i|C) for i = 1, 2.

Instead of estimating that directly, BIRD introduces a set of factors — discrete variables that together form a complete information space F, and routes the computation through them:

Read this the way you'd read any marginalization: to know how likely O_i is given partial information C, sum over every possible fully specified world f in F, weighting each world's contribution P(O_i|f) by how consistent that world is with what you actually observed, P(f|C). It's the law of total probability, and it is the one piece of real, unavoidable Bayesian machinery in this whole framework — everything else is about how to fill in its two factors without asking an LLM to compute either one in a single unreliable step.

Concept → realization: what the LLM is asked to do, twice, never three times

It is worth being precise about exactly how many LLM calls carry real risk in this pipeline, because that is what makes BIRD more trustworthy than a single forward pass, not just slower. The LLM is used for:

1. Abduction (once per scenario, reusable across every condition you'll ever apply to it) — generate the factors.

2. Coarse classification (once per factor value, also reusable) — does this factor value support O₁, O₂, or neither?

3. Entailment (once per new condition U) — which factor values does this specific condition imply?

Notice what's not on that list: the LLM is never asked “what is the probability?” directly. The arithmetic that turns factor values into a number is Equations 2, 4, 8 and 9 — fixed, auditable formulas with no LLM in the loop. This is why BIRD is controllable: the same scenario S always produces the same factor structure F, no matter how many different conditions U you later throw at it, and the final number is a deterministic function of which factor values got entailed — not of how the condition happened to be phrased.

Why route through factors at all — the computational argument

There's a second reason abduction matters, beyond controllability: it turns an intractable estimation problem into a tractable one. Directly estimating P(O_i|C) for arbitrary text C requires the LLM to implicitly integrate over every possible interpretation of C in a single forward pass — exactly the step that Chapter 0 showed produces noise. Factoring C through F means each LLM sub-call only has to answer a much narrower question (“is this one factor value implied?” or “does this one factor value support O₁?”), and narrow, single-fact questions are exactly the kind of thing LLM classification is reliable at. You're not making the model smarter. You're decomposing the question into pieces small enough that the model's existing reliability transfers.

The whole pipeline as one function

It is worth writing the entire architecture as pseudocode once, so every later chapter has a place to point back to. Read the comments as a map of which section of this session builds which line:

python
def bird_estimate(S, O1, O2, U):
    # --- computed ONCE per scenario S, reused for every future U ---
    factors = abduct_factors(S, O1, O2)        # Ch.2 — LLM: sentences, then summarize
    factors = prune_neutral(factors, O1, O2)  # Ch.2 — LLM: 3-way classify, drop all-neutral
    p_init  = {f: classify_support(f, O1) for f in factors.values()}  # Ch.4 — Eq.5, 75/50/25
    p_j     = train_probabilities(factors, p_init)          # Ch.6 — Eq.6-7, MSE + margin loss

    # --- computed FRESH for every new condition U ---
    observed = llm_entailment(S, U, factors)   # Ch.3 — Eq.8-9, hierarchy + direct prompts
    weights  = assign_weights(observed, factors)  # 1.0 on entailed value, 1/card(F_j) elsewhere

    # --- fixed arithmetic, no LLM in the loop ---
    return marginalize(p_j, weights, factors)   # Ch.5 — Eq.1-2, sum over f in F

Two lines are worth staring at. abduct_factors through train_probabilities take only S, O1, O2 as arguments — never U. That is the formal statement of “abduction is a fixed, reusable structure.” llm_entailment is the only LLM call that ever sees the specific condition U, and its entire job is a narrow, well-posed classification (“which of these already-fixed factor values does this text imply”) — nothing about it resembles the open-ended “what's the probability” question from Chapter 0.

Controllability, demonstrated with three conditions

Controllability is an abstract-sounding property until you see it hold across multiple real conditions on the same scenario. Take the charging-cord scenario again, and three different additional conditions a user might supply:

Every row maps onto the same three factors, defined once. Nothing about the second condition's completely different phrasing invents a new factor called “outlet position preference” that happens to overlap with F_B — it's literally F_B, because F_B was fixed during abduction before any of these three conditions existed. This is what “the same additional condition is consistently mapped to the same factor structure” means operationally, and it's the property that makes BIRD's outputs comparable across many different user inputs to the same underlying decision.

Two generality properties, for free

The paper notes two more consequences of this design that are easy to miss. First: because entailment can run at any point after abduction, a condition doesn't have to arrive all at once — it can be supplied incrementally, across multiple turns of a conversation, and each new sentence just updates which factors move from unobserved to observed. Second: the framework as stated handles exactly two complementary outcomes, but a decision with more than two options decomposes into this same binary machine applied hierarchically — pick outcome A vs. “everything else,” then within “everything else” pick B vs. the remainder, and so on. Neither extension requires touching Equations 1–9; both fall out of treating abduction as a one-time investment and entailment as the only per-interaction cost.

Related approaches, and specifically why BIRD differs

This isn't the first attempt to make LLM uncertainty trustworthy, and it's worth placing BIRD against the alternatives explicitly, because each one fixes a different piece of Chapter 0's problem and leaves another piece untouched.

The common thread in the first three rows: every prior approach still asks the LLM's forward pass to be the final source of the number, whether that number comes from logits, from a fine-tuned head, or from verbalization. BIRD is the odd one out in this table specifically because the arithmetic that produces the final probability lives outside the LLM entirely — Equations 1, 2, 4, 8, and 9 are ordinary Python, not a forward pass, and the LLM only ever answers the narrow sub-questions that feed those equations.

Worked example: what “hierarchical” actually looks like with three outcomes

Chapter 1 mentioned in passing that a decision with more than two options decomposes into this same binary machine, applied hierarchically. It's worth seeing that claim as actual numbers before moving on, because “hierarchically” can otherwise sound like hand-waving standing in for a real answer.

Suppose the real decision is a three-way commute choice — bike, bus, or drive — not the two-way choice this session builds everything around. The hierarchical trick: first ask a binary question, bike vs. “not bike,” then, only within “not bike,” ask a second binary question, bus vs. drive. Two applications of the same two-outcome machine this whole session is built around, chained one after the other.

Say the first-level factors (weather, distance, how much gear you're carrying) combine, by whatever process Chapters 2–6 will build, to P(bike) = 0.30, so P(not bike) = 0.70. That second number is not the end of the computation — it's the total probability mass available to split between bus and drive at the second level. A separate set of factors (is transit running on schedule, is there parking near the destination) feeds a second binary decision conditioned on already being in the “not bike” branch: say that comes out to P(bus | not bike) = 0.60, so P(drive | not bike) = 0.40.

Two completely separate two-outcome computations, each built exactly the way Chapters 2–6 build the charging-cord one, and multiplying down the tree recovers a valid three-way distribution that sums to 1. No new machinery, no N-way generalization of any equation this session derives — just two independent binary questions, one nested inside the other, which is exactly what Chapter 1 meant by “decomposes hierarchically” and exactly why the paper never needed to derive a three-outcome or N-outcome version of Eq. 4.

Chapter 2: The Factor Space

Chapter 1 left a gap: how does an LLM “imagine the relevant factors” for a scenario, and what exactly comes out? This chapter opens that box.

The direct approach, and why BIRD doesn't use it

The obvious method is to prompt the LLM once: “list the factors that would affect this decision, and their possible values.” BIRD's authors tried this and found it underperforms a two-stage alternative, for a specific, testable reason: a direct listing request makes the model reach for summarization mode — it produces the factors a textbook would list, not the ones its own training data actually associates with the scenario. It leaves the model's “imagination” — the broad associative knowledge baked into its weights by pretraining — mostly untapped.

So BIRD splits factor generation into two stages that engage a different capability at each step:

Stage 1 — generate situations, not factors. Prompt the LLM to write sentences describing situations that would make outcome O₁ more likely, and separately, situations that would make O₂ more likely — ten of each. This is a generative, low-constraint task; the model is free to draw on anything in its training distribution that resembles this scenario, without worrying about formalizing anything yet.

Stage 2 — summarize the sentences into factors. Only after twenty concrete situations exist does the LLM get asked to abstract them into a small number of named factors, each with a discrete set of values.

Pruning: not every factor survives

Not every generated factor is useful. Some — “the cell phone's model,” in the charging-cord example — turn out to have no bearing on the decision at all: every value of that factor supports neither outcome over the other. BIRD prunes these by asking the LLM to classify, for each candidate factor value, whether it supports O₁, supports O₂, or is neutral. A factor where every one of its values comes back neutral contributes nothing and is dropped before the factor space is even built. This pruning step matters enormously for what follows, and Chapter 7 will show you the exact number: dropping non-decisive factors this way improves final decision accuracy by 4.4 percentage points on average and cuts the “I don't know” rate by nearly 17 points, compared to skipping straight to the direct one-shot listing.

Building the space, by hand

Say abduction (after pruning) leaves three decisive factors for the charging-cord scenario. Each is binary — two possible values — which is the simplest case, chosen here purely for hand-computability. It follows the identical product-space logic the paper itself uses to illustrate the mechanism, just with fewer factors: the paper's own worked example is a six-factor charging-station scenario (Figure 2), where it notes plainly “there are 2⁶ elements in total” — the same multiplication rule this chapter is about to apply, scaled to twice as many factors. Nothing about the mechanism changes between three factors and six; only the arithmetic gets harder to do by hand, which is exactly why this session starts at three.

F is the product space of all three value sets — every combination of one value from each factor is one point f = (f_A, f_B, f_C) in F:

Eight fully-specified “possible worlds” — (a₁,b₁,c₁), (a₁,b₁,c₂), (a₁,b₂,c₁), and so on through all eight combinations. This is the “complete information space” from Chapter 1's marginalization formula: F is exactly this set of eight points, and P(O_i|C) is a weighted average over all eight.

Why this grows fast, and why that's a design pressure, not a footnote

Three binary factors is deliberately small for teaching. Real scenarios abduct more factors, and factors are frequently non-binary (BIRD's actual datasets use factors with anywhere from 2 to several values each). Watch what happens as N grows, holding every factor binary:

At ten factors, the “complete information space” already has over a thousand fully-specified worlds. No one hand-enumerates that, and critically, no single condition U will ever observe more than a handful of them — most scenarios entail two or three factor values, not ten. This is exactly the gap that Chapter 3's treatment of unobserved factors exists to close, and it is exactly why Chapter 4's independence assumption exists: without it, computing P(O_i|f) for a general f would require enumerating conditional probabilities over an exponentially large joint space. With it, the computation becomes a product of N independent, individually-estimable numbers — one per factor, not one per combination.

Stage 1, made concrete

It helps to see plausible output from the free-form generation step, because the two-stage design only makes sense once you can picture what stage 1 actually produces. For the charging-cord scenario, prompting for situations that support O₁ (shorter cord) versus O₂ (longer cord) separately might yield something like:

Notice these are free sentences, not yet formalized — some describe behavior (“paces while talking”), some describe environment (“outlet across the room”), some describe preference (“hates cable slack”). Stage 2 is exactly the operation of noticing that these ten-plus-ten sentences cluster into a small number of underlying dimensions — behavior, environment, preference — and formalizing each cluster into one named factor with two (or more) values. The clustering step is also where BIRD's authors report the strongest diversity gain over asking for factors directly: a direct-listing prompt tends to converge on the two or three most stereotypical factors and stop, while clustering twenty concrete situations surfaces dimensions a direct request would never think to list.

Pruning, as a loop

python
def prune_neutral(factors, O1, O2):
    kept = {}
    for name, values in factors.items():
        labels = [classify_support(v, O1, O2) for v in values]  # LLM: O1 / O2 / neutral
        if len(set(labels)) > 1 or "neutral" not in labels:
            kept[name] = values   # at least one value is decisive, or values disagree — keep it
        # else: every value came back "neutral" — e.g. "the cell phone's model" — drop it
    return kept

This is the same classification call that Chapter 4 reuses to set each surviving value's initial probability — pruning and initialization are two readings of one LLM call, not two separate passes. A factor value that gets labeled “supports O₁” here is exactly the value that receives 75% in Equation 5.

One more detail from the paper's experimental settings worth folding in here: this classification call isn't run once and trusted blindly — it's run three times at temperature 0.7, and the majority label wins. A single sample at nonzero temperature can land on “neutral” by chance even for a genuinely decisive factor value, or vice versa; three independent samples and a vote is the same self-consistency principle Chapter 3's entailment step uses, applied one step earlier, at the point where the factor space itself is still being shaped. Getting this vote wrong at the pruning stage is more consequential than getting one entailment call wrong later — a wrongly-pruned factor is gone from F for every future condition on this scenario, while a single bad entailment call only affects one condition.

The product formula, generalized beyond binary factors

Real abducted factors aren't always binary. A factor like “time of day” might have three values (morning, afternoon, evening); “room type” might have four. The size formula from earlier generalizes directly — it's the product of each factor's own value-set size, not a fixed base raised to a power:

Concretely: two binary factors and one three-valued factor gives |F| = 2 × 2 × 3 = 12, not 2³ = 8. The binary case in this session's worked examples is the simplest instance of this formula, chosen for hand-computability — not a restriction the framework itself imposes. Everything downstream (Chapters 3–5's marginalization, Chapter 4's Eq. 4 combination) is written in terms of P(O_i|f_j) for a general value f_j in a general factor F_j, and never assumes |F_j| = 2.

A flat Bayesian network, deliberately

The paper explicitly frames its factors as “similarly found in Bayesian networks” (Koller & Friedman, 2009). If you've seen a Bayes net drawn as a graph — nodes for random variables, arrows for dependencies — BIRD's factor space is the special case where every factor is a root node with no parents, feeding directly into a single outcome node O_i, and no factor has an arrow pointing to another factor. That flatness is not an oversight; it's what makes the conditional-independence assumption in Equation 2 legal, and it's why abduction's instruction to find factors that are “least relevant to each other” (Chapter 1) matters mechanically, not just stylistically: a genuine Bayes net with dependencies between factors would need a different, more expensive computation than the flat product Equation 2 uses. Chapter 9 returns to what happens when that flatness assumption is actually violated.

This also explains something about the pruning step from earlier in this chapter that's easy to miss on first read: pruning doesn't just remove factors that are useless on their own (the neutral ones). It's implicitly also the framework's only defense against factors that overlap with each other — if two candidate factors from stage 1's free-form sentences turn out to be near-paraphrases, a careful pass through the classification step in the pruning loop is where a human reviewer, or a more careful LLM prompt, would need to catch it. The paper's ablation (Chapter 7) measures the value of pruning against a direct-generation baseline, but doesn't separately measure how often near-duplicate factors slip through — a genuine open question this session's Chapter 9 will flag as an untested assumption, not a solved one.

A hand-check: how much does fixing one factor shrink the space?

One more useful arithmetic habit before moving on. Take a slightly bigger factor space than the running three-binary-factor example — say four factors with sizes 2, 3, 2, and 4 — so |F| = 2 × 3 × 2 × 4 = 48 worlds total. Now ask: if a condition entails a specific value of just the third factor (the one with 2 possible values), how many of those 48 worlds survive with nonzero weight?

Pinning one factor to a single observed value doesn't just add information — it divides the surviving space by exactly that factor's own cardinality, because every combination of the other factors is still free to vary, and every one of those combinations paired with the pinned value is still a live world. Observe a second factor (say the one with 4 values) on top of that, and the surviving count divides again: 24 ÷ 4 = 6. This is exactly the mechanical content behind this chapter's earlier claim that “most of that space always stays genuinely unknown” — each single observed fact prunes the space by a factor equal to how many values that one dimension could have taken, not by some fixed amount, and a condition would need to observe every factor before the space collapses to the single surviving world Chapter 5's Stage 3 reaches.

Chapter 3: Observed vs. Unobserved

Chapter 2 built F — eight possible worlds, fixed once per scenario. Now a specific condition U arrives: “you'll be pacing around the room while on the call, and you like your cord tidy with no slack on the floor.” The question this chapter answers: how does that one sentence turn into P(f|C), a weight over all eight worlds?

LLM entailment: mapping text onto factor values

BIRD calls this step entailment deliberately, not classification. The task given to the LLM is not “what does this condition mean” in the abstract — it's the much narrower, much more reliable question “does this condition entail (necessarily imply) this specific factor value?” Entailment is a task LLMs are well-practiced at and, crucially, one where the standard failure mode is underconfidence rather than fabrication: a well-calibrated entailment judgment says “not entailed” when the text is silent on the matter, rather than inventing an implication that isn't there.

BIRD runs two prompt styles and takes a majority vote across three samples: a hierarchy prompt (first decide which factors are touched by the condition at all, then for each touched factor, decide which value) and a direct prompt (for one specific factor-value pair, just ask “implied, yes or no”). The split exists because the two prompts trade off differently with scenario complexity: on their hardest dataset (temporal reasoning, dense with implicit timing cues) the direct prompt alone proved more reliable; on simpler datasets the hierarchy prompt, which is cheaper per factor, was favored 2-to-1 in the vote.

Turning entailment into weights: two equations, two very different shapes

For our condition — “pacing around the room” and “wants it tidy” — the entailment step finds: F_A is implied, value a₂. F_C is implied, value c₁. F_B (where the outlet sits) is never mentioned — not implied at all.

For an observed factor — one whose value the condition entails — BIRD assigns all the probability mass to that single value:

For an unobserved factor — one the condition never mentions — BIRD does something that looks almost too simple to be the answer, and is worth sitting with: it assigns equal probability to every possible value.

For our binary F_B: P(b₁|C) = P(b₂|C) = 0.5 each.

The shape of the weighting, concretely

Put the whole thing together for our example. Of the eight worlds in F, only the ones consistent with both observed values — a = a₂ and c = c₁ — get any weight at all: any world with a = a₁ or c = c₂ is weighted exactly 0, because the condition directly contradicts it. That leaves exactly two surviving worlds, distinguished only by the unobserved factor F_B:

This is Equation 2 from the paper — the conditional-independence version of the marginalization sum — made concrete: P(f|C) is a product of per-factor terms, ∏_j P(f_j|C), and because two of the three factors are pinned to a single value (probability 1), the product collapses to just the unobserved factor's contribution.

Concept → realization: what changes if entailment gets it wrong

This is a good moment to be honest about a real failure mode, because it will matter in Chapter 9. If entailment fails to detect that the condition implies a₂ — say the sentence is phrased obliquely and the LLM's majority vote lands on “not implied” — then F_A becomes unobserved by mistake, its probability spreads uniformly across a₁ and a₂, and the final P(O_i|C) drifts toward the fully-unobserved 50% baseline instead of reflecting what the condition actually said. Entailment errors don't crash the system — they silently dilute the answer toward neutral. That is a gentler failure than a confidently wrong number, but it is still a failure, and it is the single largest source of BIRD's “unknown” outputs, which Chapter 9 quantifies exactly.

The two entailment prompts, side by side

The hierarchy prompt and the direct prompt ask conceptually different questions, and it's worth seeing both in pseudocode to understand why the paper ensembles them rather than picking one:

python
def hierarchy_prompt(S, U, factors):
    # Step A: which FACTORS (not values yet) does U touch at all?
    touched = llm(f"Scenario: {S}\nCondition: {U}\n"
                  f"Which of these factors does the condition address: {list(factors)}?")
    entailed = {}
    for name in touched:
        # Step B: for each touched factor, which VALUE is implied?
        entailed[name] = llm(f"Condition: {U}\nWhich value of {name} is implied: "
                              f"{factors[name]}, or none?")
    return entailed

def direct_prompt(S, U, factor_name, value):
    # one factor-value pair at a time: yes/no, does U entail this specific value?
    return llm(f"Condition: {U}\nDoes this imply that {factor_name} = {value}? yes/no") == "yes"

The hierarchy prompt is cheaper — one call per factor plus one call per touched factor, rather than one call per factor-value pair — but it risks a compounding error: if step A wrongly decides a factor isn't touched, step B never runs and a real implication gets silently dropped. The direct prompt checks every value independently, which costs more calls but doesn't let one wrong “not touched” judgment suppress a whole factor. That's exactly the tradeoff the paper's per-dataset setting encodes.

Matching prompt style to scenario complexity

Three samples, majority vote, per condition — this is the same self-consistency principle chain-of-thought baselines use for their final answer, applied here to the much narrower entailment sub-task instead of to the final probability itself. When the three samples disagree on which value a direct prompt implies, the paper falls back to the pruning-style prompt from Chapter 2 to force a single choice.

P(f|C) as data, not just as formula

It's useful to see Equations 8 and 9 as a small piece of code, because the two-branch structure maps one-to-one onto observed versus unobserved:

python
def assign_weights(entailed, factors):
    weights = {}
    for name, values in factors.items():
        if name in entailed:                         # Eq.8 — observed
            weights[name] = {v: (1.0 if v == entailed[name] else 0.0) for v in values}
        else:                                          # Eq.9 — unobserved
            weights[name] = {v: 1.0 / len(values) for v in values}
    return weights

For our running example, calling this with entailed = {'F_A': 'a2', 'F_C': 'c1'} produces weights['F_A'] = {'a1': 0.0, 'a2': 1.0}, weights['F_B'] = {'b1': 0.5, 'b2': 0.5}, weights['F_C'] = {'c1': 1.0, 'c2': 0.0} — exactly the numbers used in this chapter's grid and Chapter 5's full computation.

Entailment across a multi-turn conversation

Chapter 1 mentioned that a condition can arrive incrementally, across turns, rather than all at once. Trace what that looks like mechanically, because it's a direct consequence of everything built so far and nothing new needs to be invented for it:

python
weights = {name: {v: 1.0/len(vals) for v in vals} for name, vals in factors.items()}  # start: everything unobserved

# turn 1: "you'll be pacing around the room"
new_entailed = llm_entailment(S, "you'll be pacing around the room", factors)
weights.update(assign_weights(new_entailed, factors))     # F_A collapses to {a2: 1.0, a1: 0.0}
print(marginalize(p_j, weights, factors))          # 0.321 — Chapter 5 Stage 1

# turn 2, later in the same conversation: "oh, and I like the cord tidy"
new_entailed = llm_entailment(S, "I like the cord tidy", factors)
weights.update(assign_weights(new_entailed, factors))     # F_C collapses too; F_A stays as turn 1 left it
print(marginalize(p_j, weights, factors))          # 0.500 — Chapter 5 Stage 2

Each turn only ever runs entailment against the new sentence and only ever updates the factors that sentence touches; a factor already pinned by an earlier turn is untouched by a later one unless the later turn explicitly revises it. This is why P(f|C) can be maintained as running state across a conversation rather than recomputed from the full transcript every time — the marginalization call at the end of each turn is cheap precisely because weights only ever tighten (unobserved → observed), never re-expand, as a conversation proceeds.

What happens when a later turn contradicts an earlier one

Real conversations aren't always monotonic — a user might say “actually, I changed my mind, I'll be sitting still the whole time” after already establishing a₂ (pacing) in turn 1. Nothing in the weight-assignment code from Chapter 3 prevents this: assign_weights is a pure function of whatever the most recent entailment call returns for that factor, so a later, contradicting entailment call simply overwrites the earlier weight — F_A flips from {a1: 0.0, a2: 1.0} back to {a1: 1.0, a2: 0.0}. There's no special-cased “revision” logic required, because P(f_j|C) was never defined as an accumulation of every sentence ever said about factor j — it's defined as the current best read of what the most complete version of C implies, and a later sentence updating an earlier one is just what a more complete C looks like. This is also, incidentally, why running entailment against the full transcript at the end of a long conversation (rather than turn-by-turn) is a legitimate implementation choice too, and would arrive at the same final weights — the incremental version is a cost optimization, not a different answer.

A worked check with a non-binary factor

Every worked example so far in this chapter used binary factors, where an unobserved factor's weight is always a clean 0.5. It's worth checking Eq. 9 on a factor with more than two values, because that's the general case the formula actually has to handle — Chapter 2 already noted real abducted factors aren't always binary — and it's exactly the kind of factor Chapter 8's aggregator widget will let you play with directly.

Suppose abduction also surfaced a fourth factor for the charging-cord scenario — F_D, room clutter, with three values: tidy (d₁), moderately cluttered (d₂), and very cluttered (d₃). A condition that says nothing about the room's state leaves F_D completely unobserved. Eq. 9 says:

Same principle as the binary case — spread the weight uniformly across every possible value — just with a denominator of 3 instead of 2. Now combine this with the original three-factor example: a condition entailing F_A=a₂ and F_C=c₁, with F_B and the new F_D both unobserved. The full factor space now has 2 × 2 × 2 × 3 = 24 worlds; the surviving, nonzero-weight ones are those consistent with a₂ and c₁, which is 1 × 2 × 1 × 3 = 6 worlds (F_B's two values times F_D's three), each carrying weight 0.5 × 0.333 ≈ 0.167.

Chapter 4: Combining Evidence

Chapters 2 and 3 built the two things the marginalization formula needs weights for — the space F and the observation weights P(f|C). The one piece left unbuilt is the other half of the sum: P(O_i|f), the probability of the outcome given a fully specified world. This chapter is about why that number is hard to get directly, and the elegant trick BIRD borrows to get it anyway.

The classic Bayesian move, and why it doesn't work here

Textbook Bayesian inference would build P(O_i|f) from Bayes' rule: start with the likelihood P(f|O_i) — “if O_i were true, how likely is this particular world f?” — multiply by the prior P(O_i), and normalize. BIRD's authors tried asking the LLM for exactly that likelihood, and it is a substantially harder question than it looks. To answer “if a shorter cord really is better, how likely is it that you're pacing the room with a taut-cord preference,” the model has to invert cause and effect and imagine a distribution over hypothetical worlds consistent with a fixed conclusion — a speculative, underconstrained judgment prone to the exact same fluency-without-computation failure from Chapter 0.

Compare that to the forward question: “given this fully-specified world (you're pacing, you want it tidy), how likely is a shorter cord the better choice?” That's a direct, well-posed judgment about a completely described situation — the kind of single-fact classification LLMs are reliable at, the same reliability Chapter 1 leaned on for entailment.

Bordley's formula, and the two assumptions that simplify it

The general form (Bordley, 1982) combines N per-factor probabilities P(O_i|f_j) into one joint probability P(O_i|f), using a per-factor weight w_j and a prior P(O_i):

BIRD makes it tractable with two choices, both justified by the same fact: there is no prior information suggesting anything other than the neutral answer. Equal weights, w_j = 1 for every factor (no factor is a priori trusted more than another — that's what “least relevant to each other, covering different angles” from Chapter 2's abduction step is for). And a 50% prior, P(O_i) = 0.5 (before seeing any factor, both outcomes are equally likely by construction — they were defined to be complementary). Substituting both collapses the formula to something you can compute on paper:

This is the single most important formula in the whole session. Everything from here to Chapter 8 is either computing its inputs or using its output.

Where the individual P(O_i|f_j) numbers come from

Before training (Chapter 6 covers the trained version), BIRD initializes each factor's probability from the same three-way classification used for pruning in Chapter 2 — does this factor value support O_i, support the opposite outcome, or sit neutral:

These exact numbers — 75%, 50%, 25% — are the real values from the paper, and they're the ones we'll hand-compute with next.

Hand-worked example 1: two pieces of agreeing evidence

Suppose two factors both support O₁ (each initialized at 75%). Plug into Eq. 4:

Two independently-moderate 75% opinions combine into a 90% joint opinion — stronger than either one alone. This is what “independent evidence compounds” looks like as arithmetic: agreement between separate sources is more convincing than either source individually, exactly the intuition behind combining two witness testimonies that corroborate each other.

Hand-worked example 2: two pieces of conflicting evidence

Now suppose one factor supports O₁ (75%) and another supports O₂ just as strongly — which means it supports O₁ at only 25%:

Equally strong, opposing evidence cancels exactly to a coin flip. Notice this is symmetric by construction: Eq. 4 is built from products, and swapping which factor “wins” a symmetric tie leaves the denominator unchanged.

Eq. 4 as three lines of Python, and a third worked example

python
def eq4(probs):                 # probs: list of P(Oi | f_j) for j = 1..N
    num  = prod(probs)
    comp = prod([1 - p for p in probs])
    return num / (num + comp)

print(eq4([0.75, 0.75]))        # 0.9   — agreement compounds (worked example 1)
print(eq4([0.75, 0.25]))        # 0.5   — conflict cancels (worked example 2)

Now a third case, mixing agreement with different strengths rather than a flat 75/25 — this is the shape real training-refined probabilities take (Chapter 6 replaces the coarse 75/50/25 buckets with continuous values like these). Three factors, all leaning toward O₁ but by different amounts: 0.60, 0.75, and 0.90.

Three moderately-to-strongly agreeing factors combine to 97.6% — stronger than the weakest (0.60), stronger than the strongest alone (0.90), stronger even than two 90%'s combined (0.9×0.9 ÷ (0.81+0.01) = 0.988 would need two 90s; three mixed factors land close to that with one much weaker contributor). Independent, agreeing evidence keeps compounding as you add more of it, even when individual pieces are only moderately confident — this is the same mechanism a spam filter or a medical test panel relies on: no single weak signal is convincing, but several independent weak signals pointing the same way are.

Why equal weights specifically, and not something learned

Bordley's derivation offers a second case beyond equal weights — one where the sum of weights exceeds 1, capturing a decision-maker who becomes more certain as more independent factors accumulate, rather than less. BIRD adopts exactly this regime with the simplest possible instantiation, w_j = 1 for every factor. The paper's stated justification is symmetry of ignorance: abduction was explicitly designed (Chapter 2) to produce factors that are “least relevant to each other” and jointly cover “all the possible individual factors/angles” — if the factor-generation step already did its job, there is no principled basis left for trusting one surviving factor's judgment over another's. A learned, unequal weighting would need its own supervision signal and its own risk of overfitting to quirks of one dataset; equal weights is the assumption that adds no additional free parameters beyond what Chapter 6's training already fits.

What the 50% prior encodes, and when it would need to change

P(O_i) = 50% is not a general default — it's a consequence of a specific setup decision. Because O₁ and O₂ are defined as complementary (Chapter 1's problem statement: “the two outcomes are complementary”), and because the framework has no information at the scenario level (before any factor is even considered) that favors one over the other, 50/50 is the only prior consistent with “no information yet.” If a deployment had a genuine base rate — say, historical data showing shorter cords get chosen 70% of the time across all past decisions regardless of condition — Equation 3's general form accepts that directly as P(O_i); Equation 4 is specifically the 50%-prior special case, chosen because BIRD's benchmark scenarios don't supply such a base rate.

What Bordley's axioms actually require, in plain language

The formal derivation (Bordley, 1982, formalized further in the paper's Appendix A.2) rests on two conditions that sound intimidating in notation but are simple statements once translated. The first, a weak ordering on odds ratios, just says: if you'd trust one set of factor-implied odds at least as much as another set in one scenario, you'd make the same comparison consistently in any other scenario with the identical odds — your ranking of “how convincing is this combination of evidence” doesn't contradict itself from one case to the next. This is barely an assumption at all; it's closer to a definition of what it would even mean to have a consistent notion of “more convincing.” The second, a noninteraction property, says that how much you learn from factor j doesn't depend on which values the other factors happen to hold — which is exactly the conditional-independence assumption Chapter 2 already built into the factor space by design. Given both, Bordley proves this multiplicative form is the only continuous formula satisfying them — not one reasonable choice among many, but the unique one. That's a stronger justification than “this formula seems to work” — it's “any formula with these two sane properties would have to reduce to this one.”

Worked example 4: what goes wrong with the obvious alternative

It's worth seeing, concretely, why Eq. 4's specific multiplicative form earns its place over the simplest alternative anyone would reach for first: just average the per-factor probabilities. Take the two agreeing 75% factors from worked example 1.

The arithmetic mean returns exactly 0.75 — the same number a single 75%-confident factor would have given on its own. That's the tell: averaging treats a second piece of agreeing evidence as if it contributed nothing new, because averaging two identical numbers just returns that number back. Two independent witnesses who both say “I'm 75% sure” about the same conclusion is genuinely more convincing than either witness alone — and a combination rule that can't move past what one witness already told you has thrown away the second witness's information entirely.

Run it the other direction, on the conflicting-evidence case (worked example 2: one factor at 75%, one at 25%). The arithmetic mean gives (0.75+0.25)÷2 = 0.50 — which happens to match Eq. 4's answer here, by coincidence of symmetry, not because averaging is secretly correct. The agreement breaks the moment the two factors aren't perfectly symmetric opposites, which the compounding case just above already demonstrated.

Chapter 5: The Full Marginalization, By Hand

Every piece is on the table: F (Chapter 2), P(f|C) for observed and unobserved factors (Chapter 3), and P(O_i|f) via Eq. 4 (Chapter 4). This chapter runs the complete pipeline, by hand, on the charging-cord scenario, and watches P(O₁|C) move as information arrives — one factor at a time.

Fixing the inputs

Recall the three factors and their initial per-value probabilities of supporting O₁ (“a shorter cord is better”), all from Eq. 5's real 75/50/25 scheme:

Stage 0: no condition at all — the prior

Before any condition U is even supplied, all three factors are unobserved. By Eq. 9, every value of every factor gets uniform weight (0.5 each), so P(f|C) = 0.5 × 0.5 × 0.5 = 0.125 for every one of the eight worlds — a uniform distribution over F. Marginalizing (Eq. 1) an Eq.-4-computed P(O₁|f) uniformly over a factor space this symmetric returns exactly the 50% prior. No condition, no information, 50/50 — the framework's floor state matches the assumption it was built on.

Stage 1: observe F_A only

Condition: “you'll be pacing around the room.” This entails a₂ only; F_B and F_C stay unobserved. Four worlds survive with equal weight (0.25 each, since two unobserved binary factors split 0.5 × 0.5). Compute P(O₁|f) via Eq. 4 for each:

One entailed fact — “you'll be pacing” — and P(O₁|C) already moved from a neutral 50% to 32.1%, correctly leaning toward O₂ (longer cord), because pacing is the one thing the condition told us and it individually points toward O₂.

Stage 2: observe F_A and F_C, leave F_B open

Condition: “you'll be pacing around the room, and you like your cord tidy with no slack.” Now a₂ and c₁ are both entailed; only F_B stays unobserved. Two worlds survive, each weight 0.5 — and we already computed both of them in Stage 1:

This is the most important row in the whole hand-computation, precisely because it is not what naive intuition predicts. We added a second piece of information — “wants it tidy,” which on its own supports O₁ at 75% — and the answer went back up to exactly 50%, not somewhere between 32.1% and 75%.

Stage 3: observe all three

Condition additionally specifies the outlet is right next to your seat: b₁ is now entailed too. Every factor is observed; exactly one world in F survives with weight 1, and we computed its P(O₁|f) back in Stage 1:

Fully observed, the framework reports 75% confidence in the shorter cord — even though the very first fact we learned (pacing around the room) on its own pointed the other way. The full picture matters more than any single factor, and BIRD's marginalization is precisely the machinery that makes that true honestly rather than by accident.

Cross-checking Stage 1 with the odds shortcut

Chapter 4's odds-multiplication shortcut is a good way to sanity-check the 0.321 result from Stage 1 without re-doing all four Eq. 4 computations. Convert each surviving world's per-factor probabilities to odds (P ÷ (1−P)), multiply, then convert back — and separately average the four resulting probabilities weighted by 0.25 each, exactly as before:

Both match the direct Eq. 4 arithmetic from before, which is exactly the point of the odds view: it's not a different formula, it's the same combination read as multiplying independent likelihood ratios — a useful mental check whenever you want to verify a hand computation without redoing every multiplication from scratch.

Reading the ledger simulation

The interactive widget below implements precisely the loop this chapter just ran by hand: for each of the eight worlds in F, compute its weight P(f|C) as a product of per-factor terms (1 or 0 for observed factors, 0.5 for unobserved ones — Equations 8–9), compute its P(O₁|f) via Eq. 4, and accumulate the weighted sum. The four stages correspond exactly to the four rows in the summary table above. Watch two things as you click through: which rows fade to near-invisible (weight-zero, contradicted by an observed factor) versus stay solid, and how the bar length of each surviving row's P(O₁|f) never changes across stages — only which rows get to vote, and how much weight each vote carries, changes. That distinction (fixed per-world probabilities, moving observation weights) is the entire mechanical content of marginalization.

Extending to a fourth factor

Everything so far used exactly three factors, matching the paper's own illustrative “2³ elements” description. Nothing about the method caps N at 3 — add a fourth binary factor, F_D (“how urgently you need to leave the room during the call”: d₁ = rarely, supports O₁ at 75%; d₂ = often, supports O₂, so P(O₁|d₂) = 25%), and F now has 2⁴ = 16 worlds instead of 8. Repeat Stage 3 (all of F_A, F_B, F_C observed at a₂, b₁, c₁) but leave the new F_D unobserved:

Two things worth noticing. First, adding a fourth fully-agreeing factor value (d₁, on top of the three already-agreeing observed values) pushed the fully-observed answer from 0.75 up to 0.90 — worked example 1's “agreement compounds” lesson, now with three agreeing factors instead of two. Second, once you average in the still-unobserved F_D, the four-factor answer (0.700) is actually lower than the three-factor Stage 3 answer (0.750) — adding a factor to the space, even one that individually favors O₁, can pull the marginalized answer either direction once you account for the uncertainty it introduces while unobserved. More factors mean more potential precision when observed, and more potential dilution when they aren't — there's no free lunch from simply abducting more factors without also having conditions that entail them.

A different branch: what if the second fact pointed the same way as the first?

Stage 2 showed a condition that adds a fact pointing opposite to the first (a₂ favors O ₂, c₁ favors O₁) landing back at exactly 50%. It's worth hand-computing the branch where the second fact agrees with the first, to see the contrast directly rather than take it on faith.

Condition: “you'll be pacing around the room, and the outlet is on the far wall from your desk.” Now a₂ and b₂ are both entailed — both individually favor O ₂ (25% each toward O₁) — and F_C stays unobserved. Two worlds survive, weight 0.5 each: (a₂,b₂,c₁) and (a₂,b₂,c₂), both already computed back in Stage 1's table.

Two agreeing facts (both pointing toward O₂) pull the answer from 50% (nothing observed) down to 14.3% — far more confident than either fact alone (32.1% from a₂ alone), and further from neutral than the opposing-evidence branch ever got. Lay the two branches side by side and the shape of marginalization becomes visible as a genuine fork, not just a formula:

Same starting point (32.1%), same-sized new fact, two opposite trajectories — one pulled back toward neutral, the other pushed further away from it — entirely determined by whether the new fact agrees or disagrees with what was already known. Marginalization has no built-in direction; it just honestly compounds whatever the observed factors actually say, which is the same point Chapter 4's worked examples 1 and 2 made in the abstract, now shown as two branches of the identical concrete scenario.

Verifying by brute-force enumeration

For anyone who wants to check any of this session's hand arithmetic without trusting a spreadsheet, the whole computation is short enough to brute-force directly — enumerate every world in F, weight it, sum:

python
from itertools import product

values = {'A': ['a1','a2'], 'B': ['b1','b2'], 'C': ['c1','c2']}
p = {'a1':0.75,'a2':0.25,'b1':0.75,'b2':0.25,'c1':0.75,'c2':0.25}
observed = {'A':'a2', 'C':'c1'}     # Stage 2: A and C observed, B unobserved

total = 0.0
for a, b, c in product(values['A'], values['B'], values['C']):
    weight = int(observed.get('A',a)==a) * 0.5**('B' not in observed) * int(observed.get('C',c)==c)
    if weight > 0:
        total += weight * eq4([p[a], p[b], p[c]])
print(total)   # 0.5 — matches Stage 2 exactly

Chapter 6: Teaching the Model to Judge Fair

Every computation in Chapter 5 used the initial factor probabilities — the blunt 75/50/25 scheme from a single three-way classification. That scheme is a reasonable starting point, but it throws away information: it can't distinguish a factor that barely supports O₁ from one that supports it overwhelmingly — both just get 75%. This chapter covers how BIRD refines those three coarse buckets into continuous, learned probabilities, and why the refinement measurably helps.

Where the training signal comes from

You cannot get ground-truth P(O_i|f_j) values from a labeled dataset — nobody annotated “factor value a₂ in isolation has probability exactly 0.23 of supporting O₁.” BIRD's trick: the LLM is reliable at judging outcomes under complete information (that's the well-posed forward question from Chapter 4). So sample 128 fully-specified instances f′ ∈ F per scenario, and for each, prompt the LLM to output a verbalized probability from a fixed seven-point vocabulary:

These 128 verbalized-and-mapped judgments become the supervision signal, denoted P_LLM(O_i|f′). The learning problem: find per-value probabilities P(O_i|f_j) such that, when combined through Eq. 4 for any sampled f′, the result matches what the LLM judged when it could see the entire f′ at once.

The loss: fit the number, but don't flip the sign

Two loss terms, added together:

Mean squared error between the Eq.-4-combined estimate and the LLM's complete-information judgment, the ordinary regression signal.

A margin-ranking term that separately protects each individual factor value's direction. It computes a “trained” per-value probability by marginalizing over the remaining factors (holding f_j fixed, averaging Eq. 4's output across the other N−1 factors' possible values) and penalizes it if that trained probability crosses to the wrong side of 50% relative to where the initial classification (Eq. 5) said the value pointed:

The paper trains with SGD, learning rate 0.01, 20 epochs, batch size 4, α = 10, ε = 0.

Does the extra machinery earn its keep?

The paper runs this as an ablation, comparing four ways of getting P(O_i|f) into Eq. 4. One bookkeeping note before the numbers: the paper only publishes this specific four-way breakdown — 1/2 assumption, 1/N assumption, fixed initial probability, and fully trained — for Llama-3.1-70B, not for the Llama-2-70B run this session has used everywhere else (Chapters 0 and 7 both quote Llama-2's headline trained average, 0.592). The two models' trained scores are close but not identical — Llama-3.1 trained lands at 0.588 — close enough that the comparison below is fully representative of the point it's making, but every row in this specific table is Llama-3.1's, not Llama-2's:

Trained probabilities win, but notice the gap to the fixed initial probabilities is only 2.0 points (0.588 vs 0.568) — the crude 75/50/25 scheme is already most of the way there, because most of BIRD's power comes from routing through explicit factors and honest marginalization at all, not from the fine-tuning of any one number. Training is a real, measurable refinement, not the source of the framework's core advantage. Notice too how much of the total climb happens before training ever enters the picture: going from the crudest baseline (1/2 assumption, 0.480) to the fixed, untrained 75/50/25 scheme (0.568) is an 8.8-point jump, more than four times the further 2.0-point gain training buys on top of it. The expensive-sounding machinery from Chapters 2 through 5 — abduction, entailment, marginalization — is doing almost all of the work; Chapter 6's training loop is a genuine but modest polish on top of it, not the reason BIRD beats the baselines in the first place.

The training loop, written out

python
# one training run per scenario S — the 3 factors from this session's example
p = {name: init_from_eq5(name) for name in all_factor_values}   # trainable, starts at 0.75/0.50/0.25
opt = SGD(p.values(), lr=0.01)

for epoch in range(20):
    for batch in sample_batches(all_128_instances(F), batch_size=4):
        for f_prime in batch:
            p_llm      = verbalize_and_map(f_prime)         # Eq: 7-word scale -> 0/20/40/50/60/80/100%
            p_est      = eq4([p[fj] for fj in f_prime])     # Eq.4 with CURRENT trainable values
            loss_mse   = (p_est - p_llm) ** 2
            loss_mr    = 0.0
            for fj in f_prime:
                p_trained = marginalize_over_others(p, fj, F)   # Eq.6 — average Eq.4 over F*
                y_target  = sign(p_init[fj] - 0.5)
                loss_mr  += max(0, -y_target * (p_trained - 0.5) + EPSILON)
            loss = loss_mse + ALPHA * loss_mr        # ALPHA = 10
            loss.backward(); opt.step(); opt.zero_grad()

Two things to notice in the loop. First, p_llm is computed once per sampled instance and never updated — it is fixed supervision, the LLM's complete-information judgment. Second, p_trained in the margin-ranking term is not the same quantity as p_est: p_est plugs the current trainable values straight into Eq. 4 for one specific sampled instance; p_trained (Eq. 6) holds one factor value fj fixed and averages Eq. 4's output over every possible completion of the remaining factors. It answers a different question — “on average, across every way the rest of the world could be, does this one value still point toward O_i?” — which is exactly what the margin-ranking term needs to check a value's direction independent of which particular instance it was sampled alongside.

A hand-sized version of Eq. 6

Suppose there are only two other factors besides the one you're checking, F_B and F_C, each binary — so ƒ* (“the remaining factors”) has 2 × 2 = 4 combinations. Holding F_A = a₂ fixed, Equation 6 says average Eq. 4's output over all four:

This is not a coincidence — it's the exact same arithmetic as Chapter 5's “observe F_A only” stage, because both computations ask the identical question: averaged over everything else being unknown, how much does a₂ alone move the needle? Equation 6 reuses the machinery Chapter 5 already built; it isn't a new algorithm, just the same marginalization aimed at one factor value in isolation, used here as a training-time consistency check rather than a deployment-time answer.

What the hyperparameters are doing

Customizing to a human preference

One more real capability worth naming here: because Eq. 4 takes each P(O_i|f_j) as a plain input, nothing stops a human expert from overriding a specific factor's learned value directly — substitute the human-preferred number in place of the trained one, and Eq. 4 combines it with the rest exactly as before. The paper offers this as one more reason BIRD doesn't use the classical P(f|O_i) likelihood formulation from Chapter 4: it would be far harder for a human reviewer to state a preference over an inverted, hypothetical-world likelihood than to simply say “no, a₂ should support O₁ at 30%, not 25%” and have that number flow straight into the same combination formula everything else uses.

Why 128 samples, and why words instead of numbers

Two design choices in this training step are worth pausing on, because both trace back to earlier chapters in this session. First: 128 is a sample, not the whole space — our toy 3-factor example only has 8 worlds total, so 128 would be redundant there, but real abducted factor spaces (Chapter 2 showed these can run into the hundreds or thousands of worlds for even a handful of multi-valued factors) are far larger than 128, and sampling is the only tractable way to get training signal without enumerating everything. Second, and more pointed: the supervision comes from a verbalized, categorical judgment (“likely,” “very unlikely”) mapped to a number, not from asking the LLM to output a raw percentage directly. This is a direct callback to Chapter 0 — that chapter's entire argument was that asking an LLM to verbalize a raw percentage produces an unreliable number. Constraining the judgment to seven coarse buckets removes the LLM's ability to manufacture false precision (there is a real difference between an LLM confidently saying “73%” with no basis for the third digit, and an LLM choosing between “likely” and “very likely”), while still being informative enough, combined across 128 samples and fit through a proper loss function, to meaningfully refine the coarse 75/50/25 starting point. The categorical scale is coarser per-judgment, but the aggregate training signal it produces is more trustworthy than 128 raw, uncalibrated percentages would have been.

How much training is this, actually?

Put a concrete number on the training run's size, since “SGD, lr=0.01, 20 epochs, batch=4” reads as boilerplate until you compute what it means: 128 instances at batch size 4 is 32 batches per epoch, times 20 epochs, is 640 total gradient steps — to fit a handful of trainable parameters (one number per factor value; three factors with two values each is only 6 numbers). This is not a large-scale training run in any sense this course has used the term elsewhere — it's closer to fitting a tiny logistic regression than training a network. That scale mismatch (640 steps, 6 parameters) is itself informative: BIRD's per-scenario training cost is small enough to redo on demand every time abduction produces a new factor set, which is part of why the framework's authors treat the trained-vs-fixed-initial-probability gap (2.0 F1 points, Chapter 6's ablation table) as a genuine but modest refinement rather than the main event — there simply isn't much capacity in 6 free numbers to overfit or to dramatically outperform a reasonable fixed starting point.

What α = 10 would look like if it were 0

It's worth imagining the counterfactual to appreciate what the margin-ranking term actually buys, since “10” is otherwise just a number. Set α = 0 and only MSE remains. Now suppose one of the 128 sampled complete-information instances happens, by chance, to be a case where the LLM's complete-information judgment (P_LLM) disagrees with what the coarse Eq. 5 classification originally said about one of its factors — not impossible, since Eq. 5's classification looks at a factor value in isolation while P_LLM judges a whole fully-specified world. Pure MSE has no reason to resist letting that one factor's learned value drift across 50% to fit this instance better, even if doing so contradicts every other instance's implied direction for that same factor. With α = 10, that drift is penalized ten times more heavily than a comparable numeric fitting error, which is precisely why the paper can afford a small, noisy, 128-instance training set without the smallness turning into instability: the margin term acts as a strong prior toward “keep the sign Eq. 5 already gave you,” and only the aggregate weight of many instances, not one outlier, is allowed to flip it.

How many times does the model revisit any one instance?

One more number worth deriving from the same hyperparameters, because it answers a natural worry: with only 128 training instances, is this training loop actually learning something, or just memorizing a handful of examples? Each epoch is a full pass over all 128 instances (32 batches of 4), and there are 20 epochs, so every one of the 128 sampled instances participates in exactly 20 gradient updates over the course of training — not the full 640 steps each, but 640 ÷ 128 = 20 visits per instance on average, since batching and epoching just partition the same 128 instances repeatedly. Twenty visits to each data point, fitting six free numbers total (one probability per factor value; three factors with two values each), is closer to repeatedly polishing a small, fixed estimate than to learning a pattern from data the model hasn't already seen — consistent with this chapter's framing of training as refinement, not as the source of BIRD's core advantage.

Chapter 7: Does It Actually Work?

The last six chapters built the machine. This one holds it to the numbers — three experiments, on Llama-2-70B-Instruct, across three datasets: Common2sense (commonsense reasoning, 216 scenarios, 3,822 instances, roughly 9 additional conditions on average supporting each outcome), Today (temporal reasoning, 1,000 instances), and Plasma (procedural planning, 279 scenarios, 1,395 instances, roughly 5 additional conditions supporting the less common outcome).

Experiment 1: does the probability actually track the evidence?

This is the F1-against-human-judgment test from Chapter 0, broken down by category. “Different1” and “Different2” are cases where the two conditions genuinely favor opposite sides; “Same” is the harder case where both conditions equally support the outcome, and the model has to recognize a tie rather than manufacture a winner:

† Explicit Comparison is an easier setting: the model sees both conditions side by side and just picks the stronger one directly, rather than scoring each independently — not a fully fair comparison, and still BIRD wins.

Look at the “Same” column: 0.300 vs 0.450 — a fifty-percent relative jump, and by far the widest gap of the three categories (Different1's gap is only 0.026; Different2's is 0.091). Recognizing that two conditions tie is exactly where a model that's pattern-matching toward “something must be stronger” fails hardest, and exactly where routing through explicit, separately-scored factors pays off most: if neither condition entails any additional factor beyond what the other already did, BIRD's marginalization correctly produces near-identical numbers for both, instead of manufacturing a winner where the human labelers themselves called it a tie.

Verify the abstract's headline claim by hand: BIRD's F1 (0.592) minus GPT-4 chain-of-thought's F1 (0.289) is 0.303, i.e. 30.3 percentage points — matching the paper's own results-section sentence, stated plainly: “our framework exhibits superior alignment with human preference, with both models achieving an F1 of 59%, more than 30% higher than GPT-4.” The paper makes a second, separate claim against the easier EC setting — and is explicit that EC is unfair to BIRD, since it lets the model see both conditions side by side and just pick the stronger one, a structural advantage no other row in this table gets: “even under this setting that is unfair to our framework, BIRD still outperforms GPT-4 by 5%.” Checked against the real numbers: 0.592 (BIRD, Llama-2) minus 0.540 (GPT-4 EC) is 0.052, a 5.2-point gap, matching that claim almost exactly. (Against Llama-2's own EC score, 0.503, the gap is 0.089 — about 9 points; the paper doesn't single this comparison out by name, but it's the identical arithmetic, checkable from the same table.) These aren't approximate — they're the paper's own arithmetic, run by hand against its own table.

Experiment 2: comparable hard-label decision accuracy, with an honest asterisk

A second question: when forced to pick a single outcome (not just rank two conditions), how does BIRD compare to chain-of-thought with self-consistency — and what happens on the instances BIRD can't resolve at all?

BIRD wins on the hardest dataset (Today) and loses on the two easier ones — the paper is direct about why: probability estimation and hard-label classification are different objectives, and CoT sometimes reaches the right label by leaning on incidental cues in the raw context that a structured, factor-based decomposition deliberately doesn't have access to (Chapter 9 shows the exact example). The real cost, though, is the “unknown” column: BIRD declines to answer at all when the condition doesn't entail any factor, 54% of the time on Today.

Put a number on what that costs in instances, not just percent: Today has 1,000 instances — 54% unknown means 540 instances need a CoT fallback, leaving 460 resolved directly by BIRD. Common2sense has 3,822 instances — 34.6% unknown is roughly 1,322 instances needing fallback, 2,500 resolved directly. Plasma's 1,395 instances at 31.4% unknown is about 438 needing fallback, 957 resolved.

Experiment 3: does abductive factor generation (Chapter 2) actually beat the direct method?

The ablation promised back in Chapter 2, verified by hand:

The accuracy figure matches the paper's own summary exactly — Section 4.2 states plainly that “our proposed method, on average, surpasses the direct method by 4.4%/3.9% in accuracy for Llama-2/Llama-3.1,” and this session's Llama-2 hand-computation reproduces that 4.4% to the decimal. (The same table's Llama-3.1 row, not shown here since this session stays on Llama-2 throughout Chapter 7, averages to the paper's other quoted figure, 3.9%.) The unknown-rate reduction, 16.6 points on average above, is this session's own computation from the same real table — the paper reports each dataset's unknown-rate drop individually but doesn't state a single headline average for it the way it does for accuracy, so treat 16.6 as an honest summary of real numbers rather than a quoted claim. Either way, the two-stage generate-then-summarize abduction process from Chapter 2 isn't a stylistic preference — it measurably produces factors that cover more of the conditions people actually write.

Two more real numbers, briefly

BIRD's probabilities also transfer as a training signal: fine-tuning a T5-large model with BIRD-estimated soft probabilities instead of hard labels raised average cross-domain accuracy from 67.9% (hard-label supervision) to 68.9% — a +1.0 point gain. A second base model tested the identical setup: PatternTime rose from 69.7% to 71.2%, a +1.5 point gain. The two gains aren't identical — averaged together they land near +1.3 points, which is the useful one-line summary, but the honest version is two different models each showing a real, positive gain of a different size, not one number reproduced twice. And when BIRD is used to generate a yes-or-no follow-up question targeting an unobserved factor, crowdworkers preferred BIRD's question over a directly-generated one 52.8% of the time, versus 32.8% for the direct question (14.4% tied) — on 250 Common2sense examples, that's 132 BIRD wins, 82 direct wins, 36 ties, and 132+82+36 checks out to exactly 250.

Trusting the human labels behind Table 1

A results table is only as trustworthy as the human judgments it's measured against, so it's worth seeing how those 350 examples were collected. The paper recruited 386 distinct MTurk annotators, restricted to workers with a 98%+ HIT acceptance rate located in the US, capped at 5 examples per annotator, and paid for at least one minute of consideration per example ($15/hour effective rate). Each example first got 5 independent labels; if fewer than 4 of 5 agreed (under 80%), 2 more annotators were added and the example was accepted only if 5 of the resulting 7 agreed (over 70%). The final 350-example set breaks down as 44.0% labeled “condition 1 wins,” 43.7% “condition 2 wins,” and 12.3% “tied” — close to balanced, which is exactly what you want from an evaluation set: a model that always guesses the majority class shouldn't be able to coast to a high score.

Reading the per-dataset unknown rates against Chapter 3's entailment settings

The unknown-rate ordering — Today worst at 54.0%, then Common2sense at 34.6%, then Plasma best at 31.4% — tracks exactly the complexity ordering Chapter 3 used to justify its per-dataset entailment settings (x=1, mostly-direct prompting for Today; x=2, mostly-hierarchy for the other two). This isn't two independent facts landing in the same order by chance: a dataset whose conditions are dense and implicit enough to need the more conservative, per-value direct entailment prompt is, for the identical reason, a dataset where more conditions will fail to entail any factor value cleanly. The unknown rate is a downstream symptom of the same underlying difficulty the entailment-prompt choice was already responding to.

Two footnotes on Table 3's columns

Table 3's five evaluation columns hide two details worth surfacing. “Today (exp)” is a variant of the Today benchmark evaluated with gold explanations supplied at test time — a strictly easier setting than the plain “Today” column right next to it, which explains why every row scores higher in the (exp) column than the plain one. And PatternTime is simply a second base model the paper reruns the identical hard-label-vs-BIRD-prob comparison against, independent of T5-large — its exact architecture isn't the point; what matters is that a real, positive gain (+1.0 for T5-large, +1.5 for PatternTime) shows up under a second, differently-built model, which is the detail that turns “soft probabilities helped once” into “soft probabilities help in general,” the standard you'd want before trusting a training-signal claim on your own model.

The distillation experiment, in more detail

The soft-label distillation result deserves one more layer: the experiment fine-tunes a T5-large model, adding Common2sense instances (only the ones where BIRD's estimated probability supports the correct outcome) on top of an existing training set, then evaluates cross-domain on two temporal-reasoning benchmarks (Matres, Tracie) and two others (Today, Plasma) the model was never fine-tuned on directly. Training with BIRD's soft probabilities (a continuous target, e.g. 0.72) instead of hard labels (a discrete 0 or 1) gives the downstream model a gradient signal proportional to how confident BIRD actually was — an easy, confidently-73%-supported instance and a borderline 51%-supported instance both count as “correct” under hard labels, but only the soft-label loss tells the smaller model to trust the first one more than the second. The result held on a second base model (PatternTime) too, with a comparable +1.5-point gain (against T5-large's +1.0) — consistency in direction, even without identical magnitude, across two different base architectures is meaningfully stronger evidence than a single result.

A refresher on what F1 rewards, and why EC is an easier question

F1 is the harmonic mean of precision and recall:

where precision is “of the times the method predicted a given label, how often was it right,” and recall is “of the times that label was actually correct, how often did the method catch it.” The harmonic mean (rather than a simple average) specifically punishes a method that's strong on one of the two and weak on the other — a method with 90% precision but 10% recall on a label scores an F1 near 18%, not 50%, because the harmonic mean is dragged down toward its smaller input. That matters here because the “Same” label is the minority-feeling case (12.3% of the human-labeled set, versus 44.0% and 43.7% for the two “different” labels) — a method that never predicts “tied” would still post a respectable-looking overall accuracy while scoring an F1 near zero specifically on that category, which is exactly the kind of failure the per-category breakdown in the table above is designed to expose.

The EC (Explicit Comparison) baseline's advantage is easiest to see as a concrete input/output contrast. Under the standard setting BIRD and the other baselines are scored in, the model sees one condition at a time and has to output an independent probability for it — call it twice, once per condition, and compare the two returned numbers afterward. Under EC, the model sees both conditions in the same prompt and just picks the winner directly — a single classification call with the comparison built in, rather than two separate probability estimates that then need to agree with each other after the fact. Seeing both options side by side and picking the better one is mechanically easier than scoring each in isolation and hoping the two scores end up correctly ordered; it's the difference between a multiple-choice question and two separate open-ended ones that happen to need consistent answers. That EC still loses to BIRD by roughly 5 points despite this structural advantage is why the paper itself calls the EC comparison unfair to BIRD, not the reverse.

How much of the dataset-by-dataset variation is signal, and how much is sample size?

It's worth being disciplined about which differences in Chapter 7's tables deserve real weight. The F1 comparison (Table 1) is measured on 350 human-labeled examples — a modest evaluation set, appropriate for the F1 gaps under discussion (0.052, 0.089, 0.303) because they're all far larger than sampling noise on 350 examples would plausibly produce, but not large enough to trust differences of a point or two between close variants. The hard-label accuracy comparison (Table 2) runs on the full datasets — 1,000 (Today), 3,822 (Common2sense), 1,395 (Plasma) instances — which is why the paper is willing to draw a real conclusion from Today's accuracy gap (73.9% vs. 71.5%, a 2.4-point BIRD win) while treating Common2sense's similarly-sized gap in the other direction (89.0% vs. 93.8%, a 4.8-point CoT win) as a genuine, not noise-level, result too — both gaps are measured against thousands of instances, not the smaller 350-example F1 set. The general rule worth carrying forward: a table's trustworthiness is bounded by its smallest evaluated subset, and the honest response to a close call on a small sample is to say so, not to round it into a confident claim.

What a 5-point F1 gap means for the downstream threshold rule

Chapter 0 opened with a concrete downstream rule: default to the long cord if P < 0.4, ask a follow-up question if 0.4 ≤ P ≤ 0.6, otherwise default to the short cord. It's worth translating BIRD's F1 advantage over the EC baseline — 0.592 vs. 0.540, a 5.2-point gap — back into that concrete frame, because an F1 difference on an abstract benchmark can otherwise feel disconnected from what actually changes for a user running that rule.

F1 measures how well a method's judgments rank against human judgments across the whole evaluation set, not how often any single number crosses a specific threshold — so a 5.2-point F1 gap doesn't mean “5.2% of decisions flip.” What it does mean, concretely: on the borderline cases near a threshold like 0.4 or 0.6 — the ones where the downstream rule's behavior actually depends on getting the number right, not just roughly in the right direction — the method with the higher F1 is measurably more likely to land on the side of the threshold a human labeler would agree with. And EC's own structural advantage (seeing both conditions at once, this chapter's earlier footnote on why EC is an easier question) makes this gap conservative, not generous, toward BIRD: EC gets to use side-by-side context BIRD is denied, and BIRD still comes out 5.2 points ahead.

This is the same argument Chapter 0 made in the abstract, now closed with a real number instead of a placeholder: a downstream system gating on a probability doesn't need that probability to be perfect, but it does need small differences near a threshold to be trustworthy rather than noise. F1's measured gap is exactly the kind of evidence that distinguishes “trustworthy near the threshold” from “might as well be random near the threshold” — the second description being precisely where Chapter 0's four sub-random direct-prompting baselines lived.

Chapter 8: Build a BIRD Aggregator (showcase)

Time to stop tracing one fixed example and build your own. This simulation is the entire pipeline from Chapters 3 and 4 running live: you define up to four factors, decide for each one whether the condition observes it and how strongly it leans, and watch P(O₁|C) recompute in real time — via the exact Eq. 4 combination and Eq. 2/8/9 marginalization you hand-computed in Chapter 5.

Three things to try, and what each one proves

1. Leave every factor unobserved, then observe just one at the extreme (90% or 10%). Watch P(O₁|C) move from exactly 50% to somewhere well past it in one step — but not all the way to 90%. A single strong factor pulls hard, but Eq. 4 still averages it against the neutral 50%-weighted mass of the other unobserved factors sitting in the sum; it doesn't dominate outright until more factors corroborate it, echoing the “agreement compounds” arithmetic from Chapter 4.

2. Set two factors to opposite extremes (one at 90%, one at 10%) and leave the rest unobserved. This is Chapter 4's conflicting-evidence example, scaled to a live UI: watch the combined answer snap back toward 50%, then nudge just one of the two slightly (say 90% → 85%) and watch how little that changes it — symmetric conflict is a stable equilibrium, not a coin flip that any small perturbation breaks.

3. Toggle a factor from unobserved to observed-neutral (50%). This should do nothing to the final number. If it visibly moves the gauge in your build, something violates Eq. 9 — a neutral, observed factor and a symmetric, unobserved factor should marginalize to the identical contribution, and BIRD's own math guarantees they do.

4. Observe all four factors at a mild, identical lean (say, all four at 60%). No single factor is individually convincing — 60% is barely more than a coin flip — but watch the combined answer climb well past any one of them. This is worked example 3 from Chapter 4, generalized to four factors instead of three: independent, weakly-agreeing evidence keeps compounding as you add more sources, the same mechanism that makes four so-so witnesses more convincing together than any one alone.

5. Reproduce Chapter 5's Stage 1 exactly, then check the widget against your own hand arithmetic. Set F_A observed at 25% (a₂, leans O₂), and leave the other two unobserved at 50%. The gauge should read 32.1% — the same number you computed by hand in Chapter 5's Stage 1. If your build shows something else, the bug is almost always a marginalization order-of-operations mistake: Eq. 4 combines the per-factor probabilities first, for each fully-specified world, and only then do you average those combined values across the unobserved worlds — averaging the raw factor probabilities before combining them is a different (and wrong) computation.

A troubleshooting table, for anyone implementing this from scratch

A worksheet: design your own scenario before you touch the sliders

The widget is most useful when you arrive with a scenario already worked out on paper, then use the sliders to check your hand computation — not the other way around. Fill in the blanks for a decision you actually care about:

Work out P(O₁|C) by hand using Eq. 4 and the marginalization pattern from Chapter 5, then enter the same setup into the widget above and confirm the gauge agrees. If it doesn't, you've either made an arithmetic slip or found a genuine bug — the troubleshooting table just above tells you which is more likely given the specific symptom. This is the single most useful exercise in the entire session for retention: everything up to this point has been watching this session's authors' arithmetic; this is the first time it's yours.

Two worked answers, so you have something to check a first attempt against before consulting the widget. Take “deciding whether to accept a job offer,” O₁ = accept: factors might be compensation (well above market → 85%), team fit from the interview loop (mixed signals → unobserved, 50%), and commute (much longer than current → 20%). With compensation and commute observed, team fit unobserved: numerator = 0.85 × 0.5 × 0.20 = 0.085; complement = 0.15 × 0.5 × 0.80 = 0.06; P(accept) = 0.085 ÷ 0.145 ≈ 58.6% — a real number, closer to a coin flip than either factor alone would suggest, which is exactly what “great pay, bad commute” should feel like. Or “whether to refactor a piece of code before shipping,” O₁ = refactor first: factors could be how much the code will be touched again (frequently → 80%), how close the deadline is (very close → 15%), and test coverage (good coverage → unobserved here, 50%). Work out your own number before checking it against the widget.

What the widget computes, underneath the sliders

python
def aggregate(factor_rows):
    # factor_rows: list of {observed: bool, lean: float}  — lean is P(O1 | this value) when observed
    p_per_factor = [row['lean'] if row['observed'] else 0.5 for row in factor_rows]
    return eq4(p_per_factor)     # same eq4() from Chapter 4 — nothing new

That is genuinely the whole widget. There is no separate “unobserved-factor logic” branch inside the aggregation step itself — an unobserved factor contributes exactly 0.5 to Eq. 4's product, the same value a fully-observed, perfectly-neutral factor would contribute. Chapters 3 and 4 built two ideas that look different on paper (marginalization over a whole information space; log-odds combination of individual factors) and this widget is the demonstration that, once you fix “unobserved → 0.5,” they collapse into one function you can fit in four lines.

Prove it generalizes: a scenario that has nothing to do with cords

Every worked example so far used the charging-cord scenario, and it would be reasonable to worry the arithmetic secretly depends on it. It doesn't — here is a completely different scenario, hand-specified the same way, that you can punch directly into the widget above to check.

Scenario: “deciding whether to bring an umbrella.” O₁ = bring it, O₂ = leave it home. Three abducted factors: F_sky (gray vs. blue), F_forecast (rain predicted vs. not), F_history (you've been caught without one before vs. you always check the forecast and it's been reliable). Suppose today's condition entails gray sky and a rain forecast, and says nothing about your history with forecasts:

Two strongly agreeing, independently-observed factors (0.80 and 0.90) compound to 97.3% confidence — even with a third factor sitting completely unobserved at its neutral 0.5. Set the widget's three rows to these exact numbers (two observed at 0.80/0.90, one unobserved) and you should see the gauge land at 97.3%, confirming that nothing in Eq. 4 or Eq. 9 was ever specific to cords, factors named F_A/F_B/F_C, or the particular scenario this session used to build intuition. The math is the scenario-agnostic part; only the factor names and their supporting sentences come from abduction (Chapter 2), which is genuinely scenario-specific.

How many agreeing factors before you're basically certain?

The umbrella example above stacked two strong, agreeing factors (0.80 and 0.90) to reach 97.3%. It's worth pushing that pattern one step further by hand, because “compounding” can otherwise sound like a vague intensifier rather than an arithmetic fact with a specific shape. Take three factors, all only moderately confident on their own — 0.80, 0.80, and 0.80 — all independently observed and all agreeing:

Three factors, none individually more confident than “fairly sure,” combine to 98.5% — past the two-factor umbrella example's 97.3%, and closing in on certainty fast. Push it to five agreeing factors at 0.80 each and the complement term shrinks to 0.2⁵ = 0.00032 against a numerator of 0.8⁵ ≈ 0.328, landing at P ≈ 0.999. Set the widget's four rows to four agreeing factors near 0.80–0.85 each and watch the gauge push hard toward one extreme — not because any single row is dramatic on its own, but because independent, agreeing evidence compounds multiplicatively, not additively, and multiplicative compounding saturates fast.

Compare this directly to Chapter 6's real ablation numbers: even the crudest variant (1/2 assumption, unanimous-agreement-only, 0.480 average F1) already clears GPT-4 chain-of-thought's 0.289 by a wide margin, precisely because factors that individually classify only coarsely still compound once several of them agree — the same multiplicative mechanism this widget lets you feel under your own fingers, not just read as a formula on a page.

The connection this course has already prepared you to see

Eq. 4 looks like decision-theory notation, but it is secretly something you already know from the rest of this course. Divide the numerator and denominator of Eq. 4 by ∏_j(1−P(O_i|f_j)):

That's exactly the algebraic form of a sigmoid: σ(x) = 1 ÷ (1 + e^−x), which equals O ÷ (O+1) when x = ln(O). And ln(O) = ∑_j ln(odds_j) — the sum of each factor's logit (log-odds). Put together:

That is precisely the shape of a logistic regression with every weight fixed at 1 and no bias term — sum the logits, squash with a sigmoid. If you've built a classifier head on top of a language model, you've already implemented this exact operation; BIRD's “novel Bayesian combination formula” is, once you strip the decision-theory framing, the single most standard operation in this entire course, applied to per-factor LLM judgments instead of learned neural weights.

This table is exactly what the widget's sliders are secretly manipulating: moving a factor's lean from 50% to 90% is moving its logit contribution from 0 to +2.197; two factors both at 75% sum to a combined logit of 2.197, and σ(2.197) = 0.9 — matching worked example 1 from Chapter 4 exactly. Unobserved factors contribute a logit of exactly 0 (neither pushing the sum up nor down), which is the algebraic reason “unobserved” and “observed-at-50%” are indistinguishable to the final answer — they contribute the identical zero to the sum.

Chapter 9: Failure Modes & Connections

BIRD is not a solved problem wearing a bow. It has two real, quantified weaknesses, and knowing exactly where they bite is what makes the difference between using this framework well and deploying it somewhere it quietly fails.

What trustworthiness actually costs, in LLM calls

Before the failure modes, one honest accounting: nothing in this session was free. Tally the LLM calls per scenario. Abduction (Chapter 2): 2 generation calls (10 sentences per outcome) + 1 summarization call = 3. Pruning and initialization (Chapter 2, this chapter's earlier note): 3 samples × N factors for the majority-vote classification. For our running 3-factor example, that's 9 calls, one-time, reusable forever. Per condition, entailment (Chapter 3) costs up to 3 samples × (1 + N) calls under the hierarchy prompt, or 3 × (values per factor × N) under the direct prompt — for our example, roughly 12 calls per new condition. Compare that to the naive baseline from Chapter 0: one call. BIRD is somewhere between 3× and 20× more expensive per scenario in raw API calls, concentrated almost entirely in the one-time abduction and pruning cost, with entailment adding a smaller, recurring cost per condition. That is the real trade this session has been making since Chapter 0: engineering and inference cost, spent once per scenario, in exchange for a probability that a downstream system can actually act on — the same “compute per request vs. engineering per capability” tradeoff that shows up whenever a single fluent guess is replaced with a structured, checkable pipeline.

Failure 1: the honest “I don't know”

You already have the numbers from Chapter 7: 54.0% unknown on Today, 34.6% on Common2sense, 31.4% on Plasma. Every one of these is a case where entailment (Chapter 3) couldn't map the condition onto any factor — not a wrong answer, an absent one. The paper's own diagnosis: this is “due to the imperfect or incomplete abductive factors generated by the LLM, and not a fundamental flaw of BIRD” — a solvable engineering gap (better factor coverage), not a limit of the math. But it is a real, present-tense cost: on Today, more than half of all instances currently need a chain-of-thought fallback, which reintroduces exactly the uncalibrated-probability problem from Chapter 0 for those cases.

Failure 2: structure can miss a side-signal that raw context catches

The paper documents a specific instance where chain-of-thought's lack of structure is an advantage. Scenario: choosing between habanero peppers and jalapeño peppers for spiciness. Condition: the recipe uses “a significant amount of habanero peppers relative to other ingredients.” The correct answer — habanero makes it hotter — follows from general knowledge about the Scoville scale, which chain-of-thought reaches directly. But the condition's actual content (a quantity claim) doesn't map cleanly onto a pre-abducted factor about which pepper is used at all; a rigid factor structure can miss a side-signal like this that raw, unstructured context handles by brute-force association.

Failure 3 (untested): what if the factors aren't really independent?

This one isn't in the paper's results tables — it's a structural risk worth naming because Chapter 2 leaned on it and Chapter 4's whole derivation assumes it away. Equation 4 treats every factor's contribution as independent evidence, multiplying odds together. If two abducted factors are secretly correlated — say “paces around the room” and “prefers an open floor plan” both got kept as separate factors, but in practice almost every scenario that entails one also entails the other — then observing both doesn't give you two independent pieces of evidence, it gives you the same piece of evidence counted twice. Eq. 4's multiplicative combination would then overstate confidence, the same way a naive-Bayes spam filter over-trusts two correlated features more than a single one warrants. Chapter 2's pruning step guards against irrelevant factors (the neutral ones); it does not explicitly test for redundant factors that pass the relevance test individually but overlap with each other. This is a gap the abduction step's “least relevant to each other” instruction is aimed at closing by prompting alone, not one the deductive math itself verifies.

Failure 3, made concrete: a hand-computed overconfidence example

Failure 3 is easiest to trust once you've watched it happen in real arithmetic, not just read the argument in prose. Suppose abduction genuinely produces two “separate” factors that are, in reality, the same underlying evidence restated — say F_pace (“you'll be pacing around the room”) and F_open (“you prefer an open floor plan”), where in this population almost everyone who paces also happens to prefer open floor plans, and vice versa. Both get classified, honestly and correctly in isolation, as each individually supporting O₂ (longer cord) at the same strength as F_A did earlier in this session — 75% toward O₂, i.e. P(O₁|value) = 0.25 each.

Treat them as independent, the way Eq. 4 always does, and observe both:

Ninety percent confidence in O₂, from what the framework treats as two independently corroborating pieces of evidence. But if F_pace and F_open are really the same fact wearing two names — if knowing one already tells you the other with near certainty in this population — then the condition only actually supplied one genuine piece of evidence, and the honest answer should stay near where a single such factor alone would land: 25% (i.e. 75% confidence in O₂), not 10% (90% confidence). That's a 15-percentage-point overconfidence artifact, purely from double-counting one piece of evidence as two — and it gets worse, not better, the more correlated near-duplicate factors survive pruning, because each additional near-duplicate multiplies the same odds ratio back in.

What would actually shrink the unknown rate?

The paper names imperfect factor generation as the root cause of Failure 1 and calls better coverage future work, without specifying a fix. A few concrete directions follow naturally from what Chapter 2 already showed matters: sampling more than ten situations per outcome in stage 1 (more raw material for the summarization step to draw distinct factors from); running entailment with a wider net — checking against near-synonyms of each factor value, not just exact implication; or allowing multi-hop entailment, where a condition implies an intermediate fact that in turn implies a factor value, rather than requiring the factor value to be implied in one step. None of these change Equations 1–9 — they're all upstream, in Chapter 2 and Chapter 3's territory, which is exactly why the paper calls the current unknown rate an engineering gap rather than a mathematical limit.

A field-guide table: when to reach for which

Where this connects on the site

BIRD's marginalization (Eq. 1–2) is the law of total probability, applied to text instead of numbers. If the phrase “sum over the complete information space, weighted by how consistent each state is with what you observed” felt familiar, it should — it's the same operation a robot performs every time it updates a belief about where it is:

That's not a loose metaphor. A Bayes filter maintains a belief distribution over where a robot is, updates it with predict (apply a motion model, spreading uncertainty — the direct analogue of Chapter 3's “unobserved factor → uniform” treatment) and correct (fold in a sensor reading, concentrating probability on states consistent with it — the direct analogue of Chapter 3's “observed factor → probability 1 on the entailed value”). BIRD's C = (S, U) plays the role of the accumulated sensor readings; F plays the role of the state space; Eq. 1's marginalization sum is, symbol for symbol, the same operation. The domains differ — centimeters and radians versus factor values and outcomes — the update rule doesn't.

• Bayes Filter — the same marginalize-then-weight recursion, over physical states instead of text factors; BIRD's P(f|C) is the direct text-domain analogue of a Bayes filter's belief update after a sensor reading.
• Bayes Nets — BIRD's factor space F, built from variables assumed conditionally independent given the context, is a flat (one-level) Bayesian network; a full Bayes net generalizes this to arbitrary dependency structure between factors, which is exactly the assumption Chapter 2 and Chapter 4 both lean on and neither ever tests.
• Bayesian Estimation — the prior-to-posterior update machinery underlying Eq. 3's general Bordley form, with the classic likelihood P(f|O_i) that BIRD deliberately routes around in Chapter 4.

Cheat sheet: every equation from this session, in one table

If you wanted to apply this to your own decision problem

The shortest path from this session to a working prototype on a scenario of your own choosing runs through four questions, in order. One: can your decision be framed as two complementary outcomes, or decomposed hierarchically into a sequence of such choices (Chapter 1)? If your options aren't naturally binary, this framework doesn't directly extend without that decomposition step. Two: can an LLM generate 8–10 free-form situations per outcome that a domain expert would recognize as plausible and distinct (Chapter 2)? If the domain is unfamiliar enough that the LLM's abduction step produces generic or repetitive factors, expect a higher unknown rate downstream, for the same reason Today's unknown rate (54%) exceeds Plasma's (31.4%) — harder-to-abduct domains propagate difficulty forward. Three: do you have, or can you cheaply construct, 350-ish human-labeled examples to validate against, in the style of Chapter 7's evaluation (not to train on — Chapter 6's 128 training instances are self-supervised from the LLM, but you still need an external check that the whole pipeline is producing sane numbers)? Four: is your deployment tolerant of an explicit “unknown,” with a fallback path, or does it require an answer every time? If the latter, budget for a CoT (or similar) fallback exactly as Chapter 7's Today/Common2sense/Plasma evaluations did, and expect the fallback rate to be substantial until your factor generation is well-tuned to the domain.

What comes next

This session showed a Bayesian model bolted onto an LLM, with the LLM supplying structure and judgments and a fixed formula doing the arithmetic. The next reasoning session in this course turns the question around: what happens when you let the model reason step by step and spend variable compute doing it — test-time compute, process reward models, and the tradeoffs between thinking longer and thinking structured. Keep BIRD's core lesson with you going in: a probability is only as trustworthy as the process that produced it, and fluency is not that process.

References

Scoring BIRD against its own stated goals

The paper opens by promising three properties — interpretable, controllable, trustworthy — and it's worth closing by checking each one against what this session actually demonstrated, rather than taking the labels on faith.

All three hold up under the numbers this session walked through by hand — not as marketing language, but as properties with specific chapters and specific arithmetic behind them. That traceability, more than any single F1 number, is the actual deliverable of routing probability estimation through abduction and deduction instead of a single fluent guess.