Spatial Memory Agent: a model that learns without learning

Chapter 1: What a Memory Is

Start with the object, because everything else in the paper is a function of it. SMA maintains a memory bank — written ℋ in the paper — which is a list of cards. Each card is one rollout, compressed. Formally:

Six fields. It would be easy to skim that line, and it is the whole design, so let us take the fields one at a time and ask what breaks if you remove it. That question — what breaks — is not rhetorical here, because the paper ablates three of the six and reports exactly how much accuracy each one is worth.

t — the source task text (the retrieval key)

The first field is the natural-language task from the problem that produced this card. Not the image. Not the answer. The question string.

Its job is to be a key. At retrieval time SMA embeds the current question, embeds every stored question, and takes cosine similarities. That means the field has a dual life: it is human-readable text that gets shown to the model as context ("here is the shape of question this lesson came from"), and it is the thing whose embedding decides whether this card is even a candidate.

Note the deliberate omission. The image is not stored and not embedded. This is a real design decision with a real consequence: two questions about totally different rooms with the same phrasing are near-neighbours, and two questions about the same room with different phrasings are not. The paper leans into this. A memory is supposed to be about the shape of the question, not the scene. The retrieval prompt says so out loud: "Match each memory to the structural shape of the current question … not to similar object nouns alone."

s — the summary

One or two sentences abstracting the task shape and the diagnosed success or failure mode. The reflection prompt in the appendix specifies it exactly: "summary: 1-2 sentences on task shape + diagnosed habit/gap (reasoning-level, not the answer)."

Remove it and RoboSpatial accuracy falls 3.2 points (68.5 → 65.3) on Qwen3.6-27B; on Omni3D it falls 1.6 points (47.6 → 46.0). So the summary earns its place, but it is the cheapest of the three text fields to lose. That is a coherent picture: the summary is context that helps the model decide whether the lesson applies, but the lesson is what actually changes the reasoning.

l — the transferable lesson

This is the payload. The reflection prompt constrains it to a single sentence in a fixed grammar:

Four slots, and each slot does a different job. Shape is the trigger condition — it tells a future reader whether this lesson is even relevant. Habit is the positive procedure. Trap is the negative — the specific error mode diagnosed in this rollout. Check is a verification step, which is what turns a piece of advice into something the model can actually execute and confirm.

Go back to the warehouse robot from Chapter 0. The lesson that episode should have produced is not "the toolbox does not fit to the left of the printer" — that is an answer, useless on a new shelf. It is something closer to: "When asked whether object A can be placed in a free-space relation to object B, estimate A's extent against the visible clearance rather than confirming that empty pixels exist, avoid answering yes from apparent gap alone, and validate by naming the obstacle or the dimension that would block the fit." That sentence transfers to any shelf, any warehouse, any pair of objects.

Remove the lesson field and RoboSpatial drops 3.5 points; Omni3D drops 5.2. On Omni3D — open-answer 3D reasoning, the hardest benchmark in the set — the lesson is worth more than three times the summary.

n, c, v — the reliability state

The last three fields are numbers, and they are the part that has no analogue in ordinary RAG.

• n_i — a visit count. How many times this card has been retrieved as guidance for some later problem.
• c_i — cumulative reward. The sum of the verifier rewards earned on those later problems.
• v_i — the Transfer Reliability Score (TRS). A number in [0, 1] estimating "if I retrieve this card for a new problem, how likely is the answer to be right?"

Read those definitions again and notice what is not in them. None of the three refers to whether the rollout that created the card was correct. A card written from a catastrophic failure and a card written from a clean success start life with identical n, c, and v. The paper is explicit: "a memory is not assigned a higher or lower initial TRS solely because the rollout that created it was correct or incorrect." Chapter 5 is entirely about why.

The asymmetry: what is stored versus what is shown

Here is the detail that separates SMA from a transcript cache, and it is one sentence in the paper: "Retrieved memories expose only the task, summary, and transferable lesson; it does not expose prior predictions or verified answers."

The card stores six fields. The prompt receives three of them, plus one derived number. Everything else is machinery the model never sees. And the paper goes further than silently omitting the prior output — the retrieval template contains an explicit placeholder line:

That last line is doing something subtle. It could simply have been left out. Instead the template shows the model a field that exists and is deliberately withheld, and names the specific failure it is guarding against. This is a prompt-engineering choice with a testable consequence, and the paper tests it: the "+ model output" ablation puts the raw rollout back in, and accuracy falls 4.4 points on RoboSpatial and 2.0 on Omni3D. More information, worse performance.

The card, laid out

Why the key is the question, not the lesson

Here is a design decision that is easy to miss because the paper states it without comment: retrieval compares ψ(t_i) to ψ(t_j) — the current question against the stored question. Not the current question against the stored lesson.

Those are different systems. Consider the alternative — embed the lesson text and match the incoming question against it. That is closer to how a document-retrieval system works, where the query and the corpus live in different spaces and you rely on the embedder to bridge them.

Question-to-question matching is symmetric retrieval: both sides of the cosine are the same kind of object, produced by the same distribution of text. That has two consequences worth naming.

• The similarity is calibrated. A cosine of 0.618 between two benchmark questions means something stable, which is why a fixed per-benchmark threshold δ works at all. Question-to-lesson cosines would be systematically lower and much more variable, since a lesson is a different genre of sentence, and a single threshold would be far harder to set.
• It retrieves on shape, which is what the lesson is about. The lesson's first slot is "when <shape>" — its applicability condition. The stored question is an instance of that shape. Matching questions to questions is therefore matching applicability conditions directly, using the most concrete available description of one.

The cost of this choice is real and shows up in the limitations. A lesson that would generalise beautifully to a differently-worded question will not be retrieved, because retrieval never sees the lesson. And nothing about the image participates — two questions about geometrically identical scenes phrased differently are far apart. The obvious extension is a multimodal or hybrid key; the paper does not take it.

The design space, field by field

To see why the card is shaped this way, walk the alternatives. Each row is a design SMA did not choose, and what it would cost.

Read down the "what breaks" column and a theme emerges: almost every alternative fails by reintroducing scene specificity or by destroying the granularity at which reliability can be measured. The six fields are close to the minimum set that supports both "this is a procedure, not a memory of an event" and "we can measure whether it works".

Concept and realisation: what actually flows

Let us make the data flow completely concrete, because "a memory bank" is the kind of phrase that hides a lot of plumbing.

Two things are worth staring at. First, the memory bank is text. Not a fine-tuned adapter, not a set of key-value caches, not activations. A few hundred JSON objects. You could open it in a text editor and read it, and the paper's qualitative appendix does exactly that.

Second, F is called with 𝒜_i as an argument. That is not decoration — it means the guidance is part of the input to a frozen function, which is precisely why nothing needs to be trained. The model's behaviour changes because its input changed. That is the entire mechanism.

What three cards look like as tokens

Abstract descriptions of "guidance" hide how small this actually is. Here is the memory context as it is composed at runtime, with the template's placeholders filled in for one card. This is the literal shape of 𝒜_i as text:

Count it. Header and footer are fixed overhead, maybe 150 tokens. Each memory block is roughly 90 to 120 tokens. Three cards is around 400 to 500 tokens prepended to a prompt with a budget of 32,768 new tokens and an image worth hundreds or thousands of vision tokens.

That is the entire intervention. A half-kilotoken of English. Against it, Chapter 8 will report a 14.4-point accuracy gain on RoboSpatial. It is worth holding both numbers in mind simultaneously, because the ratio is what makes the paper interesting.

The model is told what to do with each family's cards

The footer of the retrieval block is not generic. It gives per-family instructions for converting a lesson into an action, and reading it makes concrete what "use the memory" is supposed to mean. RoboSpatial's:

Notice the verb in the first row: turn memories into checks. That is the whole intended semantics of a card in five words. Not "recall what happened", not "match this to a previous case" — convert a sentence into a sequence of operations you perform on the current image.

And notice the escape hatch. "If no memory fits the shape, ignore them" is permission to discard the guidance entirely. Without it, a model handed three cards would feel obliged to use them — retrieved context reads as relevant by default. The instruction converts the guidance from an assertion into an offer.

The four baselines, by which fields they carry

The paper's baseline set is unusually well designed, and it is easiest to understand as an ablation ladder over the card's six fields. Each baseline drops or changes something specific.

MemRL-GT is the baseline the authors added themselves, and they say why: "Since the original MemRL uses reward-only reflection, we implement MemRL-GT, which provides ground-truth answers during reflection for fair comparison." Constructing a stronger version of your closest competitor, so that your own margin shrinks, is the right thing to do and not a common one. It is also why Chapter 8's honest reading is possible at all — MemRL-GT is the baseline that beats SMA twice.

Every field, and the number attached to it

Before moving on, put the whole card on one page with the paper's measured value for each field. This is the summary to come back to.

The last row is the most instructive line in the table. It is a field the design deliberately does not have, and the paper measured what happens if you add it: accuracy falls on both benchmarks. That is unusual and worth crediting — papers rarely ablate features they chose not to build.

Read the "Changed by" column top to bottom and the card's split personality is obvious: three fields are immutable text set once at write time, and three are mutable numbers that move on every retrieval. The bank is a set of frozen sentences with a live score attached to each. Chapter 4 is about keeping the sentences frozen; Chapter 5 is about the score.

Why the paper calls this procedure memory

There is a taxonomy in cognitive science, borrowed wholesale by the agent-memory literature, worth knowing because it is where the paper's title comes from.

The RAG baseline in this paper is exactly the episodic option: it "stores lightweight prior rollout records" and at deployment "provides the prior task and prior model output" with no reflection and no TRS. It is the control that isolates what reflection buys. And it is the baseline that sometimes loses to having no memory at all.

MemP (Fang et al. 2026), the strongest text-side prior work here, is the procedural option without the reliability machinery: it reflects rollouts into summaries and transferable lessons, then retrieves them by semantic similarity only. That makes MemP the single most informative baseline in the paper, because SMA minus TRS is approximately MemP. Every point of the gap between them is attributable to reliability-aware selection.

The bank as a file on disk

To make "the memory bank is text" completely unambiguous, here is one card as it would be serialised. Nothing in this object is a tensor.

{
  "task": "Can the storage bin fit to the right of the desk?",
  "summary": "Placement-compatibility shape; the rollout confirmed empty pixels
               but never compared object extent to available clearance.",
  "transferable_lesson": "When asked whether A fits in a free-space relation to B,
               estimate A's extent against the visible clearance rather than confirming
               that empty pixels exist, avoid answering yes from apparent gap alone,
               and validate by naming the obstacle or the dimension that would block it.",
  "n": 18,          // visits — later retrievals, not writes
  "c": 14.0,        // cumulative reward over those visits
  "v": 0.750        // = (2*0.5 + 14.0) / (2 + 18)
}

Three consequences follow from the bank being this rather than a checkpoint.

It is auditable. You can read every lesson your system has learned, in English, and disagree with any of them. Compare that with a fine-tuned model, where "what did it learn" is answerable only by probing behaviour. If a lesson is wrong, you can see it is wrong before it causes a failure.

It is portable. A few hundred JSON objects moves anywhere. Chapter 9 measures exactly this: a bank written by a 122B model gives a 27B model +9.4 points on RoboSpatial. Try that with a LoRA adapter.

It is editable by hand. Nothing prevents you from deleting a card, rewriting a lesson, or seeding the bank with human-authored entries at v = v₀ and letting visit evidence judge them alongside the machine-written ones. The paper does not explore this, and it is one of the more obvious things to try — the mechanism does not care where a sentence came from.

The states a card passes through

One card, four states, and the transitions between them are the whole system from the card's point of view.

Notice there is no dead state. A card with a terrible track record still sits in the bank, still gets compared against every query, and can still be retrieved if the candidate set is weak enough. That is Limitation D.2 — no deletion, no merging, no expiry — visible in the state machine as a missing arrow.

Chapter 2: Earning Experience

A memory bank is only as good as the experience that filled it. This chapter is about where that experience comes from, and it hinges on a split that is easy to state and easy to get wrong.

Two piles, disjoint

The paper defines 𝒳 as the environment split and 𝒟 as the deployment split, and says they are disjoint. The environment split is where memories are written. The deployment split is where all reported numbers come from, and where nothing is ever written.

Why does this matter enough to be the first sentence of the method section? Because without it the entire result is uninterpretable. If a memory could be written from a deployment problem and then retrieved for that same problem, "memory" would just be a cache of answers, and the accuracy gain would measure nothing but leakage. The disjointness is what makes the phrase "transferable lesson" checkable rather than aspirational.

The actual splits, with real counts

The construction rule is mechanical, and stating it precisely is worth the space because it is what a reproduction would need. For all non-SAT benchmarks: take the retained image-question pool, split it 50/50 per category with seed 42, and where a category has an odd count, alternate the leftover between environment and deployment, environment first.

SAT is handled differently and the paper says why: it follows the official protocol, where the test split is the circular-expanded official test set (150 items expanded to 300 rows) and serves as deployment, while the environment split is sampled from the 4001-item validation pool to match the deployment question-type distribution.

Look at the RoboSpatial row for a moment. 175 environment problems. That is the entire experience budget behind a 14.4-point accuracy gain. Not 175 thousand — 175. This is one of the quieter results in the paper and worth carrying forward: the memory bank that lifts Qwen3.6-27B from 54.1 to 68.5 on RoboSpatial is at most 175 text cards.

What one problem looks like

A spatial problem is a triple:

𝒱_i is one or more visual inputs — a single RGB image for RoboSpatial and Omni3D, one or two ordered stills for SAT, potentially several views of one scene for ViewSpatial. t_i is the natural-language task. y^⋆_i is the verified target.

The seven benchmarks were not chosen for uniformity — they were chosen to be awkwardly different from each other, which is the point. Here is what each actually asks:

RoboSpatial and Omni3D having open answer spaces is why the paper reports accuracy rather than a per-benchmark metric zoo: accuracy, it argues, "directly measures end-task success across both discrete-answer benchmarks and RoboSpatial's open pointing subset."

Why seven benchmarks, and why five in the main table

The paper evaluates seven benchmark slices but reports five in the main table, with SITE-image and ViewSpatial relegated to an appendix. That split is worth understanding rather than assuming it is arbitrary.

The five in the main table were chosen to "cover complementary spatial reasoning settings, including embodied robot perception, physical scene understanding, 3D spatial relations, abstract spatial aptitude, and instruction-grounded embodied spatial reasoning." Five settings, five benchmarks, one each. It is a deliberately spanning set rather than a convenience sample.

The two appendix benchmarks add stress rather than coverage. SITE-image is the image-only split of SITE (8,068 rows filtered to 4,449 image questions), spanning 3D information, counting, movement prediction, navigation, multi-view reasoning, localisation, and relations — a broad grab-bag. ViewSpatial is narrower and harder: multi-perspective localisation, where "each example may contain multiple views of the same scene" and the model must answer about camera-relative direction, person-relative direction, object orientation, and scene simulation.

ViewSpatial is where SMA loses twice in Chapter 8, and its structure is the reason to notice. It is the only benchmark in the set where the same scene appears in several views, which means the reasoning is a frame transformation rather than a check on a single image. Whether procedural memory is a good fit for frame transformations is a genuinely open question, and the appendix table is the only place the paper touches it.

Note also the pool sizes. EmbSpatial (3,640), SITE-image (4,449 retained), and ViewSpatial (5,712) are an order of magnitude larger than RoboSpatial (350) and ERQA (400). So the macro average over five benchmarks weights a 350-item benchmark exactly as heavily as a 3,640-item one. That is a defensible choice — it treats each setting equally rather than each question — but it means the headline averages are not question-weighted, and RoboSpatial's large gains carry a fifth of the weight despite being measured on 175 deployment items.

The acquisition loop — and its one surprising feature

Here is the loop, in the order the pseudocode gives it, for each problem in the environment split:

Step 1 is the surprising one. The agent retrieves memory while it is still acquiring memory. The paper states it plainly: "During experience acquisition, each spatial problem is solved with retrieval enabled."

Think about what this buys. If retrieval were disabled during acquisition, every card would be born with n = 0 and stay there — no card would ever be visited during writing, so TRS could only be calibrated at deployment, where the paper forbids updates. The reliability scores would be uniform forever and the whole mechanism would be dead. Retrieval-during-acquisition is what makes visit evidence exist at all.

It also creates a genuine coupling that the paper does not hide: a card written early is retrieved by later problems in the same pass, so early cards accumulate more visits than late ones. The prior strength λ in the TRS update exists partly to keep low-visit cards from being unfairly frozen at their initial value while high-visit cards race ahead.

The shuffle, and why it has a seed

Line 3 of the pseudocode is easy to skip: 𝒳̃e ← Shuffle(𝒳; seed = e). Every pass reshuffles the environment split, with the pass index as the seed.

Why reshuffle at all, if the model is deterministic at temperature 0 and the same problems are seen every pass?

Because the bank state changes between problems, and therefore order matters. On pass 0 the bank grows as you go: problem 1 is solved with an empty bank, problem 175 is solved with 174 cards available. On later passes the bank is fixed but the TRS values are moving continuously — card values after problem 50 differ from their values after problem 51. So the retrieval a given problem receives depends on where it sits in the sequence.

Reshuffling per pass means those position effects are randomised rather than baked in. A problem that happened to come first on pass 0 — and so contributed a card written with no guidance at all — will be somewhere in the middle on pass 1. Over ten passes, position-induced bias averages out instead of compounding.

Seeding by the pass index makes this reproducible: pass 3 always has the same order, so a re-run reproduces the same bank and the same TRS trajectory. Together with temperature 0 and a fixed split seed of 42, the entire pipeline is deterministic end to end — which is a stronger reproducibility property than most agent papers offer.

What "verifiable" does and does not admit

The word "verifiable spatial environment" carries a lot of weight, so it is worth marking its boundary precisely. The requirement is a function Eval(ŷ, y^⋆) → [0, 1]. That is it. In exchange, the requirement excludes several things you might want.

That last row of the left column is worth pausing on. Nothing in the memory mechanism is specific to vision. The card is text, the key is a text embedding, the verifier is a scalar function, the update is arithmetic. Spatial is the domain in which the paper demonstrates it, motivated by the observation that spatial failures are heavily procedural. The machinery would transfer to any verifiable single-turn task; whether the gains would is an empirical question the paper does not attempt.

Worked example 1 — the cost of an experience budget

Let us count model calls, because "training-free" does not mean "free". Take RoboSpatial on Qwen3.6-27B, with the paper's hyperparameters: 175 environment problems, 175 deployment problems, k = 3 retrieved cards, and the RoboSpatial pass count from the hyperparameter table.

Solver calls during acquisition. One call per problem per pass. The 122B hyperparameter table lists 6 passes for RoboSpatial:

Reflection calls. Only on pass 0, one per problem:

Deployment calls. One per deployment problem, no reflection:

Total: 1,400 calls to a frozen model. Compare that with the alternative. Post-training a 27B model on spatial data means assembling a dataset, running optimiser steps with activations and gradients and optimiser state resident in memory, and then evaluating. The paper's whole compute footprint is four H200 GPUs running vLLM inference — no backward pass appears anywhere.

Now the visit budget, which is the number that actually matters. Every solver call during acquisition retrieves k = 3 cards, and each retrieved card gets a visit update. So over 6 passes:

Eighteen visits per card. Hold on to that number — in Chapter 5 we will see that the confidence weight in the TRS estimator is n/(λ + n), so with λ = 2 and n = 18 that is 18/20 = 0.9. The score is ninety percent empirical and ten percent prior. The calibration has genuinely converged. In Chapter 4 we will compute what happens to that number under the alternative writing protocol, and it is the whole argument.

The verifier, concretely

Eval is not one function; it is per-benchmark. For ERQA, SAT, EmbSpatial, and ViewSpatial the answer is a letter or a relation label and the reward is a match: 1 or 0. For RoboSpatial's Yes/No families the same. RoboSpatial's pointing family is the interesting one: the system prompt specifies that "pointing is scored by coverage inside the reference region (convex hull by default)", and that "scattered valid free-space points beat a single guess on the object boundary". That gives a genuinely fractional reward — which is why the update rule is written for r ∈ [0, 1] rather than r ∈ {0, 1}.

This matters for the TRS arithmetic. If rewards were binary, cumulative reward c would be an integer count of successes and TRS would be a beta-binomial posterior mean. With fractional rewards the same formula still works — it is just a shrunk mean of bounded scores rather than a success count.

Algorithm 1, line by line

The paper's appendix gives the whole system as twenty-two lines of pseudocode. Every mechanism in this lesson is in there, so it is worth walking it once now — forward references and all — and then returning to it after Chapters 5 and 6 make the two unexplained lines make sense.

Inputs: a frozen VLM F, a reflection model R_φ, environment problems 𝒳, deployment problems 𝒟.
Parameters: passes T, retrieval size k, similarity threshold δ, reliability weight η, initial memory value v₀, prior strength λ.

Two observations from the shape alone.

First, the acquisition and deployment loops are nearly identical. Lines 5–7 and lines 17–19 are the same three operations. The difference is entirely in what happens after the answer: acquisition grades and updates, deployment records. This is a good property — the retrieval and prompting behaviour a card experiences during calibration is exactly what it will experience in deployment, so TRS is estimating the right quantity.

Second, line 9 sits outside the if e = 0 block and line 12 sits inside it. That single indentation difference is the whole one-pass protocol. Move line 9 inside and TRS would only ever be updated once per card. Move lines 10–13 outside and you have Continual Memory Writing. Two lines of scope determine everything Chapter 4 measures.

The verifier, per benchmark

Eval is not one function. It is a small family, and the shape of each one constrains what the reward can tell you.

The pointing row is the one that changes the mathematics downstream. Its system prompt is explicit about the scoring model: "Pointing is scored by coverage inside the reference region (convex hull by default); scattered valid free-space points beat a single guess on the object boundary."

Work an example. The model returns five points. Three land inside the reference hull for the vacant region; two land on the desk. If reward is the fraction of predicted points inside the region:

Now feed that into the TRS update from Chapter 5. A card visited on this problem receives c += 0.600, not c += 1 and not c += 0. Partial credit propagates. That is why the update rule in the paper is written for r ∈ [0, 1] rather than r ∈ {0, 1} — and it means TRS on RoboSpatial is a shrunk mean of bounded scores rather than a success count.

It also explains the prompt's insistence on several points. One point either hits or misses: reward 1 or 0, maximum variance. Five points spread across the plausible region produce a graded reward that carries information about how well the region was understood, not just whether one guess landed. The scoring rule and the prompt were designed together.

Worked example 2 — the budget on the largest benchmark

RoboSpatial is small. Run the same accounting on EmbSpatial, which has 1,820 environment problems and 1,820 deployment problems, at its tuned pass count of 4.

Solver calls during acquisition:

Reflection calls (pass 0 only): 1,820. Deployment calls: 1,820. Total 10,920 model calls.

Visit budget:

Twelve visits gives a confidence weight of 12/(2+12) = 0.857. Lower than RoboSpatial's 0.900 at 18 visits, because EmbSpatial ran four passes rather than six — and yet still firmly in the evidence-dominated regime. Note that bank size and problem count grow together, so the visits-per-card figure depends only on passes × k, not on how big the benchmark is:

Six passes at k = 3 is 18. Four passes at k = 3 is 12. Ten passes at k = 3 is 30. The whole coverage story collapses to one product — which is exactly why Chapter 4's protocol argument is about keeping the denominator at |𝒳| instead of letting it grow to |𝒳| × T.

The one design choice you should be suspicious of

The reflection model R_φ is the same frozen VLM. The paper says so directly: "For each run, the same frozen model is used both as the task-solving VLM and as the reflection model that writes procedural memories."

Sit with that. A 9B model that got a spatial question wrong is then asked to diagnose why it got it wrong and write a general lesson. There is an obvious worry: if the model could correctly diagnose its own failure, could it not have avoided the failure?

The resolution is that these are genuinely different tasks with genuinely different inputs. Solving requires reading geometry from pixels under uncertainty. Reflecting is given the model's own output and the verified answer and the reward — it is a post-hoc text task with the answer in hand, closer to "explain the difference between these two strings" than to "estimate this clearance". Verification is easier than generation, and having the target makes it easier still. That asymmetry is what the whole reflection step monetises — and it is also exactly why the reward-only ablation, which withholds the target, costs 5.5 points.

The taxonomy that makes the ablations comparable

One piece of methodology in the appendix explains how Chapter 9's atomic-ability analysis is even possible across benchmarks that share nothing. The authors annotate each benchmark's sub-categories with one or more of ten atomic spatial abilities, used as post-hoc diagnostic labels rather than runtime inputs.

The point is that RoboSpatial's compatibility category and SAT's action-consequence questions are, mechanically, completely different — different images, answer spaces, and scorers. But both require judging whether a spatial configuration supports an action. Label both Affordance and you can ask a question neither benchmark can answer alone: does SMA help with affordance reasoning?

That Omni3D exclusion is worth flagging honestly. Omni3D is the benchmark with the largest lesson-field ablation (−5.2) and one of the largest SMA gains (+6.0 on the 27B) — and it contributes nothing to the atomic-ability figure. So the ability-level conclusions in Chapter 9 are drawn from the four benchmarks where category annotations happened to exist, not from the full evaluation.

The multi-label property matters too. Because a single question feeds several ability buckets, "SMA improves all ten abilities" is a weaker statement than ten independent wins would be — the buckets overlap, so a broad improvement on compatibility questions lifts Localization, Distance/depth, and Affordance simultaneously. The paper states the non-disjointness plainly, which is the right disclosure.

Chapter 3: Verifier-Guided Reflection

The rollout is finished. The model produced 900 tokens of reasoning ending in <answer>Yes</answer>. The verifier said 0. Now what?

You have a choice about what to keep, and the choice is the entire chapter. The rollout is a few thousand tokens. The card you are about to write is two sentences. That is a compression ratio of roughly a thousand to one, and everything depends on which two sentences survive.

The reflection function

Four inputs. The raw model output o_i, the task t_i, the verified target y^⋆_i, and the reward r_i. Two outputs: a summary and a transferable lesson, returned as strict JSON.

The bolded input is where the paper differs from its closest neighbour. MemRL-style runtime memory uses reward-only reflection — the reflection step sees the output and the scalar reward but not the ground truth. The paper implements that variant as MemRL-R, and also implements MemRL-GT, which gets the ground truth, specifically so that "stronger reflection supervision" can be separated from "memory-value calibration". This is careful baseline design and it is what lets Chapter 9 attribute the gains.

Why reward-only reflection is so much weaker

Think about what a model can actually infer from each signal.

Reward-only. "You said Yes. That was worth 0." The model now knows it was wrong. It does not know what right looks like. Its diagnosis must be a guess over the space of possible errors, and a plausible-sounding guess is often the wrong one — "I should have looked more carefully at the image" is a fine-sounding lesson that teaches nothing.

Ground-truth-guided. "You said Yes. The answer was No." Now the diagnosis is a differential. There is a specific gap between two specific claims, and the reflection task becomes: what reasoning step would have moved me from the first to the second? That is a far more constrained question, and constrained questions get better answers out of language models.

The measurement: reward-only reflection costs 5.5 points on RoboSpatial (68.5 → 63.0) and 2.8 points on Omni3D (47.6 → 44.8), on Qwen3.6-27B. On RoboSpatial that is the second-largest single ablation in the paper.

The anti-leakage contract

Here are the rules, from the RoboSpatial reflection prompt in the appendix. They are labelled A through D and marked mandatory in the JSON output.

Rule C is the one that repays a second read. Rules A and B block the answer. Rule C blocks the index. Without C a model could write "when the scene contains a blue chair beside a radiator with three books on the sill, check the left clearance" — which reveals no answer but effectively pins the lesson to one image, so it is either useless (never retrieved) or a covert scene lookup.

Rule D exists because rules that only forbid produce empty output. Without an explicit permission the safest JSON is a lesson so hedged it says nothing. D tells the model what a compliant lesson looks like.

The output contract

The reflection prompt specifies the schema exactly, and the phrasing is worth quoting because it is unusually strict for an LLM prompt:

And the two fields:

• summary — "1-2 sentences on task shape + diagnosed habit/gap (reasoning-level, not the answer)."
• transferable_lesson — "exactly ONE dense sentence shaped as a procedure: 'When <structural condition>, apply <positive habit>, avoid <negative habit>, and validate by <concrete check>.' This is the part the future agent will actually use."

Plus a style clause: "Be concrete, non-redundant, and self-contained. No self-correction chatter, no apology, no meta talk about JSON." Anyone who has asked a model to reflect on its own errors will recognise what that clause is defending against — the reflexive apology paragraph that contains no information.

The prompt is written per benchmark, and the differences matter

There is not one reflection prompt; there are seven, one per benchmark, each specialised to its answer space. The skeleton is identical — possible inputs, silent diagnosis, transferable lesson, anti-leakage rules A–D, strict JSON schema — but the specifics change in ways that show what the authors were actually worried about.

Look at the "interleaved images" for ERQA and "image(s)" for SAT versus the singular "image" for RoboSpatial, Omni3D, and EmbSpatial. That is the input format leaking into the prompt, correctly — a reflector told to expect one image and handed two would describe the wrong evidence.

And notice how much more dangerous leakage is on the multiple-choice benchmarks. On RoboSpatial the answer is Yes/No or a coordinate list, so a leaked answer is at least tied to a specific spatial claim. On ERQA, SAT, and EmbSpatial the answer is a letter. A card that says "in questions like this, C is often correct" is content-free, undetectable by casual reading, and catastrophic across a benchmark where the option ordering is arbitrary. Rule A on those benchmarks is doing heavier work than its wording suggests.

One reflection per problem, and the cost of that

Reflection runs exactly once per environment problem, on pass 0. Not per pass, not multiple samples per problem, not a best-of-n over candidate lessons.

That is an interesting restraint. The obvious upgrade — sample three lessons per rollout and keep the best — requires a way to judge which is best, and the only judge available is downstream transfer, which you cannot measure at write time. You would be back at the credit-assignment problem. So the design defers the question entirely: write one lesson per rollout, then let visit evidence sort the resulting population.

That reframing is the elegant part of the whole method. Reflection quality is not optimised at write time; it is selected for at read time. A bad lesson is not prevented, it is written into the bank and then demoted by its own track record. Which means the system tolerates a fairly high rate of poor reflections, provided the good ones are distinguishable after a dozen visits.

The two-step reflection procedure

The prompt splits reflection into a silent diagnosis and a public lesson.

Notice the phrase "use the ground truth only to locate reasoning gaps". The target is a diagnostic instrument, not content. It tells the reflector where to look; it is not supposed to end up in what the reflector writes. Whether a language model reliably honours that boundary is an empirical question, and the paper's honest answer is the disjoint split: if leakage were happening at scale, deployment accuracy on unseen problems would not improve the way it does.

The prompt also enumerates the success habits worth repeating: "anchor first, multi-point spread in free space, read relation verb literally, obstacle sweep for compatibility". Reflection is not only failure analysis. A card written from a correct rollout is a card about what to keep doing — and Chapter 9 shows those cards end up with measurably higher TRS.

Reflecting on a rollout that went right

Every example so far has been a failure, which is how most people picture reflection. Half the bank is not like that. The prompt explicitly asks the reflector to "note success habits worth repeating", and names four for RoboSpatial: anchor first, multi-point spread in free space, read relation verb literally, obstacle sweep for compatibility.

Work one. The task is family 1: "Pinpoint several points within the vacant space to the left of the chair." The model bound the chair, inferred the wedge of free floor to its left, and returned six points. Four landed inside the reference hull. Reward r = 4/6 = 0.667.

Note that this is neither a clean success nor a failure — it is 0.667, and the reflection prompt receives that number alongside the target region. What should the lesson say?

Two things this example shows that a pure-failure example cannot.

Fractional rewards make the diagnosis richer, not just the score. A binary 0 would have said "you were wrong" with no information about how wrong. 0.667 with a reference region says "your approach was mostly right and your sampling was slightly too tight" — and the resulting lesson is about spread, which is a repairable habit, rather than about the whole approach.

Success habits and failure traps end up in the same sentence. The four-slot grammar has room for both, which is why a partially-successful rollout is arguably the most productive kind to reflect on: it has a habit worth keeping and a trap worth naming, from one episode.

Chapter 9 puts a number on the difference: memories written from successful source questions reach a mean TRS of 0.522 and 85.7% downstream accuracy, against 0.452 and 61.4% for failure-sourced ones. Success-sourced cards are better. But they are only better on average — and crucially, SMA does not know that at write time and does not use it. It finds out from visits.

The question families the reflector is told to recognise

One more piece of machinery, because it explains why the lessons transfer at all. The system prompt — the one used to solve the task, not to reflect — explicitly teaches the model a taxonomy of question shapes, and instructs it to "recognize the structural shape, not only the category label". For RoboSpatial:

That third row is the warehouse robot's bug, written down as a family. And the fact that the taxonomy lives in the system prompt means both the solver and the reflector share a vocabulary for describing question shapes — which is exactly what makes the "When <shape>…" slot in a lesson meaningful to a future reader.

Worked example 2 — grading three candidate lessons

Take a concrete rollout. The task is family 3: "Can the storage bin fit to the right of the desk?" The model answered Yes. The verified target was No. Reward r = 0.

Here are three lessons a reflector might produce. Grade each against the contract before reading the verdicts.

Card 3 is what the paper's grammar is engineered to produce, and card 2 is why the grammar has four slots instead of one. A single free-form sentence drifts toward card 2 — the safest thing to say is the thing that says nothing. Four named slots force specificity, because "avoid <trap>" cannot be filled without diagnosing an actual trap.

A fourth candidate, and the subtlest failure

Three candidates were not enough to cover the interesting cases. Here is a fourth, and it is the one that would slip past a careless reviewer:

Check it against the contract. Rule A: does it reveal the ground truth? Not literally — it names no specific answer for any specific question. Rule B: it quotes nothing. Rule C: no scene fingerprint. Rule D: it looks like a generic check. By the letter of all four rules it passes.

And it is poison. It encodes a prior over answers rather than a procedure over observations. "Lean toward answering no" is a bias, and once it is in the bank it will be retrieved for compatibility questions where the answer is yes, and it will push the model wrong. Worse, it will often look reliable: if compatibility questions in this benchmark happen to skew toward "no", the card will accumulate reward and its TRS will climb. The mechanism would be rewarding a base-rate exploit.

This is the failure mode that the four-slot grammar is structurally defending against, and it is worth seeing how. "When <shape>, apply <habit>, avoid <trap>, validate by <check>" has no slot for a prior over answers. There is nowhere in the template to put "lean toward no" without it landing in the habit slot, where it reads as obviously wrong — "apply leaning toward no" is not a habit, it is a guess. The grammar makes bad content awkward to express.

Whether that defence always holds is not something the paper tests directly, and it is a genuine open edge. The closest evidence is indirect: if answer-prior cards dominated the bank, cross-benchmark transfer would collapse (a prior tuned to RoboSpatial's answer distribution would be actively wrong on EmbSpatial), and Chapter 9 shows EmbSpatial → RoboSpatial transfer at +7.3.

Reflection is the only place the system gets to be clever

Step back and count the intelligence in SMA. Retrieval is a dot product and a sort. Calibration is three lines of arithmetic. Deployment is string concatenation. All of the judgement in the entire system lives in one place: the reflection call that turns a rollout into a sentence.

Everything downstream operates on whatever reflection produced. TRS can demote a bad card but cannot rewrite it. Retrieval can exclude an irrelevant card but cannot improve it. If reflection writes 175 versions of "be careful and look at the image thoroughly", the bank is worthless and no amount of calibration will make it otherwise — every card will have roughly the base rate as its TRS, the z-scores will be noise, and you will have built an expensive no-op.

Which is why the reflection prompt is the longest and most constrained artefact in the appendix, why the output grammar has four named slots, why the anti-leakage rules are enumerated A through D, and why the ground truth is supplied despite the leakage risk it creates. Every one of those decisions is protecting the only component that cannot be fixed downstream.

Why "strict JSON only" is a load-bearing instruction

The prompt says: "Return strict JSON only. No markdown, no code fence, no prose outside JSON." It is easy to read that as fussiness. It is not.

Reflection runs once per environment problem — 175 times for RoboSpatial, 1,820 for EmbSpatial — unattended, in a batch. Every output must be machine-parsed into two named fields and appended to the bank. A response wrapped in a code fence, or preceded by "Sure! Here's the memory:", either fails to parse or requires a fragile stripping heuristic. A parse failure at scale means silent holes in the bank — problems that contributed no card, discovered later as an unexplained drop in coverage.

The same discipline appears on the solve side. Every benchmark's system prompt ends by specifying exactly one tagged line, and RoboSpatial's is unusually explicit: "Yes/No output: exactly <answer>Yes</answer> or <answer>No</answer>. Pointing output: exactly <answer>[(x1, y1), (x2, y2), …]</answer> with normalized floats in [0, 1]; include several points; no explanation inside the tag."

That last clause — "no explanation inside the tag" — exists because Parse(oi) extracts the tag contents and hands them straight to Eval. Prose inside the tag is a parse failure, which scores as reward 0, which corrupts both the accuracy number and every TRS update on that problem's guidance set. A formatting slip becomes a false negative in the reliability estimate for three cards.

What the reflector may receive

The prompt lists its possible inputs, and the list is instructive: "the rollout conversation (image + turns), task text, ground truth, model output, and/or a compact trajectory JSON". The trajectory JSON is a fallback for when the full conversation is unavailable — a practical detail that tells you this ran at a scale where some conversations get dropped.

Note that the reflector sees the image. It is a vision-language model reflecting on a visual task, so it can check whether its own claimed observation was actually supported by pixels. That is a genuinely different diagnostic position from a text-only reflector reading a transcript.

The trajectory JSON, and what it tells you about scale

Every reflection prompt lists its possible inputs, and every one ends the same way: "…and/or a compact trajectory JSON." One template is explicit that this is a fallback — "a compact trajectory JSON as a fallback when the conversation is unavailable."

That clause is a small window onto the engineering reality behind the numbers. Why would a conversation be unavailable? Because reflection runs as a separate batch after solving, at a scale where holding every multi-image conversation in memory is impractical, and where some fraction of them get dropped, truncated, or fail to serialise. So the system was built to degrade: full conversation if you have it, a compact summary structure if you do not.

It also implies reflection is not a continuation of the solve call. It is a fresh call with a fresh prompt, reconstructing the rollout from stored artefacts. That matters for a reader trying to reproduce this: you cannot simply append "now reflect" to the solving conversation, because the reflection prompt is a different system prompt with different rules, and the model must be told the ground truth it did not previously have.

Notice the asymmetry in the last two rows. The ground truth is persisted — it is part of the benchmark — but it is persisted outside the memory bank, and the bank is the only artefact that survives into deployment. The separation is structural, not merely procedural: at deployment time the ground-truth store is not consulted, because the deployment loop has no reason to open it.

What makes a lesson score well, summarised

Pulling the chapter together: given a rollout, what does a good card look like? Four tests, each derived from something the prompt or the ablations establish.

A card failing the shape test is never retrieved. One failing the portability test is retrieved and misleads. One failing the no-answer test is leakage. One failing the executability test is card 2 from the worked example — harmless, retrieved often, and worth nothing.

Only the last of the four is invisible to any automated check, which is why it is also the one the grammar has to enforce structurally rather than by rule.

What the bank actually ended up containing

The qualitative appendix is the only window onto what the reflector wrote, and one sentence in it is worth more than the rest. Describing five representative successful cases, the paper says SMA "retrieves concrete spatial procedures, including size checking, coordinate localization, depth comparison, motion simulation, and background anchoring."

Read that list carefully. Every item is a verb phrase naming an operation. Not a fact, not a category, not a scene — an operation you perform on an image. That is the reflection grammar working as designed, and it is direct evidence that the "validate by <check>" slot is producing executable content rather than decorative content.

Two things follow. First, these read like a curriculum — the operations a competent human would name if you asked them how to do spatial reasoning on photographs. The model, given verified feedback and a template, rediscovered a chunk of that curriculum without being told it. Second, and less comfortably: these are all things a person could have written by hand. Which loops back to Chapter 0's objection and gives it its most honest answer — the value is not in the novelty of any single lesson, it is in the automatic discovery, scoping, and validation of a bank of them.

The paper's own conclusion from the cases is stated conservatively: these memories "steer the model toward task-relevant geometry rather than superficial semantic matches, supporting the quantitative finding that high-TRS procedures transfer reliably." Five hand-picked successful cases are an illustration, not evidence, and the paper presents them as such.

Chapter 4: Write Once, Calibrate Often

You are going to run ten passes over the environment split. On each pass the agent retrieves, solves, and gets graded. The question this chapter answers is embarrassingly simple and turns out to matter a lot: on which passes should the agent write new memories?

The obvious answer is "all of them". More experience, more cards, more coverage. The paper calls this Continual Memory Writing, tries it, and rejects it. Its default is One-Pass Memory Writing: the reflection model writes only during the first pass over 𝒳, and "later passes reuse the fixed memory bank to update the reliability state of retrieved cards."

The stated reason is duplication: "later passes often produce memories that duplicate earlier cards." Which is unsurprising once you say it out loud — pass 5 sees exactly the same 175 problems as pass 0, with a similar frozen model at temperature 0. It will diagnose similar gaps and write similar lessons.

Why duplication is not merely wasteful

If duplicates only cost disk space, nobody would care. They cost something else, and the arithmetic is the argument.

Every solver call retrieves exactly k = 3 cards. That number does not grow when the bank grows. So the total number of visit updates per pass is fixed at 3 × (number of problems), no matter how large the bank is. Meanwhile the bank size under continual writing grows linearly with passes.

Worked example 3 — TRS coverage under both protocols. RoboSpatial, 175 environment problems, k = 3, 10 passes.

One-Pass Memory Writing. Cards are written only on pass 0, so the bank ends at 175 cards. Visit updates over ten passes:

Continual Memory Writing. A card is written every pass for every problem, so the bank ends at 175 × 10 = 1,750 cards. The visit updates are the same 5,250 — k did not change:

Ten times the bank, one tenth the evidence per card. And the paper's measurement matches this arithmetic almost exactly: "By the final pass, One-Pass Memory Writing uses only one-tenth as many memories, exhibits 21% less redundancy, and achieves roughly twice the TRS-update coverage."

The one-tenth is the ratio we just derived. Note the paper says coverage — the fraction of cards receiving a TRS update — is roughly twice, not ten times, which is a healthier number than the naive average suggests: retrieval is not uniform, so under continual writing a subset of cards still gets visited repeatedly while a long tail never gets visited at all. That long tail is the problem.

What an unvisited card does to the ranking

Recall the TRS initialisation: n = 0, c = 0, v = v₀ = 0.5. A card that is never retrieved keeps v = 0.5 forever. Now look at what that does inside the ranking rule, which we will derive properly in Chapter 6 but can already sketch: the score is a blend of normalised similarity and normalised TRS.

If most of your bank sits at exactly v₀, then the TRS term is nearly constant across candidates, its z-score is nearly zero, and the blend degenerates back to pure similarity. You have paid for a reliability mechanism and received MemP. Under continual writing this is not hypothetical — it is what a bank of 1,750 cards with 5,250 scattered visits looks like.

Redundancy has a second cost, at retrieval

The paper defines redundancy as "the proportion of repeated memories in the memory bank" and reports one-pass as 21% lower. Beyond storage, duplication has a direct effect on what reaches the prompt.

Suppose the bank holds four near-identical copies of the same lesson, each with its own independently-estimated TRS. They will have near-identical embeddings, so if one clears the similarity threshold they all do. If their TRS values happen to cluster high, the top-3 can be filled by three copies of one idea — the model receives one lesson in triplicate instead of three complementary lessons. The retrieval budget has been spent on redundancy.

And their TRS estimates are individually noisier than one merged card would be: four cards with 3 visits each carry less usable evidence than one card with 12, because each of the four is shrunk hard toward v₀. Duplication splits evidence.

Where the passes actually go

If only pass 0 writes, what are passes 1 through T−1 for? Exactly one thing: calibration. Steps 1 through 4 of the acquisition loop still run — retrieve, solve, grade, update n/c/v — and step 5 is skipped.

So the passes are not "more training". They are repeated measurement of a fixed hypothesis set. Pass 0 proposes 175 lessons; passes 1–9 test them, in situ, against real problems, and record how often each one was present when the answer came out right.

The pass counts are tuned per benchmark, and the spread is informative. From the Qwen3.5-122B-A10B hyperparameter table: RoboSpatial 6, ERQA 2, Omni3D 10, SAT 9, EmbSpatial 4. Omni3D — open-answer, hardest, largest gains from the lesson field — uses the full ten. ERQA uses two. The main table reports "the best checkpoint from the 10-pass evaluation", with the same selection rule applied to every SMA row; we will return to what that selection rule costs in Chapter 8.

Why the bank stays small, and why that is a feature

One-Pass Memory Writing caps the bank at |𝒳| cards. For RoboSpatial that is 175. It is worth dwelling on how small that is, because the instinct in a retrieval system is always "more documents is better" and here the opposite holds.

The bottom row is the reductio. A memory system with a hundred thousand cards and no way to evaluate them is not a better memory system — it is a similarity search with extra steps. The reliability layer that distinguishes SMA from MemP requires evidence per card, and evidence per card is bounded by a budget you cannot escape.

This inverts the usual scaling intuition and is the most transferable engineering lesson in the chapter. In a document-retrieval system, more documents means more chance the right one exists, and documents do not need to be individually validated. In an experience-memory system, every card is a hypothesis about how to act, and an unvalidated hypothesis in the prompt is a liability, not an asset. Chapter 9's MemP numbers — negative on Tracking and Affordance — are what unvalidated hypotheses cost.

When one-pass would be the wrong choice

The paper scopes Finding 5 to ten passes over a fixed environment split. Take the scoping seriously and ask when the conclusion inverts.

The argument rests on one premise: pass 5 sees the same problems as pass 0. Under that premise, a card written on pass 5 is about a problem that already has a card, so it is a duplicate by construction. Break the premise and the argument goes with it.

So the right generalisation is not "write once" but "write once per distinct problem". One-Pass Memory Writing is what that policy reduces to when your environment is a fixed finite set you are cycling through. State it that way and it stops being a surprising empirical finding and becomes almost definitional — which is a good sign that the mechanism is understood rather than merely observed.

How writing interacts with checkpoint selection

One more consequence of one-pass writing that is easy to miss. Because cards are frozen after pass 0, every checkpoint in the 10-pass run contains exactly the same lessons. Checkpoint 3 and checkpoint 9 differ only in (n, c, v).

That makes "best checkpoint from the 10-pass run" a much narrower selection than it sounds. You are not choosing among ten different memory banks; you are choosing among ten calibration states of one bank. The lesson content is fixed by the time pass 0 ends, and the selection is over how much visit evidence has accumulated.

Under continual writing the same selection would be far more permissive — checkpoint 3 and checkpoint 9 would have different cards and different scores, and picking the best of ten would be closer to a search over ten distinct systems. Which is a second, quieter argument for one-pass: it keeps the checkpoint-selection rule honest.

It also explains why the per-benchmark pass counts vary so much (2 for ERQA, 10 for Omni3D) without that being alarming. A low pass count does not mean "we stopped learning early"; it means calibration on that benchmark reached its most useful state quickly, and additional visits started to move rankings in ways that did not help.

Watch both protocols run

Push k up and watch the continual curve improve — more retrievals per problem means more evidence to spread around. That is a real lever, and the paper swept it: the sensitivity study peaks at k = 3. Larger k dilutes the guidance with lower-ranked cards and costs prompt tokens; the evidence gain does not pay for it.

The other lever: raising k instead of T

Chapter 2 established that visits per card equals T × k under one-pass writing. So there are two ways to buy evidence, and it is worth asking why the paper spends on passes rather than on retrieval depth.

Read the last two rows and the choice is obvious. Extra passes are an offline cost that buys clean evidence. Extra k is a permanent cost that buys muddier evidence and dilutes the guidance the model actually receives. The paper's sweep confirms it empirically — accuracy peaks at k = 3 — but the asymmetry is visible before you run anything.

There is a subtlety worth naming. Raising k also worsens credit assignment quadratically-ish: with k = 3, one reward is shared three ways; with k = 6 it is shared six ways, and the chance that any individual card's average is dominated by its co-retrieved partners rises. So k is not merely a token-budget dial — it is also the dial that controls how noisy every TRS estimate in the bank is.

Pass by pass, both protocols

Put the two trajectories in a table so the divergence is visible. RoboSpatial numbers: |𝒳| = 175, k = 3. "Coverage" here is the average visits per card, which we showed in Chapter 2 reduces to T×k for one-pass and to exactly k for continual.

The continual column never moves. Not once in ten passes. Every pass adds 525 updates and 175 cards, and 525/175 = 3, forever. Meanwhile one-pass climbs linearly to 30.

Translate that into confidence weights. At pass 10, one-pass cards sit at n/(λ+n) = 30/32 = 0.938 — almost entirely empirical. Continual cards sit at 3/5 = 0.600 — forty percent prior, permanently. And that 0.600 is the average; retrieval is not uniform, so the median continual card is worse and a long tail sits at exactly n = 0, confidence 0.000, TRS pinned at 0.5.

The paper's measured coverage ratio is "roughly twice", not ten times, which tells you the real distribution is more forgiving than these averages — popular cards get visited repeatedly under both protocols. But the direction and the mechanism are exactly what the arithmetic predicts.

The pass counts are themselves a result

The Qwen3.5-122B-A10B hyperparameter table lists a different pass count per benchmark, and the spread is not arbitrary noise — it is the outcome of the checkpoint-selection rule.

The ordering is coherent: benchmarks with more diverse procedures and more room to improve keep benefiting from additional calibration passes, while saturated or narrow benchmarks stop early. Omni3D wanting all ten also raises an unanswered question the paper does not chase — would it have kept improving at fifteen?

The honest reading of Finding 5

The paper's Finding 5 says One-Pass Memory Writing "is far more efficient than continual rewriting: it maintains a smaller, less redundant memory bank while increasing TRS-update coverage." Note what is claimed and what is not.

Claimed: smaller bank, less redundancy, better coverage — all directly measured across ten passes on five benchmarks with Qwen3.6-27B. Not claimed: that one-pass wins for any number of passes, any k, or any environment split size. The paper is careful: "the trajectories therefore show that continual writing produces a larger, more redundant memory bank and lower TRS-update coverage over the ten passes considered here."

That hedge is correct and you should keep it. If your environment split is enormous and highly diverse, pass 0 may never see a large region of the problem space, and a second writing pass over fresh problems is not duplication — it is coverage. The paper's finding is about repeated passes over the same problems, which is a specific regime, not a universal law.

What "redundancy" is measuring

The paper reports that one-pass writing exhibits "21% less redundancy (the proportion of repeated memories in the memory bank)". The parenthetical is the whole definition, and it is worth thinking about what "repeated" can mean for text.

Two cards written from the same problem on different passes will not be byte-identical — the model is at temperature 0, but the guidance set it saw differs between passes (the bank has grown, the TRS values have moved), so the rollout differs and so does the reflection. What you get is not duplication but near-duplication: the same procedure phrased three slightly different ways.

Near-duplicates are worse than exact duplicates for this system, for a reason worth spelling out. Exact duplicates could be detected and merged with a hash. Near-duplicates require a similarity threshold, and their embeddings are close enough that they cluster in retrieval — if one clears δ, they all do. So they compete for the same three slots while carrying one idea's worth of information, and each one's TRS is estimated from a third of the evidence a merged card would have had.

Same total evidence, strictly worse outcome on every row. That is the cost of redundancy stated precisely, and it is why the 21% figure is reported alongside bank size rather than as an afterthought.

It also names the missing piece. SMA prevents near-duplicates from being written; it has no mechanism to merge them if they exist. A bank assembled from several sources — say, the cross-benchmark transfer setting in Chapter 9, where you concatenate two banks — would have duplicates that one-pass writing never guarded against. The paper's own Limitation D.2 names merge and compress as unimplemented lifecycle operations, and this is where their absence would first bite.

Diagnosing a bank that is not calibrating

If you implement this and the TRS term is doing nothing, the symptoms are specific and the causes are few. This is the debugging table the chapter has been building toward.

That last row is worth adopting as a standing metric. The paper reports macro-average retrieved similarity falling 0.792 → 0.698 as evidence that TRS changes selection. It is cheap to compute, requires no accuracy measurement, and tells you immediately whether your reliability machinery is live. If retrieved similarity is unchanged, the mechanism is off, whatever the accuracy says.

And the first row names the failure this chapter is about. A bank ten times too large produces exactly the "everything at v₀" symptom, and the fix is not a better estimator — it is writing fewer cards.

The number to watch: visits per card

One metric summarises the whole protocol argument, and it is worth computing before you run anything, because you can predict it in advance.

Look at the last two rows together. Continual writing at ten passes gives you the same per-card evidence as one-pass writing at one pass — while costing ten times the model calls and ten times the storage. That is the finding, stated as sharply as the arithmetic allows.

Chapter 5: TRS — Visit Evidence

This is the chapter the paper is named for. Everything so far has been about producing good candidate lessons. This is about deciding which of them to believe.

The wrong answer, and why it is tempting

Here is the obvious design. A card written from a rollout the model got right came from good reasoning, so score it high. A card written from a failure came from bad reasoning, so score it low. Rank by that.

The paper explicitly refuses this, and gives two reasons in one sentence: "A source rollout can be correct but produce a lesson that is too specific, and an imperfect rollout can still produce a useful spatial procedure after verifier-guided reflection."

Take both halves seriously. A correct rollout tells you the model got that problem right; it says nothing about whether the sentence the reflector distilled from it generalises. A model can answer correctly by luck, or by a scene-specific shortcut, and the resulting lesson can be narrow or simply wrong-headed. Conversely, a failed rollout with the ground truth in hand is often the richest reflection material there is — there is a specific, diagnosable gap, and "avoid <trap>" has something concrete to point at.

The five properties the estimator must have

The appendix lists the requirements before giving the formula, which is a good way to read it — you can almost derive the update from the list.

Properties 3 and 4 together are quite restrictive. Order invariance kills EMAs. Low-visit conservatism kills the plain mean. What survives is a mean pulled toward a prior, with the pull weakening as evidence accumulates — and property 5 fixes how it must weaken.

The estimator

Initialisation, for every newly written card:

And whenever card m_j is selected into the guidance set for problem ξ_i, and that problem's answer receives reward r_i ∈ [0, 1]:

Two constants: v₀ = 0.5 (the neutral prior value) and λ = 2.0 (the prior strength). Both are held at those values across every benchmark in the hyperparameter tables.

Check the properties. Uniform init: every card starts at v₀, full stop. Visit dependence: v moves only when n and c move, which happens only on retrieval. Order invariance: the formula reads only n and c, and addition commutes. The last two properties need the rewrite.

The rewrite that explains everything

The appendix gives the algebraically identical form, and it is the one to memorise:

Verify it in one line: expand and the second term's n_i cancels, giving (λv₀ + c_i)/(λ + n_i). Same formula.

But now read what it says. TRS is a weighted average of two things: the neutral prior v₀, and the empirical success rate c_i/n_i. The weights sum to one. And the weight on the empirical term,

is a confidence term: zero at n = 0, and approaching 1 as n grows. This is a shrinkage estimator — the empirical mean shrunk toward a prior by an amount that decays with evidence. It is the same shape as a beta-binomial posterior mean, as James–Stein shrinkage, as Laplace smoothing. The paper offers the cleanest reading: "a prior with λ virtual visits whose average reward is v₀. With the default neutral prior v₀ = 0.5, λ = 2 corresponds to two virtual visits with one success and one failure."

Every card is born having already been tried twice — once successfully, once not. Real visits then have to outvote those two ghosts.

Worked example 4 — calibrate a card by hand

A card is written. Over the next passes it gets retrieved five times, and the problems it guided scored: 1, 0, 1, 1, 1. Take λ = 2, v₀ = 0.5, so λv₀ = 1.0.

Initial. n = 0, c = 0, v = 0.500.

Visit 1, r = 1. n = 1, c = 1.

The empirical rate is 1.000 — a perfect record. TRS says 0.667. The prior is holding it back, correctly: one success is not evidence of reliability.

Visit 2, r = 0. n = 2, c = 1.

Empirical rate 0.500, prior 0.500, and at n = 2 the weights are equal, so v lands exactly on 0.500. The card is back where it started — which is the right thing to say about a card that is 1-for-2.

Visit 3, r = 1. n = 3, c = 2.

Visit 4, r = 1. n = 4, c = 3.

Visit 5, r = 1. n = 5, c = 4.

Now cross-check with the convex-combination form. Empirical rate c/n = 4/5 = 0.800. Confidence = 5/(2+5) = 5/7 = 0.714. So:

Both routes agree. The card's honest track record is 80%; TRS reports 71.4%, because five visits is not yet enough to fully trust 80%.

Now verify order invariance by hand. Reorder the rewards as 1, 1, 1, 1, 0 — the failure last instead of second. After five visits: n = 5, c = 1+1+1+1+0 = 4. Identical. So v = (1.0 + 4)/(2 + 5) = 5/7 = 0.714. The same number.

Compare that to an exponential moving average with rate 0.3, which many memory systems use. Sequence 1,0,1,1,1 ends near 0.83; sequence 1,1,1,1,0 ends near 0.24. A three-and-a-half-fold difference from reordering the same five outcomes. Order invariance is not a nicety — retrieval order is an artefact of the shuffle seed, and an order-sensitive score would be reading noise.

Worked example 5 — fractional rewards from pointing

Binary rewards make the estimator look like a success counter. RoboSpatial's pointing family breaks that, so work one through.

A card about multi-point spread in vacant regions is retrieved for four pointing problems. The coverage rewards come back as 0.60, 0.80, 0.40, 1.00. Same constants: λ = 2, v₀ = 0.5, λv₀ = 1.0.

Visit 1, r = 0.60. n = 1, c = 0.60.

Visit 2, r = 0.80. n = 2, c = 1.40.

Visit 3, r = 0.40. n = 3, c = 1.80.

Visit 4, r = 1.00. n = 4, c = 2.80.

Cross-check with the convex form. Empirical mean = 2.80/4 = 0.700. Confidence = 4/6 = 0.667.

Notice what fractional rewards buy. Under binary scoring, visit 1 would have been a coin flip — either the single point landed in the hull (r = 1) or it did not (r = 0), and the card's estimate would swing to 0.667 or 0.333 on the strength of one guess. With coverage, r = 0.60 says "mostly right, partially wrong", and TRS moves to 0.533 — a small, well-earned step. Fractional rewards are variance reduction in the reliability estimate, and they cost nothing extra to compute.

This is also why the pointing prompt insists on several points. More points means the coverage fraction is a finer-grained measurement, which means each visit carries more information about the card. The scoring rule, the prompt instruction, and the TRS estimator are one design.

The estimator you already know, under a different name

If the formula looks familiar, it should. Line up three standard estimators and the resemblance is exact.

So λ = 2, v₀ = 0.5 is exactly a Beta(1, 1) prior — the flat, uninformative one — and the update is its posterior mean. The paper does not frame it Bayesianly, and it does not need to; the design properties in the table above are sufficient justification on their own. But the correspondence tells you the estimator is not ad hoc: it is the standard answer to "estimate a bounded rate from few, noisy, exchangeable samples", arrived at by listing requirements rather than by invoking a prior.

The one place the correspondence breaks is fractional rewards. A Beta-binomial posterior assumes successes are integer counts of Bernoulli trials; c = 2.80 from coverage scores is not that. The formula still computes a sensible shrunk mean of bounded observations, but the exact posterior interpretation is a convenient analogy rather than a derivation.

What λ buys, and what it costs

The paper is direct: "A larger λ makes the calibration more stable and conservative; a smaller λ makes it more reactive to early feedback."

Run the extremes on the same 1,0,1,1,1 sequence. With λ = 0 the estimator is the raw mean: after visit 1 it reads 1.000, after visit 2 it reads 0.500, and a card whose first visit happened to fail reads 0.000 — effectively deleted from ranking on the strength of one coin flip. With λ = 20 the final value is (10 + 4)/(20 + 5) = 0.560, barely off the prior after five real visits; you have built a mechanism that refuses to learn.

λ = 2 sits at the point where two real visits are enough to weigh as much as the prior. Given the coverage numbers from Chapter 4 — 18 to 30 visits per card under one-pass writing — that puts most cards deep in the evidence-dominated regime while still protecting the tail.

The λ sweep, computed

Take a single card with a fixed record — 8 visits, 6 successes, so an empirical rate of 0.750 — and read off what different prior strengths would say about it. v₀ = 0.5 throughout.

The two ends are both degenerate and they fail in opposite directions. At λ = 0 the estimator is maximally reactive and maximally fragile — single observations become confident assertions and the ranking chases noise. At large λ the estimator refuses to update and you have paid for machinery that does nothing.

Now check λ = 2 against the coverage numbers. RoboSpatial cards reach n = 18, confidence 0.900. EmbSpatial cards reach n = 12, confidence 0.857. Omni3D reaches n = 30, confidence 0.938. In every case the operating point is well into the evidence-dominated regime, while a card that somehow got only two visits still sits at confidence 0.500 and is protected from a bad pair of draws. The constant is chosen so that the typical card is trusted and the rare card is not.

What an adversarial visit sequence does

A useful way to stress an estimator is to ask what the worst realistic sequence does to it. Two cases.

Case 1 — a genuinely good card, unlucky early. Rewards 0, 0, 1, 1, 1, 1, 1, 1. True rate is 0.750.

The card recovers. Its worst reading is 0.250 after two failures, not 0.000 — and crucially, 0.250 is still inside the range where it can win a slot in a weak candidate set, so it gets the chance to prove itself. Under λ = 0 it would have read exactly 0.000 for two visits, near the bottom of any ranking, and might never have been retrieved again. Shrinkage is what keeps the estimator from being self-sealing.

Case 2 — a card that rides other cards' success. Suppose a useless lesson keeps getting co-retrieved with two excellent ones on easy questions. It accumulates c += 1 repeatedly and its TRS climbs toward 1.0, entirely undeserved.

The estimator has no defence against this. It is the credit-assignment hole in its purest form, and the only thing working against it is that retrieval is not that correlated — over 30 visits a card is pulled into a varied set of problems with varied partners, so a passenger's average drifts toward the base rate rather than toward its lucky partners'. That is a statistical argument, not a guarantee, and it is exactly why Limitation D.1 exists.

Drive the calibrator

The whole mechanism, in code

Concept and realisation. Everything in this chapter is four lines of Python plus a dataclass. Here it is, so you can see there is no hidden machinery.

from dataclasses import dataclass, field

LAMBDA = 2.0   # prior strength — two virtual visits
V0     = 0.5   # prior value — one virtual success, one virtual failure

@dataclass
class Memory:
    task:    str            # t — the retrieval key AND prompt content
    summary: str            # s — exposed
    lesson:  str            # l — exposed. the payload
    key:     list           # ψ(t), precomputed once
    n: int   = 0          # visits — never exposed
    c: float = 0.0        # cumulative reward — never exposed
    v: float = V0         # TRS — uniform init, regardless of source outcome

    def visit(self, reward: float) -> None:
        """One retrieval outcome. reward ∈ [0, 1]."""
        self.n += 1
        self.c += reward
        self.v  = (LAMBDA * V0 + self.c) / (LAMBDA + self.n)

    @property
    def confidence(self) -> float:
        """How much of v comes from evidence rather than the prior."""
        return self.n / (LAMBDA + self.n)

Three lines in visit. That is the Transfer Reliability Score in its entirety.

And here is the property that makes it correct, expressed as a test you should actually write:

def test_order_invariance():
    rewards = [1, 0, 1, 1, 1]
    a = Memory("q", "s", "l", [])
    b = Memory("q", "s", "l", [])
    for r in rewards:           a.visit(r)
    for r in reversed(rewards): b.visit(r)
    assert a.v == b.v == 5 / 7   # 0.714…  — the worked example above
    # An EMA implementation fails this test. That is the point.

If you are porting this and you reach for v = alpha * reward + (1 - alpha) * v because it is one line shorter, this test is what catches you. The EMA gives 0.832 forward and 0.240 reversed at α = 0.3 — same evidence, wildly different conclusion, determined by shuffle order.

The credit-assignment hole, named early

There is a real weakness here and the paper puts it in its own limitations section, so we will name it now rather than pretend.

When a guidance set of three cards produces a correct answer, all three get c += 1. But maybe only one of them mattered. Maybe none did — the model would have answered correctly anyway. Maybe one helped and one actively hurt, and the net was positive. The update rule cannot tell these apart. The paper: "when a final answer improves or fails, the current framework cannot precisely determine whether the outcome should be attributed to memory writing, reflection, retrieval, semantic filtering, or the model's final use of the retrieved memory."

What saves it in practice is scale and randomness. Across thousands of retrievals, a card that never actually helps is co-retrieved with a broad, roughly random assortment of partners; its expected reward converges toward the base rate of the problems it gets pulled into. A card that genuinely helps lifts that average. The estimator is noisy per-visit but not biased in the limit — provided a card is retrieved for a reasonably diverse set of problems, which is another argument for keeping the bank small enough that each card gets used broadly.

Chapter 10 covers the work the paper points to for fixing this properly — attribution-guided process feedback, local rerollouts, provenance DAGs.

Chapter 6: Filter, Then Rank

You have a bank of a few hundred cards, each carrying a lesson and a calibrated TRS. A new spatial problem arrives. You can afford to put three cards in the prompt. Which three?

The default answer everywhere in retrieval is "the three most similar". SMA does something else, and the paper's most quotable diagnostic is that doing something else means retrieving less similar memories and getting more accuracy.

Stage one: the semantic filter

Embed the current task string and every stored task string with ψ = text-embedding-3-large, then take cosines:

Keep only what clears a threshold:

This is a hard cut, not a soft weight. A card below δ is discarded outright and cannot be rescued by a high TRS.

That asymmetry is deliberate and it is the most important structural fact in the retrieval design. TRS is a global quantity — "this lesson tends to help" — with no reference to the current question. Left unconstrained, a globally excellent lesson about pointing geometry would be injected into every question in the benchmark, including ones it has nothing to do with. The semantic filter is what keeps TRS honest: it says first establish topical relevance, then argue about reliability.

Which is exactly why removing the filter is the single most expensive ablation in the paper: −5.8 points on RoboSpatial (68.5 → 62.7) and −7.2 points on Omni3D (47.6 → 40.4). Larger than removing the lesson field. Larger than reward-only reflection. A high-TRS lesson delivered to the wrong question is worse than no lesson at all.

The threshold is tuned per benchmark

δ is not a universal constant, and the values are informative. From the Qwen3.5-122B-A10B hyperparameter table:

The Omni3D value is the interesting one. At δ = 0.488 you are admitting fairly loose matches, which sounds sloppy — until you remember stage two exists to sort them out. A low δ and a strong ranker is a different, and here better, allocation of work than a high δ and a weak ranker.

Stage two: combined ranking

Among the survivors, score each candidate:

where z(·) is clipped z-score normalisation and η weights the TRS term. Take the top k. The paper's sweeps land on η = 0.5 and k = 3, and those values are used for every benchmark in the hyperparameter tables.

Two details deserve unpacking.

Why z-scores at all? Because the two quantities live on incomparable scales. Within a candidate set, similarities might span 0.62 to 0.71 — a range of 0.09 — while TRS spans 0.34 to 0.92, a range of 0.58. Add them raw with equal weight and TRS dominates by a factor of six, for no reason other than that its numbers happen to be more spread out. Z-scoring maps both to "standard deviations above this candidate set's mean", which makes η = 0.5 actually mean "count both equally".

The paper is careful about what the normalisation does and does not do: "Z-score normalization is used only to combine similarity and TRS on a comparable scale within the candidate set; it does not replace the low-visit shrinkage provided by λ, because shrinkage is part of the reliability estimate before candidate-level ranking."

Read that twice. Shrinkage happens first, at update time, and is about how much to believe a card's record. Z-scoring happens second, at rank time, and is about putting two numbers on one axis. If you z-scored raw empirical means instead, a card that is 1-for-1 would z-score as spectacularly reliable. Shrinkage is what stops that.

Why z-score within the candidate set? Because it makes the comparison local. The question "is this card's TRS high?" is answered relative to the other cards competing for the same slot, not relative to the whole bank. If every candidate is mediocre, the least mediocre still wins — which is correct, because you are choosing among these, not among all.

Worked example 5 — rank five candidates by hand

A RoboSpatial compatibility question arrives. δ = 0.618, η = 0.5, k = 3. Six cards clear the similarity floor in the raw bank; one falls below and is discarded before ranking.

Step 1 — similarity z-scores. The five survivors have rel = 0.780, 0.740, 0.700, 0.660, 0.620.

Deviations: +0.080, +0.040, 0.000, −0.040, −0.080. Squares: 0.0064, 0.0016, 0, 0.0016, 0.0064, summing to 0.0160. Divide by 5 and take the root:

So z(rel) = +1.414, +0.707, 0.000, −0.707, −1.414 for A through E.

Step 2 — TRS z-scores. v = 0.42, 0.88, 0.50, 0.79, 0.61.

Deviations: −0.220, +0.240, −0.140, +0.150, −0.030. Squares: 0.0484, 0.0576, 0.0196, 0.0225, 0.0009 → sum 0.1490.

So z(v) = −1.275, +1.390, −0.811, +0.869, −0.174.

Step 3 — blend at η = 0.5. S = 0.5·z(rel) + 0.5·z(v):

Pure similarity would have picked A, B, C. The combined ranking picks B, D, A. Card C — middle similarity, exactly-average TRS of 0.50, which is the value of a card that has never been visited — is dropped for card D, which is noticeably less similar but has a demonstrated track record.

Now compute what the swap did to average retrieved similarity. Similarity-only picks A, B, C: mean rel = (0.780 + 0.740 + 0.700)/3 = 2.220/3 = 0.740. Combined picks B, D, A: mean rel = (0.740 + 0.660 + 0.780)/3 = 2.180/3 = 0.727.

Average similarity went down by 0.013. That is the mechanism behind the paper's headline diagnostic, at the scale of one query. Across the whole macro average, the same effect measures as 0.792 → 0.698, a drop of 0.094 in retrieved similarity, alongside 66.8% → 69.8% in accuracy. The paper's Finding 4 states it as well as anyone could: "The best memory is not always the nearest memory; transfer reliability turns retrieval from semantic matching into evidence-weighted procedure selection."

And look at card F once more. TRS 0.95 — the most reliable lesson in the entire bank — discarded at the filter for a similarity of 0.590 against a threshold of 0.618. That is the filter doing precisely its job. Whatever F is brilliant at, it is not this question.

How you would tune δ on a new task

δ is the one hyperparameter the paper varies per benchmark, so it is the one you would actually have to set yourself. There is no sweep reported for it, which means we should reason about it from what it does rather than pretend the paper prescribes a value.

δ controls the size of the candidate set, and both failure directions are visible in the simulation above.

A sensible procedure, given what the paper reports: embed your environment split's questions, compute the full pairwise cosine distribution, and pick δ near a percentile that leaves comfortably more than k candidates for the median question. The observed values are consistent with exactly that. RoboSpatial's three tightly-templated families produce a high-similarity distribution, so δ = 0.618 still leaves plenty of candidates. Omni3D's free-form questions produce a much lower distribution, so the same percentile lands at 0.488.

Which reframes δ usefully: it is not a semantic constant meaning "similar enough". It is a percentile on the question-phrasing distribution of your particular task, and it has to be re-estimated whenever that distribution changes.

Drive the ranker

Push δ to 0.80 and watch the candidate set empty out — with fewer than k survivors, the model gets fewer than three cards, or none. Push η to 1.0 and the selection stops responding to the question at all: the same three globally-reliable cards get injected regardless of what is being asked. Both extremes are worse, which is why the sweep peaks in the middle at η = 0.5.

The same five cards at the two extremes

Keep the worked example's numbers and re-rank at η = 0 and η = 1. Nothing else changes — same filter, same candidates, same z-scores.

Trace the mean-similarity row across the columns: 0.740, 0.727, 0.673. It falls monotonically in η, which is exactly the mechanism behind the 0.792 → 0.698 macro measurement. And the mean-TRS row rises: 0.600, 0.697, 0.760. η is a dial that trades one for the other, and the paper's sweep says the accuracy peak is halfway along.

Look at what η = 1 does to card E. It has the worst similarity of the five survivors, and it gets into the prompt purely because its TRS is not the worst. At η = 1 the ranking has stopped looking at the question — it is selecting globally good cards from whatever the filter happened to admit. That is the degenerate mode the paper's −5.8-point no-filter ablation is a stronger version of.

And look at what η = 0 does to card C: it gets a slot despite a TRS of exactly 0.50, which is the value of a card that has never been visited. Similarity-only retrieval cannot distinguish "reliably useless" from "completely unknown" from "reliably useful", because it does not look. That is MemP, and Chapter 9 shows MemP going negative on two atomic abilities.

Why the z-scores are clipped

The paper specifies "clipped z-score normalization", and the clipping is not cosmetic. Consider a candidate set where one card's TRS is a wild outlier — say four cards near 0.50 and one at 0.98 with three visits behind it. Its z-score could exceed 4 or 5, which after blending would dominate S no matter what the similarity term says, and it would take both remaining slots' worth of margin with it.

Clipping bounds any single term's contribution, so a blend at η = 0.5 really is a blend rather than a lexicographic sort with a tiebreak. It is the ranking-time analogue of what λ does at update time: both are mechanisms for refusing to let thin evidence become a strong assertion.

There is a second reason clipping matters here specifically. Candidate sets are small — often a handful of cards — and z-scores computed over five samples are noisy by construction. The standard deviation in our worked example was estimated from five points; with a different five it could easily halve, doubling every z-score. Clipping caps how much that estimation noise can propagate into the selection.

What retrieval costs

Concept and realisation: what does a query actually execute?

The entire retrieval stack is a dot product and a sort. There is no vector index, no ANN structure, no reranker model — at a few hundred to a few thousand cards, brute force is faster than anything clever. The dominant cost is the prefill on 500 extra prompt tokens, which against an image-bearing prompt at a 32,768-token budget is noise.

Compare that with route two from Chapter 0: running a depth estimator and a 3D reconstruction pipeline per question. The asymmetry in inference cost between "call a monocular depth model" and "do a 175×3072 dot product" is several orders of magnitude, and it is the practical reason the parameter-update-free route is interesting even where a tool agent would score higher.

Two-stage retrieval, in code

The other half of the realisation. Twenty lines, no dependencies beyond NumPy.

import numpy as np

DELTA = 0.618   # RoboSpatial. a percentile on YOUR question distribution
ETA   = 0.5     # sweep peak
K     = 3       # sweep peak

def z(x, clip=3.0):
    """Clipped z-score. Computed WITHIN the candidate set, not globally."""
    x = np.asarray(x, dtype=float)
    sd = x.std()
    if sd < 1e-9:                 # all candidates identical on this axis
        return np.zeros_like(x)     # → contributes nothing to the blend
    return np.clip((x - x.mean()) / sd, -clip, clip)

def retrieve(query_key, bank):
    # --- stage 1: semantic filter (a HARD cut — TRS cannot override it) ---
    rel  = np.array([float(query_key @ m.key) for m in bank])  # keys are unit-norm
    keep = np.flatnonzero(rel >= DELTA)
    if keep.size == 0:
        return []                       # memory block omitted entirely → No-memory behaviour

    # --- stage 2: combined ranking ---
    cand = [bank[i] for i in keep]
    S = (1 - ETA) * z(rel[keep]) + ETA * z([m.v for m in cand])
    order = np.argsort(-S)[:K]
    return [cand[i] for i in order]

Four details in that code are load-bearing, and three of them are easy to get wrong.

1. z is called on rel[keep], not rel. Normalising over the whole bank instead of the candidate set would make the similarity z-scores measure "how similar is this compared to everything, including the cards we already rejected" — which compresses the surviving candidates into a narrow band near the top and effectively kills the similarity term. Local normalisation is the point.

2. The zero-variance guard. If every candidate has the same TRS — which is exactly the situation early in pass 0, when every card sits at v₀ — the standard deviation is zero and a naive z-score is a division by zero or a NaN that silently poisons the sort. Returning zeros makes the TRS term contribute nothing and the ranking fall back to pure similarity, which is the correct behaviour when there is no reliability information yet.

3. The empty-candidate early return. It returns a list, not an error, and the caller omits the memory block. Chapter 7's floor property depends on this branch existing.

4. Keys are unit-norm, so cosine is a dot product. Normalise once at write time and retrieval is a single matrix-vector product. At a few hundred cards this is faster than any index you could build.

What the ranking cannot do

One structural gap worth naming. The top-k is computed independently per card — each candidate gets a score and the best three win. There is no term penalising redundancy within the selected set. If the three highest-scoring cards happen to carry the same lesson in three phrasings, all three go in the prompt.

Retrieval literature has standard answers to this — maximal marginal relevance, submodular selection, clustering before ranking. SMA uses none of them, and instead attacks the problem upstream: One-Pass Memory Writing exists partly so that near-duplicates never enter the bank in the first place. That is a legitimate design choice, and it makes Chapters 4 and 6 two halves of one argument. It is also the place a follow-up paper would most obviously push.

Where this sits in the retrieval literature

Strip away the spatial framing and Chapter 6 is a retrieval-ranking design. It is worth naming what it is and is not, because retrieval has a long history of two-stage systems.

The PageRank comparison is the most illuminating. Web search learned decades ago that pure query–document matching is exploitable and insufficient, and that blending in a query-independent quality signal — how often does anyone link to this page — is what makes ranking robust. TRS is the same move: how often did this card help, independent of what is being asked.

And the same caveat applies. A query-independent prior must be gated by relevance, or your highest-authority page gets returned for every query. That is precisely why the semantic filter is a hard cut and why removing it is the most expensive ablation in the paper. Chapter 6 rediscovers, in a memory-agent setting, a lesson search engines learned about ranking a long time ago.

What is genuinely new is where the quality signal comes from. PageRank harvests link structure that already exists. TRS has no such structure to harvest — there are no links between memory cards — so it manufactures the signal by using each card and watching what happens. The retrieval system generates its own authority scores by acting.

Chapter 7: Read-Only Deployment

Acquisition is over. The bank is written and calibrated. Now the agent meets problems it has never seen, and this chapter is about the contract it operates under — which is defined mostly by a list of things it is not allowed to do.

The three freezes

The paper: "Deployment does not write new memories and does not update any memory-value state: the visit count n_i, cumulative reward c_i, and TRS value v_i remain frozen even when a memory is retrieved."

The third freeze is the one that would be easy to skip, and it is the strictest.

Suppose TRS were updated during deployment. Updating requires a reward. A reward requires the verified target. So the system would be reading answer keys on the test set at inference time, and every deployment question would improve the ranking for the questions after it. Accuracy would then be a function of evaluation order, and the number would no longer mean "how well does this memory bank transfer" — it would mean "how well does this memory bank transfer, plus how much online supervision leaked in".

Freezing (n, c, v) closes that door completely. Deployment needs no targets at all — the pseudocode's deployment loop takes ξ_i = (𝒱_i, t_i), with no y^⋆ field — and every question is answered by a system in exactly the same state.

What one deployment call actually contains

Trace the tokens end to end for a single question.

Count the defensive instructions in that stack. "Treat these as procedural notes, not answer keys." "High task similarity does NOT license copying prior coordinates or Yes/No; the current image is always new." "Current image wins over any memory; never copy prior coordinates or Yes/No labels." "Extract at most one check or one trap per memory, then re-derive from the current image." "If no memory fits the shape, ignore them."

Five separate hedges against one failure mode. That density tells you something honest about the method: the biggest risk in in-context memory is not that the model ignores the memory, it is that the model over-trusts it and copies. The design is layered against this — anti-leakage rules at write time, the hidden-output line at render time, and these instructions at read time.

A full deployment trace, end to end

Walk one RoboSpatial compatibility question all the way through, naming every object as it appears. This is the whole system, once.

Step 10 is the whole chapter. In acquisition, step 10 was four separate mutations. In deployment it is a no-op.

And notice step 1: the deployment tuple genuinely has no y^⋆ in the pseudocode — for all ξi = (𝒱i, ti) ∈ 𝒟. The target exists in the benchmark, of course, and is used afterwards to compute the reported accuracy. But it is not available to the system while the system is answering, which is the difference between evaluating a model and training one.

Where the guidance sits relative to the image

One placement detail with a real consequence. The paper says the guidance set "is prepended to the current user prompt" and that the composed memory context "is prepended to the current benchmark task". So the order is: system prompt, then memory block, then the task with its image.

Prepending rather than appending means the model reads the procedures before it reads the question and looks at the image. For a reasoning model that produces its chain of thought after ingesting the full prompt, this ordering primes the approach rather than second-guessing a conclusion already forming. It also means the memory block is at a stable position, which matters for prefix caching — though with three cards varying per query, the memory block itself is not cacheable across questions.

The alternative — task first, memory after — would put the advice in the position of a correction. Given how hard the prompt works to prevent the model from treating memories as answers, arriving before the question is the safer position: a procedure you read before seeing the problem is a lens; one you read after is a hint.

The empty-candidate case

Worth stating because it is a real branch: if no card clears δ, "this block is omitted". The model gets the plain benchmark prompt, exactly as in the No-memory baseline.

This is a good property. It means the floor of SMA on any individual question is the no-memory behaviour, not something worse. A question sufficiently unlike anything in the environment split simply gets answered without help, rather than being handed an irrelevant lesson. It also explains part of why the whole-benchmark gains look modest on saturated benchmarks — on EmbSpatial, where no-memory is already 85.7%, most questions have little room to move and some get no memory at all.

The composed prompt, in full

Here is the whole thing, assembled, for a RoboSpatial compatibility question. Fixed text is quoted from the paper's templates; runtime-filled fields are marked.

══ SYSTEM ══════════════════════════════════════════════════
You solve RoboSpatial-Home open-answer questions from a single indoor RGB image.
…
Spatial-intelligence question families (recognize the *structural shape*…):
 1) Vacant-region localization via normalized pointing
 2) Object-object spatial configuration verification
 3) Placement compatibility and affordance
Cross-cutting checks:
 - Image-plane directions ("left of") follow the camera view unless stated.
 - For pointing: spread several valid points; coordinates in [0, 1].
 - For Yes/No: inspect *both* named objects and the exact relation phrase.
Reasoning protocol (apply silently; do not narrate the protocol):
 1. Single-image grounding.   2. Shape-specific validation.   3. Decision discipline.
Input / output constraints:
 - Yes/No output: exactly `<answer>Yes</answer>` or `<answer>No</answer>`
 - Pointing output: exactly `<answer>[(x1, y1), …]</answer>`, floats in [0, 1]

══ MEMORY CONTEXT (omitted entirely if 𝒞ᵢ is empty) ════════
Relevant memories from prior RoboSpatial rollouts (different images):
Treat these as procedural notes, not answer keys.
 - Match each memory to the *structural shape*… not to similar object nouns alone.
 - High task similarity does NOT license copying prior coordinates or Yes/No.
 - Extract at most one check or one trap per memory, then re-derive.

[Memory 1] task_similarity=0.741            ← runtime
 - prior_task_shape:     {task}              ← runtime
 - transferable_lesson:  {transferable_lesson} ← runtime
 - abstract_summary:     {summary}           ← runtime
 - prior_model_output:   [hidden; do not reuse prior coordinates, Yes/No, or wording]
[Memory 2] …
[Memory 3] …

Memory-use (silent):
 - Pointing: turn memories into checks — anchor object, correct side, multiple
   points in vacant pixels, stay in [0, 1].
 - Configuration: bind both objects, test the stated relation verb against pixels.
 - Compatibility: estimate clearance and obstacles before Yes/No.
 - If no memory fits the shape, ignore them.
 - Re-derive the answer from the current image only.
 - End with exactly one line: `<answer>…</answer>`

══ USER ════════════════════════════════════════════════════
<image>
Can the storage bin fit to the right of the desk? Answer yes or no.

Three structural observations you can only make with the whole thing in view.

The memory context is sandwiched, not appended. It sits after the system prompt and before the image. So the model reads the general procedure, then the specific procedures, then the problem — general to specific to concrete, which is the order a human would want.

The system prompt and the memory block say overlapping things. "Inspect both named objects and the exact relation phrase" is in the system prompt; "bind both objects, test the stated relation verb against pixels" is in the memory footer. That redundancy is deliberate. The system prompt is the general scaffold present in every condition including No-memory; the memory block adds specifics learned from experience. The reported gains are what the specifics buy on top of a good general prompt.

The three hardest instructions all appear in the memory block, not the system prompt. "Not answer keys", "does NOT license copying", "re-derive from the current image only". None of these are needed when there is no memory. Their presence is a map of the risks memory introduces.

The three states of the system

It is worth laying the whole lifecycle out as a state machine, because "acquisition" is really two states wearing one name and conflating them causes confusion.

State C is the only one you could run in production, because it is the only one that does not need answers. States A and B are an offline preparation phase — the analogue of training, executed entirely with forward passes.

Read the "Model calls" row and the economics become clear. Writing costs two calls per problem. Calibrating costs one. Deploying costs one, forever. So the entire investment is (2 + (T−1)) × |𝒳| calls, amortised over every deployment query thereafter. For RoboSpatial at T = 6 that is 7 × 175 = 1,225 calls, once. Every deployment query after that costs exactly what a No-memory query costs, plus ~450 prompt tokens.

The engineering budget, honestly

If you were putting this in front of a robot, what would you actually be paying?

The one row worth flagging is decode length. Prompting a model to apply an explicit check-and-validate procedure plausibly lengthens its chain of thought, and at temperature 0 with a 32,768-token budget there is room for that to be substantial. The paper reports accuracy but not latency or token counts, so the true inference-time cost of SMA relative to No-memory is not established. It is almost certainly far below a depth-estimation pipeline, but "almost certainly" is doing work there.

Why temperature zero matters more than it looks

Every run uses temperature 0 and top-p 1 — greedy decoding. For the headline results this is standard practice. But it interacts with the memory mechanism in a specific way worth noticing.

Under greedy decoding, a given (image, question, guidance) triple maps to exactly one output. So the reward attributed to a guidance set is not an average over samples — it is a single deterministic outcome. All the stochasticity in the TRS estimate comes from which problems a card gets retrieved for, not from sampling noise in the model.

That is why the visit count n is a meaningful sample size: each visit is an independent draw from the distribution of problems that retrieve this card, and the reward is a deterministic function of that draw. It is also why the ten-pass repetition is about coverage rather than variance reduction — re-running the same problem with the same bank state would return the same answer.

Threats to the read-only claim, checked one by one

"Read-only" is a strong claim, and the value of the whole paper depends on it holding. Here is every route by which deployment information could leak backwards, and what closes it.

Three of the six are fully closed by construction; three are prompt-enforced or protocol-level and carry residual risk. That is a normal profile for an empirical paper, and the authors are transparent about the checkpoint rule. The one to keep in mind when reading Chapter 8 is that the SMA rows are best-of-ten calibration states while the baselines are reported at whatever their own protocols specify — the comparison is fair in that the same rule is applied to all SMA rows, but it is not a blind single-shot number.

What happens when the memory is wrong

A question the design has to answer: what is the damage when a retrieved card is confidently irrelevant?

Chapter 8's delta table gives the empirical answer — across all four base models and five benchmarks, SMA never falls below the no-memory baseline in any cell. So the layered defences are, in aggregate, sufficient. But it is worth naming what each layer is holding back, because they are the reason.

Compare this with RAG, which has layer 2 and nothing else — and which goes negative on two of four base-model blocks. The layering is the difference between a mechanism that helps on average and one that never hurts.

What "best checkpoint" means, and why to flag it

One methodological detail belongs here rather than buried in Chapter 8. The main table reports, for SMA, "the best checkpoint from the 10-pass One-Pass Memory Writing run", with "the same selection rule applied to all SMA rows".

A "checkpoint" here is a state of the memory bank after some number of passes — the cards are fixed after pass 0, so what varies across checkpoints is the calibration state (n, c, v), and therefore the ranking. Selecting the best one is a legitimate hyperparameter choice, consistently applied. It is also a choice that consumes some of the evaluation signal, and the honest framing is: the reported number is the best of ten calibration states, not the state you would get by fixing the pass count blindly. The per-benchmark pass counts in the hyperparameter table (RoboSpatial 6, ERQA 2, Omni3D 10, SAT 9, EmbSpatial 4) are the outcome of that selection.

What would have to change to run this on a robot

Everything so far is a benchmark protocol. The paper's motivation is embodied agents, so it is worth being explicit about the gap between "held-out deployment split" and "in the aisle".

One row is already solved and it is the important one. The hardest thing about deploying a learned system is usually that learning needs supervision and deployment does not have any. SMA's read-only phase is designed for exactly that asymmetry: acquisition happens where verifiers exist, deployment happens where they do not, and nothing in the deployment path needs a reward signal.

The remaining rows are honest open work, and the paper's limitations section names two of them (drift and lifecycle under D.2, credit assignment under D.1). The single-turn constraint is not named as a limitation, and it is arguably the largest gap between the demonstrated system and the motivating application.

What "no verifier at deployment" buys you

Worth dwelling on, because it distinguishes this from the test-time-training literature it superficially resembles.

A system that updates at test time — test-time training, online RL, adaptive prompting with feedback — needs a reward at test time. Sometimes you have one: a compiler, a unit test, a simulator. Often you do not: there is no oracle standing behind a robot telling it whether the bin actually fits. Systems in that second category either fake a reward (self-consistency, model-as-judge, confidence heuristics) or do not adapt.

SMA takes the third option: do all the adapting where rewards are free, then stop. The environment split is a place where rewards are free, because it is a labelled dataset. Deployment is a place where they are not, so deployment does not use them. The system is not "online learning with a weak reward"; it is "offline learning, then a fixed artefact".

That is a less exciting position than continual adaptation, and it is a far more shippable one. It also means the failure mode is legible: if the deployment distribution drifts away from the environment split, performance degrades quietly rather than adapting wrongly. A frozen artefact fails predictably.

Chapter 8: The Numbers, Honestly

Twenty evaluations: five benchmarks by four frozen base models, six methods each. Here is the whole table, and then the parts of it that are less flattering than the abstract.

The main table

Accuracy (%), measured on the held-out deployment split. Avg. is the macro average over the five columns.

The three readings that hold up

1. SMA has the best macro average in all four blocks. 68.8, 66.7, 69.8, 63.5. And the margins over the strongest non-SMA baseline in each block are +2.6 (over MemRL-R's 66.2), +2.9 (over MemRL-R's 63.8), +1.7 (over MemRL-GT's 68.1), +2.8 (over MemP's 60.7). Note the strongest baseline is a different method in three of the four blocks — there is no single runner-up.

2. Gains are not concentrated in one benchmark type. On Qwen3.6-27B: RoboSpatial 54.1 → 68.5 (+14.4), Omni3D 41.6 → 47.6 (+6.0), EmbSpatial 85.7 → 87.9 (+2.2). Open pointing, open 3D reasoning, and closed relation labels all move up.

3. Memory is not automatically good. This is the most useful thing in the table and it comes from the baseline rows. Count the cells where a memory method is worse than no memory at all: RAG loses on the 122B block (64.2 vs 65.3) and on the 9B block (59.2 vs 60.6). MemRL-R loses on the 9B block (58.7 vs 60.6). MemRL-GT loses on the 9B block (60.0 vs 60.6). On Qwen3.5-9B, every single baseline is at or below the no-memory average, and only SMA beats it.

Where SMA does not win

The abstract says "best accuracy among the evaluated methods in most of the 20 evaluations". Here are the cells it does not win, and the appendix block where it loses outright.

In the main table. On Qwen3.6-27B SAT, SMA and MemRL-GT tie at 87.0. On Qwen3.5-9B RoboSpatial, SMA's 58.5 is only 0.4 above no-memory's 58.1 — and no-memory beats every other baseline on that cell. On Qwen3.5-9B ERQA, SMA's 52.0 is below MemP's 53.0. So "most of 20" is doing real work in that sentence.

In the appendix. Table 7 evaluates two further benchmarks, SITE-image and ViewSpatial, on all four models:

On the Qwen3.5-122B-A10B block, MemRL-GT beats SMA on both the ViewSpatial column (62.8 vs 59.7) and the two-benchmark average (66.9 vs 65.6). That is a clean loss, reported by the authors, in their own appendix. On the 27B block MemRL-GT also takes ViewSpatial (64.3 vs 63.2) though SMA takes the average.

ViewSpatial is the benchmark about multi-perspective localisation — camera-relative direction, person-relative direction, object orientation, scene simulation. It is the one where the same scene is shown from several views. A plausible reading is that perspective-transformation questions reward a strong reflection signal (which MemRL-GT has, since it also gets ground truth) more than they reward reliability filtering — but the paper does not analyse this and neither should we pretend to. The honest statement is: on ViewSpatial with a large base model, ground-truth-guided reflection without TRS was better, twice.

The training-based comparison, scoped correctly

Table 3 compares SMA on Qwen3.5-9B against SpatialEvo-7B, a public training-based self-evolving spatial baseline.

A sixteen-point macro average gap, higher on every benchmark. It is the paper's most dramatic table and also the one to be most careful with, and the authors are: they note SpatialEvo was chosen because it "releases a public 7B baseline", and conclude only that "an external procedure-memory route can be competitive with training-based spatial self-evolution under this evaluation scope."

Three reasons to keep that hedge. The base models differ (7B versus 9B, different families, different pretraining). The evaluation is the paper's own splits, not SpatialEvo's reported protocol. And it is a single training-based system — SpatialEvo, SAGE, and AtlasVA are all named in the related work, but only one had a public checkpoint to run. "Competitive with one public 7B training-based baseline on our splits" is what the table supports.

"Most of the 20 evaluations", counted

The abstract's central claim deserves an exact tally rather than a vibe. Twenty evaluations means five benchmarks × four base models. For each cell, does SMA hold the highest accuracy among the six methods?

18 outright wins, 1 tie, 1 loss. "Most of the 20" is an understatement, and it is interesting that the authors chose the weaker phrasing when 18/20 was available. The likely reason is the appendix: extend to seven benchmarks and the 122B block loses ViewSpatial to MemRL-GT and the 27B block loses it too, taking the tally to 20 wins, 1 tie, 3 losses out of 28. "Most" is accurate under either counting, and safe under both.

Now do the same for the block averages, which is the claim the abstract makes separately: "the highest macro average in every base-model block." 68.8 > 66.2, 66.7 > 63.8, 69.8 > 68.1, 63.5 > 60.7. Four for four in the main table. In the appendix's two-benchmark average, SMA takes three of four blocks — losing the 122B to MemRL-GT, 65.6 against 66.9.

What is missing from the reporting

Since we are being honest about the numbers, here is what a reader cannot check.

None of these undermine the main result — the effect sizes on RoboSpatial and Omni3D are far too large to be noise, the ablations are directionally consistent across two benchmarks, and the transfer probes are all positive. They do mean that the narrower margins should be read as suggestive rather than settled, and that the single-family evaluation is the most obvious limit on how far to generalise.

Explore the results

Everything as a delta against no memory

Absolute accuracies make it hard to see who is actually helping. Subtract each block's No-memory row and the picture sharpens considerably. Negative cells are memory methods that made the frozen model worse.

Count the negative cells. Across the four blocks and the five benchmarks, the four baselines fill 80 cells between them, and 28 of those 80 are negative — more than a third. SMA fills 20 cells and none of them is negative. Not one cell in the main table where SMA is worse than doing nothing.

The distribution of those 28 is itself informative: 8 on the 122B block, 5 on the 35B, only 2 on the 27B, and 13 on the 9B — nearly half of all the damage lands on the smallest model, which is the one least able to weigh a suspect instruction against the evidence in front of it.

That is a different and arguably better claim than any margin. "Never hurts" is a property; "wins by 2.6 on average" is a measurement. And it is the property you actually want from a memory system you are about to deploy, because a method that helps by six points on average while occasionally costing five is much harder to ship than one that helps by three and never costs anything.

Cross-checking the tables against each other

A paper's tables should agree. When they do, it is weak evidence the pipeline is sound; when they do not, it is worth knowing. Four checks are available here without any extra experiments.

Seven for seven. Every cross-reference resolves, including the similarity analysis, which quietly confirms that the MemP baseline used for the 0.792 → 0.698 measurement is the same MemP run reported in the main table. That kind of consistency is not proof of anything, but its absence would be a red flag and it is worth having checked.

One arithmetic check you can also do yourself. The paper claims average gains over the strongest non-SMA baseline of 2.6, 2.9, 1.7, and 2.8. Take the 27B block: the best non-SMA average is MemRL-GT at 68.1, and 69.8 − 68.1 = 1.7. Correct. And the 35B: best non-SMA is MemRL-R at 63.8, and 66.7 − 63.8 = 2.9. Correct. Note that the runner-up is a different method in three of the four blocks — MemRL-R, MemRL-R, MemRL-GT, MemP — which means no single baseline is consistently second, and SMA's margin is measured against a moving target each time.

Reading the benchmarks against each other

The per-benchmark spread says something the macro average hides.

EmbSpatial is where every method clusters within about three points, and it drags the macro average toward flatness. If you wanted to make SMA look better you would drop it; if you wanted to make it look worse you would weight it higher. Reporting it as one of five equally-weighted columns is the neutral choice.

What the appendix benchmarks add

SITE-image and ViewSpatial are relegated to an appendix, and the temptation is to skip them. They are the two most informative tables in the paper for anyone deciding whether to build this, because they are where the method is stressed.

Compute the deltas over no-memory the way we did for the main table:

Look at the ViewSpatial column. Every memory method except RAG produces a huge gain — MemRL-GT reaches +15.4 on the 27B, SMA +14.3. No-memory sits in the high 40s on a benchmark where memory methods reach the low 60s. This is the benchmark where having procedural memory at all matters most, and where the difference between memory methods matters least.

That reframes SMA's two losses here. It is not that TRS fails on multi-perspective reasoning; it is that on a benchmark with 15 points of headroom for any reflected procedure, the marginal value of reliability filtering is small, and a one-point difference between two strong methods is inside the noise of a single split.

Meanwhile on SITE-image, SMA takes the column on all four models — 71.4, 74.0, 69.5, 63.9. Four for four, including beating MemRL-GT everywhere. So across the two appendix benchmarks, SMA wins SITE 4/4 and loses ViewSpatial 2/4, taking the two-benchmark average on 3 of 4 models.

What the macro average hides

Every headline number in this paper is a macro average over benchmark columns. That choice deserves scrutiny because it drives the four-for-four claim.

It equalises benchmarks of wildly different size. RoboSpatial contributes 175 deployment items and EmbSpatial contributes 1,820, weighted identically. A question-weighted average would be dominated by EmbSpatial, where every method scores between 81.8 and 87.9 and the whole spread is six points. Under question weighting, SMA's margins would shrink dramatically — not because the method got worse, but because the benchmark with the least headroom would carry most of the weight.

It equalises benchmarks of wildly different difficulty. Omni3D's ceiling in this table is 47.6; EmbSpatial's floor is 81.8. Averaging a 47 and an 88 produces a number that describes neither. The delta table earlier in this chapter is a more honest summary for that reason: it is scale-free per column.

But it is the right choice given the goal. The paper's claim is about spatial reasoning across settings, and each benchmark is one setting. Weighting by question count would say that whichever benchmark happened to release the most items is the most important kind of spatial reasoning, which is clearly wrong. Macro-averaging is standard, defensible, and stated plainly — and the per-column numbers are all published, so a reader can reweight however they like.

Chapter 9: What the Ablations Prove

Main tables tell you a system works. Ablations tell you why, and they are where a paper is most likely to overclaim. This chapter goes through every one, states what it measures, and states what it does and does not license.

Component ablations, on two benchmarks

Both on Qwen3.6-27B. Remove or add one component at a time.

Four things fall out of reading both columns together rather than either alone.

The semantic filter is the most valuable component on both benchmarks. −5.8 and −7.2, larger than any other single change. This is the strongest quantitative argument in the paper for the two-stage design: without a relevance gate, TRS becomes a global popularity score and injects well-performing but topically wrong lessons everywhere.

The lesson field matters more on the harder benchmark; the summary matters less. On RoboSpatial the two are near-equal (−3.5, −3.2). On Omni3D they diverge sharply (−5.2, −1.6). Omni3D is open-answer 3D reasoning — metric estimates, occlusion, containment, counterfactual placement — where an explicit procedure is doing real work, and abstract framing is nearly redundant with the system prompt.

Adding the raw model output hurts on both, and hurts more where answers are constrained. −4.4 on RoboSpatial versus −2.0 on Omni3D. That direction is consistent with the copying story: RoboSpatial's answer space is Yes/No and coordinate lists, which are short, concrete, and eminently copyable. Omni3D's open answers are more varied and copy less cleanly.

Reflection signal quality is worth more on RoboSpatial than on Omni3D. −5.5 versus −2.8. Where the answer is Yes/No, the reward alone tells you almost nothing about the failure mode — you know you said the wrong one of two things. Where the answer is a number or a phrase, even a scalar reward is somewhat more diagnostic.

The hyperparameter sweeps

Swept on RoboSpatial with Qwen3.6-27B: the reliability weight η and the retrieval depth k. The paper reports the sweep "peaks at η = 0.5 and k = 3", which are the values used throughout.

η = 0.5 sitting at the peak is the whole thesis restated as a hyperparameter. η = 0 is MemP — similarity only. η = 1 is TRS only, ranking a filtered candidate set purely by global track record with no regard for which candidate best matches this question. That the optimum is exactly in the middle says the two signals are close to equally informative and largely complementary.

k = 3 is a budget the paper does not dramatise but is worth noting: three cards is roughly three sentences of summary plus three of lesson. A tiny prompt addition. The gains in Chapter 8 are bought with a handful of sentences.

Reading the two ablation columns as a single experiment

The paper runs the same six-row ablation on two benchmarks. Most readers will look at one column. Putting them side by side is where the information is, because the ordering changes and the ordering is diagnostic.

Only the first and last positions agree. Everything in between reorders, and the reordering tracks the answer space.

RoboSpatial's answers are Yes/No and coordinates. Short, enumerable, copyable. So the two ablations that touch what the model is tempted to copy or guess — adding the prior output, and degrading the reflection signal so lessons become vaguer — do the most damage after the filter. The lesson field matters less because RoboSpatial's system prompt already carries a detailed family taxonomy and cross-cutting checks; the card is adding to an already-strong scaffold.

Omni3D's answers are open. A number, a Yes/No, or a short phrase, about metric extent, occlusion, containment, or counterfactual placement. There is much less to copy, so the prior-output ablation costs only 2.0. But there is far more procedure required, and no strong per-family scaffold in the prompt to lean on — so removing the lesson field costs 5.2, more than anything except the filter.

One more thing the paired columns establish: the filter is first in both, by a clear margin, on benchmarks that otherwise disagree about everything. That is the most robust single finding in the ablation set, and it is an argument about retrieval design rather than about spatial reasoning — relevance gating before value ranking is not a spatial insight, it is a retrieval one.

Does TRS actually predict anything?

This is the load-bearing question — TRS is only worth computing if it correlates with downstream success. The paper measures it three ways.

1. Binned by mean retrieved TRS. Bin deployment questions by the mean TRS of their retrieved memories, then measure accuracy per bin. Accuracy rises from 19.3% in the [0.2, 0.3) bin to 97.3% in the [0.9, 1.0] bin.

That is a very large spread, and the paper immediately hedges it: the trend "may also reflect differences in benchmark difficulty and question composition". Take the hedge seriously, because it is the right one. TRS is built from rewards. Easy questions produce high rewards, so cards retrieved for easy questions accumulate high TRS. Then at deployment, high-TRS cards are retrieved for — easy questions. Part of that 19.3-to-97.3 sweep is a difficulty gradient dressed as a reliability gradient. It is a pooled correlation, not a controlled comparison.

2. By source-question outcome. Stratify Qwen3.6-27B memories by whether the environment rollout that wrote them was correct.

A 24.3-point accuracy gap. This is a genuinely interesting result to sit with, because it complicates Chapter 5. TRS was deliberately initialised uniformly so that source correctness would not bias the score — and then visit evidence discovered, on its own, that success-sourced memories are in fact better. The mechanism was not told; it found out.

Note the TRS gap is small (0.522 vs 0.452, both close to v₀ = 0.5) while the accuracy gap is large. So source outcome is a real signal that TRS partially recovers, but TRS is not a clean sorter of it. There is genuine headroom here for something better — which is more or less what the credit-assignment limitation says.

3. By retrieval composition. Group deployment questions by the source outcomes of all three retrieved memories.

Monotone in both accuracy and TRS. And look at the counts: 603 all-failure retrievals out of 24,093 total — 2.5%. The ranking has already pushed failure-sourced cards down; they rarely fill all three slots. The mechanism is doing quiet triage before this table is even computed.

The same difficulty confound applies to this table as to the bins. It is consistent evidence, not clean evidence, and the paper's own framing — "confirming that memories from successful questions transfer more effectively" — is about as strong as the data supports.

Does the bank transfer off its home turf?

The strongest generalisation result in the paper, and the one with the clearest practical consequence. Two probes: write a bank with one model and read it with another; or write on one benchmark and read on a different one.

Every probe is positive. Two things follow.

Model transfer is what makes this deployable. Look at RoboSpatial: a bank written by the 122B model gives the 27B model +9.4 points. That is 65% of the +14.4 the 27B gets from its own bank. So a memory bank is an asset that outlives the model that wrote it — the thing that is emphatically not true of a fine-tuned checkpoint. Upgrade the base model and your lessons come with you. It also opens an obvious recipe the paper does not pursue: write the bank with the strongest model you can afford, once, then deploy it on a cheap one.

Benchmark transfer proves the lessons are procedures. EmbSpatial → RoboSpatial gives +7.3. EmbSpatial is closed-set relation labels; RoboSpatial is open-answer pointing and compatibility on different images with a different answer format. Nothing scene-specific or format-specific could survive that crossing. What transfers has to be procedural — bind both referents before judging a relation, read the relation verb literally, check the image-plane convention. Exactly the content the reflection grammar was built to produce.

The paper's hedge: "the magnitude still depends on source–target similarity". True and visible in the table — the gains into EmbSpatial and Omni3D are much smaller than the gains into RoboSpatial. But RoboSpatial also has the most headroom, so target difficulty and source–target distance are entangled here.

Atomic spatial abilities

The finest-grained analysis. Ten capability labels, applied post-hoc to benchmark sub-categories as diagnostic annotations rather than runtime inputs. Each question can carry several labels, since the labels are multi-label and non-disjoint. Gains are averaged across four base models over the common RoboSpatial / ERQA / SAT / EmbSpatial scope. Omni3D is excluded because its released annotations expose answer_type (float, int, str) rather than a spatial taxonomy.

SMA improves all ten on average. The pattern in the extremes is coherent with everything else in the paper.

Correspondence tops the list at +11.2, and correspondence is the most procedural of the ten — "bind the named entity to the right pixels before answering" is a checkable habit, exactly what the lesson grammar produces. Distance/depth is near the bottom at +2.6, and that is the ability least amenable to advice: no sentence tells you how many centimetres of clearance are in the image. You either read metric extent from pixels or you do not. The gap between +11.2 and +2.6 is the boundary between "procedure helps" and "perception is the bottleneck".

And the MemP numbers are the sharpest single data point in the chapter. MemP is negative on Tracking (−3.0) and Affordance (−1.9) — the paper's reading is that "unfiltered procedural memory can hurt when retrieved experience does not transfer". Same reflection machinery, same lesson grammar, no reliability filter, and two abilities get worse. The filter and the ranking are what turn a mechanism that sometimes hurts into one that helps everywhere.

Putting the two TRS tables together

Tables 5 and 6 look like the same measurement twice. They are not, and the difference is worth extracting.

Table 5 groups memories by their own source outcome and reports the mean TRS each group reached. The gap is small: 0.522 versus 0.452, both hugging the prior. Table 6 groups evaluation questions by the composition of their three retrieved cards and reports mean TRS. That gap is enormous: 0.909 versus 0.480.

Why the difference? Because the second table is conditioned on having been selected, and selection is a strong filter. A question whose three retrieved cards are all success-sourced did not get those cards by accident — they won a combined ranking in which TRS carries half the weight. So "all success" and "high TRS" are two names for largely the same event, and Table 6's monotone TRS column is partly the ranking rule described back to itself.

Table 5 is the cleaner measurement of the two, because it conditions only on how a card was written, not on whether it was chosen. And it says the effect is real but modest — a 0.07 TRS separation. Read together, the honest summary is: source outcome predicts transfer; TRS recovers part of that signal from visit evidence alone, without ever being told.

The counts tell a fourth story. 13,226 all-success retrievals, 10,264 mixed, 603 all-failure. That is 24,093 evaluation questions total across five benchmarks — consistent with the deployment split sizes (175 + 200 + 250 + 300 + 1,820 = 2,745 per pass, times the multi-pass evaluation). And all-failure retrievals are 2.5% of the total. Given that failure-sourced cards are roughly half the bank by construction, an all-failure triple would occur about 12.5% of the time under random selection. Observing 2.5% means the ranking is suppressing them by a factor of five before the analysis begins.

What a controlled version of the TRS analysis would look like

Since the paper names its own confound, it is worth being concrete about what would remove it — both because it sharpens what the current evidence means, and because it is the obvious follow-up.

None of this means the TRS result is wrong. It means the reported 19.3 → 97.3 sweep is an upper bound on the effect size, and the true causal contribution of TRS is somewhere below it. The ablation evidence — η sweeping to a peak at 0.5, MemP trailing SMA by 3.0 macro points, the similarity-down-accuracy-up pairing — is the causally cleaner support, and it points the same way.

The atomic abilities, read as a boundary

Ten abilities, all improved on average, with a spread from +11.2 to +2.6. That spread is more informative than the fact that all ten are positive, because it marks where the method stops working.

Sort the named numbers by what kind of cognitive work each ability requires.

The gradient is monotone in "how much of this is a step you can be reminded to take". That is the clearest empirical statement in the paper of what procedural memory is for, and it is also the ceiling: memory converts available perception into correct answers more reliably. It does not create perception.

The paper's failure cases say the same thing from the other side. Three cases where "both the baseline and SMA fail despite high-quality, semantically relevant retrieved memories (TRS ≥ 0.6)", with responses that "attempt to apply the retrieved procedure, but still misread critical visual evidence such as movement direction, object count, or spatial connectivity." The procedure was retrieved. The procedure was applied. The pixels were read wrong. Nothing in the mechanism can help.

MemP going negative is the most important baseline number

Buried in the atomic-ability discussion is a result that deserves top billing: MemP has negative gains on Tracking (−3.0 pp) and Affordance (−1.9 pp).

Sit with what MemP is. It reflects rollouts into summaries and transferable lessons — the same card format, produced by the same kind of reflection. It retrieves them by semantic similarity. The only thing it lacks relative to SMA is the reliability layer.

And on two of ten abilities, that identical card format, retrieved by similarity alone, makes the model worse than having no memory. The paper's reading: "unfiltered procedural memory can hurt when retrieved experience does not transfer."

So the claim SMA is actually supporting is not "procedural memory helps". Procedural memory sometimes hurts, measurably, on the same benchmarks with the same cards. The claim is narrower and more interesting: procedural memory helps once you can tell which procedures have earned their place. Every other result in the paper — the η = 0.5 peak, the similarity-down-accuracy-up pairing, the 28 negative baseline cells against SMA's zero — is a restatement of that same point from a different angle.

It also explains why the paper bothers with the No-memory baseline at every cell rather than only reporting method-vs-method comparisons. Without that column you could not see that several memory methods are net harmful, and the contribution would look like a margin rather than a qualitative difference.

The five findings, and what each one is worth

What model transfer actually means operationally

Of the nine transfer probes, the one with the clearest practical consequence is the 122B → 27B row on RoboSpatial: 54.1 → 63.5, a gain of +9.4. Compare that with the 27B's gain from its own bank: 54.1 → 68.5, +14.4.

Now count what that saves. Writing the 27B's own bank costs, from Chapter 2's accounting, roughly 1,225 calls to the 27B. Writing the 122B's bank costs 1,225 calls to a much larger model — more expensive per call, but it is a one-time cost that can then be amortised across every model you deploy on.

The fourth row is the one the paper does not run and the one an engineering team would care most about. If a single bank written once by your strongest model retains most of its value on every smaller model in your fleet, then procedural memory becomes infrastructure rather than a per-model artefact. The single 122B → 27B probe is suggestive of that, not proof of it.

Note also the direction that was tested: large to small. The reverse — a bank written by a 9B model and deployed on a 122B — is not reported. It is the more interesting question in some ways (can a cheap model generate lessons that help an expensive one?) and its absence is worth noticing.

Why benchmark transfer is the stronger evidence

Model transfer could, in principle, be explained without any procedural content: two Qwen models share a tokenizer, a pretraining lineage, and probably many failure modes, so lessons diagnosing one might transfer for uninteresting reasons.

Benchmark transfer has no such escape. Consider EmbSpatial → RoboSpatial, +7.3 on the 27B:

Nothing scene-specific, format-specific, or answer-specific can survive that crossing. A card that leaked an answer letter is useless. A card that fingerprinted a room is useless. A card that encoded a base rate over EmbSpatial's six labels is worse than useless. The only content that can produce +7.3 is content about how to look — bind both referents before judging a relation, read the relation verb literally, check which coordinate frame the question means.

So the transfer table doubles as a leakage audit. If the anti-leakage rules from Chapter 3 were failing at scale, cross-benchmark transfer would be near zero or negative. It is positive on every probe. That is indirect evidence, but it is the best evidence the paper offers that the cards contain procedures rather than answers.

Chapter 10: Limits & Lineage

The paper devotes an appendix to its own weaknesses and a set of figures to its failures. Both are unusually candid and both are worth reading before you decide what to build on top of this.

Limitation 1 — credit assignment

Named in Chapter 5, stated properly here. When an answer improves or fails, the framework "cannot precisely determine whether the outcome should be attributed to memory writing, reflection, retrieval, semantic filtering, or the model's final use of the retrieved memory."

The paper notes this is especially acute for spatial reasoning, "where a successful answer may depend on several coupled operations, such as identifying the target object, estimating scale, transforming viewpoints, and applying a previously learned placement rule". A single scalar reward at the end of that chain is a very blunt instrument for deciding which of three cards deserved it.

Three lines of work are named as the fix. AttriMem (Li et al. 2026b) introduces attribution-guided process feedback to assign localised rewards to memory construction rather than relying on outcome-level signals. Memory-R2 (Yan et al. 2026) uses local rerollouts and a global objective to distinguish the effects of memory operations across sessions. MemQ (Liao et al. 2026) models memory dependencies with provenance DAGs and propagates value through memory-generation chains, so credit can reach earlier memories that indirectly supported later decisions.

The local-rerollout idea is the most directly applicable here: answer the same problem with and without a given card and difference the rewards. Expensive — k times as many model calls — but it would turn the TRS update from "was present when the answer was right" into "changed the answer for the better", which is the quantity you actually wanted all along.

Limitation 2 — long-term memory maintenance

SMA writes, retrieves, and reliability-weights. It does not curate. The paper: "As the memory bank grows, old lessons may become redundant, partially conflicting, overly specific to early environment examples, or stale with respect to later deployment distributions… SMA does not explicitly decide when to delete, merge, compress, expire, or rewrite memories under a storage or latency budget."

TRS can downweight an unreliable card, and the filter can exclude an irrelevant one, but neither ever removes anything. A bad card sits in the bank forever, costing an embedding comparison on every query. And two cards giving contradictory advice can both be retrieved for the same question, with nothing to detect the conflict.

Named prior work again: MemRefine (Kim et al. 2026) studies storage-budgeted long-term memory with LLM-guided decisions to delete, merge, or preserve. TRUSTMEM (Yang et al. 2026) focuses on trustworthy consolidation, with the sharp observation that write, revise, and delete operations "can themselves introduce omission, corruption, or hallucinated persistent state". memorywire (Munirathinam 2026) standardises remember / recall / forget / merge / expire as an interface-level concern.

The TRUSTMEM point deserves a beat: a memory system that edits itself can corrupt itself. SMA's write-once design is unsophisticated, and that is also the reason it cannot degrade — the bank after pass 9 contains exactly the sentences written on pass 0. Whatever else is true, nothing rotted.

The failure cases, split into two kinds

The qualitative appendix separates failures deliberately, and the separation is the useful part.

Benchmark-side ambiguity (five cases). Questions that are "underspecified, depend on unavailable metric or depth information, contain only partial visual evidence, or rely on hidden simulator geometry". These are cases where the observable RGB evidence does not determine a single answer — no memory system can fix them, because there is nothing to reason from.

Base-model visual grounding limits (three cases). Both baseline and SMA fail despite relevant retrieved memories with TRS ≥ 0.6. "The responses attempt to apply the retrieved procedure, but still misread critical visual evidence such as movement direction, object count, or spatial connectivity."

That second category is the boundary of the whole method, stated by the authors in one line: "memory can guide the reasoning process but cannot replace accurate visual grounding."

Should you build this? A decision table

The most useful thing to take from a systems paper is a rule for when it applies. Score your task on these six rows.

Roughly: verifiable, procedural, shaped, with headroom. Four conditions. Spatial reasoning happens to satisfy all four, which is why the paper's gains are as large as they are, and it is worth being suspicious of anyone who reports similar gains on a task that satisfies two.

The lineage, in one table

Where this sits. Every row is cited by the paper.

Read down the "contribution" column and the trajectory is clear: from retrieve documents, to write your own documents, to write procedures instead of episodes, to measure which procedures work. Each step narrows what a memory is and adds a way to judge it. The obvious next step, which the limitations section names, is measure which procedure in a retrieved set deserved the credit.

Six things that would be worth trying next

The five sentences worth remembering a year from now

Papers decay into one or two ideas. Here are the ones from this paper that will still be useful when the benchmark numbers are stale.

Notice that none of the five is about spatial reasoning specifically, and none depends on the accuracy table being reproduced. They are claims about the structure of experience-grounded memory, demonstrated in a spatial setting because that setting happens to satisfy the four conditions from the decision table above.

What would falsify this

A useful last question for any paper. What result would show the mechanism is not doing what it claims?

Three of six are answered, and answered well. The three that are not — the shuffled-TRS control, a non-Qwen model, and any measure of variance — are the three things a reviewer should ask for, and the first one is nearly free to run.

Glossary — every term this lesson bolded

The cheat sheet — every equation, every symbol

The five equations, in order.

The numbers worth remembering

Where to go from here

Build it yourself — the weekend recipe

Every piece of SMA is implementable without a GPU-hours budget. Here is the checklist, with the decision that matters at each step.

References

1. Zhang, H., Ding, Y., Zhou, Y., Du, X., Zhang, S., Zhao, Z., Xi, Y., Chen, H. "Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence," 2026 — arXiv:2608.12743. The paper this lesson is built on. Project page.
2. Fang, R. et al. "MemP: Exploring Agent Procedural Memory," Findings of ACL 2026 — the similarity-only procedural-memory baseline. SMA minus TRS is approximately MemP.
3. Shinn, N. et al. "Reflexion: Language Agents with Verbal Reinforcement Learning," NeurIPS 2023 — arXiv:2303.11366. The ancestor of verifier-guided reflection.
4. Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020 — arXiv:2005.11401. The RAG baseline.
5. Song, C. H. et al. "RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics," 2026 — arXiv:2411.16537. Benchmark and post-training exemplar.
6. Chen, B. et al. "SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities," CVPR 2024 — the instruction-data route.
7. Ray, A. et al. "SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models," 2025 — arXiv:2412.07755.
8. Jia, M. et al. "OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models," 2026 — arXiv:2506.03135.
9. Du, M. et al. "EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks," ACL 2024.
10. Li, D. et al. "SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments," 2026 — arXiv:2604.14144. The training-based comparison in Table 3.
11. Dai, Y. et al. "S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence," 2026 — arXiv:2606.20515; Chen, S. et al. "SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL," 2026 — arXiv:2512.04069. The tool-agent route.
12. Li, Q. et al. "AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning," 2026 — arXiv:2607.21106; Liao, J. et al. "MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs," 2026 — arXiv:2605.08374. The credit-assignment fixes.
13. Kim, M. et al. "MemRefine: LLM-Guided Compression for Long-Term Agent Memory," 2026 — arXiv:2606.13177; Munirathinam, T. "AMP: A Vendor-Neutral Wire Format for Agent Memory Operations," 2026 — arXiv:2606.01138. Memory lifecycle.
14. Chhikara, P. et al. "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory," 2025 — arXiv:2504.19413; Packer, C. et al. "MemGPT," 2024 — arXiv:2310.08560.
15. Kwon, W. et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM), SOSP 2023. The serving layer for all four base models.