Harness the Memory: Eleven Substrates, No Winner

Chapter 1: Eleven Methods, Seven Families

Before comparing substrates on any benchmark, you need a map of what a "memory substrate" even is. The paper's first move is a taxonomy, and it is worth taking slowly, because the two-way split at the top of it explains half of everything that follows.

External versus internal

Every memory method stores information in one of two places. External memory keeps information in a data structure that lives outside the model — a vector index, a graph, a set of text files. The model's weights never change; at read time, something is fetched from that structure and stuffed into the prompt. Internal memory keeps information inside the model itself — either baked into its weights via fine-tuning, or held in its activations (the running key/value cache that a transformer builds as it processes tokens).

That distinction is not decoration. It determines who can inspect the memory (external: you can read the file; internal-weights: you cannot, short of probing), who can edit a single fact without retraining (external: yes; internal-weights: no), and what the read-time cost looks like (external: an index lookup; internal-activations: potentially a full re-prefill of a transformer's attention cache).

The seven families

Within external memory, five families differ in how much structure they impose on what gets stored. Within internal memory, two families differ in whether the information lives in weights or in activations.

Why three methods do not appear everywhere

Not every method runs on every benchmark. Three exclusions matter and will resurface in later chapters:

Keep that last row in mind. It is a fact about the world, not a limitation of the paper's harness: activation memory that keeps everything necessarily runs out of room, which is exactly the pressure that motivates M11's alternative — keep everything organized into episodes, and re-attend to only the one you need.

Reading the taxonomy as a spending decision, not a category label

It helps to stop thinking of the seven families as seven different technologies and start thinking of them as seven points on one spending curve. At write time, a method can spend nothing (Flat Index: append a vector, done) or spend an LLM call per turn (Structural, Refinement) or spend an entire fine-tuning run (Weight). At read time, a method can spend nothing beyond a similarity search (Flat Index, most of Structural) or spend an LLM call to synthesize what got retrieved (M5's serialization step, Chapter 3) or spend a full re-prefill of the transformer's attention cache (M11, Chapter 5).

Every method in this taxonomy is a different point on that two-axis spending plane — write cost against read cost — and no method sits at the origin with real capability. M1 and M2 are as close to the origin as the taxonomy gets, and Chapter 2 shows exactly what that buys and what it gives up. Everything else in this chapter's table is a deliberate move away from the origin, in one direction or the other, made in exchange for a specific capability the origin does not have: cross-fact synthesis (Structural), noise reduction (Refinement), or a store that does not grow forever (Refinement, again).

One more distinction worth making explicit before moving on: within External memory, families differ in whether they accumulate (Flat Index, Text Record, Structural, Hierarchical all keep growing as more turns arrive) or refine (Refinement methods actively shrink or bound what they keep). That accumulate-versus-refine split cuts across the write-cost axis independently of it — Structural methods spend heavily at write time and still accumulate; Refinement methods spend at write time specifically in order to stop accumulating. Chapter 10's scalability results are, in large part, a direct measurement of that second distinction.

Chapter 2: The Retrieval Floor

Start with the two simplest substrates, because everything more elaborate in this paper is measured against them. M1 Dense Vector and M2 Sparse Vector are both Flat Index methods: no LLM call happens at write time, none happens at read time. They are pure similarity search, and the paper is explicit that they exist to "establish the retrieval floor" — the baseline every structural or refinement method has to beat to justify its extra cost.

What "flat" means

Every other family in this taxonomy imposes some organization on memory — a graph, a tree, a distilled strategy. Flat Index imposes none. Every turn becomes one entry in one unstructured collection, and retrieval is a single similarity computation over that collection. Dense Vector embeds each turn with a text embedding model; Sparse Vector represents each turn as a weighted bag of terms (the vector is mostly zeros, hence "sparse"). Both skip the step every other family spends compute on: deciding how information relates to other information.

Concept + Realization: what a write call and a read call actually do (M1)

"Store it in a vector database" is a sentence that hides two very different operations. Here is what each one is, concretely, for M1 Dense Vector.

M1 write call — one new conversation turn arrivesdef write(turn_text, turn_id, timestamp):
    vec = embed(turn_text)              # text -> float32[D], e.g. D=1024. ONE forward pass, no LLM call
    store.append({
        "id": turn_id,
        "vec": vec,                    # the only thing indexed
        "text": turn_text,              # kept verbatim, never summarized
        "t": timestamp,
    })
    # cost: 1 embedding call. no reasoning, no restructuring, no LLM.

M1 read call — the agent needs context for the current querydef read(query_text, k):
    qvec = embed(query_text)            # float32[D]
    scored = [(cos_sim(qvec, e["vec"]), e) for e in store]
    topk = sorted(scored, reverse=True)[:k]
    return "\n".join(e["text"] for _, e in topk)   # raw turn text, concatenated, into the prompt
    # cost: 1 embedding call + k dot products. no reasoning, no synthesis.

Two things to notice about that pair of functions, because they explain both M1's speed and M1's ceiling. First, nothing in either function reasons about the content — there is no step that asks "what does this turn mean" or "how does this relate to what I already know." Second, the read call returns the raw text of whatever turns score highest, unedited. If the fact the agent needs is split across three separate turns that never scored in the same top-k together, M1 will never assemble them into one coherent answer — it can only hand back what was literally said, verbatim, in isolation.

That is precisely the gap that structural methods like M5 (Chapter 3) are built to close — at a cost you can measure directly once you have M1's near-zero cost as the reference point. In Chapter 7's numbers, M1 and M2 read-and-write cycles complete in a fraction of a second; the methods that add structure will run into the tens of seconds.

Dense versus sparse: two ways to be flat

Both M1 and M2 skip organization, but they disagree about what "similarity" means. M1's dense embedding captures meaning — "the dog barked" and "a canine made noise" land close together even though they share almost no words. M2's sparse representation captures literal term overlap — it is closer to keyword search, weighting rare, distinctive words heavily and common words lightly, so it excels when the query and the answer share exact vocabulary (a proper noun, a specific error code, a name) and struggles when they are paraphrases of each other.

Which one wins depends on the benchmark's vocabulary structure, and the paper's numbers (Chapter 7) show it is genuinely task-dependent: M1 and M2 trade places across benchmarks rather than one dominating the other everywhere. That inconsistency is itself informative — it means neither dense nor sparse similarity, alone, is "solved" retrieval. Both are the floor, not the ceiling, and both are cheap enough that there is rarely a reason not to run one of them as a baseline in any real system.

What flat indexing cannot do, made concrete with an example

Take a dialogue spread across three sessions. Session one: "I just switched to the marketing team." Session four: "my new manager is pretty hands-off." Session seven: "marketing hits a big product launch next month, it's going to be a lot." A question at session eight — "is her new role stressful right now?" — requires connecting all three turns: the role (marketing), the manager style (hands-off, which on its own is not stressful), and the upcoming deadline (which, combined with a new role and limited oversight, plausibly is).

Run M1's read call on that question. Each of the three turns will likely score reasonably on similarity to "stressful" or "role" individually — "hands-off manager" is at best loosely related to "stressful," "product launch" more directly so. If k is large enough, M1 may well retrieve all three turns. But retrieving all three is not the same as connecting them: the read call in Chapter 2's pseudocode returns them concatenated, unordered relative to each other beyond similarity rank, with no step that reasons "new role + hands-off manager + imminent launch = plausible near-term stress." Whether the downstream policy model manages to make that connection from three raw, juxtaposed sentences depends entirely on the policy's own reasoning at generation time — M1 has done nothing to help beyond surfacing the raw material.

This is not a flaw specific to M1's implementation. It is the direct consequence of the taxonomy in Chapter 1: Flat Index spends zero compute at write time and zero at read time beyond similarity ranking. There is no step in the pipeline where connecting turns to each other was ever going to happen. Structural memory (Chapter 3) exists specifically to add that step.

When the floor is actually the right choice

None of this makes M1 or M2 a bad default. Quite the opposite — Chapter 7's numbers will show M1 and M2 completing a full write-and-read cycle in well under a second on every benchmark tested, while structural and refinement methods routinely take tens of seconds. For any task where facts do not need connecting across turns — single-fact lookups, recent-turn recall, anything closer to keyword search than reasoning — the extra machinery of every other family in this taxonomy buys nothing and costs a great deal. The honest framing, which the rest of this lesson will keep returning to: Flat Index is the right answer whenever the task does not need what only a more expensive substrate can provide, and knowing which is which is exactly what the harness in Chapter 6 was built to measure.

Sparse vector's specific failure mode

M2's term-overlap similarity has a predictable weak spot worth naming directly, because it resurfaces in Chapter 8's ALFWorld results. Sparse retrieval rewards literal vocabulary overlap between the query and the stored text. On a benchmark where the useful signal is phrased differently across the query and the memory (a paraphrase, a synonym, an implicit reference), sparse retrieval can systematically miss what dense retrieval would catch — and conversely, it can surface text that shares surface vocabulary with the query while being about something else entirely (matching on a common word like "shelf" or "kitchen" without matching on the actual relevant object or action). That second failure mode — high lexical overlap, low actual relevance — is a source of exactly the kind of retrieval noise Chapter 8 shows actively hurting agent-centric tasks.

Chapter 3: Memory That Restructures Itself

Flat Index answers "what looks similar to my query." It cannot answer "who is connected to whom" or "what changed since I last checked." Three families exist because those are different questions: Text Record compresses, Structural connects, and both spend LLM calls at write time to make read time cheaper or better-informed.

M3 Gist Index: compress, then select pages

M3 adds one idea on top of Flat Index: instead of indexing raw turns, it writes a gist — a short LLM-generated compression of what the turn contained — and at read time it selects whole pages of gisted text rather than ranking individual vectors. The write side costs an LLM call per turn (or per batch of turns); the read side trades vector math for a selection step that can reason about which page, as a unit, is relevant. It sits between Flat Index and the fully structural methods: some write-time compute, no persistent structure connecting one gist to another.

M4 Evolving Notes: memory that gets rewritten

M4 keeps memory as a set of interlinked notes, and the distinguishing move is evolution-triggered rewrites: when new information contradicts or extends an existing note, the note itself gets edited, not just appended to. If session one says "she lives in Austin" and session six says "she moved to Denver," a Flat Index method stores both turns and leaves it to the read-time context to sort out which is current. M4 is designed to actually update the note, so a later read returns "lives in Denver (moved from Austin)" as one coherent, current fact rather than two contradictory raw turns competing for the same retrieval slot.

M5 Dual-Level Graph: the structural family's most elaborate member

M5 builds an entity-relation graph from the conversation, and it does so at two levels: a fine-grained local level (specific entities — "Biscuit", "the half-marathon" — connected by specific relations) and a coarser global level (communities of related entities summarized into higher-level themes). This dual structure is what lets M5 answer both narrow questions ("what is Biscuit") and broad ones ("what has she been dealing with lately") from the same store.

Concept + Realization: M5's write and read calls, contrasted with M1

Where M1's write call was one embedding call and its read call was a sort, M5's write and read calls each involve multiple LLM invocations — and seeing exactly where those calls happen is what makes the latency gap in Chapter 7 (an 85× multiplier, worked out with real numbers) make sense as an engineering fact rather than an abstract cost.

M5 write call — one new conversation turn arrivesdef write(turn_text, turn_id, timestamp, graph):
    triples = llm_extract_entities_relations(turn_text)
        # LLM CALL 1: turn_text -> [(entity_a, relation, entity_b), ...]
        # e.g. [("she", "trains_for", "half-marathon"), ("Biscuit", "is_a", "dog")]

    for (a, rel, b) in triples:
        graph.local.upsert_node(a); graph.local.upsert_node(b)
        graph.local.upsert_edge(a, rel, b, source_turn=turn_id)
        # no LLM call — pure graph mutation, but the graph itself now has structure M1 never had

    if graph.local.community_touched_enough_to_resummarize():
        community_summary = llm_summarize_community(graph.local.get_community(a))
            # LLM CALL 2 (amortized — not every write triggers this)
        graph.global_level.upsert_summary_node(community_summary)
    # cost per write: 1 required LLM call, sometimes a 2nd. M1's write call: 0 LLM calls.

M5 read call — the agent needs context for the current querydef read(query_text, graph):
    query_entities = llm_extract_query_entities(query_text)
        # LLM CALL 3: "how's the training going?" -> ["training", implicit: her half-marathon]

    local_hits = graph.local.traverse(query_entities, hops=2)      # graph walk, no LLM
    global_hits = graph.global_level.match_communities(query_text)  # embedding match, no LLM
    subgraph = merge(local_hits, global_hits)

    context_text = llm_serialize_subgraph(subgraph)
        # LLM CALL 4: turns a subgraph of nodes/edges back into readable prose for the prompt
    return context_text
    # cost per read: 2 required LLM calls (extract + serialize) + 2 graph traversals.
    # M1's read call: 0 LLM calls, 1 embedding call, k dot products.

Count the LLM calls. M1's full write-and-read cycle touches an LLM exactly zero times — it uses an embedding model, which is orders of magnitude cheaper per call than a full generative pass. M5's cycle touches an LLM three to four times: once to extract triples on write, sometimes once more to re-summarize a community, once to extract entities from the query on read, and once to turn the retrieved subgraph back into prose the policy model can use. Every one of those is a real forward pass through a real model, and every one of them adds latency that a flat vector lookup simply does not pay.

Two levels, revisited: why not just one big graph

It is worth asking why M5 bothers with two levels instead of one flat entity-relation graph. A single-level graph answers narrow, entity-specific questions well — "who does she report to" is a one-hop or two-hop traversal from the "she" node. But a broad question — "what's been going on with her lately" — has no single entity to anchor a traversal from. It needs something closer to a summary of an entire neighborhood of the graph: several related entities, several related edges, compressed into a theme.

The global level exists to answer exactly that second kind of question without forcing every read call to traverse and re-synthesize the entire local graph from scratch every time. Community summaries are pre-computed (amortized across writes, per the pseudocode's "LLM CALL 2" comment, not recomputed per query) and then matched against broad queries directly, the same way a table of contents lets you jump to a chapter without reading a book's full index first. This two-level design is a specific, deliberate answer to the narrow-versus-broad question tension — and it is also exactly why M5's write pipeline has two possible LLM calls (extraction, always; resummarization, only sometimes) rather than one.

M3 and M4, worked the same way M5 was

It is worth walking M3 and M4's write/read shape briefly too, because the family shares a pattern that M5 makes most visible. M3 Gist Index's write call is gist = llm_compress(turn_text) — one LLM call, producing something shorter than the raw turn but with none of M5's relational structure. Its read call selects whole gisted pages (groups of gists, typically corresponding to a session or time window) rather than individual entries, trading M1's fine-grained ranking for coarser, cheaper, page-level relevance. M4 Evolving Notes' write call includes a conditional the others do not: if new_info contradicts existing_note: existing_note = llm_rewrite(existing_note, new_info) — an LLM call that fires only when new information actually conflicts with or extends something already stored, which is what lets M4 return one current, coherent note about a fact instead of a growing pile of individually-true-at-the-time, collectively-stale statements.

Notice the trend: as you move down this table, write-time cost rises and the amount of read-time reasoning the substrate itself performs (as opposed to leaving reasoning to the downstream policy model reading raw retrieved text) also rises. M5 is the family's most expensive member because it is also the family's most capable member at the specific job this family exists to do — represent relationships between pieces of information, not just the pieces themselves.

Chapter 4: Distill, Don't Accumulate

Every substrate so far grows monotonically. Every turn adds a vector, a gist, a note edit, a graph triple. That is fine until the history gets long — then the store keeps getting bigger and read time keeps getting more expensive to search, no matter how clever the index. Two families take a different bet: instead of keeping raw or lightly-processed history, distill it into something small and reusable, and let the raw history go.

M6 Hierarchical Tree: summarize, then summarize the summaries

M6 recursively clusters and summarizes memory into a multi-level tree. Leaf nodes hold something close to raw turns; each level up groups related leaves and replaces them with a summary; the top level is a small number of high-level summary nodes covering the whole history. At read time, the search can walk a collapsed, dense version of the tree — searching a handful of summary nodes instead of every raw turn — and only descend to raw text if the summary match is strong enough to warrant it.

M6 sits at the boundary of this chapter's theme: it does compress, but it still keeps every level, including the raw leaves, so memory size still grows with history — just more slowly, and more searchably, than a flat list. The two methods that actually shrink are the Refinement family.

M7 Distilled Strategies: judge, then extract the lesson

M7 is agent-centric only (Chapter 1's exclusion table explained why: it needs an action trajectory with an outcome to judge, and dialogue has neither). Its write-time pipeline looks nothing like appending a turn to an index. It looks at a completed trajectory — a sequence of actions and their results — and asks an LLM to judge whether the trajectory succeeded or failed, and if it did (or failed instructively), to extract a reusable strategy from it: not "here is what happened" but "here is the general rule that would have made this succeed, or that did make it succeed."

M7 write call — one trajectory just finisheddef write(trajectory, outcome):
    verdict = llm_judge_trajectory(trajectory, outcome)
        # LLM CALL: was this trajectory successful, and why or why not?
    if verdict.worth_keeping:
        strategy = llm_extract_strategy(trajectory, verdict)
            # LLM CALL: distill into a general, reusable rule —
            # e.g. "when the target object is not visible, check the shelf ABOVE eye level first"
        strategy_store.append(strategy)     # small, does NOT grow with raw trajectory length
    # the raw trajectory itself is never stored for retrieval

Two write-time LLM calls per trajectory sounds expensive turn-by-turn, but it happens once per episode, not once per turn — and what gets stored afterward is a short, general strategy, not the whole episode. A hundred episodes do not produce a hundred episodes' worth of retrievable text; they produce, at most, a hundred short strategies, and many of those will be near-duplicates that a real system would deduplicate further. This is the structural reason Refinement methods behave differently at scale, which Chapter 10's scalability results will show directly.

M8 Skill Bundles: cluster experiences, keep the cluster

M8 takes a related but distinct approach: instead of judging and extracting one strategy per trajectory, it clusters experiences into skill bundles — groups of related past experiences that get consolidated into a reusable unit. Where M7 produces one crisp rule per trajectory, M8 groups many related experiences and keeps the bundle, trading some of M7's precision for coverage of a wider range of situations per stored unit.

Refinement versus Structural: a different kind of "expensive"

It is tempting to lump M4/M5 (Structural) and M7/M8 (Refinement) together as "the substrates that use LLM calls at write time," but the shape of what they produce is opposite. Structural methods keep more information than they started with, organized (a graph has more retrievable structure than the raw text it was built from). Refinement methods keep less information than they started with, on purpose (a distilled strategy discards the specific trajectory that produced it). Both cost LLM calls; only one of them controls how big the memory gets as history grows.

What gets lost when you distill

Refinement's bet is not free of downside, and it is worth naming the downside precisely rather than treating M7 and M8 as strictly better than the substrates that keep raw history. A distilled strategy is, definitionally, a generalization — it discards the specific circumstances of the trajectory it came from in favor of a rule meant to transfer. That means M7 and M8 are structurally unable to answer a question that depends on a specific past detail rather than a general pattern. "What strategy tends to work when the target object isn't visible" is exactly M7's kind of question. "What happened during episode 14 specifically" is not — the raw trajectory that would answer it was never stored for retrieval in the first place.

This is why M7 has no dialogue analog (Chapter 1) and why it would make a poor substrate for the companion-app job from Chapter 0: a user does not want their memory system to have "distilled a general strategy" out of the fact that their dog is named Biscuit. Biscuit's name is not a lesson to generalize; it is a specific fact to retain exactly. Refinement memory is a strong fit precisely for tasks that reward generalization — repeated categories of decision, where the specific episode matters less than the pattern it represents — and a poor fit for tasks that reward exact recall of a specific, non-repeating fact.

M6's middle position, worked through

M6 Hierarchical Tree deserves a second look here because its position in the taxonomy is genuinely in-between, not just nominally so. Its recursive summarization looks, on the surface, like it is doing what M7/M8 do — compressing raw material into something smaller. The difference is that M6 keeps every level of the tree, including the raw leaves. Nothing is ever discarded; the summary levels are an additional index layered on top of the full raw history, not a replacement for it. That means M6's memory footprint still grows linearly with history (like Flat Index and Structural), even though its typical search cost grows more slowly, because most queries can be resolved by walking the smaller, denser summary levels without ever touching the (still-present, still-growing) raw leaves underneath.

Put differently: M6 optimizes read-time search cost without controlling write-time storage growth. M7 and M8 optimize both, by refusing to keep the raw material that would need summarizing in the first place. That distinction is exactly what Chapter 10's scalability sweep measures directly — and it is why M6 is grouped as its own family (Hierarchical) rather than folded into Refinement, despite the surface-level similarity of "these methods summarize things."

Chapter 5: Memory Inside the Model

Every substrate in Chapters 2 through 4 shares one architectural assumption: information lives outside the model, in a data structure you retrieve from, and gets stuffed into the prompt as text. Internal memory rejects that assumption. There is no "retrieve and inject" step at all — the information is already part of what the model computes over, either because it was trained into the weights or because it is sitting in the transformer's own attention state.

M9 Adapter Tuning: knowledge as parameters

M9 bakes new facts into the model's weights via fine-tuning — typically a lightweight adapter layered on top of the frozen backbone, trained on the accumulated interaction history. At read time there is no retrieval step whatsoever: the model just answers, the way it answers questions from its original pretraining, because the new information is now indistinguishable (to the model's own forward pass) from anything else it learned.

That sounds elegant until you ask the two questions every other substrate answers for free. Can you inspect what the model "remembers"? No — short of probing techniques, a fact baked into weights is not a line in a file you can read. Can you query it selectively, the way M1's read call fetches only the top-k relevant turns? No — fine-tuned knowledge is not addressable; it is diffused across parameters and surfaces (or fails to surface) however the forward pass happens to activate it for a given prompt.

M10 Full Context: keep everything, retrieve nothing

M10 is the naive baseline for activation memory: put the entire interaction history in the context window, every time, and let the model's own attention mechanism figure out what matters. No retrieval, no compression, no judgment calls about what to keep — just more tokens.

Chapter 1 already explained why M10 is absent from the agent-centric benchmarks: ALFWorld and BigCodeBench-Hard trajectories run long enough that cumulative history exceeds every context window tested, so "keep everything" is not a weaker option there, it is simply not an option. On the user-centric benchmarks, where sessions are shorter, M10 is very much a contender — and Chapter 7 will show it winning outright on one of them, cheaply, because reading a fixed context does not require the query-time reasoning that retrieval-based methods pay for.

M11 Episode-Clustered Re-prefill: keep everything, organized, and re-attend to only what matches

M11 is the paper's answer to the pressure that makes M10 unworkable at length: cluster the interaction history into episodes as it arrives, and at read time, identify which episode matches the current situation and re-prefill — recompute the transformer's key/value attention cache — using only that episode's turns, discarding the rest of the accumulated history from the active context entirely.

Concept + Realization: M11's write and read calls, contrasted with M1 and M5

M11 write call — one new turn or action-observation pair arrivesdef write(turn, timestamp, episodes):
    current = episodes.last_open()
    boundary = detect_episode_boundary(turn, current)
        # similarity/rule-based check: does this turn continue the current episode
        # or start a new one (e.g. task-relevant topic/location shift)? cheap, not a full LLM call.
    if boundary.is_new_episode:
        current.close(summary=cheap_summarize(current.turns))
        current = episodes.open_new(start=turn)
    current.turns.append(turn)                # raw turns kept, but scoped to ONE episode, not one flat list
    # cost: cheap boundary check per turn, no required full-LLM call — unlike M5's write call

M11 read call — the agent needs context for the current stepdef read(current_observation, episodes, model):
    best = max(episodes.closed, key=lambda ep: sim(current_observation, ep.summary))
        # match against episode SUMMARIES, not every raw turn — cheap
    kv_cache = model.prefill(best.turns)
        # RE-PREFILL: a fresh forward pass over ONLY this one episode's raw turns,
        # rebuilding the attention key/value cache from scratch — this is the expensive step
    return model.generate(current_observation, kv_cache=kv_cache)
        # the policy attends to the matched episode's turns, not the full trajectory buffer

Compare the three substrates on one axis: what does the policy actually attend to at the moment it has to act? M1 hands back a handful of retrieved text snippets, unconnected. M5 hands back a synthesized paragraph built from graph traversal. M11 hands back nothing as text at all — it re-shapes the model's own attention cache so that, internally, the model is attending to exactly one coherent episode's worth of raw turns, with everything from other episodes fully excluded rather than merely down-weighted.

That exclusion is the mechanism behind a result Chapter 8 works out in full: on ALFWorld, M11 roughly doubles the no-memory baseline's task success rate, because re-prefilling only the matched episode keeps the policy from having to sift the entire trajectory buffer for the one relevant precedent. The same exclusion is also why M11 is among the worst performers on LoCoMo dialogue (P4 0.307 on Qwen3-8B, similar to M9): a multi-session conversation's relevant facts are often scattered across many episodes, not concentrated in the single best-matching one, so picking exactly one episode and discarding the rest throws away information a dialogue answer actually needed.

Why "re-prefill" is the expensive part, in engineering terms

It is worth being precise about what makes M11's read call costly, because it is a different kind of cost than anything in Chapters 2 or 3. When a transformer processes a sequence, it builds a key/value cache — per-layer, per-token representations that every subsequent token's attention computation reads from. Normally, a cache is built once, incrementally, as tokens are generated or appended. M11's read call does something structurally different: it discards the accumulated cache state relevant to non-matching episodes and rebuilds a fresh cache from scratch over just the matched episode's raw turns, before the policy can generate anything for the current step.

That rebuild is a full forward pass over every token in the matched episode — not a lookup, not a similarity computation, an actual pass through every transformer layer for every token being re-prefilled. For a short episode, that is cheap. For a long one, it approaches the cost of just running M10 Full Context over that one episode's length. M11's efficiency bet depends entirely on episodes staying short enough that this rebuild remains fast — which is exactly why M11 performs comfortably on ALFWorld's naturally bounded episode lengths (a single embodied task, tens of steps) and would degrade if episodes were allowed to grow toward the length of an entire multi-session dialogue.

Comparing all three internal-adjacent options directly

Notice that M9 and M10 sit at two extremes of "how much read-time work happens": M9 does none, because the knowledge is baked in; M10 does none either, because everything is already present in the prompt. M11 is the only one of the three that does real read-time work — matching, then re-prefilling — and that work is exactly what buys it the doubling result on ALFWorld that neither of the zero-read-time-work alternatives achieves. Chapter 5's pseudocode made that work visible; this comparison makes its payoff visible.

A note on what "internal" memory gives up, architecturally

Step back and notice what all three internal-adjacent methods share, in contrast to every external substrate from Chapters 2 through 4: none of them produce a human-readable artifact you could open in a text editor and inspect. M1 through M8 all produce something — a vector store, a set of gists, a graph, a tree, a strategy library — that a developer can dump, read, and debug directly when a memory-agent behaves unexpectedly. M9's weights cannot be inspected this way at all. M10's context is inspectable but is exactly the raw conversation log, offering no additional structure to debug with. M11's episodes are inspectable as discrete units, which is a real advantage over M9 and part of why it remains a reasonable production choice for agent-centric tasks despite being an "internal" substrate in the taxonomy's strict sense.

Chapter 6: Building an Honest Harness

Chapter 0's landscape survey found that most of the field's evidence is confounded: one backbone family, one or two datasets, no cost numbers. This chapter is about the specific choices that keep this paper's comparison from repeating those mistakes — because every worked number in Chapters 7 through 10 depends on the harness actually holding everything but the memory substrate constant.

Three backbones, deliberately not all the same family

The paper runs every applicable substrate on three backbones, served with vLLM on 4×H200 GPUs:

Notice what this buys: any finding that holds across all three backbones (like "M5 wins recall-heavy dialogue" in Chapter 7) is not an artifact of one model's specific attention pattern or context-handling quirks — it survived a change in size, a change in quantization, and a change in architecture family.

One fixed auxiliary LLM, for every method that needs one

Several substrates — M3's gisting, M4's note evolution, M5's triple extraction and summarization, M6's clustering, M7's judging and strategy extraction, M8's bundling — need an auxiliary LLM to do the write-time (and sometimes read-time) reasoning that Chapters 3 and 4 walked through in pseudocode. The paper fixes that auxiliary model to gpt-4o-mini across every method that needs one.

Two regimes, four benchmark suites

The benchmarks split deliberately into two regimes with opposite retrieval demands — the split that Chapter 9's central finding turns on.

The regime names are not just labels — they describe what kind of question each benchmark is asking. A LoCoMo question ("what's her dog's name") has one correct answer sitting somewhere in the dialogue history; the model's job is to find it, and the more of the relevant history it can see, the better its odds. An ALFWorld step ("what do I do next, standing here, holding this mug") needs the model to act correctly right now; retrieved history is useful only insofar as it informs that action, and if it crowds out the model's attention to the actual room in front of it, retrieval has become a liability rather than a help. Chapter 9 turns that intuition into a measured mechanism.

What each MAB sub-capability actually isolates

MemoryAgentBench is worth unpacking further, because its four factored capabilities are not four random tasks — each one isolates a distinct failure mode a memory substrate can have, and Table 1's per-column numbers only make sense once you know what each column is actually testing.

Reading Table 1 column by column with this key in hand changes what the numbers mean. M4's relatively modest LoCoMo score next to M5's makes more sense once you notice M4's design target (contradiction resolution) is not what LoCoMo's five question categories primarily test — M4's real strength shows up more directly on MAB's CR column, which Chapter 10 reuses as the scalability probe specifically because it is the column most sensitive to whether a substrate can track what is current as history accumulates.

Twenty-six metrics, in two families

Every method on every benchmark is scored on the same instrumented set: performance metrics (exact match, token F1, BLEU-1, an LLM-judge score called P4 that appears throughout Table 1, compression ratio, recall@k, and benchmark-specific scores like ALFWorld task success rate, ALFWorld goal-condition success, steps-to-goal, BigCodeBench Pass@1, and SubEM) and efficiency metrics (memory size, inference time, write latency, retrieval latency, write/retrieved/read token counts, and write/read/management call counts). E15 in Table 1, which every latency comparison in this lesson uses, is per-query wall-clock latency in seconds.

The ablations, previewed

Two additional experiments sit on top of the main harness and drive Chapters 9 and 10. A retrieval-breadth sweep varies how many entries a method retrieves per query — k ∈ {1, 2, 5, 10, 20} on LoCoMo, k ∈ {1, 2, 3, 4, 5} on ALFWorld — against a no-memory baseline that receives the identical prompt template with an empty retrieved block, so that any curve you see attributes quality change to retrieval itself, not to the presence of a memory-shaped scaffold in the prompt. A context-length stress test sweeps MAB's Conflict Resolution benchmark across 6K, 32K, and 262K token histories to see which substrates keep working as the raw amount of history grows by more than 40×.

Why the harness bothers with a no-memory baseline at all

It might seem obvious that a system with memory should beat a system without one, but Chapter 8's results (M2 scoring below the no-memory baseline on Qwen3-32B-AWQ ALFWorld) show this is not guaranteed. A no-memory baseline is not a strawman here — it is the harness's control condition, the same way a placebo is a control condition in a drug trial. Without it, you cannot tell whether a memory substrate's benchmark score reflects the substrate actually helping, or reflects the substrate simply not actively hurting relative to whatever floor performance the backbone achieves on its own. Every "M11 doubles NoMem" or "M2 falls below NoMem" claim in this lesson depends on that baseline being measured under the exact same prompt template as every memory-bearing method, which is precisely what Chapter 6's empty-retrieved-block design guarantees.

What the harness deliberately does NOT vary

It is worth being explicit about the things this harness holds fixed, because each one is a potential confound the paper chose to rule out by design rather than by argument: the auxiliary LLM (always gpt-4o-mini, so no method benefits from a stronger helper), the hardware (always 4×H200 via vLLM, so no method benefits from faster serving infrastructure), the prompt template shape (memory-bearing and no-memory conditions share the same template with only the retrieved-block content varying), and the metric definitions (the same 26 metrics, computed the same way, for every method on every applicable benchmark). What the harness DOES vary — deliberately, as the independent variables of the whole study — is the substrate, the backbone, the benchmark, and (in the two ablations) the retrieval breadth and the context length. Every result in Chapters 7 through 10 is a controlled comparison along exactly one of those four axes at a time, which is what makes the worked numeric examples in this lesson actual apples-to-apples comparisons rather than numbers pulled from differently-configured runs.

Chapter 7: When Structure Wins

User-centric results are where the paper's most repeatable finding lives: across three backbones with different sizes, quantization, and architecture, M5 Dual-Level Graph is the consistent winner on recall-heavy dialogue. It is also, by a wide margin, the most expensive method on the board. This chapter works both facts out with real numbers from Table 1, because the size of that cost is the whole reason Chapter 6 insisted on logging it.

Worked example 1 — the 85× question

On Qwen3-8B, LoCoMo scores (P4, the LLM-judge metric, on a 0–1 scale; E15, latency, in seconds per query):

Work the two gaps by hand. The accuracy gap is a subtraction: 0.648 − 0.470 = 0.178 — call it +0.18 P4 points. The latency gap is a division: 27.84 ÷ 0.33. Scale both sides by 100 to clear the decimal: 2784 ÷ 33 = 84.36. So M5 costs roughly 84–85× M2's latency for a P4 gain of 0.178.

Worked example 2 — the cost of a substrate that cannot be queried

Compare M9 Adapter Tuning (Chapter 5's weight-based substrate) against the best performer on each of two benchmarks, same backbone (Qwen3-8B):

The relative-drop arithmetic: on LoCoMo, 0.269 ÷ 0.648 = 0.415, a 41.5% relative shortfall. On LME-S, 0.287 ÷ 0.537 = 0.534, a 53.4% relative shortfall — more than half again as bad as the best available method on that benchmark. M9 is the single most consistent underperformer in the entire table, on every backbone it runs on. Chapter 5 explained the mechanism: new facts baked into weights are not inspectable or selectively addressable at read time the way an index entry or a graph node is, so the model has no equivalent of "retrieve exactly the fact that answers this query" — it can only hope the right knowledge activates.

Worked example 3 — M5 wins on every backbone, not just one

This is the result Chapter 6's three-backbone design was built to test for: M5 does not just win on one dense Qwen model, it wins on the quantized 32B and on the mixture-of-experts Gemma-4 too, and its margin over the closest rival (M10 Full Context) actually widens slightly on Gemma-4. That is a substrate finding, not a backbone-specific artifact.

Worked example 4 — the same substrate, opposite verdicts on two "user-centric" columns

M5's dominance is not universal even within user-centric benchmarks — and neither is any single substrate's, including M10. Compare M10 Full Context on two of Table 1's four MAB/LoCoMo columns, same backbone (Qwen3-8B):

On MAB Long-Range Understanding, M10 posts the single best score in the Qwen3-8B band — beating M5's own MAB-LRU score (0.578) while running roughly 48× faster (313.18 ÷ 6.58 ≈ 47.6). On LME-S — the column built specifically to test finding one precise fact among a large volume of distractor history — that same M10 collapses to 0.120, one of the worst scores anyone posts in the whole table, far below M5's 0.537 and even M9's 0.250. Same substrate, same "keep everything, retrieve nothing" mechanism, opposite verdicts: a task that rewards seeing a wide, unfiltered span of history rewards M10's total recall; a task that rewards finding one needle among a lot of hay punishes it, because nothing in M10's mechanism ever narrows the model's attention toward the single fact that matters. The lesson is not "M5 wins" or "M10 wins" as a blanket statement — it is that the winning substrate tracks what kind of information the task actually needs recovered.

Reading the scatter

Two shapes matter more than any single point. First, there is no method that is both fastest and most accurate — the frontier runs from M1/M2 in the bottom-left (fast, moderate accuracy) up through M6 and M10 to M5 in the top-right (slow, best accuracy) with M9 sitting off to the side, below the frontier entirely, because it does not even buy accuracy for its cost the way the others do. Second, that frontier's shape barely moves when you switch backbones — M5's cost is architectural (four LLM calls per read-write cycle, Chapter 3), not a quirk of one model's inference speed.

The middle of the pack: M3, M4, M6, M8 on LoCoMo

It is easy to let the M5-versus-M2 comparison dominate the picture, but the methods sitting between those two extremes tell their own story. On Qwen3-8B LoCoMo, M3 Gist Index reaches 0.562 at 4.18 seconds — meaningfully cheaper than M5 (27.84s) while still beating both flat-index methods (M1 0.540, M2 0.470) on accuracy. M6 Hierarchical Tree reaches 0.556 at just 1.39 seconds — nearly M3's accuracy at roughly a third of the latency, making it, by a simple accuracy-per-second measure, one of the more efficient non-flat methods on this specific benchmark. M4 Evolving Notes, by contrast, actually falls below the flat-index floor here (0.435 versus M1's 0.540) despite costing more (6.36s) — a reminder that LoCoMo's question categories do not primarily test contradiction-resolution, M4's specialty, so its extra write-time cost is not well matched to what this particular benchmark rewards.

That last column — P4 divided by E15 — is a crude but useful lens the paper's own 26-metric harness makes possible: it is not a metric the paper reports directly, but it is exactly the kind of derived number a systems engineer building against a latency budget would compute from the raw numbers Chapter 6 insisted on logging. M2 dominates it because its latency is so low; M5 is worst on it despite having the highest raw accuracy, because its cost grows so much faster than its accuracy gain. Neither ranking is "correct" in isolation — which one matters depends entirely on whether your deployment is latency-constrained or accuracy-constrained, which is precisely the judgment call Chapter 10's design rule is built to help you make explicitly rather than by default.

What stays true and what doesn't, across the three backbones

Worked example 3 showed M5 winning LoCoMo on all three backbones. It is worth being precise about what does NOT stay constant: the absolute gap between M5 and its nearest rival shrinks as backbone capability increases. On Qwen3-8B, M5 beats M10 by 0.648 − 0.589 = 0.059. On Qwen3-32B-AWQ, the gap is 0.683 − 0.669 = 0.014 — more than 4× smaller. A larger, more capable backbone appears to extract more value out of a simpler substrate's raw retrieved context on its own, narrowing (though not eliminating) the advantage that M5's explicit structure provides. That is a genuinely useful piece of engineering intuition: the case for paying M5's cost is strongest with a smaller, less capable backbone doing the reasoning, and weakens — without disappearing — as the backbone gets more capable.

Chapter 8: When Noise Is Toxic

Table 2 covers ALFWorld (embodied planning, task success rate as TSR) and BigCodeBench-Hard (code generation with retrieval, Pass@1). The winners here are not the same substrates that won Chapter 7, and the reason why is the whole setup for Chapter 9's central finding: in these tasks, retrieval that is not precisely relevant does not just fail to help — it actively competes with the information the policy needs to act correctly right now.

Worked example 5 — M11 roughly doubles the no-memory baseline

On ALFWorld, Qwen3-8B:

Divide: 11.9 ÷ 5.7 = 2.088. M11 is a 2.09× multiplier on the no-memory baseline — close enough to call it doubling, and the paper does. Chapter 5's mechanism explains why: M11 re-prefills attention over exactly the one matched episode and structurally excludes every other episode from the active context, so the policy is never competing for attention against irrelevant history the way M10 (Full Context) would be if it could even run here at all.

Worked example 6 — the refinement family's best single result

On Qwen3-32B-AWQ, ALFWorld:

Subtract: 32.1 − 22.4 = +9.7 percentage points. As a ratio: 32.1 ÷ 22.4 = 1.433, a 43.3% relative improvement. Chapter 4's mechanism explains this one too: M7 does not hand the policy a pile of retrieved history to sift through — it hands back a short, pre-judged, general strategy that was already distilled from a past trajectory's success or failure. There is no competing-for-attention problem when what gets injected is one clean rule instead of a stack of raw context.

Worked example 7 — same family, opposite sign

This is the result that most sharply previews Chapter 9. Still Qwen3-32B-AWQ, ALFWorld, comparing the two Flat Index methods (M1 and M2 — Chapter 2's zero-structure baseline) against the no-memory floor:

BigCodeBench-Hard: retrieval helps more broadly, until it doesn't

Code retrieval behaves differently from embodied planning, and understanding why sharpens the picture rather than complicating it. Retrieved code snippets extend a prompt rather than competing with it — a past solution sitting in the context does not crowd out the current task description the way retrieved dialogue history can crowd out a robot's current observation. So on BigCodeBench-Hard, unlike ALFWorld, most methods sit at or above the no-memory baseline.

That crossover on the 32B backbone is worked example 8: M5's Pass@1 advantage evaporates and inverts — M2 beats it by 19.6 − 16.2 = 3.4 percentage points, while running at a small fraction of M5's cost. Across backbones on this benchmark, the paper reports M5 paying 2 to 6× the inference cost of flat substrates for a lead that is not even consistent. The paper's own framing: on BigCodeBench-Hard, M5 becomes "a strictly dominated choice when latency matters" — not because its mechanism stopped working, but because a cheaper method reached comparable or better accuracy without the overhead.

The Gemma-4 ALFWorld numbers, read carefully

Gemma-4-26B-A4B-IT's ALFWorld column deserves its own look, because it complicates the "bigger backbone always narrows the gap" intuition Chapter 7 established. NoMem on Gemma-4 is 7.5% — noticeably lower than Qwen3-32B-AWQ's 22.4% NoMem floor, suggesting Gemma-4's baseline embodied-planning competence (with zero memory assistance at all) is weaker on this task specifically. Against that lower floor, M1 (10.4%) and M3 (10.4%) both provide a genuine relative lift, while M5 actually falls BELOW the NoMem floor here (4.5% versus 7.5%) — the only backbone/benchmark combination in Table 2 where M5, the paper's most consistent user-centric winner, is a net negative relative to having no memory at all.

This is a genuinely useful data point precisely because it breaks the pattern rather than confirming it cleanly. M5's structural overhead — synthesizing a subgraph into prose via an LLM call before injecting it — appears to interact badly with Gemma-4's specific attention behavior on this particular benchmark, producing worse results than simply retrieving flat, unstructured text (M1) or a hierarchical summary (M6). The honest lesson here is not "M5 is bad on MoE backbones" (Chapter 7 showed M5 winning LoCoMo decisively on this same backbone) — it is that a substrate's fit to a task is not a fixed property of the substrate alone; it depends on the specific interaction between the substrate's output shape, the task's attention demands, and the backbone processing both. No single-axis story — "M5 is the best substrate," "bigger backbones close the gap" — survives contact with every cell in these two tables, and that irreducible complexity is itself part of what the paper is arguing: there is no universal ranking, only regime-conditional ones.

Steps-to-goal: a metric that separates "succeeded" from "succeeded efficiently"

Table 2's headline numbers are task success rate, but the harness also logs ALFWorld's steps-to-goal metric — how many actions the agent took before completing (or failing) the episode. This matters because two methods can post similar success rates while differing sharply in how many wasted actions they take along the way, and a memory substrate that helps an agent succeed via a long, meandering trial-and-error path is doing something different from one that helps it succeed directly. The paper's broader efficiency framing from Chapter 6 — memory size, latency, token and call counts — extends naturally to this benchmark-specific metric: a full accounting of a substrate's value on an agentic task has to include not just whether it got the job done, but how much wasted action and wasted attention it took to get there.

Chapter 9: The Central Reversal

Chapters 7 and 8 built up a pile of evidence that felt, at times, like two unrelated stories: dialogue rewards broad, structured retrieval; embodied and code tasks reward narrow, distilled retrieval. This chapter is where the paper shows those are not two stories. They are one mechanism, viewed from two different tasks, and the paper isolates it directly with an instrumented attention probe plus the retrieval-breadth sweep that Chapter 6 previewed.

The reversal, stated precisely

The paper's own framing: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Read that twice, because the operative word is "the same." It is not that dialogue tasks have a good retrieval method and agent tasks have a bad one. It is that retrieving more is the identical operation in both cases, and it has opposite effects because of what it does to where the model's attention goes.

The retrieval-breadth ablation

To isolate the effect, the paper sweeps k — the number of entries retrieved per query — independently on each regime: k ∈ {1, 2, 5, 10, 20} on LoCoMo, k ∈ {1, 2, 3, 4, 5} on ALFWorld (a narrower range, because an embodied task's admissible-action list is short enough that k=5 already means "retrieve a meaningful fraction of everything available"). Every point on the curve is compared against the no-memory baseline with an empty retrieved block, so the shape of the curve as k grows is attributable to retrieval content itself.

On LoCoMo, task performance climbs as k grows across most of the sweep — more retrieved context means more chances the answer-bearing passage is included. On ALFWorld, performance peaks early and then degrades as k continues to grow past a small value — more retrieved context means more competition for the policy's limited attention against the one thing it actually needs to look at: the current observation and the list of admissible actions.

The attention probe: where does the model actually look

To go from "performance curves diverge" to "here is why," the paper instruments attention allocation directly. It measures, at the model's last prompt token (Qwen3-8B, upper-half transformer layers, attention mass summed across heads, under M1 retrieval), how much attention mass falls into each of four labeled regions of the prompt: System (instructions), Retrieved (the memory block), Context (the dialogue transcript, or the trajectory/observation/action list for ALFWorld), and Cue (the actual question or the action prompt the model must respond to right now).

The finding, on both tasks: as k grows, attention mass is funneled out of Context and into Retrieved. That is a structural, near-mechanical consequence of adding more retrieved tokens to the prompt — more tokens competing for the same fixed attention budget necessarily pull some share of it away from wherever it was previously concentrated. What differs between the two tasks is not the shift itself. It is which region carries the answer.

Why this is not a fixable bug

It is tempting to think a smarter model could just learn to ignore irrelevant retrieved content. The attention probe argues against that being a simple fix: the shift is a property of how a fixed attention budget gets redistributed as prompt length and prompt composition change, not a failure of judgment the model is making that better training could remove outright. You could mitigate it — by retrieving less (lower k), by filtering harder before injection (which is exactly what M7's distillation and M11's single-episode match do), or by architecturally separating "retrieved" tokens from "current state" tokens so they do not compete for the same budget. But the underlying tension — more retrieved tokens necessarily draw attention away from something else in the prompt — is a fact about how attention is a finite, shared resource, not a bug in any one substrate's implementation.

Reconciling the reversal with Chapter 8's specific numbers

It is worth explicitly connecting this chapter's mechanism back to two of Chapter 8's worked examples, because the attention probe is what makes those results predictable rather than merely observed. M11's doubling of NoMem on ALFWorld (5.7% → 11.9%) is exactly what the reversal predicts for a substrate that narrows attention to one matched episode rather than broadening it — M11's read call structurally prevents the attention-drain-from-Context failure mode this chapter describes, by never presenting the policy with more than one episode's worth of retrieved material to compete against the current observation. M7's peak result (32.1% on Qwen3-32B-AWQ) fits the same story from a different angle: a single distilled strategy is about as far from "broad retrieval" as a memory read can get, so it barely taxes the Context/Cue attention budget at all.

Conversely, M2 falling below NoMem on the same benchmark (21.6% versus 22.4%) is the reversal's failure mode in miniature: whatever M2's sparse-vector search happened to surface at whatever k that run used, it was broad and unfiltered enough to compete for attention against the current observation without being precise enough to be worth that cost. None of these four results needed a separate explanation before this chapter — they are four instances of one mechanism, and that convergence is the strongest evidence the paper has that the reversal is real rather than a benchmark-specific coincidence.

What the reversal implies about prompt engineering, not just substrate choice

The reversal has a consequence beyond "pick the right substrate for the regime": it also implies something about how retrieved content should be formatted and placed within a prompt, independent of which substrate produced it. If attention is a finite budget that retrieved tokens compete for, then two substrates producing the same retrieved content can still behave differently depending on how much of it is included and where it sits relative to the current-state information the policy needs. This is part of why M7 and M11's design choices — return one short, precise unit; scope attention to exactly one matched episode — are not just "compress more" as a vague good practice, but a specific, mechanistically justified response to a finite, competed-for attention budget that this paper is among the first to measure directly rather than assume.

A boundary condition worth stating explicitly

The reversal is a tendency, established with a probe on one backbone (Qwen3-8B) and generalized across the retrieval-breadth sweep — it is not a law that guarantees every increase in k will hurt every agent-centric task or help every user-centric one. Chapter 8's Gemma-4 ALFWorld result (M5 underperforming NoMem, Chapter 8) shows the interaction between substrate output shape and backbone attention behavior can produce exceptions even within a regime that mostly follows the reversal's pattern. The mechanism explains the dominant trend convincingly; it is not a substitute for measuring the specific substrate-backbone-benchmark combination you actually care about, which is precisely why Chapter 6 built a harness that measures rather than assumes.

Chapter 10: Scaling Limits & the Design Rule

Everything so far has held history length roughly fixed and varied the substrate. This final chapter does the opposite: fix a substrate's regime (MAB's Conflict Resolution benchmark, the paper's scalability probe from Chapter 6) and grow the history from 6,000 tokens to 32,000 to 262,000 — a more than 40× increase — on Qwen3-32B-AWQ. Which substrates keep working, and which ones hit a wall, turns out to sort cleanly along the family lines Chapters 1 through 5 laid out.

Worked example 9 — refinement scales, and pays for it predictably

Quality alone makes it look like every method except M9 keeps improving with more history — more context genuinely does contain more of the information Conflict Resolution needs to reconcile. The story changes once you look at what each of those quality gains cost. M8's latency moves from roughly 1 second to roughly 10 seconds across the sweep — a 10× latency cost for a 59% relative quality gain. M5 and M10 both "face steep latency costs at 262K," in the paper's own words — M10 in particular "scales well in quality... but pays linear read-time cost," meaning its latency grows in direct proportion to how much history it has to read through, with no compression cushioning that growth the way M8's bundling does.

The design rule

Put every chapter of this lesson together and the paper lands on a single, actionable sentence: trade read breadth for write depth. Retrieve fewer entries per query. Invest more compute distilling and structuring memory at write time, so that what gets retrieved is already precise enough that you do not need to retrieve much of it.

Every result in this lesson is an instance of that rule, viewed from a different angle. M7 and M8 (Chapter 4) spend LLM calls at write time specifically so the read side hands back one clean unit instead of a pile to sift. M11 (Chapter 5) spends a cheap boundary check at write time so the read side re-attends to exactly one matched episode instead of the full trajectory buffer. M5 (Chapter 3) spends the most at write time of any external substrate, and it is not a coincidence that it also wins the benchmark where read-time precision (which specific fact, connected to which other fact) matters most. The substrates that skip write-time investment — M1, M2, M10 — are cheap and fine when the task rewards raw breadth over precision, and they are exactly the ones that either collapse (Chapter 7's M10-on-LME-S result) or actively hurt (Chapter 8's M2-below-NoMem result) once the task instead punishes broad, unfiltered retrieval.

The broader implication: route, don't commit

No single substrate wins every regime in this paper's own results — and the paper's conclusion follows that fact to its logical end. Production agent memory should not be built by picking one substrate at design time and hoping it generalizes. It should be a multi-substrate system that routes between substrates per regime: recall-heavy, dialogue-shaped interaction routed to graph or hierarchical memory; long-horizon agentic execution routed to refinement or episode-clustering memory. The routing decision itself becomes part of the system design, not an afterthought bolted onto whichever substrate a team happened to reach for first.

Where production systems already sit

Three widely deployed memory systems — Zep, Mem0, and MemGPT — are notably absent from the main controlled comparison in Tables 1 and 2. The paper reports them separately, in an auxiliary-cost appendix, specifically because their production pipelines introduce auxiliary-LLM budgets that differ from the fixed gpt-4o-mini budget Chapter 6 insisted on for a fair comparison. They are useful precisely as a contrast: production memory systems in the wild already tend to be hybrids — combining elements of several of this paper's eleven controlled substrates — which is itself informal evidence for the routing conclusion, arrived at independently by engineers building real deployed systems before this paper measured why.

Return one last time to the two jobs from Chapter 0. A production system serving both the companion app and the warehouse robot is not choosing between M5 and M11 — it is running both, routed by which regime the current interaction belongs to, with the routing decision itself informed by exactly the mechanism Chapter 9 measured: does this task need broad recall, or does it need attention protected for the current, action-critical state?

A worked routing decision, end to end

Put the whole lesson together on one concrete design problem: a single deployed assistant that both remembers ongoing user relationships (the companion-app job) and executes multi-step tool-use tasks on the user's behalf (closer to the warehouse-robot job, minus the physical embodiment). Chapter 9's mechanism gives you a routing rule, not a guess: classify each incoming turn by whether it is building durable, cross-session context (route to a Structural or Hierarchical substrate, spending write-time compute because read-time precision across sessions is what the task rewards) or executing a bounded, current-state-dependent action sequence (route to a Refinement or episode-clustering substrate, keeping retrieved context narrow because the policy's attention needs to stay on the current tool call's actual state, not on a backlog of loosely related past sessions).

The routing signal itself does not need to be complicated — the two regimes in this paper's own benchmark suite (dialogue-shaped versus trajectory-shaped input) are already a reasonable first-pass classifier, and a real system can refine it further with its own task-specific signal. What matters is that the decision is made explicitly, per-turn or per-task, rather than baked in once at design time as "we use a vector database for everything" — which was, per Chapter 0's landscape survey, close to the default the field actually shipped in 62% of the 52 surveyed systems' benchmark choices.

What this paper does not settle

Being precise about the paper's boundaries matters as much as being precise about its findings. The harness runs three backbones, not every backbone in production use; it runs four benchmark suites, which is far more than most prior work but still a finite sample of possible task shapes; and its scalability probe covers one benchmark (MAB Conflict Resolution) at three context lengths, not a fully general characterization of every substrate's asymptotic behavior. The design rule — trade read breadth for write depth — is the paper's best generalization from the evidence it does have, not a law derived from first principles. Treat it the way this whole lesson has tried to treat every number in it: as a strong, mechanistically grounded prior to start from, not a substitute for measuring your own specific deployment.

Connections to the rest of the site

This lesson assumed familiarity with what a vector embedding is and what retrieval-augmented generation does. If either felt shaky, the site's RAG and vector-database Gleams cover that ground from zero. The Agent Memory Gleam in the OpenClaw series covers one concrete, file-backed implementation of hybrid vector-plus-keyword memory with consolidation — a real system occupying roughly the Text-Record-to-Structural boundary of this paper's taxonomy, useful as a worked example of what M3/M4-adjacent design looks like outside a research harness. For the broader question of how a harness like this one gets built and instrumented in the first place — choosing metrics, controlling confounds, designing ablations that isolate one variable at a time — the site's evaluation and analytics family of lessons builds that methodology up from first principles, independent of any one paper's specific substrates.