Context & Streaming — What the Model Sees, and How You See It Back (OpenClaw 5/7)

Chapter 0: The Two Pipes — Context In, Tokens Out

This lesson in one sentence. Lesson 4 drew the agent loop as a row of stages and waved a hand at two of them: “context assembly” (build the prompt) near the front and “streaming replies” (send the answer) at the back. This lesson opens both boxes. They are two pipes running in opposite directions, and each is a formatting-under-constraints problem in its own right.

Picture the gateway in the middle of a turn. A human just typed something into WhatsApp. Before the model can think, the gateway has to build the exact block of text the model will read — who the agent is, what tools it has, what's in its workspace, what the user said. That's the first pipe: context in. Text flows toward the model.

Then the model produces an answer, and the gateway has to get that answer back onto the screen. But here is the catch that makes this hard: the model emits its reply token by token, a steady drip of word-fragments. A chat app like WhatsApp has no idea what a token is. It knows how to show a message — a finished chunk of text. So the gateway has to repackage a token stream into something a chat surface can actually display. That's the second pipe: tokens out. Text flows away from the model.

The shape of the whole lesson. Two pipes, two constraints. Context in is constrained by a token budget: the prompt must FIT inside the model's context window, and you want the stable part to be cacheable so it's cheap. Tokens out is constrained by the surface: a chat app can't take a raw token drip, so you must group tokens into clean, valid, well-paced chunks. Same gateway, two formatting problems pointing in opposite directions.

Notice these are genuinely different problems. The inbound pipe is about selection and ordering under a size cap — what to include, what to drop, what to put where. The outbound pipe is about buffering and chunking under a display constraint — how much to hold before you send, where it's safe to cut. We'll spend Chapters 1 through 5 on the first pipe and Chapters 6 and 7 on the second, then watch both run together in the showcase.

Why care this much? Because these two pipes are where an agent gateway feels either professional or broken. Get context assembly wrong and the model forgets its own name halfway through a long workspace, or you burn money re-sending the same unchanging text every single turn. Get streaming wrong and your assistant either spams the channel with one-line fragments or freezes for thirty silent seconds and then dumps a wall of text. The whole art is here.

Why does the gateway need a dedicated “tokens out” pipe at all — why not just hand the model's output straight to the chat app?

Because chat apps are too slow to receive text. Because the model output is encrypted and must be decoded. The model emits a token-by-token drip, but a chat surface only knows how to display finished messages — the gateway must repackage the token stream into chunks the surface can show. Because tokens are larger than messages and must be compressed.

Chapter 1: System-Prompt Assembly — Three Layers

Let's follow the inbound pipe. The single most important piece of text in the whole system is the system prompt — the instructions and context the model reads before the user's message. It tells the model who it is, how to behave, what tools exist, what's in the workspace. Everything downstream depends on getting this block of text right.

You might imagine the gateway building this prompt by stuffing variables into one giant template string. It doesn't — and the reason it doesn't is a clean lesson in software design. OpenClaw splits prompt-building into three layers, each with a strictly narrower job than the last.

The three layers, outside-in.

buildAgentSystemPrompt — a pure renderer. Hand it explicit inputs (the owner's name, the tool list, the mode) and it returns the prompt text. Same inputs in → same text out, every time. It reads no files, calls no network, touches no clock. This is the part you can test in isolation.
resolveAgentSystemPromptConfig — the config resolver. It decides the knobs the renderer will use: the owner's display name, text-to-speech hints, model aliases, whether memory should be cited, which delegation mode is active. It turns settings into concrete values.
runtime adapters — the fact gatherers. They reach into the live system and collect what's true right now: which tools are actually connected, the sandbox's current state, the node capabilities available, the workspace context files. These are the impure, moment-to-moment facts.

Read the layers from the inside out and the logic is: the runtime adapters gather live facts, the config resolver folds in settings, and the pure renderer turns both into text. The renderer is deliberately dumb about where its inputs came from. That separation is what lets you reason about the prompt at all — the only layer that produces words is the one with no side effects.

Why a PURE renderer matters. If the function that produces the prompt also read files and queried the network, you could never be sure why a given prompt came out the way it did — the output would depend on the state of the disk and the clock at that instant. By forcing all the messy fact-gathering into separate adapter layers and feeding the renderer only explicit arguments, OpenClaw makes the prompt reproducible: give the renderer the same inputs and you get a byte-for-byte identical prompt. That is the difference between a prompt you can debug and one you can only pray about.

There's one more flourish. The prompt is built from named sections — an interaction-style section, a tool-call-style section, an execution-bias section, and so on. Because each is named, a provider plugin can swap a small named section for a different model without rewriting the whole prompt. A model that likes terse tool calls gets a different tool_call_style section; the rest of the prompt is untouched. Named, swappable pieces — not one frozen block.

Why is buildAgentSystemPrompt deliberately written as a PURE renderer that reads no files and calls no network?

Because pure functions run on more CPUs at once. So the prompt is reproducible — same explicit inputs always yield the same text — which makes it debuggable; all the impure fact-gathering is pushed into the adapter layers. Because the model refuses prompts built by impure functions. So the prompt can be encrypted before sending.

Chapter 2: Prompt Modes & Token Budgets

Not every agent run needs the full, lavish system prompt. A small sub-agent spun up to do one narrow job doesn't need the whole personality, the memory-recall instructions, the messaging etiquette, the voice hints. Sending it all that is wasted tokens — and tokens are both money and space. So the renderer takes a mode.

Three prompt modes.

full — the default. Every section is rendered: identity, memory recall, self-update, model aliases, user identity, output directives, messaging, silent replies, heartbeats, and the rest. The complete agent.
minimal — for sub-agents. It omits a specific set of sections that a focused helper doesn't need: Memory Recall, Self-Update, Model Aliases, User Identity, Output Directives, Messaging, Silent Replies, Heartbeats. What's left is the core instructions to do the job.
none — just the base identity line. The barest possible prompt, for when you want the model almost raw.

Modes are not just about cost; they are about fit. Every model has a hard ceiling on how much text it can read — its context window, measured in tokens. The prompt plus the conversation plus the room for the answer all have to live inside that ceiling. OpenClaw enforces model-specific token limits: the assembled prompt is checked against the target model's budget, and the prompt must fit. There is no “mostly fits.”

What “must fit” forces. If the assembled sections would blow past the budget, something has to give — sections get truncated or dropped. This is why assembly can't be naive concatenation. The assembler has to (a) skip sections the mode doesn't want, (b) keep each section under its own cap, and (c) stop adding sections once the running total would overflow the budget. Three rules, every one of them load-bearing. Let's build exactly that assembler.

Below is the core of the inbound pipe in miniature: an assemble function that respects the mode and the budget. The numbers are scaled down so you can see the behavior on a few short strings, but the logic is the real logic — omit by mode, truncate per-section, stop at the total cap.

A sub-agent is built in minimal mode. What is true of its system prompt compared to a full one?

It is identical — mode only changes the model, not the text. It is longer, because sub-agents need more instruction. It omits a defined set of sections (Memory Recall, Messaging, Heartbeats, and others) the focused helper doesn't need — smaller, cheaper, and it still must fit the model's token budget. It ignores the token budget entirely.

Chapter 3: Bootstrap Context Files

So far the prompt has been instructions about the agent. But the agent also needs to know about its world — the project it's working in, its own personality, who its owner is. That world lives in a set of workspace files, and the gateway injects them into the prompt when the agent boots. These are the bootstrap context files.

Each file has a job, and the gateway pulls them in by the lifetime they describe:

The workspace files.

AGENTS.md — the project's working agreement: how this codebase or workspace expects an agent to behave.
SOUL.md — the agent's personality: tone, character, voice.
TOOLS.md — notes on the tools available here.
IDENTITY.md & USER.md — who the agent is, and who the owner is.
HEARTBEAT.md — what to do on a timed beat.
BOOTSTRAP.md — first-run instructions, injected only for new workspaces.
MEMORY.md — the long-term memory of the workspace.

Now the constraint returns, because these files can be huge. OpenClaw enforces two caps. Each individual file is limited by bootstrapMaxChars — twenty thousand characters — and the files together are limited by bootstrapTotalMaxChars, sixty thousand. A workspace with a sprawling MEMORY.md would blow past both. So when a file is over its cap, the gateway truncates it and leaves a marker where the cut happened.

The key reframe: truncation is NOT data loss. It would be alarming if a truncated MEMORY.md meant the agent simply forgot the rest. It doesn't. The truncated prompt is only the opening view. The model can reach the full MEMORY.md whenever it needs, by calling memory_search or memory_get — tools that read the complete file on demand. The prompt holds a bounded preview; the tools hold the rest. Truncation trades a smaller, cheaper prompt for one extra tool call when (and only when) the model actually needs the deeper content.

This is the same pattern you'll see again and again in the inbound pipe: put a small, bounded thing in the prompt; keep the full thing reachable by tool. The prompt is an index, not the library. And it scales down for sub-agents too: a sub-agent doesn't get the whole workspace — it's injected only AGENTS.md and TOOLS.md, the bare minimum to do focused work, keeping its context small on purpose.

A workspace's MEMORY.md is far larger than the per-file cap, so the prompt only contains a truncated slice of it. Has the agent lost the rest of its memory?

Yes — anything past the cap is gone for that run. No — truncation only bounds the prompt; the model reaches the full MEMORY.md on demand via memory_search / memory_get. The prompt is an index, not the whole library. Yes, but only until the workspace is rebooted. No — the cap is never actually enforced.

Chapter 4: Skills Injection — The Menu, Not the Meal

Bootstrap files told the agent about its world. Skills tell it about its abilities — reusable procedures it can follow, each documented in a SKILL.md file. A workspace might have dozens of them. If the gateway pasted every full SKILL.md into the prompt, the budget would evaporate before the user even spoke.

So OpenClaw does the same trick as with memory, sharpened. Into the prompt goes a compact list of available skills: for each, just its path and a small content-version marker. That's it — a menu. The full SKILL.md is read on demand, only when the model decides to use that skill.

The menu-not-the-meal pattern. The prompt lists skill paths plus a version marker per skill. To actually use a skill, the model reads the full SKILL.md at that path. The version marker is the clever part: if the model has read a skill before but its marker has changed, the model knows the skill was edited and re-reads it. Unchanged marker → trust the cached understanding. Changed marker → refresh. A tiny string keeps the model honest without re-injecting the whole file.

There is still a budget here — even the menu can grow too long. OpenClaw caps the skills list with skills.limits.maxSkillsPromptChars (plus a per-agent allowance). But because the list is just paths and markers, that cap buys a lot of skills: hundreds of one-line entries fit where a handful of full files would not. This is the whole reason on-demand loading exists — it keeps the base prompt small while leaving an enormous library one tool-call away.

The recurring shape of the inbound pipe. Notice the pattern has now appeared three times. Memory: index in the prompt, full file by tool. Bootstrap files: truncated preview in the prompt, full content by tool. Skills: path-plus-version in the prompt, full SKILL.md by tool. The inbound pipe's governing idea is lazy loading under a budget: the prompt carries pointers and previews; the model pulls the heavy content only when it needs it. That's how a finite context window holds an unbounded workspace.

The prompt lists each skill as just a path plus a content-version marker. What is the version marker FOR?

It encrypts the skill so other agents can't read it. It tells the model how many tokens the skill costs. It lets the model tell whether a skill changed since it last read it — a changed marker means re-read the full SKILL.md; an unchanged one means the cached understanding still holds. It is a random ID with no behavioral meaning.

Chapter 5: The Cache Boundary — The Money Idea

Here is the single most valuable idea in the inbound pipe, and it's about money. Every turn, the gateway sends the model a prompt. Most of that prompt — the identity, the project context, the skill menu — is the same as last turn. Re-processing identical text from scratch, turn after turn, is pure waste. LLM backends know this, and they fix it with prefix caching.

What prefix caching does. The backend remembers the work it did on a stable prefix of the prompt — the longest run of text from the very start that hasn't changed since last time. If your new prompt begins with the exact same bytes, the backend reuses that cached work instead of recomputing it. The reused prefix is cheaper and faster. But it only works on a contiguous run from the front: the moment the text differs from what was cached, the cache stops — everything after the first change is recomputed.

That last sentence is the whole game. Caching reuses the longest unbroken stable prefix. So a single volatile section placed near the top of the prompt poisons everything below it — even content that never changes. The fix is almost embarrassingly simple, and it's the trick: order the prompt by volatility.

Stable above the line, volatile below it. OpenClaw puts large stable content — the Project Context — at the TOP, above an imaginary cache boundary. Everything that changes turn-to-turn goes BELOW it: the Control UI state, Messaging context, Voice settings, Group Chat info, Reactions, Heartbeats, the live Runtime status, even the current clock. Because all the volatile stuff sits at the bottom, the entire top of the prompt forms one long, unbroken, cacheable prefix that gets reused every turn. Re-order it badly — drop one live clock near the top — and the cacheable prefix collapses to almost nothing.

Let's make this measurable. In the lab below, sections are tagged stable or volatile, and you'll compute how many characters the cache can actually reuse for two layouts: a GOOD one (all stable first) and a NAIVE one (a big volatile section sitting near the top). The gap between them is real money saved per turn.

And here is the same idea as a picture. Drag the cache boundary up and down a stacked prompt; sections above are reused (cached), sections below are recomputed. Flip on the naive layout to drop a volatile section high and watch the reuse number collapse.

The cache boundary — drag the line, watch reuse

A prompt is a stack of sections; teal = stable, orange = volatile. The dashed line is the cache boundary: everything above it is reused from the cache; everything below is recomputed this turn. Drag the line, and toggle naive layout to slip a volatile section up high — the reuse percent reacts. The real win comes from putting every stable section above the highest volatile one.

cache boundary55

Why does OpenClaw push volatile sections (live clock, runtime status) to the BOTTOM of the prompt and keep stable content (project context) at the top?

Because the model reads bottom-to-top, so important things go last. To make the prompt look tidier in logs. Prefix caching reuses only the longest unbroken stable run from the front; a volatile section placed high breaks the prefix and forces everything below it to be recomputed. Ordering by volatility maximizes the cached (cheaper, faster) prefix. Because volatile sections are larger and must go last to fit.

Chapter 6: Streaming I — Block Streaming

Switch pipes. The model is now producing an answer, token by token, and we have to get it onto a chat surface. The obvious dream is true token-by-token streaming — the words appearing one at a time, like a person typing. The problem: most channels can't take that. A WhatsApp or Telegram message is an atomic thing you send once; you don't dribble a single message into existence one character at a time. So OpenClaw streams in coarse blocks instead of token deltas.

The EmbeddedBlockChunker. The chunker watches the growing reply and decides when to cut off a block and send it. It's governed by two bounds:

minChars — a low bound. The buffer must fill to at least this many characters before the chunker is even allowed to emit. This stops it from sending a one-word fragment.
maxChars — a high bound. If the buffer reaches this size, the chunker is forced to split, even if it would rather wait. This stops any one block from getting too big.

Between those bounds, the chunker looks for a graceful place to cut.

What's a graceful place? The chunker prefers to break where text naturally breathes, walking down a break-preference ladder: first try to end at a paragraph break; if none, a newline; if none, a sentence end; if none, any whitespace; and only as a last resort, a hard break mid-word at maxChars. The ladder means blocks almost always end on a clean boundary — a finished thought — rather than slicing a word in half.

The one rule it never breaks: don't split a code fence. Markdown code blocks are delimited by triple backticks. If the chunker cut a reply in the middle of a fenced code block, the first block would have an opening fence with no closing one — broken Markdown, a mangled render on the surface. So the chunker is fence-aware: if a split would land inside a code block, it closes the fence at the end of the block and reopens it at the start of the next, keeping every emitted block valid Markdown on its own. (We'll watch this exact behavior in the showcase canvas.)

The controls expose all of this: blockStreamingDefault (off unless you opt in), blockStreamingBreak (whether to break at text_end or only at message_end), and blockStreamingChunk — the {minChars, maxChars, breakPreference} triple that tunes the bounds and the ladder. Let's build the core of it: the chunker that honors both bounds and the break ladder.

The block chunker has filled past minChars but not yet reached maxChars. How does it decide where to cut the block?

It always cuts at exactly minChars. It walks a break-preference ladder — paragraph, then newline, then sentence end, then whitespace, then a hard cut at maxChars — so blocks end on clean boundaries; and it never splits inside a code fence. It cuts at a random position between the two bounds. It waits for the entire reply to finish, then cuts once.

Chapter 7: Streaming II — Previews, Coalescing, Pacing

Block streaming gets clean chunks out. But raw chunks, fired as fast as they're ready, still feel wrong — a burst of five tiny messages, or a robotic instant reply. The outbound pipe has a second half whose entire job is to make the stream feel human and legible. Three mechanisms do it.

First, preview streaming. Some surfaces — Telegram, Discord, Slack — let you edit a message after sending it. OpenClaw exploits this to show progress in a single message that updates in place, rather than spamming new ones. It has modes:

Preview modes.

off — no live preview; the final answer arrives when it's done.
partial — one preview message is posted, then replaced by the final answer.
block — the message is grown by appending chunks as they arrive (chunked appends).
progress — a status line (“thinking…”) is shown, then swapped for the final answer.

Second, coalescing. Even with clean blocks, consecutive small chunks can arrive in a flurry. Coalescing merges them: it waits a short idleMs for the next chunk and, if one comes, glues them together, capped by maxChars. On chatty surfaces — Signal, Slack, Discord — the default minChars is bumped up to fifteen hundred so blocks are bigger and the channel doesn't fill with one-liners. Merge the dribble; send fewer, fuller messages.

Third, human pacing — and the silent token. humanDelay inserts a natural pause — roughly eight hundred to twenty-five hundred milliseconds — between block replies, so a multi-block answer arrives like a person typing in turns rather than a machine dumping text. And there's a special escape hatch: if the model decides it should say nothing, it emits the silent token NO_REPLY, which the gateway filters out entirely — no empty message reaches the surface. Finally, while a tool is running, the gateway can show a tool-progress preview (“searching the web…”) so the user isn't staring at silence during a long tool call.

Stack these and the outbound pipe is complete: blocks are cut cleanly (Chapter 6), merged so they aren't spammy (coalescing), paced so they feel human (humanDelay), shown live where the surface allows (previews), and suppressed entirely when there's nothing to say (NO_REPLY). The model emits a token drip; the user sees a calm, well-timed conversation.

On Signal/Slack/Discord, OpenClaw bumps the default minChars up to ~1500 and waits idleMs to merge consecutive chunks. What problem do coalescing and that higher floor solve?

They make the model think faster. They stop the gateway from spamming the channel with many tiny one-line messages — merging consecutive chunks and raising the size floor sends fewer, fuller messages. They encrypt the messages before sending. They let the model exceed the token budget.

Chapter 8: Showcase & Connections

Here is the outbound pipe in motion, with the one rule that's hard to believe until you see it: the chunker never splits a code fence. Below is a long Markdown reply — prose, then a fenced code block, then more prose. Press Stream and watch the chunker emit blocks that honor minChars and maxChars and the break ladder, and that close and reopen the code fence rather than ever cutting it open. Slide the bounds and replay.

The stream inspector — blocks, bounds, and the unbroken fence

The full reply is the faint text; each emitted block is outlined as the chunker cuts it. Teal outlines end on a clean boundary; the orange region is inside a code fence — notice the chunker closes the fence at a block's end and reopens it at the next, so every block is valid Markdown. Tune minChars / maxChars and press Stream.

minChars60

maxChars160

Press Stream to chunk the reply.

Both pipes in one breath. Context in: a three-layer assembler (pure renderer ← config resolver ← runtime adapters) builds named sections, under a mode (full / minimal / none) and a model token budget; bootstrap files and skills go in as bounded previews with the full content reachable by tool; and the prompt is ordered by volatility so a stable prefix sits above the cache boundary and is reused cheaply each turn. Tokens out: the model's token drip is cut into clean blocks (minChars / maxChars / break ladder, never splitting a code fence), then coalesced, paced by humanDelay, shown live by preview modes, and suppressed by NO_REPLY when there's nothing to say.

Where this connects

← Lesson 4 — The Agent Loop. The stage we zoomed into: “context assembly” at the front and “streaming replies” at the back are exactly the two pipes of this lesson.
Lesson 6 — The Command Queue → Keeping overlapping turns from colliding — what happens when a second message arrives while one is still streaming out.
Prompt Engineering — the broader craft of shaping what a model reads; this lesson is that craft made systematic and budget-aware.
Retrieval-Augmented Generation — the cousin of context assembly: RAG fetches the right chunks to put in the prompt, exactly the lazy-loading-under-a-budget idea we used for memory and skills.

The Feynman test. Close the lesson and explain to a friend: the two pipes and why each is a formatting-under-constraints problem; the three assembly layers and why the renderer is pure; the three prompt modes and why the prompt must fit a token budget; why truncating a bootstrap file is not data loss; the cache boundary and why ordering by volatility saves money; and the block chunker's two bounds, its break ladder, and the one rule it never breaks (don't split a code fence). If you can, you own both directions of the gateway's text flow.

In one sentence, what are the “two pipes” this lesson opened?

Input encryption and output decryption. The WhatsApp pipe and the Telegram pipe. Context in — building the exact prompt the model reads under a token budget and a cache boundary — and tokens out — repackaging the model's token-by-token reply into clean, paced blocks a chat surface can show. The training pipe and the inference pipe.