Self-Healing & Reliability — When Things Go Wrong (OpenClaw 9/11)

Chapter 0: Why — Failure Is the Normal Case

The whole lesson in one sentence. A real agent gateway runs for weeks against flaky networks and busy providers, so it is built around a single stance: something is always failing — expect it, classify it, and recover automatically. This lesson builds the four self-healing reflexes that make that stance real: retry, failover, compaction, and diagnostics.

By Lesson 8 your gateway can hold a conversation, run an agent, call a model, use tools, and remember. Every one of those steps assumed the happy path: the network was up, the model answered, the context fit. On your laptop, for five minutes, that assumption holds. Run the same gateway for a week and it stops holding the way "the sun is up" stops holding — reliably, on a schedule you didn't pick.

Here is the uncomfortable arithmetic. Suppose any single model call succeeds 99.9% of the time — one failure in a thousand, which sounds great. A busy agent that makes ten thousand calls a day will then see roughly ten failures every day. Not "if." When. At scale, rare events are not rare; they are a steady drizzle. The question is never "will something fail?" but "what does the system do when it does?"

A toy agent answers that question by crashing, or by handing the user a stack trace, or by silently producing garbage. None of those is acceptable for a thing you talk to from your phone all day. The fix is a change of stance: stop treating failure as an exception you didn't plan for, and start treating it as the normal case — an input the system is designed to absorb.

The design stance: expect, classify, recover. A self-healing system does three things on every failure. Expect it — assume the call might fail and wrap it in machinery, don't bolt that on later. Classify it — not all failures are alike; a momentary network blip and a malformed request need opposite responses. Recover it — pick the reflex that matches the class, automatically, without a human in the loop for the routine cases.

Those reflexes map cleanly onto the four ways a gateway tends to break. A transient hiccup — a dropped packet, a one-second outage — calls for retry: try again, a little later. A whole provider going down — your model vendor is overloaded — calls for failover: switch to a different model. A conversation growing past the model's memory — the context window overflows — calls for compaction: shrink what the model sees. And a run that seems frozen calls for diagnostics: figure out whether it is genuinely wedged or just doing slow honest work.

The rest of this lesson is those four reflexes, one or two chapters each, built up from the simplest version to the exact rules OpenClaw uses in production. By the end you'll have a map: this kind of failure, that reflex. Let's start with the most common failure of all — the transient blip — and the most misunderstood fix for it.

Why does a self-healing gateway treat failure as "the normal case" rather than an exception?

Because the code is poorly written and fails often. Because failures are illegal and must be reported. Because at scale even a tiny per-call failure rate means many failures per day — rare events are a steady drizzle, so the system must be designed to absorb them, not surprised by them. Because retrying is always faster than succeeding the first time.

Chapter 1: Retry — Backoff and Jitter

The first reflex is the simplest one to state and the easiest to get subtly wrong. When a call fails for a transient reason — a momentary network drop, a provider that was busy for half a second — the right move is often just to try again. The failure was a flicker; a second attempt sails through. But "try again" hides three decisions: how many times, how long to wait, and whether everyone should wait the same amount.

Start with how many times. Not forever — an infinite retry loop on a genuinely-down service is a way to hang your whole system. OpenClaw's default is 3 attempts: try, and if that fails, try twice more, then give up and let a different reflex (failover, next chapter) take over. Three is a sweet spot — enough to ride out a brief blip, few enough to fail fast when the blip is actually an outage.

Now how long to wait between attempts. Retrying instantly is a mistake: if the service is briefly overloaded, hammering it again immediately just adds to the load that caused the failure. So we wait — and we wait longer each time, because a failure that survived one short wait probably needs a longer one. This is exponential backoff: the delay doubles each attempt. One second, then two, then four, then eight. The idea is to back off the busy service gently, giving it room to recover.

Backoff with a cap. Doubling forever would soon mean waiting minutes between tries, which is useless — nobody wants to wait four minutes for attempt seven. So the delay grows exponentially up to a ceiling. OpenClaw caps the wait at 30 seconds: it doubles — one, two, four, eight, sixteen — and then flattens, never waiting more than half a minute between attempts. Exponential early, constant late.

Here is the decision people miss, and it is the whole reason this chapter exists. Imagine a provider hiccups and a thousand clients all get the same failure at the same instant. If every client uses the exact same backoff schedule — all wait one second, all retry — then a thousand retries land on the recovering service at the same moment, one second later. That synchronized wave is called a thundering herd, and it re-causes the very outage it was trying to recover from. The fix is jitter: add a small random wobble to each client's delay — OpenClaw uses about 10% — so the thousand retries spread out across a window instead of stacking on one instant.

The retry schedule — backoff, the cap, and the thundering herd

The bars show the delay before each attempt: doubling early, then flattening at the 30 s cap (the dashed line). Turn on thundering herd to release many clients at once: without jitter their retries pile onto the same instants (tall red spikes); with jitter they spread into a smooth band. Drag the jitter slider and watch the spikes melt.

jitter %10

So the full reflex is: try up to three times; wait an exponentially-growing delay between tries, capped at 30 seconds; and stir in 10% jitter so a crowd of clients doesn't retry in lockstep. Let's build the schedule itself — the function that turns an attempt number into a delay — and watch the curve double, flatten at the cap, and wobble.

Why does OpenClaw add ~10% random jitter to each client's retry delay?

To make retries take longer overall. Because random delays are easier to compute than fixed ones. So a crowd of clients that failed together don't retry in lockstep — jitter spreads their retries across a window instead of stacking them on one instant (a thundering herd) and re-causing the outage. To hide the retry from the provider.

Chapter 2: Classify — What Deserves a Retry

Retry is powerful, but applied blindly it is also harmful. The previous chapter quietly assumed every failure was a transient blip worth a second try. Most aren't. Retrying the wrong failure wastes time at best and corrupts data at worst. So before the retry reflex fires, the gateway runs a quick triage: error classification — is this the kind of failure that a retry could fix, or not?

The dividing line is simple to state. Transient failures are the fault of the moment, not the request: the network reset, the provider was momentarily busy, a request timed out. Try the same request again and it may well succeed. Permanent failures are the fault of the request itself: you asked for something that doesn't exist, or sent a malformed body, or weren't authorized. Trying the identical bad request a second time will fail in the identical way — you've just wasted a round trip.

Retry the transient; surface the rest immediately. OpenClaw retries on the transient signals — a request timeout (status 408), a conflict (409), being rate-limited (429), any server-side error (the 5xx family), and raw timeouts or connection resets. It does not retry the other client errors (the rest of the 4xx family — bad request, not found, forbidden): those mean you sent something wrong, so it surfaces the error to the caller right away instead of burning three attempts on a request that can never succeed.

The status numbers are worth a moment, because they encode who is at fault. The 4xx range means "the client made a mistake" — the request was bad. The 5xx range means "the server made a mistake" — the request was fine but something broke on their end. That single distinction is most of the classifier: a server-side error (5xx) might clear up on its own, so retry; a client-side error (4xx) won't, so don't. The exceptions are the few 4xx codes that are actually transient — 408 (a timeout dressed as a client error), 409 (a conflict that may resolve), and 429 (rate-limited, which is explicitly "try again later").

That last one, 429, often comes with a gift: a Retry-After header, where the provider tells you exactly how long to wait before trying again. Honor it — it overrides your computed backoff, because the provider knows its own recovery time better than your formula does. But put a ceiling on your patience: if Retry-After says "wait more than 60 seconds," OpenClaw stops retrying and surfaces, so a different reflex can take over. A minute-long wait isn't a blip anymore; it's a sign this provider is down, and the right move is to fail over to another one (next chapter), not to sit and wait.

Retries are per-request, not per-flow. This is the subtle one. A single user action — "send this message" — might be several steps under the hood: look up a contact, then post the message, then mark it delivered. Retry operates on each individual request, never on the whole flow. Why does that matter? Because if step two (post the message) succeeds but step three fails, you must not re-run the whole flow — that would post the message twice. Retry only re-attempts the one request that failed. And the only reason re-attempting a request is safe at all is idempotency (Lesson 2): each request carries a key so the provider can recognize "I've already done this one" and not duplicate it. Retry without idempotency is a duplication bug waiting to happen.

So the classifier is the gatekeeper standing in front of the retry reflex. Transient signal? Hand it to retry. Permanent client error? Surface it now. Rate-limited with a short Retry-After? Wait exactly that long. Rate-limited with a long one? Stop and let failover step in. Triage first, then react — which brings us to what happens when retry gives up entirely.

A request returns a 404 Not Found (a 4xx client error). Should the retry reflex try it again?

Yes — retrying anything is harmless and might work. No — a 4xx client error means the request itself is wrong; the identical request will fail identically, so the gateway surfaces it immediately instead of wasting attempts (unlike 408/409/429 and 5xx, which are transient). Yes, but only after waiting an hour. No — 4xx errors are illegal to retry by law.

Chapter 3: Failover — Walking the Candidate Chain

Retry handles a flicker. But sometimes the thing flickering is the whole provider — your model vendor is overloaded, or your API key got rate-limited for the next hour, or billing got disabled. No number of retries against that provider will help; you need a different one. That is the second reflex: failover — when a whole provider is unusable, switch to another model.

For failover to work, the gateway needs a list of which models to try, in order. OpenClaw builds this candidate chain from three pieces, and the order is deliberate:

The candidate chain: requested → fallbacks → primary, deduped.

The requested model goes first — you asked for it, so we try your choice before anything else.
Then the configured fallbacks, in their listed order — the backups you set up in advance.
Finally the primary model is appended at the end — so after exhausting the alternatives we can settle back to the house default.
The whole list is deduplicated while preserving order: if the requested model is the primary, it appears once at the front, not again at the back.

With the chain built, failover walks it. Try the first model. If it answers, you're done — return that model's result. If it fails, the question is whether to advance to the next candidate or stop walking entirely. That decision is the heart of failover, and it depends on why the model failed.

Advance when the failure is the provider's fault and another provider might do better: an auth failure (this provider rejected our credentials), a rate limit (this provider is throttling us), being overloaded or busy (this provider is swamped), a timeout (this provider didn't answer in time), or billing disabled (this provider's account is frozen). All of those say "this provider can't help right now — try the next one."

When NOT to advance. Three failures must stop the walk instead of advancing. An explicit abort — the user or operator cancelled the run — means stop; there's nothing to fail over to. A context-overflow error means the conversation grew past the window — but that is not the provider's fault, and switching models won't fix it (the next model has a window too). That's a job for compaction (Chapter 5), so failover steps aside. And when no candidates remain — we walked off the end of the chain — there is simply nothing left to try, so we report exhaustion. Advancing on a context overflow would be a bug: it would burn every model in the chain on a problem none of them can solve.

Let's build the walker exactly. Two pieces: assemble the candidate chain (with the dedup that keeps a requested-equals-primary model from appearing twice), then walk it — returning on success, advancing on a provider failure, and stopping cold on a context overflow or an abort.

A model call fails with a context-overflow error. Why does failover STOP rather than advance to the next model in the chain?

Because context overflow crashes the whole gateway. Because the next model is always slower. Because overflow isn't the provider's fault — every model has a window, so switching won't help; the fix is compaction (shrink the context), not a different model. Advancing would burn the whole chain on a problem none can solve. Because the requested model must always win.

Chapter 4: Cooldowns & Auth Rotation

Failover walks past a broken provider once. But it would be foolish to walk right back to that same broken provider on the next request, a second later, only to fail over again. A provider that just rate-limited you is going to rate-limit you again immediately. So when failover skips a provider, the gateway also puts that provider in a cooldown: a timeout during which we don't even try it. Stop hammering the thing that's down.

And the cooldown grows if the provider keeps failing — the same exponential idea as backoff, applied to whole providers. The first failure earns a short rest; repeated failures earn longer ones, because a provider that's failed three times in a row is probably going to keep failing for a while.

Exponential cooldowns. A failed provider's cooldown climbs through roughly one minute, then five, then twenty-five, then an hour — each step about five times the last, so we back off harder the more it misbehaves. Billing-disabled failures are treated as more serious (the account is frozen, not just busy), so they start much longer — around five hours — then double on repeat, capped at 24 hours. The principle is the same throughout: the worse and more persistent the failure, the longer we leave the provider alone.

There's a second knob in the same spirit. A provider failure isn't always the provider's fault — sometimes it's the specific credential you're using. One API key got rate-limited; another might be fine. So alongside cooldowns, the gateway can rotate auth profiles: it holds several ways to authenticate and picks a different one when the current credential is failing.

The rotation order is deliberate. OpenClaw prefers OAuth profiles before raw API keys (OAuth tokens tend to carry higher limits and cleaner identity), and among equals it picks the oldest-used first — the credential that's had the longest rest, so usage spreads evenly instead of pounding one key.

Pin the profile for cache warmth. Here's a non-obvious wrinkle. Once a profile is chosen for a session, it is pinned — the same credential is reused for the rest of that session rather than re-picked every message. Why? Because providers keep a prompt cache keyed partly to the credential: re-sending a conversation under the same profile lets the provider reuse its cached work on the shared prefix, which is faster and cheaper. Switching profiles mid-session would throw that warmth away. The pin only releases on a reset, a compaction, or a cooldown — the moments when the cached prefix is invalid anyway. Reliability and efficiency, pulling the same direction.

So between failover (skip the broken provider now) and cooldown (don't come back too soon) and auth rotation (try a fresher credential before giving up on the provider entirely), the gateway has a graded response to provider trouble — from "wait a beat" all the way to "leave it alone for a day." Next we turn to a failure that no amount of switching providers can fix: the conversation that outgrows the model's memory.

Why is a chosen auth profile pinned for the rest of a session instead of re-picked on every message?

Because re-picking is illegal. Because providers keep a prompt cache keyed partly to the credential — reusing the same profile lets the provider reuse cached work on the shared prefix (faster, cheaper); switching mid-session throws that cache warmth away. The pin releases only on reset, compaction, or cooldown. Because there is only ever one profile. Because rotating profiles uses too much memory.

Chapter 5: Compaction — Context-Overflow Recovery

The third reflex answers a failure unique to language-model agents. Every model can only read a fixed amount of text at once — its context window (Lesson 5). A long conversation keeps growing: every user message and every agent reply is appended to the transcript the model sees. Eventually the transcript brushes up against the window, and the next call returns a context-length-exceeded error. The model has, in effect, run out of memory mid-thought.

We saw in Chapter 3 that switching providers won't help — the next model has a window too. The real fix is to make the transcript smaller. That's compaction: when the transcript nears the window (or the model explicitly says it overflowed), summarize the older turns into one compact entry and keep the recent tail verbatim. The distant past becomes a paragraph of summary; the recent exchanges stay word-for-word, because that's what the current reply depends on.

The model's view shrinks; the real history doesn't. Compaction only changes what the model sees on the next call. The full, untouched transcript stays on disk — nothing is deleted. If you scroll back through the conversation later, every word is still there. Compaction is a lossy view for the model's benefit, layered over a lossless record on disk. You trade some of the model's memory of the distant past for the ability to keep going at all.

So far so simple: cut the transcript at some boundary, summarize everything before it, keep everything after. But there is a trap that makes a naive cut corrupt the conversation, and it's the whole reason compaction needs care.

Recall how tool use works (Lesson 4). When the agent calls a tool, the transcript records two entries that belong together: the tool call ("I'm calling the weather tool") and, right after it, the tool result ("the weather tool returned: 72°F sunny"). The model's format requires these to come as a pair — a tool result with no preceding tool call is malformed, and the API will reject it outright.

Never orphan a tool result. Now picture the cut landing between a tool call and its result: the call gets swept into the summary, but the result stays in the kept tail. You've just created an orphaned tool result — a result with no call in front of it — and the very next API call fails, defeating the whole point of compacting. The fix: if the boundary would split a pair, shift it earlier so the tool call stays in the tail with its result. The tail may grow a little to protect the pair; it must never split one.

Let's build compaction with exactly that safety. Choose a boundary so the last few entries are kept verbatim; but if that boundary falls between a tool call and its result, move it back to keep the pair whole. Summarize everything before the (possibly adjusted) boundary into one marker, and watch the orphan bug appear in the naive version and vanish in the careful one.

During compaction the cut boundary would fall between a tool call and its tool result. What does compaction do, and why?

Delete both so neither is in the way. Keep only the result and drop the call. Shift the boundary EARLIER so the tool call joins the tail with its result — a tool result with no preceding call is malformed and the API rejects it, so the pair must never be split (the tail may grow, never orphan). Summarize both into the marker.

Chapter 6: Diagnostics — Hung, or Just Busy?

The first three reflexes handle failures that announce themselves: an error code, an overflow message, a refused connection. The fourth reflex handles the sneakier case — a run that just sits there. Nothing has errored. No exception was thrown. The agent is simply… not finishing. Is it wedged, or is it doing slow honest work?

This matters because the two cases demand opposite responses. A genuinely stuck run should be killed and restarted — leaving it pins resources and blocks the lane. But a run that's legitimately slow — a big tool call, a long generation — must be left alone; killing it throws away real progress and frustrates the user. Get this wrong and you either let zombie runs accumulate, or you murder good work. The gateway needs to tell them apart.

Three diagnoses for a run that hasn't finished. OpenClaw classifies a non-finishing run into one of three states:

Stalled — making no progress: no new output, no tool activity, no tokens, nothing changing. This is the one that looks wedged.
Stuck — blocked waiting on something external: a tool that hasn't returned, a lock it can't acquire, a dependency that's hanging. It's not idle by choice — it's parked, waiting.
Long-running — legitimately slow: progress is happening, just slowly. Tokens are still streaming; the tool is still working. This run is healthy; it just needs more time.

The distinction the classifier is really drawing is between absence of progress and slowness of progress. A long-running run shows movement — the token count climbs, the tool reports steps — so the gateway leaves it be. A stalled run shows a flat line: the same state, message after message, with no forward motion. Stuck sits in between: not moving, but for an identifiable external reason (it's waiting on a named thing), which is diagnosable in a way pure stall is not.

Why classify instead of just timing out? A blunt timeout — "kill any run older than five minutes" — can't tell a wedged run from a long-but-healthy one, so it murders good work to catch the bad. Classification gives the operator a reason: this run is stalled (no progress — safe to kill), or this one is long-running (still producing — let it finish). The point isn't to automate the kill; it's to surface an honest diagnosis so a human can act without throwing away legitimate work.

So diagnostics is less a recovery action and more an observability reflex: it makes the invisible state of a run legible. Stalled, stuck, or long-running — three words that turn "why is this thing taking forever?" into an answerable question. And once a system can inspect its own runs, the natural next step is letting it inspect — and carefully repair — its own configuration.

A run hasn't finished, but its token count is still climbing and its tool is reporting steps. How does diagnostics classify it, and what's the response?

Stalled — kill it immediately. Stuck — force-unblock it. Long-running — progress is happening, just slowly, so leave it alone; killing it would throw away legitimate work. (Stalled = no progress; stuck = blocked waiting.) Crashed — restart the gateway.

Chapter 7: Self-Update & Audit — Repair, with Guardrails

The reflexes so far heal runtime failures — a flaky call, a down provider, a full context. But some failures are baked into configuration: a risky default left on, a setting that should have been flipped, a tool that needs adjusting. The most ambitious self-healing reflex lets the agent inspect and repair its own config — carefully, through the gateway, with the brakes wired in.

The agent does this through a dedicated gateway tool rather than by editing files directly. Two operations matter. config.schema.lookup lets the agent read the configuration schema — ask "what settings exist, what do they mean, what are the valid values?" — before touching anything. And config.patch lets it change a specific setting through the same audited path. Look up first, then patch: the agent never pokes at a config it doesn't understand.

The security audit: flip risky defaults back to safe. A separate command, openclaw security audit --fix, scans the configuration for risky settings — defaults that are convenient but unsafe — and flips them back to their safe values. It's a one-shot hardening pass: find the loosened bolts and tighten them. Run it after setup, or whenever you suspect the config has drifted toward "convenient but exposed."

The word doing the heavy lifting in this whole chapter is guardrails. Letting an agent rewrite its own configuration is genuinely dangerous — a careless patch could open a security hole or break the gateway. So self-update runs are wrapped in the same machinery as everything else: they go through the audited tool path (not raw file edits), and updates run with status checks — the system verifies the change took effect and the gateway is still healthy, rather than blindly assuming success.

Self-repair is a privilege, not a free-for-all. The point of routing self-update through config.schema.lookup / config.patch and the audit command — instead of handing the agent a shell and a text editor — is that every change is legible and reversible. The agent can read the schema to know what's valid, patch one setting at a time through an audited seam, and let the audit pass undo anything risky. Self-healing config is powerful precisely because it is fenced: the agent gets to fix itself, but only along the rails the gateway lays down.

That's the fourth corner of self-healing filled in. Retry rides out a flicker; failover routes around a down provider; compaction recovers an overflowed context; diagnostics makes a frozen run legible; and self-update lets the agent harden its own config — each one fenced so the cure is never worse than the disease. Now let's watch all of them work together on one request.

Why does the agent change its own configuration through config.schema.lookup / config.patch and the security audit --fix path instead of editing config files directly?

Because direct file editing is impossible. So every change is fenced and legible — the agent reads the schema to know what's valid, patches one audited setting at a time with status checks, and the audit pass can flip risky defaults back to safe. Self-repair with guardrails, not a free shell. Because config files are encrypted. Because the agent isn't allowed to read its own config.

Chapter 8: Showcase & Connections

Here is the whole lesson in one moving picture. A request flows left to right through the gateway's stages. Press a button to inject a failure mid-flight and watch the matching reflex take over: rate-limit triggers retry with backoff; provider down triggers failover walking the candidate chain; context overflow triggers compaction shrinking the transcript. The request ends green when a reflex recovers it, or amber when the gateway surfaces the failure with a reason. This diagram is the map of which reflex handles which failure.

The failure-recovery playground — one request, four reflexes

The dot is your request, moving through intake → classify → provider → context → reply. Inject a failure at the provider or context stage and watch the reflex that owns it kick in — the dot loops back (retry), hops models (failover), or shrinks the context (compaction). Green = recovered; amber = surfaced with a reason.

Idle. Inject a failure to see a reflex respond.

The four reflexes, in one breath. Retry rides out a transient blip — up to three attempts, exponential backoff capped at 30 s, 10% jitter so crowds don't stampede — but only for the transient error classes (408/409/429/5xx/timeouts), surfacing the rest. Failover walks a candidate chain (requested → fallbacks → primary, deduped) when a whole provider is down, advancing on provider faults and stopping on a context overflow or abort, with exponential cooldowns and auth rotation behind it. Compaction recovers an overflowed context by summarizing the older turns and keeping the recent tail — never orphaning a tool result. Diagnostics tells a stalled run from honest slow work, and self-update lets the agent harden its own config behind guardrails.

Where each thread connects

Lesson 2 — The Wire Protocol. Idempotency keys are what make retry safe: a re-attempted request carries a key the provider recognizes, so retrying a half-completed flow doesn't duplicate the completed step. Retry without idempotency is a duplication bug. Revisit the wire protocol.
Lesson 5 — Context & Streaming. Compaction is the reliability arm of the context budget: when the budget the gateway carefully built overflows anyway, compaction shrinks the model's view back under the window. Revisit context & streaming.
Lesson 10 — Tools & Sandboxing. The next reflex of reliability: when the agent's tools can run code, how do you fence that code so a failure (or a bad actor) can't hurt the host? On to tools & sandboxing.

Related lessons on this site

AI Evaluation — how you'd measure whether these reflexes actually keep success rates up under load and failure injection.
Agents & Tool Use — the tool-call/result pairing that compaction must protect comes straight from how agents call out to the world.
MCP (Model Context Protocol) — the tool seam that self-update audits, generalized to any agent's toolset.

The Feynman test. Close the lesson and explain to a friend: why failure is the normal case at scale; the retry reflex (three tries, exponential backoff with a 30 s cap, and why jitter); which errors deserve a retry and which get surfaced; how failover walks the candidate chain and which failures make it advance versus stop; what compaction does and why it must never orphan a tool result; and the three diagnoses for a run that won't finish. If you can name which reflex handles which failure, you own self-healing — and a gateway that stays standing when the world misbehaves.

In one sentence, what is the design stance behind all four self-healing reflexes?

Prevent every failure so none ever happens. Crash loudly so a human always intervenes. Treat failure as the normal case — expect it, classify it, and recover automatically with the reflex that matches its class (retry, failover, compaction, diagnostics). Retry everything forever until it works.