QuoteBench: When Scores Match and Meaning Does Not

Chapter 1: One Boundary, Two Parsers

Chapter 0 asserted that adding one parser destroys most working commands. This chapter earns that assertion by hand, on real bytes, because the mechanism is completely mechanical and you should never have to take it on faith.

What a shell actually does to a string

When Bash reads a command line it performs quote removal and several expansions, and the rules depend on which kind of quoting you are inside. There are exactly three states, and you only need these three.

A complete trace, character by character

Take a task of the shape QuoteBench actually uses: write a file whose exact content is

followed by one trailing newline. This is a legitimate literal-preservation task — a dollar sign, a backtick pair, a nested double quote. A competent model replies with a single command, and this one is correct:

the model's reply, call it R
printf '%s\n' 'cost: $5 `now` "x"' > out.txt

Read it once on the raw path. The outer printf gets format %s\n and one argument. Both are single-quoted, so $5, the backticks and the inner double quotes are all inert. printf's %s conversion does not interpret backslashes in its argument (that is %b), so the payload passes through untouched. The file receives exactly nineteen bytes plus a newline. Pass.

Now the nested path. The harness does not execute R. It builds the string bash -c "R" — R pasted verbatim between two double quotes — and executes that. So an outer Bash reads this line first:

what the outer shell sees
bash -c "printf '%s\n' 'cost: $5 `now` "x"' > out.txt"

Walk it left to right in the outer shell, using only the three-state table above.

Adjacent quoted and unquoted fragments concatenate into one word, so what bash -c actually receives as its script is:

The inner shell now parses that perfectly happily. It has a valid printf, a valid redirection, and it writes the file. The file contains cost:, three spaces, x, newline. The expected content was cost: $5 backtick now backtick "x".

The real task from the paper

The paper prints one worked task in full — the write-file family, hazard tier 3. The instruction asks for a file out.txt whose content is exactly:

The paper also prints the naive command, the thing a model reaches for without thinking:

naive — fails even on the raw path
echo "back\slash and `date` here
…" > out.txt

Three separate defects, and the paper names all three: the backticks execute date, $(pwd) expands locally, and the payload's inner double quote terminates the string early. This is the tier-3 variant — the benign tier-0 control of the same family uses ordinary text and the naive command passes. That contrast is deliberate and it is how the benchmark separates "cannot do the operation" from "cannot preserve a literal."

The passing command the paper describes is "a single-quoted printf with an embedded-quote splice." The splice is the standard idiom for putting a single quote inside a single-quoted string, since there is no escape for it:

the machine oracle, structurally
printf '%s\n' 'back\slash and `date` here
'"'"'single'"'"' "double" $(pwd) 100%' > out.txt

# the splice: close single quote, emit a double-quoted apostrophe, reopen
#   '   "'"   '     ->  one literal ' with nothing else changed

Now count what the outer double-quoted parser does to that. The splice contains " characters — three of them per apostrophe — and each one flips the outer quote state. The $(pwd) expands in the outer shell. The backticks around date run date in the outer shell. The \s survives, because s is not one of the four characters backslash protects inside double quotes. The model wrote a correct program and the channel shredded it.

Is the nested transport a strawman?

This is the obvious objection and the paper answers it with an experiment rather than an argument. The authors took each stored raw reply and replayed it through an actual remote command — ssh localhost "R" inside a container running a real sshd — and compared the loss to the synthetic nested transport. Zero model calls; pure replay.

Seven of eight match to the decimal. The eighth, Gemini-3.1-Pro, differs by 1.8 points, which is exactly one task: −71.4 is −40 tasks and −73.2 is −41 tasks out of 56. The synthetic condition is not a caricature of a remote command — on this benchmark it is numerically the same thing, because it is the same parser.

Two repairs, both trivial, both total

If the mechanism is a missing escape, the fix is an escape. The paper measures both obvious repairs on all 448 public replay pairs (8 configurations × 56 tasks).

So why is there a paper? Because both repairs require the caller to control the boundary. The authors surveyed six public agent systems at fixed commits and found the boundary applied downstream of whatever contract the model was given — a command string that later becomes shell -c/-lc R, an action that enters a persistent Bash session, a structured command written to a shell's stdin, key strings pushed through a tmux session. The system that states the contract is frequently not the system that adds the parser.

The four transports, in code

The abstraction is easier to hold once you have written it. Here are all four ways a harness can get a reply R to a shell, as the paper describes them, with the difference between them reduced to one line each.

python
import shlex, subprocess, tempfile, os, pathlib

def run_raw(R, cwd):
    # the raw transport: R IS the program. One parser, and it is Bash's.
    return subprocess.run(["bash", "-c", R], cwd=cwd)

def run_nested(R, cwd):
    # the nested transport: R is INTERPOLATED into a double-quoted word.
    # This is the whole intervention. One extra parser, no other change.
    outer = 'bash -c "' + R + '"'
    return subprocess.run(["bash", "-c", outer], cwd=cwd)

def run_nested_escaped(R, cwd):
    # REPAIR 1 — quote at the interpolation point. shlex.quote wraps R in
    # single quotes and splices any embedded apostrophes, so the outer
    # shell hands the inner one exactly the bytes the model wrote.
    outer = "bash -c " + shlex.quote(R)
    return subprocess.run(["bash", "-c", outer], cwd=cwd)

def run_script(R, cwd):
    # REPAIR 2 — never cross the boundary. The program lives in a file,
    # so no second parser ever reads its text. Costs a file lifecycle.
    fd, path = tempfile.mkstemp(suffix=".sh")
    os.write(fd, R.encode()); os.close(fd)
    try:
        return subprocess.run(["bash", path], cwd=cwd)
    finally:
        os.unlink(path)

Read the diff between run_nested and run_nested_escaped. It is one function call. That single call is worth 292 commands across the eight configurations in Chapter 4, and its absence is what the entire paper measures.

Three more hazard families, traced

Chapter 2 will list all 14 families. Three of them are worth walking now, because each breaks in a structurally different way and together they explain why the damage is not concentrated in one mechanism.

Expansion timing (the find / glob family). A glob is not a character problem, it is a timing problem: the question is which shell expands it. Consider a reply intended to delete backup files in a subtree:

the reply
find . -name '*.bak' -delete

On the raw path, the single quotes stop Bash expanding *.bak and find receives the pattern itself, matching at every depth. Under bash -c "R", single quotes are inert — but so are globs inside double quotes, so this particular reply survives. Now consider the same intent written without quoting, which many models do:

a fragile variant
find . -name *.bak -delete

Unquoted, the pattern expands against the current directory before find ever runs. If there is exactly one matching file at the top level, find silently searches for that one name and misses the rest; if there is none, Bash passes the literal pattern and it works by accident. The outcome depends on the fixture, which is precisely why the benchmark builds fixtures without invoking a shell. This is also why find-glob is the family where disclosure makes things worse — adding escapes changes when expansion happens, and the model is being asked to reason about ordering rather than about characters.

Argument boundaries (the hostile-filename family). A filename can contain a space, a newline, a leading dash, or a glob character — POSIX permits everything except the null byte and the path separator. The classic hazard is the leading dash:

the fixture contains a file literally named  -rf
rm -- -rf          # correct: -- ends option parsing
rm ./-rf           # also correct: the path is no longer dash-initial
rm -rf             # catastrophically wrong: parsed as flags

Here the second parser does not change the characters at all — it changes the word splitting, which changes the argument vector, which changes which strings are options and which are operands. That is why the validators inspect the received argv for this family rather than file bytes: the failure is in argument structure, not content.

Embedded languages (the sed, grep and AWK families). These stack a third grammar on top of the two shells. A sed replacement has its own escaping rules for the delimiter, for &, for backreferences and for newlines, and those rules are unrelated to Bash's:

python
# the model must satisfy THREE grammars at once:
#   1. sed's substitution syntax    (delimiter, &, backrefs)
#   2. the inner shell's quoting    (what the model is thinking about)
#   3. the outer shell's quoting    (which nobody told it about)
sed -i 's|old|new \& co|g' cfg.txt

The paper's family-level compensation numbers say something precise about this stack. sed-replace recovers +46.9 points under disclosure and json-write +50.0, while find-glob recovers −12.5. Explicit payload-quoting hazards, where the model can see a string that needs protecting, are the ones disclosure helps with. Implicit hazards — expansion order, argument structure — are not.

Chapter 2: 56 Tasks, Exact Final State

You now know the mechanism. Turning it into a measurement is a separate craft, and QuoteBench is a good place to learn it, because almost every design choice exists to close a specific way a benchmark can lie to you.

The shape of the benchmark

Each task is a fixture (a directory prepared without ever invoking a shell), an instruction, and a final-state validator. A model returns exactly one Bash program. It runs in a fresh working directory with a trimmed environment and a 15-second timeout. Then the validator looks at what is on disk — and only at what is on disk.

The tier structure is the load-bearing part. Within a family, the operation is held fixed and only the payload changes: the control is ordinary text; the hazardous tiers add quotes, expansion characters, multiline data, leading dashes, or parser-boundary conflicts. So the control answers "can this model do the operation at all?" and the difference between control and hazardous answers "can it preserve a literal while doing it?" Those are different competences and a single aggregate score confuses them.

Where the families came from

The families were not invented at a whiteboard. Two surveys fed the design, and the counts are worth memorising because they show what the evidence does and does not support.

Check the public arithmetic: 17 retained + 10 harness + 7 excluded = 34 read in full, out of 412 screened. Retained reports span Claude Code, Codex, Gemini CLI, Warp, and OpenHands — broken heredocs, over-quoted operators, repeated repair attempts. The authors are careful with the inference this licenses: the surveys "document mechanism coverage. Prevalence and complete shell coverage remain outside their purpose."

Auditing the validators

A benchmark's validators are code, and code has bugs. The failure mode here is well documented in the literature the paper cites: permissive validators that accept incorrect coding-agent patches. So QuoteBench audits every validator in four directions.

The mutation classes and their counts add up exactly, which is a nice sanity check to run yourself: delete a changed or required file 60/60, flip one byte in a changed file 60/60, insert an unexpected collateral file 56/56, restore a file that should have been removed 17/17, amend a Git-only final state 4/4. Sum: 60 + 60 + 56 + 17 + 4 = 197.

And a solvability control for the nested arm specifically, which matters more than it looks: three configurations pass all 56 tasks under the nested transport. So every task has a feasible nested solution, and the nested arm is not a degenerate condition where the benchmark is simply impossible.

What the benchmark deliberately does not cover

Scope statements are how you avoid being cited for things you did not measure. QuoteBench focuses on POSIX/Bash command construction. Outside the benchmark: PowerShell, Windows CMD, authentication, network failures, interactive terminal state, and multi-turn recovery. The public core also excludes real SSH networking and authentication — the SSH-like families are local two-shell simulations.

Two more design details you should copy if you ever build an evaluation like this.

Contamination handling. Publishing the frozen 56-task core creates a contamination risk. The authors treat it as a versioned audit set (core-v1), hold out regenerated private variants, and embed a fixed canary GUID in every released task file so downstream contamination checks have a known token to grep for.

The query date is part of the result. Hosted deployments can change under the same model identifier. So the paper records that the same-window mechanism sweep was queried on 2026-07-31 and the effort ladders earlier in July 2026. If you ever cite a number from a hosted model without a date, you have cited a moving target.

One family, across all four tiers

The tier structure is what turns a pile of shell tasks into an instrument, so it is worth walking one family from the benign control up to the hardest variant. Take write-file, whose tier-3 instance the paper prints in full. The operation is identical at every tier — create out.txt with exactly this content and one trailing newline — and only the payload changes.

That last column is a validated property of the benchmark, not an assumption: the audit confirms benign naive probes pass and their hazardous counterparts fail on the raw path. So a model that scores 14/14 on controls and 13/42 on hostile tasks — Haiku-4.5 at its best observed setting — is not failing at shell. It is failing at literal preservation, and the two-column split makes that legible in a way a single 37.5% cannot.

What a final-state validator actually looks like

The phrase "final-state validator" hides some real engineering, so here is its shape for the write-file family. Three checks, and the third is the one people forget.

python
import os, pathlib

EXPECTED = (b"back\\slash and `date` here\n"
            b"'single' \"double\" $(pwd) 100%\n")

def validate(workdir):
    # 1. the artefact exists
    p = pathlib.Path(workdir) / "out.txt"
    if not p.is_file():
        return False
    # 2. the bytes are EXACT — including the single trailing newline.
    #    Not a substring match, not a strip()ed compare, not a decode.
    if p.read_bytes() != EXPECTED:
        return False
    # 3. no collateral. A command that got the file right by writing
    #    three helper files on the way has not solved the task.
    if set(os.listdir(workdir)) != {"out.txt"}:
        return False
    return True

Every design decision in those fifteen lines answers a way the measurement could have been wrong. Byte comparison rather than text comparison, because an encoding round trip would silently normalise a payload the task exists to preserve. No stripping, because the trailing newline is part of the specification — echo adds one and printf '%s' does not, and the difference is exactly the kind of thing a benchmark should notice. And the collateral check, because the paper's own recovery story is "write a temporary script", which leaves workspace artefacts; a validator blind to collateral would grade that as clean.

Notice also what is not in there: the command text. No regular expression over the reply, no oracle-string comparison, no check that the model used printf. That is what makes "any semantically correct implementation receives credit" true rather than aspirational, and it is what allows the disclosed-boundary arm to rewrite its commands completely and still be scored fairly.

The mutation audit as a debugging technique

The mutation audit deserves one more paragraph because it is a technique you can lift wholesale. The procedure: run the oracle, capture the resulting valid state, then corrupt it in every way the validator is supposed to catch and confirm each corruption is rejected.

The counts also tell you about the benchmark's shape. Only 17 tasks involve removing a file, and only 4 have Git-only final state — so the Git family is small, and most tasks are about producing bytes rather than deleting them. That is the kind of thing you can read off an audit table if you look.

Chapter 3: The Crossed Design

Everything so far has been setup. This chapter is the paper's actual contribution, and it is a piece of experimental design rather than a piece of engineering. The question is: given that a command can fail either because the model wrote it wrong or because the channel broke it, how do you tell which happened?

Two axes that everybody conflates

QuoteBench separates two things that agent systems normally bundle together.

The paper evaluates two observed model-facing contracts and adds one controlled intervention.

The two Study-A contracts are printed verbatim in the appendix, and the diff between them is the entire experimental manipulation. The raw clause:

The disclosed-boundary clause:

The 2×2

Now cross them. Let G be the generation contract and T the execution transport, each taking value R (raw) or N (nested / disclosed-boundary). Four cells, written contract-then-transport:

The diagonal cells, RR and NN, are what any normal benchmark reports: each was generated and executed under its own consistent setup. The off-diagonal cells are the new information, and they cost zero model calls — RN is the RR reply replayed through a different transport, NR is the NN reply replayed through a different transport. Same bytes, different plumbing.

From the four cells, three quantities, all in percentage points averaged over the 14 families:

And the identity that makes the whole thing work — add and subtract Y_RN:

Play with the cells

Flip to the second mode on GPT-5.6-sol. Two numbers survive, 94.6 and 91.1, and the story they tell is "robust model." Flip back and the same model has lost 36 tasks and rebuilt 34 different ones. Both descriptions are true. Only one of them predicts what happens when your harness changes.

Why fixing the reply is the whole trick

There is a large literature showing that scores move when you change a nuisance variable — prompt formatting, scaffold choice, harness. The paper is careful about what makes its design different, and the distinction is worth internalising because it applies to any experiment you will ever run on a model.

The paper puts it plainly against the closest concurrent work: because that work "replaces the harness wholesale, including context handling, retry, and verification, it measures variance but cannot attribute a reversal to a mechanism; QuoteBench fixes the model output and changes a single parser."

What real harnesses actually do

The paper inspected six public agent systems at fixed commits, recording separately what each system asks the model to produce and what it subsequently does with the reply. Both contract styles are in use, and the downstream boundaries vary a lot.

Notice the bottom two rows. OpenHands tokenizing to argv and AutoGen writing to a temporary file are, structurally, the two repairs from Chapter 1 — those systems have already removed the boundary. Codex's row is the one that matters for this paper: a command string that later becomes shell -c R. And the contract the model was given said nothing about that.

The whole experiment, in twenty lines

The design is easier to trust once you see how little code it is. Generation happens once; everything after that is replay.

python
CONTRACTS  = {"R": RAW_CLAUSE, "N": DISCLOSED_CLAUSE}   # differ by ONE sentence
TRANSPORTS = {"R": run_raw,     "N": run_nested}      # differ by ONE parser

# ---- phase 1: generate once, in ONE serving window, order randomized ----
replies = {}
for task in TASKS:                       # 56 tasks
    for g in CONTRACTS:                    # 2 contracts
        replies[(task.id, g)] = model(CONTRACTS[g], task.instruction)

# ---- phase 2: replay. Zero model calls from here on. ----
Y = {}
for task in TASKS:
    for g in CONTRACTS:
        R = replies[(task.id, g)]
        for t in TRANSPORTS:               # SAME bytes, two paths
            work = task.fresh_fixture()     # built without a shell
            TRANSPORTS[t](R, cwd=work, timeout=15)
            Y[(task.id, g + t)] = task.validate(work)

# ---- phase 3: the decomposition, per family, in percentage points ----
def rate(cell, fam):
    ts = [t for t in TASKS if t.family == fam]
    return 100 * sum(Y[(t.id, cell)] for t in ts) / len(ts)

for fam in FAMILIES:                     # 14 families = the inferential units
    damage = rate("RN", fam) - rate("RR", fam)
    comp   = rate("NN", fam) - rate("RN", fam)
    gap    = rate("NN", fam) - rate("RR", fam)
    assert abs(gap - (damage + comp)) < 1e-9   # the identity, per family

Three details in that loop are the difference between a measurement and an anecdote, and each corresponds to a sentence in the paper.

And the economics are the reason so many robustness checks exist. Phase 1 is 56 × 2 = 112 model calls per configuration. Phase 2 is free. Every extra transport — escaped, temporary script, real ssh, JSON serializer, single-quote wrapper — is another pass over the stored replies at the cost of container time. That is how one paper affords a real-ssh grounding, a grammar crossover, a JSON boundary study, two userlands, and a script-bypass replication.

Why there is a third contract at all

The native contract sits slightly outside the 2×2 and it is easy to skip past. Its purpose is to isolate the model-facing representation from the execution path: QuoteBench extracts the command field from the provider's shell-tool call and runs that string on the same raw path as the raw contract. Same transport, different way of asking.

The paper is upfront that the native comparison is not as clean as the crossed one — the native contract also mentions a Bash script payload, a difference it folds into a declared total-effect estimand rather than pretending the prompts are otherwise identical. Chapter 9 reports what that study found; the point here is that "how you ask" and "how it runs" are two knobs, and this paper turns each one separately.

Chapter 4: Transport Damage

Now the numbers. This chapter is one table and its consequences, worked slowly, because every claim in the paper's abstract lives inside it.

The table

Eight configurations, collected in one randomized serving window with the effort field omitted, each contributing 56 raw-contract replies and 56 disclosed-boundary replies. Every reply replayed through both transports. Cells are pass rates in percent; effects are percentage points.

Convert the whole RR column back to counts to see what you are looking at. Multiply by 56 and round:

And the RN column: 17, 16, 17, 14, 16, 15, 17, 11, totalling 123. So of 415 commands that worked when executed directly, 123 survived the added parser and 292 did not. The 33 commands that already failed on the raw path (448 − 415) stayed failed — the parser rescues nothing, which is exactly what a clean one-directional intervention should look like.

Per-configuration retention, which the paper reports as 25.0–35.4%, is a division you can do in your head at both ends:

Why the weak models lose the least

That last observation is worth a paragraph because it is counter-intuitive and it explains the shape of the damage column. Damage ranges from −55.4 (Qwen3.5-27B) to −73.2 (Gemini-3.1-Pro), and it is anti-correlated with raw ability: the highest raw scorers take the biggest hits.

The mechanism is not mysterious. A model that solves hostile payloads correctly does so by quoting them properly — single quotes, splices, printf with a literal format. Those are precisely the constructions that contain $, backticks, backslashes and embedded quotes, which are precisely what the outer double-quoted parser consumes. Competence on the raw path is exposure on the nested path. A model that solves only 48 of 56 tasks had fewer correct-but-fragile commands to lose.

The controls, again

Chapter 0 quoted the control-task loss; here is why it is the most alarming number in the paper. The 14 controls are the benign tier — no hostile characters, ordinary operations. Under the added parser the eight configurations lose 28.6–57.1 points on the controls alone, which in counts is 4 to 8 tasks of 14.

Gemini-3.5-Flash makes it concrete. At the fixed same-window setting used for the crossover it passes 54 of 56 tasks in RR — near perfect — and only 8 of 14 control tasks in NN. Six of fourteen ordinary, hazard-free operations, failed on the deployed path, by a model that had essentially solved the benchmark on the direct path.

The paper's one-line explanation: "models emit double-quote-active characters even for ordinary commands." You do not need a hostile payload to write grep "TODO" src/*.py or echo "done". You just need to write shell.

Is it one weird family?

The obvious worry is that a single pathological family is carrying the entire effect. The paper closes this two ways.

Leave-one-family-out. Drop each of the 14 families in turn and recompute. Every one of the 8 × 14 = 112 leave-one-family-out estimates of transport damage remains negative. There is no family whose removal makes the effect disappear, or even change sign.

An exact enumerated test. Because the families are purposively constructed rather than sampled, the paper uses a family-sign symmetry null: conditional on the observed effect magnitudes, positive and negative signs are exchangeable. With 14 families that is 2¹⁴ = 16,384 sign assignments, enumerable exactly — note that the smallest reported p-value, .000122, is 2 ÷ 16,384, the two-sided probability of the most extreme assignment. Holm's step-down correction is then applied across the eight models.

The intervals come from a percentile bootstrap of the mean over families as units, 10,000 replicates. The largest adjusted p is .001465. Every interval excludes zero by a wide margin — the closest, Qwen3.5-27B, still has its upper end at −41.1 points.

The interaction, and why it is enormous

There is a fourth quantity you can build from the four cells: the generation-by-transport interaction, (NN − NR) − (RN − RR). It asks whether the disclosed contract helps more on the nested path than on the raw path. Work it for GPT-5.6-sol in tasks first, since the arithmetic is cleaner:

An interaction of a full hundred points. The paper reports the range across the eight configurations as −7.1 to +119.6, with the maximum at Gemini-3.1-Pro:

and the minimum at Gemini-3.1-Flash-Lite, which is the only configuration where disclosure makes things slightly worse in both directions:

A large positive interaction means the boundary-aware contract does not make the model generically better at shell — it makes it better at that boundary specifically, and Chapter 5 shows it makes it measurably worse everywhere else.

The Holm correction, done by hand

Eight models means eight tests, and running eight tests at α = .05 gives you roughly a one-in-three chance of at least one false positive by luck alone. Holm's step-down procedure fixes that, and it is simple enough to do on paper — which is worth doing once, because you will then recognise a Holm-adjusted column forever.

Sort the raw p-values ascending. Multiply the smallest by m = 8, the next by 7, the next by 6, and so on. Then enforce monotonicity by taking a running maximum, so an adjusted value can never be smaller than one before it.

Which reproduces the paper's two distinct reported values exactly: .000977 for the two smallest and .001465 for the rest. The running-max step is doing real work here — without it, rank 2 would report .000854, smaller than rank 1, which is incoherent because a less extreme observation cannot be more significant.

Note also what the raw p-values are made of. With 14 families there are 2¹⁴ = 16,384 sign assignments, so the finest achievable two-sided p is 2 ÷ 16,384 = .000122 — the case where the observed assignment and its exact mirror are the only ones at least as extreme. The value .000244 is 4 ÷ 16,384, and .000488 is 8 ÷ 16,384. Every p-value in the table is a small integer over 16,384, which is what an exact enumerated test looks like when you meet one.

Reading the damage column as an engineer, not a statistician

Statistical significance here is almost beside the point — every effect is 55 points or more, and no reasonable amount of noise produces that. The engineering content is in the ordering and the spread. Sort the eight configurations by damage and put raw score beside it:

The "fragile commands lost" column is just RR minus RN in tasks, and it tracks RR almost perfectly: the correlation is visible by eye. The retention column, which normalises by how many the model had to lose, is much flatter — 25% to 35% for every configuration in the table, weak and strong alike. That is the honest summary of what the parser does: roughly seven in ten working commands die, regardless of who wrote them.

Chapter 5: What One Sentence Buys

Damage is the depressing half. This chapter is the surprising half: tell a capable model where its command is going to run — one sentence, no advice — and it rewrites its commands into a form that survives. Sometimes it recovers nearly everything it lost.

The size of the effect

Compensation is NN − RN: hold the nested transport fixed and compare replies written under the two contracts. Six of eight configurations recover between 30.4 and 60.7 points, every one with a family-bootstrap interval excluding zero. Two do not: Qwen3.5-27B recovers exactly 0.0, and Gemini-3.1-Flash-Lite goes backwards by 5.4.

A configuration joins the supported positive-compensation set when three things hold at once: the effect is positive, its Holm-adjusted p is at most .05, and every leave-one-family-out estimate stays positive. Six qualify. Gemini-3.5-Flash squeaks in at .041016 — a good reminder that a criterion stated in advance is worth more than a criterion chosen after seeing the numbers.

Note also what a zero-width interval means. Qwen3.5-27B is [0.0, 0.0]: it recovers exactly zero on every family, in every bootstrap resample. It did not partially adapt. It did not adapt at all.

Proof that this is behaviour change, not generic robustness

The sceptical reading of compensation is that the disclosed contract simply makes models write more careful shell in general, and the nested arm is where careful shell pays. The off-diagonal NR cell refutes this in one step: take those same boundary-aware replies and run them on the raw path, where there is no boundary at all.

Gemini-3.1-Pro is the sharpest case: it gains 55.4 points on the nested path and loses 64.3 on the raw one. Its behaviour is now strongly conditioned on a claim about the environment. If that claim is wrong, it is worse than a model that ignored it.

Conditioned on the grammar, not on a memorised fix

Is the model actually reasoning about the stated boundary, or has it learned one reflex — "when someone mentions wrapping, add backslashes before double quotes"? The paper settles this with a second crossover, this time on the grammar.

Disclose either a double-quote wrapper or a single-quote wrapper, then replay every stored reply through both wrappers. A model with a memorised double-quote reflex will do well only on the double-quote wrapper. A model that reasons about the declared grammar will do well on whichever wrapper it was told about — the matched diagonal — and badly on the other.

The denominator here is 112, since each row aggregates two arms of 56. GPT-5.6-sol:

Look at the single-disclosed column: 53 of 56 on the single-quote wrapper it was told about, 10 of 56 on the double-quote one. Nothing about a memorised double-quote defence explains that. And look at the bottom row: Gemini-3.1-Flash-Lite has a negative diagonal advantage — it does slightly better on the wrapper it was not told about, which is what noise looks like when there is no adaptation to detect.

Disclosure versus instruction

One more decomposition, and it is the practically useful one. The disclosed contract states the boundary but gives no quoting advice. What happens if you also tell the model how to escape? The paper regenerates both the advice-free and advice-bearing contracts within one serving window, so there is no cross-window drift to confound the comparison.

Three distinct regimes, and the shape is one you should expect to see again in other capabilities.

Compensation is uneven across families

Aggregates hide structure here too. Averaged over the eight configurations — so each family effect is over 4 tasks × 8 configurations = 32 cells, and one cell is 3.125 points — the recovery concentrates where the hazard is explicit and stalls where it is implicit.

Verify one: +46.9 points is 15 ÷ 32 = 0.46875. And find-glob going negative is the instructive one. Glob and expansion-timing hazards are about when something expands rather than what characters are in it, so a model told "you will be wrapped in double quotes" adds escaping that changes expansion timing in the wrong direction. Being told about the boundary is not the same as understanding it.

The families that recover about half their damage are the ones where the model can see the thing that needs protecting: a JSON string literal, a sed replacement between delimiters. The families that stay broken are the ones where the hazard is a property of the evaluation order or the argument vector rather than a visible span of characters. That split is a genuine finding about what disclosure elicits, and it is more informative than the aggregate: a system builder who only writes files and edits JSON gets most of the benefit, and one who does bulk renames and recursive searches gets almost none.

What the model actually does differently

The paper does not publish reply diffs, but the numbers constrain the answer tightly, and it is worth reasoning it through because it explains both the gain and the cost.

On the raw path, the winning strategy is single quotes: they make everything inert in one step, and they are what a correct solution to a hostile payload looks like. Under a disclosed double-quote wrapper that strategy is worthless, because single quotes do not survive the outer context. The model must instead produce a string that, after the outer shell has performed its expansions and quote removal, still constitutes the intended program. Concretely that means pre-escaping every character the outer parser treats as special — $, backtick, backslash, " — so they arrive at the inner shell as literals.

the same intent under two contracts
# raw contract — single quotes make everything inert
printf '%s\n' 'cost: $5 "x"' > out.txt

# disclosed contract — the reply must SURVIVE one round of double-quote
# parsing first, so the specials are escaped for the OUTER shell
printf '%s\n' 'cost: \$5 \"x\"' > out.txt

Now the NR result is not surprising at all. Run that second command on the raw path and the backslashes never get consumed by anyone: the file receives cost: \$5 \"x\". Wrong bytes. The reply is not a more careful version of the first — it is a different program, correct against a different machine.

Why advice can make a top model worse

GPT-5.6-sol loses 8.9 points — five tasks — when the contract adds escaping advice on top of the boundary disclosure. That is the single most counter-intuitive number in the paper and it has a clean reading.

The advice-free disclosed contract states a fact about the environment and leaves the strategy open. A model that can already reason from "I will be inside double quotes" to a correct encoding chooses whichever encoding fits the payload — and different payloads want different encodings. Adding prescriptive escaping advice narrows that search: the model now follows the recommended pattern even where a different one would have been correct. The paper's framing is exactly this: "Disclosure alone already elicits the adaptation," and at the top of the ladder the added instruction "barely moves matched nested success."

Meanwhile the middle tier gains the most — Sonnet-4.6 +25.0 points, fourteen tasks. Those models can execute a stated procedure but were not deriving it from the environment description alone. And the bottom tier gains 1.8 points, one task, from both contracts, which is the signature of a capability ceiling rather than an information gap.

That table is the practical distillation of the whole chapter, and note where it lands: for two of the three tiers, the answer involves changing your own code rather than your prompt.

Chapter 6: Masking

Two large effects with opposite signs. This chapter is about what happens when they are nearly equal, which is where the paper's title comes from and where the practical damage to evaluation happens.

A reading aid, stated as a rule

The paper offers a descriptive criterion — explicitly "one illustrative reading aid (not a fitted criterion)" — for calling a configuration masked: the matched gap is small while both components are large.

GPT-5.6-sol at the same-window setting meets all three: |−3.6| ≤ 5, −64.3 ≤ −40, +60.7 ≥ +30. And the identity closes exactly: −3.6 = −64.3 + 60.7.

Across the 30 rung-level crossovers the paper measured, ten meet the same cut — and all ten sit at the tops of their ladders. That is not an accident. Masking requires large compensation, compensation is a frontier capability, and frontier capability shows up at high effort settings.

The clearest case in the paper

Opus-4.8's effort ladder is the demonstration, because it is the case where matched success genuinely improves and the improvement means something different from what it looks like. Two rungs, low and max, with the disclosed contract:

Reconstruct those from the identity to convince yourself. At low, matched nested success is 39.3 and the matched gap is −48.2, so RR = 39.3 + 48.2 = 87.5. Damage is −58.9, so RN = 87.5 − 58.9 = 28.6. At max, NN = 94.6 and the gap is −3.6, so RR = 98.2; damage is −67.9, so RN = 98.2 − 67.9 = 30.4.

Now read the RN column. It moved from 28.6 to 30.4 — one task. Turning the effort dial from low to max moved the matched gap by 44.6 points and moved cross-path portability by 1.8.

Who is masked and who is merely bad

Run the criterion across all eight same-window configurations and it separates them into three genuinely different situations. This is worth doing explicitly, because "masked" is not a synonym for "has a large damage number" — every configuration has a large damage number.

Look at the bottom two rows and something clarifying appears: for a model that does not compensate, the matched gap is an honest report of transport damage. Qwen3.5-27B's −55.4 gap is exactly its −55.4 damage, because compensation is precisely zero. The benchmark tells the truth about the weak models and hides the truth about the strong ones. Masking is a frontier phenomenon.

Why any threshold is the wrong fix

The obvious response is to adopt the criterion as a rule: flag any configuration meeting those three inequalities. The paper deliberately does not do that, and its reason is worth adopting. The cut points — 5, −40, +30 — are arbitrary. Opus-5's gap is 7.1, which is four tasks; is that meaningfully different from GPT-5.6-sol's two tasks? The criterion says yes and reality says barely.

The real fix is not a better threshold. It is to stop reducing three numbers to one. If you report damage, compensation and the matched gap side by side, nobody needs a masking rule, because there is nothing left to mask.

Watch it across a whole ladder

The paper's summary of this figure is one sentence and it is worth memorising: across the observed trial-0 effort rungs, the unconditional nested-replay pass rate stays between 23.2% and 33.9%, and moves by at most 5.4 points within any one ladder. Five point four points is three tasks. That is the entire range of portability improvement available from every effort dial the providers expose.

Check it on GPT-5.6-sol, whose four rungs give RN values of 28.6, 30.4, 32.1, 30.4. Highest minus lowest is 3.6 points, or two tasks. On Opus-5's five rungs: 28.6, 32.1, 30.4, 32.1, 32.1 — again 3.6. On Gemini-3.1-Pro's three: 25.0, 26.8, 26.8 — 1.8 points, one task.

Why the matched gap can narrow without anything improving

Put the two observations together and the mechanism of masking is fully explained. The matched gap is NN − RR. Damage is roughly constant across a ladder, so RN is roughly constant. Therefore:

And the interior of these ladders is not monotone, which is a separate warning. Opus-5 at low is 60.7, medium 91.1, high 96.4, xhigh 100.0, max 98.2 — the max rung is worse than xhigh by one task. GPT-5.5 runs 94.6, 92.9, 91.1, 100.0: three rungs of decline and then a jump. If you tune effort by picking the highest label you will sometimes be picking a worse operating point.

Every rung, for the three complete ladders

The claim is easier to believe when you can see all four cells at every rung rather than a summary. Asterisks mark rungs meeting the descriptive masking criterion.

Three readings, top to bottom. GPT-5.6-sol is masked at three of its four rungs — its reported gap is −1.8, +0.0, +0.0 while damage sits at −64 to −68 the whole way. Opus-5 is unmasked at its two cheapest rungs and masked at its three most expensive, which is the "masking lives at ladder tops" pattern in a single row group. And Gemini-3.1-Pro is masked at none of them, because its compensation never quite catches its damage — its NN is pinned at exactly 91.1 across all three rungs while RR sits at 98–100.

Now scan the NR column, which is the one nobody would think to look at. For GPT-5.6-sol it runs 53.6, 48.2, 48.2, 51.8 — the model's boundary-aware replies get worse on the raw path as effort rises from low to medium, then recover. Effort makes the targeting sharper, and sharper targeting is more wrong when aimed at the wrong platform.

Chapter 7: Effort Is Not Compute

Every provider now exposes a knob labelled with words like low, medium, high, xhigh, max. It is enormously tempting to treat those labels as a common axis — to compare "model A at high" with "model B at high" as though the two were spending the same budget. QuoteBench measured what the labels actually do, and the answer is: something different for every provider, sometimes non-monotone, sometimes nothing at all.

The ladders

Matched-nested success on the 56-task core under the disclosed-boundary contract, trial 0, with the provider-reported mean output tokens per task alongside — including reported hidden reasoning.

Read down the token column before the success column, because the token column is the one that destroys cross-model comparison. GPT-5.6-sol at high spends 773 tokens. Gemini-3.1-Pro at low spends 5,267 — nearly seven times more — and Opus-4.8 at max spends 6,960, which is nine times GPT-5.6-sol's high.

Four ways a ladder misbehaves

1. It goes down. Haiku-4.5: 32.1, 37.5, 32.1, 26.8 across low, medium, high, xhigh. The best setting is medium and the worst is the highest label. Its token means are also non-monotone — 5,432, 4,717, 4,708, 4,783 — and because those means are strongly right-skewed the paper also reports the medians: 1,206, 1,560, 936, 1,283. The non-monotone budget ordering survives the robust summary, so it is not one runaway generation dragging an average.

2. It goes down and then jumps. GPT-5.5 declines through low, medium, high (94.6 → 92.9 → 91.1) and then reaches 100.0 at xhigh. Two of its labelled rungs are worse than its cheapest one. A tuner that tries low and medium and concludes "effort does not help here" would stop one rung before the perfect score.

3. It does nothing. Gemini-3.1-Pro is 91.1 at every one of its three rungs, at 5,267 to 6,308 tokens. You can pay 20% more tokens for identically zero improvement.

4. It is not real. Gemini-3.1-Flash-Lite returns byte-identical replies at all four settings, at 20 tokens each. The paper retains it in the tables "only to document that the provider did not expose a usable ladder." Four labels; one behaviour. Qwen3.5-27B's think toggle is the mirror image: 20 tokens versus 7,489 tokens — a 375× difference in spend — for the same 28.6% score.

Two knobs that are not effort at all

Two of the eleven rows in that table do not expose an effort ladder in any meaningful sense, and both are instructive about what the label is hiding.

The think toggle. Qwen exposes a binary rather than a multi-rung parameter, and the measurement is startling: 28.6% non-thinking at 20 mean output tokens, 28.6% thinking at 7,489. Identical score, 375 times the spend. Twenty tokens is barely a command; 7,489 is a long deliberation. Whatever the model does with those 7,469 extra tokens, it does not change how many of the 56 tasks reach the correct final state. And the crossover table backs this up from a different direction — Qwen3.5-27B is one of the two configurations with exactly zero contract-conditioned compensation, so more thinking is not buying boundary reasoning either.

The rung that is barely a generation. Sonnet-4.6's low setting averages 28 output tokens and scores 23.2%. Its high setting averages 4,337 and scores 64.3%. A 28-token mean means most replies are a single short command with no deliberation at all — which is a perfectly legitimate operating point, and one you would never guess from a label that reads "low" rather than "essentially no reasoning budget."

The unset field is not a neutral rung

Most production code never sets the effort field at all. So what do you get? The paper calibrated the unset arm against each model's labelled ladder, and the answer is different for every provider.

Opus-4.8's default sits near xhigh; Opus-5's sits near medium; Gemini-3.1-Pro's falls below its entire measured ladder, 10.7 points under its own lowest rung. Three defaults, three completely different places on three different ladders. If you compare two models with the effort field omitted, you are comparing two arbitrary and undisclosed operating points.

Points per thousand tokens

If the labels are meaningless, the tokens are not. Divide the improvement by the spend and the ladders sort themselves into shapes that are actually decision-relevant. Take each model's cheapest and most expensive measured rungs:

Two rows are worth staring at. Opus-4.8 gains 55.3 points, the largest movement in the table, and pays 6,125 extra tokens for it — but Chapter 6 already showed that essentially all of that gain is contract-conditioned compensation, not portability. You are buying a number, not a capability, unless your deployment discloses the boundary. And Gemini-3.1-Pro's row is the cleanest waste in the paper: 1,041 extra tokens, zero points, three times over.

The two-axis mistake

There is one more trap, and it combines this chapter with Chapter 6. Suppose you tune effort on the matched nested score, which is the natural thing to do if that is your deployment configuration. You will find a rung that maximises it — and Chapter 6 showed that the maximising rung is very likely the masked one, since masking happens where compensation is largest. So the standard tuning procedure systematically selects operating points whose reported number is the least informative about cross-path behaviour.

The design discipline behind that restriction

It is worth spelling out, because it is the difference between a paper you can trust and one you cannot. QuoteBench runs many campaigns — a same-window sweep, ladder sweeps, three-draw repeats, a native-tool campaign of 8,736 generations, private-payload crossovers, a real-ssh grounding replay, an advice arm, a grammar crossover, a JSON boundary replay. Each has a declared design and a declared role.

Every replay campaign is a zero-call execution of stored replies in the pinned container. That is what makes so many robustness checks affordable: the expensive part, generation, was paid once.

Chapter 8: The Leaderboard Reorders

If damage and compensation differed by a constant across models, none of this would matter for model selection — every model would shift by the same amount and the ranking would hold. They do not. Compensation ranges from −5.4 to +60.7. So the ranking moves.

Kendall's τ, computed by hand from the table you already have

Rank correlations tend to arrive as a number with no visible derivation. This one is small enough to do completely, and doing it recovers every claim the paper makes about the reordering. Label the eight configurations and write down their two scores in tasks:

A pair is concordant if the two columns order it the same way, discordant if they disagree, and tied if either column gives the pair the same value. Go through all 8 × 7 ÷ 2 = 28 pairs. Ties first, because there are only two:

Now the disagreements. A is behind B, C, D and E on RR and ahead of all four on NN — four discordant pairs. C is behind D on RR (54 versus 55) and ahead on NN (50 versus 45) — one more. Every other pair agrees.

And Kendall's τ is the concordant-minus-discordant difference over the number of pairs:

Which is the paper's reported value, and the five discordant pairs are exactly the five reversals it names: GPT-5.6-sol against Gemini-3.5-Flash, Gemini-3.1-Pro, GPT-5.5 and Opus-5, and Opus-5 against Gemini-3.1-Pro. The "26 of 28 strictly comparable" figure is the two ties removed. Every published number in this analysis falls out of eight pairs of integers.

Two orderings of the same eight models

Take the same-window crossover table and sort it twice: once by RR, the direct-path score, and once by NN, the deployed-path score. Same models, same tasks, same window, same validators.

The headline reversal is GPT-5.6-sol versus Gemini-3.5-Flash, and it is worth doing in counts because the asymmetry is startling:

Eighteen tasks is 32.1 percentage points. A one-task deficit on the path you measured becomes a thirty-two-point lead on the path you deploy. If you selected on RR you did not make a marginal mistake; you inverted a large difference.

How much of the ranking survives?

The paper quantifies the agreement between the two orderings with Kendall's rank correlation: τ = 0.57, with a task-cluster bootstrap 95% interval of [0.32, 0.82] that excludes perfect agreement. With eight models there are 8 × 7 ÷ 2 = 28 pairs, and τ is the concordant-minus-discordant fraction, so τ = 0.57 corresponds to roughly 22 concordant against 6 discordant pairs.

Two further statements, and the difference between them matters:

You can locate the two tied pairs yourself from the crossover table. Under RR, Opus-5 and Gemini-3.5-Flash both score 54 of 56. Under NN, GPT-5.5 and Opus-5 both score 50 of 56. That is 2 of 28 pairs non-comparable, leaving 26 — the arithmetic closes.

The cost of ignoring the path

Turn it into a decision. You must pick one model. Two selection rules:

The paper is careful not to oversell this: "The regret is small at the saturated frontier, but the reversed top rank." That is the right size of claim. The point is not that path-blind selection is catastrophic; it is that the number you selected on did not contain the information you needed, and you had no way to know.

The best-observed scorecard, and how to read a jackknife

Separately from the mechanism analysis, the paper reports each base model at its best measured operating point — a within-model maximum over single-trial rungs. The Hostile LOFO column is a family jackknife, not a confidence interval: the minimum and maximum hostile success across the 14 leave-one-family-out slices.

Verify one row and the jackknife stops being mysterious. GPT-5.6-sol misses exactly one hostile task out of 42. Dropping a family removes 3 hostile tasks, leaving 39:

which is exactly the reported [97.4, 100.0]. Now Sonnet-4.6, which passes 27 of 42 hostile tasks and therefore misses 15. If the dropped family had all three of its hostile tasks passing, the slice is 24/39 = 61.5%; if all three failing, 27/39 = 69.2%. Reported: [61.5, 69.2]. So some family fails Sonnet-4.6 completely, and removing it lifts the score by 4.9 points. That is what a jackknife is for — it tells you how concentrated the failures are.

One more thing worth noticing in that table: Qwen3.5-4B at 21.4% beats Qwen3.5-9B at 17.9%. The smaller model wins. And GPT-5.6-sol at 98.2% is behind three models at 100.0%, which contradicts the same-window crossover where it had the best matched-nested score — because these are different selections from different serving windows. The paper flags this explicitly: the scorecard supports operating-point selection, the crossover diagnoses path sensitivity, and "neither is a controlled compute ranking."

Controls versus hostile, read as a diagnosis

Split every row of the scorecard into its two components and the aggregate stops being a single skill. The control column measures "can you do the operation"; the hostile column measures "can you preserve a literal while doing it". Compute the ratio and models sort into recognisably different failure modes.

Opus-4.8's row is the strange one and it is a good illustration of why you should always look. It passes 41 of 42 hostile tasks and only 12 of 14 benign ones. A model that can preserve a multiline payload with mixed quotes but drops two ordinary operations is telling you something about its failure distribution that no aggregate can — and its Hostile LOFO of [97.4, 100.0] is identical to GPT-5.6-sol's, despite a 3.6-point difference in the total.

Sonnet-4.6's flat profile is the other instructive one: 64.3% on both halves. The hazard tier is supposed to be harder, and for this configuration it is not, which means its 20 failures are distributed across operations rather than concentrated on literals. Two models at exactly 64.3% overall — Sonnet-4.6 and Gemini-3.5-Flash — and their splits differ, their jackknives differ, and a user would experience them differently.

How to state a leaderboard result honestly

Put the chapter together into a template. Three claims of decreasing strength, each supported by a different piece of the analysis:

That third block is the one most papers skip. Reporting "five reversals" invites the reader to treat five as an estimate with negligible error, when the interval spans one to eight. Reporting the interval costs one sentence and prevents a citation chain from hardening a soft number.

Chapter 9: Not Really About Bash

Everything so far concerned one shell and one wrapper. If that were the whole story, the fix would be "stop using ssh host quotes" and we could go home. This chapter shows the same four cells appearing on a boundary with no shell anywhere in it, and gathers the robustness checks that say the effect is not an artefact of the published payloads, the userland, or a single sampled reply.

The same mechanism with no shell involved

Replace the shell wrapper with a JSON tool call. The paper replays each stored raw reply through a JSON boundary two ways: a correct serializer that round-trips the string, and a naive embedding that pastes the reply into a JSON string field without escaping.

Two things to read off. The correct-serializer column is identical to the raw column, every row. Damage exactly zero. And the naive column loses 51.8 to 66.1 points — the same magnitude as the shell nested transport, on a boundary that contains no shell at all.

What a structured tool call actually buys

The natural conclusion is "use structured actions instead of shell strings." The paper measures that too, in a separate observational campaign: six provider-hosted models, both the raw contract and the provider's native shell tool, pooled over each model's measured effort ladder and three trials per cell.

The attempt counts are readable if you decompose them: 56 tasks × 3 trials × the number of rungs. Opus models expose 5 rungs, so 56 × 3 × 5 = 840. GPT-5.6-sol and Gemini-3.5-Flash have 4 rungs: 56 × 3 × 4 = 672. Gemini-3.1-Pro has 3: 56 × 3 × 3 = 504. The six sum to 4,368 raw plus 4,368 native = 8,736 generations, each replayed in both userlands for 17,472 execution outcomes.

The native-minus-raw change spans +2.6 to −10.0 points, and the family-sign sensitivity analysis returns twelve Holm-adjusted values all above .05, minimum .18750. So this study is explicitly descriptive and exploratory — the paper says so. But its central comparison is unambiguous: the native effect is smaller than the controlled nested loss for every model. A structured call removes one quoting surface. It does not remove sixty points of anything.

And the typed-operation pilot puts a number on the residual. Replacing shell-string construction with structured operations on 18 private tasks across two models — 36 attempts — passes 25 of 36, or 69.4%, against 35 of 36, or 97.2%, for raw bash -c and temporary scripts. Of the eleven typed failures, ten reach the wrong final state and one fails during execution. The most common cause the paper names: the model copies the instruction's delimiter markers into the payload.

The silent-failure ledger

Study B is also where the exit-code claim gets its numbers, and the full table is worth reading because each row partitions exactly 4,368 executions into five outcomes.

Check one row, then compute the headline. GNU native: 4146 + 20 + 22 + 128 + 52 = 4,368. Failures on that row are 4,368 − 4,146 = 222, of which 52 exited zero:

And BSD native, where the ratio is worst: failures are 4,368 − 4,236 = 132, of which 62 exited zero:

So the abstract's "23.4–47.0%" is the span across the four conditions, and the extremes are both on the native contract. That is not a coincidence: the adherence column shows why. The native arm has 20 invalid one-call invocations per userland and far fewer syntax errors — 9 versus 31 on BSD, 22 versus 39 on GNU — because a schema-validated tool call rules out malformed shell more often. Removing loud failures does not remove failures. It converts them into quiet ones.

Paired transitions, and where the wire was checked

Aggregate deltas hide churn. The paper reports paired pass-to-fail and fail-to-pass transitions for the native-minus-raw comparison, and the pairs are far larger than the nets.

Opus-5's net is −7 executions out of 840, which reads as "no effect" — but 25 individual task outcomes changed. Twenty-five different tasks behaved differently depending on how the model was asked, and the aggregate cancelled most of that. Verify one net against its percentage: Gemini-3.5-Flash loses 67 net of 672 attempts, and 67 ÷ 672 = 9.97% exactly.

The paper also does something rare and worth naming: it checks the wire. For three configurations from two providers, the raw native-tool arguments were retained, allowing byte-level verification between the decoded command field and what actually reached the executor — 660 records for Gemini-3.5-Flash, 497 for Gemini-3.1-Pro, and 672 for GPT-5.6-sol, with 19 adherence failures containing no usable argument. For the other three models the stored artefact begins at the decoded command string, so their analysis starts at that boundary and the paper says so rather than implying the check covered everything.

Static analysis does not catch it

If the failures were malformed shell, a linter would find them. The authors ran ShellCheck over the replies and the result is a clean demonstration that this is not a syntax problem.

It misses two thirds — 191 of the 292 — and its base rate on working commands is 11.4%, so the signal-to-noise is poor even where it fires. The reason is structural, and the paper states it: "each command is individually well-formed and the fault is in the downstream interpolation." There is nothing wrong with the program. The program never reached the executor.

Not the published payloads, and not one lucky draw

Two ways the result could be an artefact: the released tasks could be special, or a single sampled reply per cell could be unrepresentative. Both are tested.

Held-out payloads. A private crossover on 42 unpublished hostile payloads — three per family, hostile-only, manifest and sample hash fixed before inference, both contracts interleaved in one serving window:

Denominator 42, so one task is 2.38 points. GPT-5.6-sol: 39 of 42 on the direct path, 8 of 42 through the parser — damage −31 ÷ 42 = −73.8 — and 41 of 42 under disclosure, compensation +33 ÷ 42 = +78.6. The matched gap is positive, +4.8, which would read on a leaderboard as "the deployed path is slightly better for this model." It hides a 73.8-point loss and a 78.6-point recovery. Same story, different payloads, and here the masking is even more complete.

Repeated draws. Three generations per contract for all eight public configurations: damage stays negative for every configuration and every draw, with per-configuration ranges of 1.8–7.1 points. No compensation changes sign; the largest spread is 12.5 points (Opus-4.8), and Qwen3.5-27B realizes exactly zero compensation in every draw. On the private tasks, three extra generations for eight tasks produce different reply text in 9 of 16 task-contract cells for GPT-5.6-sol and 10 of 16 for Opus-4.8 — the models really did write something different — and every draw still shows negative damage and positive compensation. Mean over four draws: GPT-5.6-sol −78.1 damage and +81.3 compensation; Opus-4.8 −84.4 and +46.9.

The userland is a second systems axis

One more nuisance variable, and it is the one command benchmarks most often leave undeclared: GNU/Linux coreutils versus BSD/macOS coreutils. The primary replay is a pinned, network-disabled GNU container; the analysis is repeated in a BSD userland.

For the crossed design, the userland barely matters: corresponding crossover cells differ by at most 3.6 points for six of the eight configurations and by 7.1–12.5 for the other two, fixed-reply damage stays negative in both, and the masked set is unchanged. But for absolute shell competence it matters a lot. An earlier BSD-elicited campaign, replayed unchanged on GNU, shows model-specific dialect affinity in both directions:

Shifts reach 8.9 points and they reorder models: the raw top three go from Gemini-3.5-Flash / Fable-5 / Gemini-3.1-Pro on BSD to Gemini-3.1-Pro / Gemini-3.5-Flash / Fable-5 on GNU, and the nested ordering likewise swaps Fable-5 and Gemini-3.1-Pro at the top. Note the correct interpretation, which the paper insists on: because every command was elicited in BSD sessions, this measures cross-userland transfer of fixed commands, not what a model would write if told to target GNU.

Chapter 10: What To Report

The paper ends with a prescription rather than a technique, which is unusual and correct. This chapter is that prescription, the fixes, the limits, and the connections to the rest of what you already know about evaluation.

Five things, or the number means nothing

The abstract's closing sentence is the whole paper compressed: evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator "rather than treat a matched score as an intrinsic model property."

The repair, chosen by who controls the boundary

The fixes are boring and total. What is not boring is deciding which one you are allowed to use, and the deciding factor is ownership of the interpolation point.

The private replay makes the accounting concrete. Three models, 42 tasks each, one fixed raw reply per task, three transports:

Add the recoveries: (41 − 8) + (40 − 7) + (30 − 9) = 33 + 33 + 21 = 87 commands that fail only under the wrapper. Add the residual raw failures: (42 − 41) + (42 − 40) + (42 − 30) = 1 + 2 + 12 = 15, unresolved by any transport. The script column reproduces the raw column exactly, row by row.

What the paper does not know

The limitations section is short and every line of it constrains a claim you might otherwise make.

And the artefacts, because a benchmark you cannot rerun is a claim rather than a measurement: the harness, all 56 tasks, validators, contract prompts, and offline verification commands are released, along with a sanitized rollout archive of 12,999 records across 33 arm files carrying each generation with its replays, prompt, reply, identifiers, usage, and final-state outcomes, plus a SHA-256 manifest and a REPRODUCE.md that reconciles campaign-level counts. Private payloads and replies are withheld to preserve held-out evaluation. Every released task file embeds a fixed canary GUID for contamination checks.

One more design detail worth copying: because the harness executes untrusted model output, every attempt runs in a fresh fixture inside a timeout-bounded, network-disabled container, and the incident evidence is released only as de-identified mechanism classifications.

Instrumenting your own system this week

The measurement is cheap enough that there is no reason not to run it on whatever you are building. Here is the whole thing, reduced to the four steps the paper actually performs.

python
# STEP 1 — store every reply. Not the score. The BYTES.
# This is the only step that costs model calls, and it is the only
# step you cannot go back and do later.
rollout = []
for task in TASKS:
    R = model(SYSTEM_PROMPT, task.instruction)
    rollout.append({"task": task.id, "contract": "raw", "reply": R,
                    "model": MODEL_ID, "queried": TODAY, "effort": EFFORT})

# STEP 2 — replay through every transport you might ever deploy on.
# Zero model calls. Add transports freely; each one is just container time.
TRANSPORTS = {"direct": run_raw, "nested": run_nested,
              "escaped": run_nested_escaped, "script": run_script}
for rec in rollout:
    for name, fn in TRANSPORTS.items():
        work = fresh_fixture(rec["task"])
        fn(rec["reply"], cwd=work)
        rec[name] = validate(rec["task"], work)   # FINAL STATE, not exit code

# STEP 3 — the number that tells you whether you have a problem.
direct = sum(r["direct"] for r in rollout)
deploy = sum(r[DEPLOYED_TRANSPORT] for r in rollout)
print(f"portability: {deploy}/{direct} of working commands survive")

# STEP 4 — if portability is bad, check whether it is YOUR bug.
# If "escaped" reproduces "direct" exactly, the model was never wrong.
assert all(r["escaped"] == r["direct"] for r in rollout), \
       "escaping does not fully restore — there is a second boundary somewhere"

Step 4 is the diagnostic worth internalising. If escaping at the interpolation point restores every direct-path success, then the model produced correct programs and your harness destroyed them — and that is a one-line fix in your code, not a model-selection problem. In the paper that assertion holds for all 448 public pairs. If it fails in your system, you have more than one boundary, and you have just found the second one.

Three things that make the difference between this being a real measurement and a comforting one:

Bridges

Cheat sheet

References

1. S. Li, Y. Zhang, V. Tresp, Y. Yang. "QuoteBench: How Matched Scores Can Hide Command-Path Failures." arXiv preprint arXiv:2608.13547, 2026. arXiv
2. Free Software Foundation. "Bash Reference Manual, version 5.3." 2025. — the quoting and expansion rules traced in Chapter 1.
3. D. A. Wheeler. "Fixing Unix/Linux/POSIX Filenames." 2010. — the hostile-filename hazards behind one mechanism group.
4. V. Holen. "ShellCheck: a static analysis tool for shell scripts." — flags only 34.6% of the nested-only failures.
5. X. V. Lin, C. Wang, L. Zettlemoyer, M. D. Ernst. "NL2Bash: a corpus and semantic parser for natural language interface to the Linux operating system." LREC, 2018. arXiv
6. M. Agarwal et al. "NeurIPS 2020 NLC2CMD competition: translating natural language to Bash commands." arXiv:2103.02523, 2021. arXiv
7. F. Westenfelder, E. Hemberg, S. Moskal, U. O'Reilly, S. Chiricescu. "LLM-supported natural language to Bash translation." NAACL, 2025.
8. L. Yu et al. "BashCoder-R1: towards robust and explainable Bash code generation with robustness-aware group relative policy optimization." arXiv:2606.27733, 2026. arXiv
9. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, O. Press. "SWE-agent: agent-computer interfaces enable automated software engineering." NeurIPS, 2024.
10. M. A. Merrill et al. "Terminal-Bench: benchmarking agents on hard, realistic tasks in command line interfaces." arXiv:2601.11868, 2026. arXiv
11. Y. Zhang, J. Wang, Y. Ge, W. Xu, J. Hamm, C. K. Reddy. "Stop comparing LLM agents without disclosing the harness." arXiv:2605.23950, 2026. arXiv
12. Z. Wang, B. Yu, J. Xu, Z. Li. "Action boundary blindness: when LLM agents cannot tell where one action ends and another begins." ACL, 2026.
13. B. Yu, Y. Zhu, P. He, D. Kang. "UTBoost: rigorous evaluation of coding agents on SWE-bench." ACL, 2025. — permissive validators accepting incorrect patches.
14. M. Sclar, Y. Choi, Y. Tsvetkov, A. Suhr. "Quantifying language models' sensitivity to spurious features in prompt design." ICLR, 2024. — the input-side analogue that regeneration cannot decompose.
15. W. Zhang et al. "CARE: pre-execution command verification for shell-executing LLM agents." arXiv:2607.21642, 2026. arXiv
16. X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, H. Ji. "Executable code actions elicit better LLM agents." ICML, 2024. — CodeAct, changing the action language itself.