Evaluate Your Agents
Rigorously

Chapter 1: What IS an Agent?

Before we can evaluate agents, we need to agree on what one is. An agent is an LLM running in a loop, equipped with tools and instructions, that can take actions in an environment. Three components define it:

The agent harness is the software system that enables a model to act as an agent. It manages the loop: send the context to the model, parse its output for tool calls, execute those tools, feed results back, repeat. When we evaluate an agent, we're evaluating the harness AND the model working together — not the model in isolation.

How Tool Calling Works

Modern LLMs are trained to emit special structured output when they want to call a tool. The pattern looks like this:

json
// 1. You give the model a list of available tools:
{
  "tools": [
    {
      "name": "lookup_order",
      "description": "Look up order details by order ID",
      "parameters": {
        "order_id": { "type": "string" }
      }
    }
  ]
}

// 2. The model responds with a tool call:
{
  "tool_call": {
    "name": "lookup_order",
    "arguments": { "order_id": "ORD-12345" }
  }
}

// 3. Your harness executes the function, returns result:
{
  "tool_result": {
    "status": "shipped",
    "item": "Blue Widget",
    "tracking": "1Z999AA10123456784"
  }
}

// 4. Model sees result, decides next action or responds to user

This parse-execute-continue cycle is the heartbeat of every agent. The harness parses the model's output for tool calls, executes them in the real environment, and feeds results back into the context. The cycle continues until the model produces a final response (no tool call) or hits a maximum iteration limit.

The Autonomy Spectrum

Not everything that uses an LLM is an agent. There's a spectrum from simple to fully autonomous:

As you move down this spectrum, evaluation gets harder. A single-turn LLM can be evaluated with a simple input-output dataset. A multi-agent system requires evaluating coordination, delegation, and emergent behaviors across multiple interacting components.

The Harness Loop in Detail

Let's look at a concrete harness implementation. This is the actual control flow that runs every agent:

python
def run_agent(user_message, system_prompt, tools, max_iters=25):
    """The core agent loop. Every agent harness is a variant of this."""
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]

    for i in range(max_iters):
        # Step 1: Send context to model
        response = llm.generate(
            messages=messages,
            tools=tools,
            temperature=0.0  # lower = more deterministic
        )

        # Step 2: Check if model wants to call a tool
        if response.has_tool_call:
            # Step 3: Parse and execute the tool call
            tool_name = response.tool_call.name
            tool_args = response.tool_call.arguments
            result = execute_tool(tool_name, tool_args)

            # Step 4: Feed result back into context
            messages.append(response.to_message())
            messages.append({
                "role": "tool",
                "content": json.dumps(result)
            })
            continue  # back to Step 1

        # No tool call = final response to user
        messages.append(response.to_message())
        return messages  # the full transcript

    # Safety: hit max iterations without finishing
    raise MaxIterationsError("Agent stuck in loop")

Notice the key design decisions embedded in this code:

• Max iterations: Without this, a confused agent could loop forever, calling tools that don't help. The iteration limit is a safety valve.
• Temperature: Lower temperature means more deterministic output, but reduces the agent's ability to explore alternative strategies. This directly affects evaluation — higher temperature means more variance across trials.
• Context accumulation: Every tool call and result is appended to the messages list. This is why context management matters — after 20 tool calls, you've added 40+ messages to the context.
• No retry logic: If a tool call fails, the raw error is fed back to the model. The model decides whether to retry, try a different tool, or give up. This is a harness design choice that affects evaluation results.

What Makes a Good Tool Definition?

The quality of tool definitions dramatically affects agent performance. Compare these two definitions for the same tool:

bad tool def
{
  "name": "search",
  "description": "Search",
  "parameters": {
    "q": {"type": "string"}
  }
}

good tool def
{
  "name": "search_orders",
  "description": "Search orders
  by customer email, order
  ID, or date range. Returns
  up to 10 matching orders.",
  "parameters": {
    "query": {
      "type": "string",
      "description": "Email,
      order ID, or YYYY-MM-DD"
    }
  }
}

The good definition tells the model exactly what the tool does, what inputs it accepts, and what format to use. When evaluating agents, changing tool definitions can shift pass rates by 10-20%. This is part of why we evaluate the harness as a whole, not just the model.

Chapter 2: Multi-Agent Systems

Sometimes a single agent isn't enough. When your system prompt grows to thousands of lines, or you have dozens of overlapping tools, or different tasks require fundamentally different reasoning strategies — you need multiple agents working together. But multi-agent systems are harder to build, debug, and evaluate. So the first rule is: start with a single agent. Only add complexity when the single agent demonstrably fails.

When to Go Multi-Agent

Three signals tell you a single agent is struggling:

Manager vs Decentralized

There are two main patterns for multi-agent systems:

Sub-agents and Context Protection

A critical benefit of multi-agent systems is context protection. When a sub-agent handles a tool-heavy subtask, all those tool calls and results fill up the sub-agent's context window, not the main agent's. The main agent only sees the sub-agent's final summary. This keeps the main agent's context clean and focused.

Think of it like delegation at a company. The CEO doesn't sit in on every engineering standup. They delegate to a VP, who reports back a summary. The CEO's "context window" stays clear for strategic decisions.

The Manager Pattern in Code

python
# Manager agent with specialized sub-agents

class ManagerAgent:
    def __init__(self):
        self.router = LLM(system_prompt="""You are a routing agent.
        Given a user request, determine which specialist to delegate to:
        - 'returns': for return/refund requests
        - 'booking': for flight/hotel changes
        - 'billing': for payment/invoice questions
        Respond with ONLY the specialist name.""")

        self.specialists = {
            "returns": Agent(tools=return_tools, prompt=return_policy),
            "booking": Agent(tools=booking_tools, prompt=booking_policy),
            "billing": Agent(tools=billing_tools, prompt=billing_policy),
        }

    def handle(self, user_message):
        # Step 1: Route to specialist
        specialist = self.router.generate(user_message).strip()

        # Step 2: Delegate (sub-agent gets its own clean context)
        result = self.specialists[specialist].run(user_message)

        # Step 3: Manager only sees the summary, not all tool calls
        return result.final_response

Notice the key property: the sub-agent runs in its own context. If the returns specialist makes 8 tool calls consuming 4,000 tokens, the manager never sees those tokens. It only gets the final response — maybe 200 tokens. This is the context protection in action.

Evaluating Multi-Agent Systems

When evaluating a multi-agent system, you need to test at three levels:

1. Individual agent evaluation: Does each specialist work correctly in isolation? Test the returns agent on return tasks, the booking agent on booking tasks.
2. Routing evaluation: Does the manager route to the correct specialist? Test with clear-cut cases AND ambiguous cases (e.g., "I want to return an item I booked" — returns or booking?).
3. End-to-end evaluation: Does the full system work? This catches integration failures: the manager routes correctly, the specialist handles it correctly, but the response doesn't get properly relayed back.

A common mistake is skipping level 2 (routing evaluation). Teams test each specialist and test the full system, but never specifically evaluate whether the router makes correct decisions. A routing error sends the user to the wrong specialist, which wastes time and often fails spectacularly because the specialist doesn't have the right tools for the task.

Chapter 3: Context Engineering

An agent's context window is its working memory. Everything the agent knows — the system prompt, conversation history, tool results, retrieved documents — lives in this finite window. And here's the problem: context rots.

Context rot is what happens when an agent runs for many steps. Each tool call adds its result to the context. Each reasoning step adds tokens. After twenty iterations, the context is stuffed with old tool results, irrelevant intermediate reasoning, and stale information. The model's attention is spread across thousands of tokens of noise, and the signal — the user's original request, the current state of the task — gets buried.

Static vs Dynamic Context

There are two strategies for getting information into the context window:

The modern best practice is progressive disclosure: give the agent tools to discover context rather than pre-loading everything. Let the agent search a knowledge base, look up a policy document, or query a database when it needs information. This way, only relevant information enters the context, and the agent stays in control of what it knows.

Compaction Strategies

Even with progressive disclosure, context eventually fills up. Three strategies fight context rot:

Each strategy involves a tradeoff: compaction loses detail but reclaims space. The art is knowing when to compact and how much to keep. Compact too aggressively and the agent forgets important context. Compact too little and you hit the token limit.

A Concrete Context Budget

Let's do the math for a typical agent task. Say you're using a model with a 128K token context window:

At 50 tool calls, you've consumed 36% of the context window. That sounds manageable, but here's the catch: the attention quality of a Transformer degrades well before you hit the limit. Research shows that models struggle to attend to information in the middle of long contexts (the "lost in the middle" problem). By the time you're at 50% utilization, the model may already be failing to recall its original instructions.

Progressive Disclosure in Practice

Here's a concrete example of progressive disclosure vs static context. Imagine an agent that helps with HR policies:

python
# BAD: Static context — dump everything into system prompt
system_prompt = """You are an HR assistant.
Here are ALL company policies:

[3,000 tokens of vacation policy]
[2,000 tokens of leave policy]
[4,000 tokens of benefits policy]
[2,500 tokens of expense policy]
...
"""
# Result: 15,000+ tokens wasted if user only asks about vacation

# GOOD: Progressive disclosure — agent discovers what it needs
system_prompt = """You are an HR assistant.
Use the search_policy tool to look up relevant policies before
answering any question."""

tools = [{
    "name": "search_policy",
    "description": "Search company HR policies by topic",
    "parameters": {"query": {"type": "string"}}
}]
# Result: only loads the 800 tokens about vacation when needed

The static approach consumes 15,000 tokens upfront for policies the user may never ask about. The progressive approach uses 800 tokens only when needed. Over a 20-step conversation, that 14,200-token savings compounds — leaving more room for actual tool results and reasoning.

A Context Rot Failure: Worked Example

Here's a real failure pattern caused by context rot. An agent is handling a complex customer request that requires multiple tool calls:

transcript (simplified)
Turn 1: User: "I have two orders. Cancel ORD-100 and check status of ORD-200."
Turn 2: Agent calls lookup_order("ORD-100") # 500 tokens of result
Turn 3: Agent calls check_cancel_policy("ORD-100") # 400 tokens of result
Turn 4: Agent calls cancel_order("ORD-100") # 300 tokens of result
Turn 5: Agent calls lookup_order("ORD-200") # 500 tokens of result
Turn 6: Agent calls get_tracking("ORD-200") # 600 tokens of result

# By Turn 6, context has ~3,500 tokens of tool results.
# Agent's response:
Turn 7: "I've cancelled ORD-200 and your order ORD-100 is in transit."
# ^^^^^^^^ WRONG! Mixed up the two order IDs!

The agent correctly executed all the tool calls but then confused the two orders in its final response. It cancelled ORD-100 (correct) but then said it cancelled ORD-200 (wrong). This is classic context rot: the model's attention, spread across 3,500 tokens of tool results, failed to correctly attribute which results belonged to which order.

This failure would be caught by a code grader that checks the response text for correct order IDs. But it would NOT be caught by a pure outcome grader that only checks database state (the cancellation was correct). You need both.

Chapter 4: The Evaluation Framework

Now we get to the core. Every agent evaluation system, no matter how sophisticated, boils down to five components. Learn these five, and you can evaluate anything.

1. Task

A task is a predefined input paired with success criteria. "Given this user message and this database state, the agent should do X." A task includes:

• The input: user message, conversation history, initial environment state
• The success criteria: what counts as a correct outcome
• Optional ground truth: the expected final state, expected tool calls, or reference output

Good tasks are specific, solvable, and have clear success criteria. "Handle a customer complaint" is a bad task. "User wants to return order ORD-789. Policy allows returns within 30 days. Order was placed 15 days ago. Agent should process the return and confirm to the user" is a good task.

2. Trial

A trial is one attempt at completing a task. Because agents are non-deterministic, running the same task twice produces different trials. This is why we need multiple trials per task — a single trial tells you almost nothing about reliability.

3. Transcript (Trajectory)

The transcript is the complete record of everything the agent did during a trial: every message, every tool call, every tool result, every reasoning step. This is your forensic evidence when things go wrong. A good evaluation system always saves the full transcript.

transcript
# Example transcript structure:
[
  {"role": "user",     "content": "I want to return order ORD-789"},
  {"role": "assistant", "tool_call": "lookup_order(ORD-789)"},
  {"role": "tool",      "result": "{status: delivered, days_ago: 15}"},
  {"role": "assistant", "tool_call": "check_return_policy(ORD-789)"},
  {"role": "tool",      "result": "{eligible: true, window: 30 days}"},
  {"role": "assistant", "tool_call": "process_return(ORD-789)"},
  {"role": "tool",      "result": "{return_id: RET-456, refund: $49.99}"},
  {"role": "assistant", "content": "Your return has been processed..."}
]

4. Outcome

The outcome is the final environment state after the agent finishes. Did the database change correctly? Was the right API called? Does the final response contain the right information? Outcomes can be checked by inspecting the environment, the agent's final output, or both.

5. Grader

The grader takes a transcript and/or outcome and produces a score. "Did this trial succeed?" Graders can be human reviewers, automated code checks, or LLMs acting as judges. We'll dive deep into grader types in the next chapter.

Environment Isolation

A subtle but critical requirement: each trial must run in a clean, isolated environment. If Trial 1 cancels an order in the database, Trial 2 must start with the order still active. Otherwise, Trial 2 might fail not because the agent is broken, but because the order was already cancelled by Trial 1.

Three isolation strategies:

Docker containers are the most robust approach — they guarantee complete isolation. Mock environments are fastest but risk missing bugs that only appear in real tool interactions. Most production eval systems use a hybrid: mocked tools for fast iteration, Docker containers for full regression runs.

Saving and Analyzing Transcripts

Every trial should save the complete transcript in a structured format. This is your debugging lifeline. When pass rates drop, you read the transcripts to understand why. A good transcript record includes:

json
{
  "task_id": "cancel-order-01",
  "trial_id": "trial-007",
  "timestamp": "2026-05-18T14:30:00Z",
  "model": "claude-opus-4-20250514",
  "temperature": 0.0,
  "messages": [/* full message sequence */],
  "tool_calls": [
    {"name": "lookup_order", "args": {"id": "ORD-123"}, "latency_ms": 45},
    {"name": "cancel_order", "args": {"id": "ORD-123"}, "latency_ms": 120}
  ],
  "total_tokens": 3847,
  "total_latency_ms": 4521,
  "db_state_before": {/* snapshot */},
  "db_state_after": {/* snapshot */},
  "grading": {
    "code_grader": {"passed": true, "checks": ["db_state", "tool_calls"]},
    "model_grader": {"score": 4, "reasoning": "..."}
  },
  "result": "PASS"
}

With this structure, you can query your evaluation database: "Show me all failing trials for task 'cancel-order-01' where the agent called the wrong tool." This turns debugging from guesswork into data analysis.

A Concrete Example

Let's put it all together with a worked example:

Notice that Trial 1 and Trial 2 took different paths (3 steps vs 4 steps) but both arrived at the correct outcome. This is normal for agents — the path varies, but we care about the destination.

Trajectory-Based vs Outcome-Based Evaluation

This brings us to a fundamental design choice: do you grade the trajectory (how the agent got there) or the outcome (where it ended up)?

In practice, the best approach combines both: outcome-based grading for "did it work?" plus lightweight trajectory checks for safety constraints: "did the agent avoid calling the delete_all_data() tool?" "did it confirm with the user before making changes?"

Statistical Significance

How many trials do you need per task? This depends on how precise you need your estimates to be. Here's a rule of thumb:

These are approximate — the actual confidence interval depends on the true pass rate. But as a planning heuristic: 10 trials per task is the sweet spot for most iterative development. Use 20+ for final deployment decisions.

Chapter 5: Types of Graders

You have a transcript. Now you need to decide: did the agent succeed? This is the job of the grader. There are three families of graders, each with distinct strengths and weaknesses. The best evaluation systems use all three.

1. Human Graders

Human graders are the gold standard. A subject-matter expert reads the transcript, evaluates the outcome, and assigns a score. Nothing beats a human who deeply understands the domain.

But human grading has problems. It's slow (minutes per transcript), expensive (expert time), and doesn't scale. You can't have a human review 10,000 trials overnight. And humans disagree with each other — inter-annotator agreement (how often two humans give the same score) is often surprisingly low, especially for subjective quality judgments.

Variants of human grading:

• SME review: Domain experts review transcripts. Highest quality, most expensive.
• Crowdsourced: Many non-expert reviewers. Cheaper, but noisier. Need redundancy (3+ reviewers per item) and agreement filters.
• A/B testing: Show human reviewers outputs from two agent versions side by side. "Which is better?" Relative judgments are easier and more consistent than absolute scores.

2. Code-Based Graders

Code-based graders are deterministic functions that check specific properties of the outcome or transcript. They're fast, cheap, and perfectly reproducible. But they're brittle — they can only check things you can express as code.

python
# Example code-based graders:

def check_string_match(transcript, expected):
    """Did the final response contain the expected string?"""
    final_msg = transcript[-1]["content"]
    return expected.lower() in final_msg.lower()

def check_tool_called(transcript, tool_name):
    """Was the required tool called at least once?"""
    return any(
        msg.get("tool_call", "").startswith(tool_name)
        for msg in transcript
    )

def check_database_state(db, order_id, expected_status):
    """Did the database end up in the correct state?"""
    return db[order_id]["status"] == expected_status

def check_no_forbidden_tool(transcript, forbidden):
    """Did the agent avoid calling a forbidden tool?"""
    return not any(
        msg.get("tool_call", "").startswith(forbidden)
        for msg in transcript
    )

Code-based graders are perfect for objective criteria: "Was the database updated?" "Did the response include the order ID?" "Was the forbidden tool avoided?" They fail for subjective criteria: "Was the tone professional?" "Did the explanation make sense?"

3. Model-Based Graders (LLM-as-Judge)

Model-based graders use another LLM to evaluate the agent's output. You provide a rubric and the transcript, and the judge LLM scores it. This combines the flexibility of human judgment with the scalability of automation.

python
# LLM-as-Judge grader (simplified)

def llm_judge(transcript, rubric):
    prompt = f"""You are an expert evaluator.

Given this agent transcript:
{transcript}

Score the agent on the following rubric:
{rubric}

Respond with a JSON object:
{{"score": 1-5, "reasoning": "..."}}"

    response = llm.generate(prompt)
    return json.loads(response)

Three common patterns for model-based grading:

The catch: model-based graders are themselves non-deterministic. Run the same judgment twice and you might get different scores. They also have biases — they tend to prefer longer outputs, outputs that look like their own writing style, and outputs listed first in pairwise comparisons. You need to calibrate model-based graders against human judgments before trusting them.

Calibrating LLM Judges

How do you know your LLM judge is actually reliable? The standard approach is calibration against human labels:

1. Have human experts score 50-100 transcripts on your rubric.
2. Run the LLM judge on the same transcripts.
3. Compute agreement: what percentage of the time do human and model agree?
4. If agreement is <80%, revise your rubric — it's probably ambiguous.
5. Check for systematic bias: does the model consistently score higher or lower than humans?
6. Re-calibrate periodically as you update the agent (the distribution of outputs changes).

A well-calibrated LLM judge typically agrees with human experts 80-90% of the time on binary (pass/fail) judgments. For 5-point Likert scales, expect within-1-point agreement about 70-80% of the time. If you're below these numbers, the judge is too noisy to trust.

Known Biases in LLM Judges

Chapter 6: Metrics — Pass@K and Pass^K

You've run your agent on 50 tasks, 10 trials each, and collected 500 scores. Now what? You need metrics that capture both capability (can the agent solve the task at all?) and reliability (can it solve it consistently?). Two metrics do this: Pass@K and Pass^K.

Pass@K: Can It Succeed At Least Once?

Pass@K measures the probability that at least one out of K attempts succeeds. If you give the agent K tries, what are the odds it gets it right at least once?

Where n is the total number of trials you ran, c is the number of successful trials, and C(a, b) is the binomial coefficient "a choose b." Let's unpack this with a worked example.

Pass^K: Does It Succeed EVERY Time?

Pass^K flips the question: do ALL K attempts succeed? This measures reliability — can you trust the agent to get it right every time, not just sometimes?

The gap between pass@K and pass^K reveals a critical truth about your agent. High pass@K means the agent can solve the task. Low pass^K means it does so unreliably. For production systems, reliability matters more than capability. A user doesn't get 3 tries — they get one.

How Pass^K Drops With K

Even for agents with respectable per-trial accuracy, pass^K drops dramatically as K increases. Let's compute for an agent with 80% per-trial success (c = 8 out of n = 10):

At K = 5, pass@5 is essentially 1.0 (the agent can almost certainly solve it in 5 tries). But pass^5 is only 0.222 — it only succeeds on all 5 attempts about 22% of the time. For an 80% accurate agent! This is why pass^K is the metric that matters for production deployment.

Deriving the Formulas

Where do these formulas come from? Let's derive them from first principles.

Pass@K derivation: We have n total trials, c of which succeeded. We want to pick K trials at random. What's the probability that at least one is a success?

It's easier to compute the complement: the probability that all K picks are failures. There are (n - c) failures total. The number of ways to pick K failures from (n - c) is C(n - c, K). The total ways to pick K trials from n is C(n, K). So:

Pass^K derivation: Same setup, but now we want the probability that ALL K picks are successes. The number of ways to pick K successes from c is C(c, K). So:

Both formulas use the hypergeometric distribution — sampling without replacement from a finite population. This is important: we're not assuming each trial is independent (which would give a simpler binomial model). The hypergeometric approach accounts for the fact that we ran a fixed number of trials and observed a fixed number of successes.

When Approximate Counts Are Enough

If you assume trials are independent with success probability p = c/n, you get simpler approximations:

These are good approximations when n is large relative to K. For small n (say n = 10, K = 5), use the exact hypergeometric formulas. For large n (n = 100), the approximations are close enough.

Worked Example: Comparing Two Agents

Agent A achieves 70% per-trial success (7/10). Agent B achieves 50% per-trial success (5/10). But Agent B uses a retry mechanism that auto-retries on failure. Which is better for production?

Agent A is better on every metric. The retry mechanism helps Agent B's pass@3 (0.917 is decent), but its pass^3 (0.083) means it reliably succeeds on 3 consecutive attempts only 8% of the time. Retries mask capability problems — they don't fix them.

Chapter 7: Case Study — τ-bench

Theory is great, but what does a real agent evaluation look like? Let's study τ-bench (tau-bench), one of the most well-designed agent benchmarks. It evaluates agents on dynamic, multi-turn customer service conversations in two domains: retail and airline.

The Four Components of τ-bench

The magic of τ-bench is that the user is also an LLM. This means conversations are dynamic — the simulated user reacts to the agent's responses, asks follow-up questions, provides clarifications, and can even get confused or frustrated. No two conversations are identical, even for the same task.

How a τ-bench Task Works

Each task specifies a scenario. For example: "User wants to change a flight from economy to business class. The fare difference is $350. The user has $200 in credit. Policy: credit can be applied to upgrades." The user simulator drives the conversation, and the agent must navigate the tools and policies to resolve it correctly.

python
# Simplified τ-bench agentic loop

def run_tau_bench_trial(task, agent, user_sim, db, tools, policies):
    # Initialize conversation with user's opening message
    user_msg = user_sim.generate_opening(task.scenario)
    conversation = [user_msg]
    db_snapshot = copy.deepcopy(db)  # snapshot for grading

    for turn in range(25):  # max 25 turns
        # Agent sees: system prompt + policies + conversation
        agent_response = agent.generate(
            system=policies,
            tools=tools,
            messages=conversation
        )

        # If agent wants to call a tool:
        if agent_response.has_tool_call:
            result = execute_tool(agent_response.tool_call, db)
            conversation.append(agent_response)
            conversation.append(tool_result(result))
            continue  # agent gets to see result and decide next

        # If agent responds to user:
        conversation.append(agent_response)

        # Check if conversation is done
        if user_sim.is_satisfied(conversation, task):
            break

        # User simulator responds
        user_reply = user_sim.generate_reply(conversation, task)
        conversation.append(user_reply)

    # Grade: check database + output against ground truth
    return grade(db, task.expected_db_state, conversation)

Outcome Verification

τ-bench grades outcomes in two ways: checking that the database was modified correctly (e.g., flight was actually changed, credit was applied) AND checking that the agent's text responses contain the right information (e.g., confirming the new flight details to the user). Both must pass for a trial to succeed.

Why Non-Determinism Matters Here

There are two sources of randomness in τ-bench. The agent samples from its LLM, producing different reasoning and tool call sequences. And the user simulator also samples, producing different follow-up questions and phrasings. This means every trial of the same task generates a genuinely different conversation. Pass@K and pass^K become essential because a single trial is statistically meaningless.

Policy Compliance: The Hard Part

The most challenging aspect of τ-bench isn't the tool calling — it's policy compliance. The agent receives policy documents like this:

policy excerpt
## Return Policy
- Electronics: 14-day return window, 15% restocking fee
- Clothing: 30-day return window, no restocking fee
- Sale items: final sale, no returns
- Defective items: 90-day window, full refund, no restocking fee

## Exceptions
- If customer has Gold status: waive restocking fee
- If order was delayed >7 days: extend return window by 14 days
- Items over $500: require manager approval for return

The agent must navigate these rules while conversing with a user who may not know the policies. When a Gold-status customer asks to return a $600 electronic item purchased 16 days ago, the agent needs to: (1) recognize the item is past the 14-day window, (2) check if the order was delayed >7 days, (3) note the item is over $500, (4) check Gold status to waive restocking. Missing any one of these conditions produces a wrong outcome.

This is where agents typically fail. The model might apply the wrong policy, miss an exception clause, or hallucinate a policy that doesn't exist. Policy compliance failures account for roughly 40% of all errors in τ-bench.

The Extensions: τ²-bench and τ³-bench

τ-bench spawned two important extensions:

What Makes τ-bench a Good Benchmark?

Several design choices make τ-bench particularly well-designed:

• Realistic complexity: Tasks mirror real customer service scenarios, not toy problems.
• Dynamic conversations: The user simulator creates genuine back-and-forth, not scripted exchanges.
• Multi-signal grading: Checking both database state and output text catches agents that say the right thing but do the wrong thing (or vice versa).
• Built-in non-determinism: Two sources of randomness (agent + user simulator) force the use of multiple trials and proper statistical metrics.
• Domain specificity: Retail and airline policies are complex enough to be challenging but well-defined enough to have clear ground truth.

Chapter 8: Case Study — Terminal-Bench

While τ-bench evaluates conversational agents, Terminal-Bench tackles a different challenge: agents that solve real tasks in real terminal environments. No simulated users. No conversation. Just a Docker container, a task instruction, and a set of automated tests to verify the result.

The Harbor Framework

Each Terminal-Bench task is packaged as a Harbor — a self-contained evaluation unit with four components:

The key design principle is outcome-oriented evaluation. Terminal-Bench doesn't care HOW the agent solves the task — it only checks the final state of the Docker container. Did the right files get created? Does the script work? Are the tests passing? This means the agent can use any approach: write a script, install packages, copy files, whatever works.

Task Quality: The 7-Step Audit

Not all tasks are good tasks. Terminal-Bench enforces quality through a rigorous 7-step audit process for every task:

Step 3 — integrity checking — is especially important. An agent might learn to exploit the test suite rather than solve the task. For example, if the test checks "does file output.txt contain the word 'success'?", a lazy agent could just write "success" to the file without doing any real work. The audit process includes adversarial exploit detection: a human tries to find shortcuts that pass the tests without solving the task, then fixes the tests to block those shortcuts.

Adversarial Exploit Detection: A Deep Dive

Let's walk through a concrete integrity failure and how to fix it. Original task and test:

example
# TASK: "Write a Python script that sorts a CSV file by the 'date' column"

# TEST (naive):
def test_csv_sorted():
    output = read_file("output.csv")
    dates = [row["date"] for row in csv.DictReader(output)]
    assert dates == sorted(dates)  # just checks output is sorted

An agent could exploit this test by simply writing a hardcoded sorted CSV file without implementing any sorting logic. Or it could read the test file, figure out the expected output, and write it directly. To fix this, the audit adds:

example
# TEST (robust):
def test_csv_sorted_robust():
    # 1. Check output file exists and is valid CSV
    output = read_file("output.csv")
    dates = [row["date"] for row in csv.DictReader(output)]
    assert dates == sorted(dates)

    # 2. Check a Python script exists (not just a data file)
    assert os.path.exists("sort_csv.py")

    # 3. Run the script on a DIFFERENT input to verify generality
    create_random_csv("test_input.csv", n=100)
    subprocess.run(["python", "sort_csv.py", "test_input.csv", "test_output.csv"])
    test_dates = [row["date"] for row in csv.DictReader(open("test_output.csv"))]
    assert test_dates == sorted(test_dates)

    # 4. Verify the script doesn't just copy the expected output
    assert "hardcoded" not in open("sort_csv.py").read().lower()

The robust test runs the agent's solution on a new, random input that didn't exist during the agent's execution. This is the gold standard for integrity checking: the test should verify that the agent created a general solution, not a task-specific shortcut.

Scale and Coverage

Terminal-Bench contains 89 tasks across 15+ categories: software engineering, data processing, system administration, scientific computing, networking, databases, and more. Each task takes a capable agent 5-30 minutes to attempt, making the full benchmark a substantial test of real-world terminal skills.

Task Category Breakdown

Terminal-Bench Results Across Models

Performance on Terminal-Bench varies dramatically by model and agent framework. Some findings:

The last finding is counterintuitive: you'd expect more time to help, but agents with generous time budgets sometimes perform worse because they make more tool calls, fill up the context, and start making confusion errors. This connects directly back to context engineering — a well-designed agent knows when to stop exploring and commit to an answer.

Designing Your Own Task Harbors

If you're building a Terminal-Bench-style evaluation for your own domain, follow these principles:

1. Dockerfile first: Define the exact environment. Pin package versions. Make the setup reproducible.
2. Oracle solution second: Solve the task yourself before asking the agent. If you can't solve it, it's not a fair task.
3. Tests third: Write tests that verify the outcome, not the path. Test on new inputs when possible.
4. Adversarial audit fourth: Try to break your own tests. Can you pass them without solving the task? Fix any exploits.
5. Difficulty calibration last: Estimate task difficulty based on number of required commands, conceptual complexity, and domain specificity.

Chapter 9: The Swiss Cheese Model

No single evaluation method catches everything. Human reviewers miss things. Code-based graders are brittle. Model-based graders have biases. Automated benchmarks have blind spots. If you rely on any one method alone, failures will slip through.

The solution is the Swiss cheese model, borrowed from safety engineering. Imagine each evaluation method as a slice of Swiss cheese — it blocks most failures, but has holes. Stack multiple slices together, and the holes in one slice are covered by solid cheese in the next. Enough layers, and almost nothing gets through.

What Each Layer Catches

Notice the pattern: no single layer has all green. But together, they cover everything. This is why you need all three — not just your favorite.

Implementation: What Each Layer Looks Like

Layer 1 — Automated Evals: A CI pipeline that runs your benchmark suite on every pull request. Takes 10-30 minutes. Reports pass@1 and pass^3. Blocks merge if pass@1 drops by more than 5% from baseline. This is your first line of defense.

yaml
# .github/workflows/agent-eval.yml (simplified)
name: Agent Evaluation
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python run_eval.py
               --tasks eval/tasks/*.json
               --trials-per-task 5
               --output results.json
      - run: python check_regression.py
               --current results.json
               --baseline eval/baseline.json
               --max-regression 0.05

Layer 2 — Manual Review: Weekly, a domain expert reviews 20-30 randomly sampled transcripts from the eval suite. They score on a rubric and flag any new failure modes. This catches "the agent technically passed the code grader but gave bad advice" situations.

Layer 3 — Production Monitoring: In production, track these signals:

• Resolution rate: What percentage of conversations result in the user's issue being resolved?
• Escalation rate: How often does the agent hand off to a human? Rising escalation rate = agent is struggling.
• User satisfaction: Post-conversation thumbs up/down or CSAT score.
• Latency: How long does the agent take? Long conversations may indicate the agent is stuck or confused.
• Error rate: Tool call failures, API errors, max-iteration hits.

Chapter 10: Build Your Own Agent Eval

You now have all the concepts. Let's put them together into a practical, step-by-step roadmap for building your own agent evaluation system. Seven steps, from scratch to production-ready.

Step 1: Define Success Criteria

Before writing a single test, answer: what does "good" look like? Be specific. Not "the agent should help the user" but "the agent should resolve the user's issue, update the database correctly, and respond in a professional tone within 30 seconds." Your success criteria become the inputs to your graders.

Step 2: Collect a Small Task Set

Start with 10-20 tasks that cover your most common scenarios. Don't try to be comprehensive yet. Include a mix of easy tasks (single tool call), medium tasks (3-5 tool calls), and hard tasks (ambiguous instructions, policy edge cases). Real user conversations from logs are the best source.

Step 3: Create Useful Tasks

Good tasks are specific (unambiguous instructions), solvable (the agent has the tools and information to succeed), and representative (they reflect real usage patterns). Include the full context: user message, initial database state, available tools, and applicable policies.

Step 4: Provide Ground Truth

For each task, define the expected outcome. This might be: expected database state after completion, required tool calls, reference output text, or a combination. The more specific your ground truth, the easier grading becomes.

Step 5: Configure Graders

Set up a layered grading system. Start with code-based graders for objective checks (database state, tool calls). Add a model-based grader for subjective quality (tone, helpfulness, accuracy of explanations). Schedule periodic human review to calibrate the model-based grader.

python
# Example: composite grader

def composite_grade(transcript, outcome, ground_truth):
    scores = {}

    # Code graders (fast, deterministic)
    scores["db_correct"] = check_database_state(
        outcome.db, ground_truth.expected_db
    )
    scores["tools_correct"] = check_required_tools(
        transcript, ground_truth.required_tools
    )
    scores["no_forbidden"] = check_no_forbidden_tool(
        transcript, ground_truth.forbidden_tools
    )

    # Model grader (flexible, non-deterministic)
    scores["quality"] = llm_judge(
        transcript,
        rubric="Rate 1-5: professional tone, accurate info, concise"
    )

    # Overall: code graders must pass, quality ≥ 3
    passed = (all([
        scores["db_correct"],
        scores["tools_correct"],
        scores["no_forbidden"]
    ]) and scores["quality"] >= 3)

    return {"passed": passed, "scores": scores}

Step 6: Build the Evaluation Harness

The evaluation harness is the software that runs tasks, collects transcripts, applies graders, and stores results. It needs to handle:

• Running N trials per task (for pass@K / pass^K computation)
• Isolating each trial (fresh database state, clean context)
• Saving full transcripts for debugging
• Parallelization (running multiple trials concurrently)
• Result storage (database or structured files)

python
# Skeleton evaluation harness

import asyncio, json, copy
from datetime import datetime

class EvalHarness:
    def __init__(self, agent, graders, trials_per_task=10):
        self.agent = agent
        self.graders = graders
        self.trials_per_task = trials_per_task
        self.results = []

    async def run_task(self, task):
        """Run N trials for one task."""
        trial_results = []
        for i in range(self.trials_per_task):
            # Fresh environment per trial
            env = copy.deepcopy(task.initial_env)

            # Run agent
            transcript = await self.agent.run(
                user_message=task.user_message,
                tools=task.tools,
                env=env
            )

            # Grade
            scores = {}
            for name, grader in self.graders.items():
                scores[name] = grader.grade(transcript, env, task.ground_truth)

            trial_results.append({
                "trial": i,
                "transcript": transcript,
                "scores": scores,
                "passed": all(s["passed"] for s in scores.values())
            })

        # Compute metrics
        c = sum(1 for t in trial_results if t["passed"])
        n = len(trial_results)
        return {
            "task_id": task.id,
            "pass_at_1": c / n,
            "pass_at_3": compute_pass_at_k(n, c, 3),
            "pass_up_3": compute_pass_up_k(n, c, 3),
            "trials": trial_results
        }

This is a minimal but functional harness. Production systems add logging, progress bars, error handling, cost tracking, and result dashboards — but the core loop is always the same: run, grade, store, compute metrics.

Reporting Results

When presenting evaluation results, include enough detail for informed decision-making. A good eval report looks like this:

text
========================================
Agent Evaluation Report — v2.4.1
Date: 2026-05-18 | Model: claude-opus-4
Tasks: 50 | Trials/task: 10 | Total runs: 500
========================================

METRICS (all tasks):
  pass@1:  0.820  (410/500 trials passed)
  pass@3:  0.978
  pass^3:  0.574
  pass^5:  0.389

METRICS (by difficulty):
  Easy   (20 tasks):  pass@1=0.940  pass^3=0.821
  Medium (20 tasks):  pass@1=0.815  pass^3=0.520
  Hard   (10 tasks):  pass@1=0.640  pass^3=0.267

METRICS (by category):
  Order lookup:     pass@1=0.960
  Cancellation:     pass@1=0.880
  Returns:          pass@1=0.780
  Policy edge cases: pass@1=0.600

TOP FAILURE MODES:
  1. Policy hallucination (12 failures)
  2. Wrong order ID in response (8 failures)
  3. Premature action without confirmation (6 failures)

VS BASELINE (v2.3.0):
  pass@1: 0.820 vs 0.790 (+3.0%) ✓
  pass^3: 0.574 vs 0.510 (+6.4%) ✓
  No regressions in any category. SAFE TO SHIP.
========================================

This report gives you everything needed to decide whether to ship: overall metrics, difficulty breakdown, category breakdown, top failure modes, and comparison to baseline. The category breakdown is especially important — a 82% aggregate can hide a 60% pass rate on policy edge cases, which might be unacceptable for your use case.

Step 7: Inspect, Iterate, Maintain

Run your evaluation. Read the failing transcripts. Understand why the agent failed. Fix the agent (better prompts, better tools, better instructions). Run again. This cycle never ends. The best agent evaluations evolve continuously through new failure cases and ongoing maintenance.

A Concrete Task Set Example

Here's what a small but well-designed task set looks like for a customer service agent:

Notice the distribution: 2 easy, 4 medium, 3 hard, 1 edge case. This gives you a nuanced view of agent capability. An agent that passes 7/10 might still be failing all the hard tasks — the average masks the weakness.

Common Failure Patterns

After running hundreds of evaluation trials, certain failure patterns appear again and again. Knowing these helps you write better tasks and graders:

For each pattern, add specific test cases to your eval suite. The infinite loop pattern, for example, is caught by a simple grader that checks whether the same tool was called 3+ times with identical arguments.

Chapter 11: Connections & Cheat Sheet

Benchmark Landscape

Here's the lay of the land — every major agent benchmark, what it tests, and how it works:

Choosing the Right Benchmark

No single benchmark is comprehensive. Choose based on your agent's domain:

Benchmark Limitations

No benchmark is perfect. Be aware of these common limitations:

• Contamination: If the benchmark tasks leaked into the model's training data, scores are artificially inflated. Always check for contamination when using public benchmarks.
• Distribution mismatch: Benchmark tasks may not reflect your actual usage patterns. An agent that scores 90% on τ-bench might score 60% on your real customer service tasks.
• Ceiling effects: As agents improve, benchmarks saturate. When every agent scores 95%+ on a benchmark, it stops being useful for comparison.
• Gaming: Systems can be specifically tuned to perform well on a known benchmark without generalizing. This is why custom evals matter.
• Static nature: Most benchmarks are fixed task sets. Real users bring novel, ever-changing requests that no fixed benchmark anticipates.

Evaluation Cheat Sheet

The 7-Step Roadmap

Decision Checklist: When to Ship

Use this checklist before deploying an agent to production:

"Must pass" items are blockers — if any fail, do not ship. "Should pass" items are strong recommendations — shipping without them adds risk that should be acknowledged.

Formula Reference

Every formula from this lesson in one place:

Glossary of Key Terms

The Evolution of Agent Evaluation

The field is evolving rapidly. A few trends to watch:

• Dynamic benchmarks: Benchmarks that regenerate tasks to prevent memorization/contamination.
• Agentic evaluation: Using agents to evaluate other agents (recursive, but effective when calibrated).
• Reward-model graders: Training specialized reward models as graders, combining the speed of code with the flexibility of LLMs.
• Real-time monitoring: Evaluating agents in production with live grading, not just offline benchmarks.
• Multi-turn evaluation: Benchmarks specifically designed for long conversations (50+ turns) where context management becomes critical.

Related Lessons

Continue your learning path:

• Reward & Alignment — RLHF, DPO, and making agents do what you want
• microGPT — Understand the underlying model that powers agents
• Transformer — The attention mechanism behind every agent's brain
• CS224N Lec 10: Agents, Tools & RAG — ReAct loops and tool calling in depth
• CS224N Lec 11: Benchmarking & Evaluation — LLM evaluation fundamentals

References

1. Anthropic. "Demystifying evals for AI agents." 2025.
2. Anthropic. "Effective context engineering for agents." 2025.
3. OpenAI. "A practical guide to building agents." 2025.
4. Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023. arXiv
5. Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023. arXiv
6. Yao et al. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." 2024. arXiv
7. Fourney et al. "τ²-bench: Benchmarking Multi-Agent Systems." 2025.
8. Shen et al. "Terminal-Bench: Evaluating Terminal Automation Agents." 2025. arXiv
9. Wolfe, Cameron R. "Agent Evaluation: A Detailed Guide." Deep Learning Focus, May 2026.