Tool Use & Function Calling

Last time, the lecture built retrieval-augmented generation (RAG: finding documents relevant to a question, then handing them to a model before it answers). It looks like a cure for everything a model does not know. Hand it more documents, and it should know more. Today’s lecture opens by taking that promise apart.

The slide draws three columns of ways RAG breaks, one for each stage of the pipeline. On the retrieval side: a recall ceiling (the retriever simply misses evidence the agent needs), a stale index (the stored documents were never updated), and a permission leak (the system hands back content the user was not allowed to see). On the reader side, once documents are found: lost in the middle (where in the list a document sits changes whether the model uses it), distraction (similar-looking but irrelevant text pulls attention away), and evidence conflict (two retrieved sources disagree). On the loop side, across an agent’s many steps: compounding error (an early mistake shapes every decision after it), no stopping rule, and prompt injection (retrieved text itself gives the agent new, unwanted instructions).

Lost in the middle, made concrete

The second column has a name and a paper: lost in the middle (Liu et al., 2023). Give a model twenty documents and ask a question whose one true answer sits in exactly one of them. Vary only where in the list that document sits, first, last, or somewhere in between, and watch accuracy change even though every document the model sees is identical.

Models read the start and the end of a long context more reliably than the middle. An agent that stuffs in every document it can find, hoping more is safer, can bury the one document that mattered at exactly the position the model reads worst.

The citations can lie too

RAG’s one unique benefit over a plain chatbot is that it can cite its source: here is the sentence, here is the document it came from. But the slide points out the citation itself is written by the same language model that might hallucinate the answer. So a 2023 study (Liu et al.) rated outputs on perceived usefulness, a 1 to 5 scale, and separately measured whether the cited sources actually said what the model claimed. Usefulness scored high, an average around 4.5 across four systems the slide names. Citation precision and recall, whether the citation is accurate and whether every claim that needed one got one, both averaged well under 75%. A 2025 ICLR study found a similar pattern: of accepted papers with confirmed citations, many still carried at least one hallucinated reference. An answer can look well-sourced and still not be.

Three kinds of job a language model cannot do alone

So retrieval alone will not save you. The slide sorts the actual gaps into three groups, and a tool is the fix for every one of them.

The 59% multiplication figure from the hero above is the first group made concrete, and the slide right after it (slide 16) sharpens the same point with a separate study: task accuracy fell as problem size grew, across digit-count multiplication, a logic puzzle with more houses and attributes, and a longer dynamic-programming sequence (Dziri et al., 2023). The model is not confused about what multiplication means. It is simply doing long arithmetic with no scratch space and no way to check its own work, one token at a time, and small slips compound. A calculator never has that problem. That is the whole argument for this lecture: stop asking the model to be the thing it is bad at, and let it call something that already is good at it.

Picture a model answering a question with a calendar on its desk, out of reach. It cannot pick the calendar up. It can only describe, in words, what it would do with it: “I would look up today’s date.” A tool is what puts the calendar in its hand.

The lecture quotes a precise definition from a 2024 survey (Wang, Cheng, Zhu, Fried and Neubig): a tool is “a function interface to a computer program that runs externally to the LM, where the LM generates the function calls and input arguments in order to use the tool.” Unpack that sentence slowly. External: the tool is not part of the model, it is a separate program, a calculator, a web search, a database. Function interface: the model does not operate the program directly; it writes out a function call, a name plus arguments, the same shape as calling a function in any programming language. The model generates the call: deciding to use a tool, and with what arguments, is itself something the model predicts, one token at a time, exactly like any other text it writes.

One shape, many jobs

The slide sorts real tools into four families by what they are for. Knowledge access tools answer or retrieve or search: search_engine(query) -> document. Computation tools run code or do arithmetic: calculator(formula) -> value. Tools for interaction with the world read or write real state: get_weather(city) -> weather, email.verify(address) -> bool. And tools for non-textual modalities handle images or audio the model cannot read directly: image.caption(image) -> str. Every one of these, however different the job, has the identical outer shape: a name, typed inputs, and a typed output. That shared shape is what makes “tool use” one topic instead of a hundred unrelated ones.

Watch the loop, once, slowly

Here is the full round trip for a tool call, the loop every later chapter assumes you already understand.

Four steps, in a fixed order, every single time. The model writes a call in a format it was taught to use, calculate(17423 × 892). Something outside the model, the harness (the program wrapping the model, reading its output, running real code) parses that call and actually runs the calculator. The calculator returns a plain value, 15,541,316, with no opinion about what it means. And the model, seeing that value appended to its context, writes the final answer in words. Notice what the calculator never does: it never writes a sentence. It returns a value back to the model, and language is still the model’s job alone.

Why this split matters

Splitting the job this way, let the model decide and describe, let the program execute, buys you two things at once. First, correctness: a calculator that runs real multiplication hardware is never going to misplace a digit the way a language model guessing token by token can. Second, auditability: the call is a line of text you can log, replay and check before you let it run, which Chapter 9’s failure modes and the later lectures on agent safety both depend on. None of that works if the model is allowed to skip writing the call and just assert an answer; the entire benefit of a tool comes from forcing that one extra, checkable step.

It is worth saying plainly what the harness is, since every later chapter leans on it. The harness is ordinary code you write and control: it reads the model’s raw text output, recognises the shape of a tool call inside it, runs the matching real function, and appends whatever comes back to the context before asking the model to continue. The model never touches a network socket, a file system, or a database directly; it only ever writes words, and the harness is the one piece of the system actually allowed to act on the world. Get the harness wrong, skip validating a call before running it, trust a result without checking its shape, and every protection the rest of this lesson builds, Toolformer’s loss filter, Gorilla’s hallucination check, MCP’s required human consent, has nothing underneath it to enforce.

Chapter 1 showed the loop. It did not say how a model learns to use it, when to call a tool, what to call, and with which arguments, without a human writing thousands of worked examples by hand. Toolformer (Schick et al., 2023) answers that with a trick: let the model write its own training examples, then keep only the ones that actually helped.

The lecture frames it as self-supervised learning (the model grades its own attempts against plain, unlabelled text, instead of a person labelling each one by hand) from a handful of demonstrations per API: a calculator, a question-answering system, a search engine, a translator, a calendar. Five tools, almost no labelled data, one finetuned model, GPT-J (6B parameters). Three steps turn that into a model that knows when to reach for a tool.

Step 1: sample candidate calls

Take ordinary text, say “Pittsburgh is known as the Steel City.” At every position i where the model’s own probability of inserting an API call there, p_i, is above a threshold τ_s, sample several candidate calls the model might make at that spot: [QA(“What other name is Pittsburgh known by?”)]. Nothing has executed yet. The model is only guessing where a tool call would plausibly fit and what it might say.

Step 2: execute, and see what comes back

Now actually run each candidate call against the real API. The question-answering system returns “the Steel City”, and that text gets spliced into the sequence right after the call, exactly where it would sit if the model had looked the answer up mid-sentence.

Step 3: keep only the calls that helped

This is the step that makes the whole scheme self-correcting. For every candidate, compute a loss (a single number measuring how surprised the model is by the real next words, lower means it predicted them more confidently), a weighted cross-entropy loss, L_i, over the tokens that come after it, three ways: with no API call at all, with the call written but not executed (no result spliced in), and with the call and its real result both present. Call the first two losses L⁻ (the better of “no call” and “call, no output”) and the third L⁺ (call plus real output). The usefulness of that candidate call is simply L⁻ minus L⁺: how much easier the rest of the text became to predict once the real answer was in view. Keep the call only if that gap clears a threshold, τ_f.

L_a

The loss on the text that follows, with no API call written at all.

L_b

The loss with the call written but not executed, so no result is spliced in; a call the model never gets to see answered.

L_c

The loss with the call executed and its real result spliced into the text.

The slide’s own worked numbers: for one position, L_a (no call) is 2.5, L_b (call, no execution) is 2.1, and L_c (call, executed) is 0.8. The usefulness is min(2.5, 2.1) − 0.8 = 2.1 − 0.8 = 1.3: seeing the real answer dropped the loss by 1.3, this lesson’s plain unit for how much easier the rest of the sentence became to predict, than either not calling at all or calling without an answer. Set your own threshold against that same example below.

Every call that survives this filter becomes a real training example: ordinary text with one extra API call woven in, exactly where it already helped. Finetune GPT-J on a dataset built entirely this way, and the model learns, from its own graded experience, which positions deserve a tool call and which tool to reach for.

Did it work?

The slide reports Toolformer (6B) against two baselines across five task families: fact completion, question answering, math, temporal reasoning, and multilingual question answering. On most of these, the finetuned 6B model closes most of the gap to GPT-3 at 175B parameters, nearly 30 times larger, and clearly beats the un-finetuned GPT-J it started from. The one place scale still wins outright is multilingual question answering, where GPT-3’s broader training data gives it roughly 11 points of accuracy against Toolformer’s roughly 4: a tool cannot hand the model knowledge of languages it barely saw in training. A follow-up chart confirms the loss filter is doing real work, not just adding noise: GPT-2 models trained with Toolformer’s calls climb steadily above the same models with tool use disabled, as parameter count grows.

Toolformer writes a call as ordinary text: a function name, a bracket, an argument, more text. Every one of those pieces costs the model several predictions, several chances to get the syntax slightly wrong. ToolkenGPT (Hao et al., 2024) asks a different question: what if calling a tool were not a sentence to write, but a single word to say?

A tool as a word

The lecture’s slide calls this “viewing tools as words.” Give every tool its own toolken (a tool token): one new entry added to the model’s vocabulary, right alongside “the” and “area”, with its own learned embedding (the vector of numbers the model uses to represent it). The model keeps predicting one token at a time exactly as it always has. The only change is that one of the possible next tokens, at every position, is now [square] or [add] or whatever toolken names a tool.

P(t_i | t_<i)

The probability of the next token, a word or a toolken, given every token so far. Exactly the same quantity every language model predicts (Lecture 2, Chapter 1); nothing about the model’s own machinery changes.

[W_v; W_τ]

The ordinary vocabulary’s output weights, W_v, with the new toolken weights, W_τ, stacked right alongside them. Only W_τ is trained; W_v and the whole rest of the frozen model are untouched.

h_i−1

The model’s hidden state after reading everything up to position i − 1, the same summary of context it always carries forward.

Watch one tool call collapse into a token

The lecture’s own training example is a word problem ending “the area is 256 square feet.” Line the original text up against the training target the model actually learns from.

In the original text, positions 4, 5 and 6 spell out “2”, “5”, “6”, the already-computed answer. The training target, s′, replaces that whole run with one toolken, [square], and marks the three result tokens as [N/A]: they are skipped in the loss entirely, because the model is never asked to predict an already-known numeric result token by token. The loss function makes this exact: it sums −log P(t′_i | t_<i) only at positions where t′_i is not [N/A], so training spends its signal entirely on learning when to reach for a tool, never on re-deriving a number the tool will supply.

Why this trade is worth making

Once the model predicts [square], generation does not simply continue; it switches into a tool mode, moving the context to a new prompt built for that one tool, with a few demonstrations of its calling syntax. Filling in the actual arguments, the side length, is easy work for in-context learning (Lecture 2, Chapter 10): the model has already decided that a tool is needed, and only has to copy the right numbers into the right slots of a format it just saw demonstrated. The tool executes, its return value re-enters the sequence, and the model resumes normal reasoning mode as if nothing had interrupted it.

The lecture’s comparison table makes the payoff plain. Toolformer must finetune its whole 6B model for every new API; ToolkenGPT trains only the new toolken embeddings, leaving the frozen base model untouched, and reaches 234 tools in the lecture’s report, roughly 50 times Toolformer’s five. A results chart confirms accuracy holds up as the number of distinct tools in play grows, which is exactly the property you want from something meant to scale: add tool number 235 and you train one new embedding, not a whole model.

Toolformer learns five tools. ToolkenGPT reaches 234. Neither approach says what happens at 1,645, or 16,000. Gorilla (Patil et al., 2023) and ToolLLM (Qin et al., 2023) are both attempts to answer that, and both run into the same new problem once the tool count gets large: the model starts making tools up.

Gorilla: an API retriever, and a dataset of 1,645 real calls

Gorilla finetunes LLaMA into Gorilla-7B on 1,645 real API calls, collected from three machine-learning hubs: Hugging Face’s Model Hub, PyTorch Hub, and TensorFlow Hub. Each call is stored as a JSON record with a fixed set of fields: domain, framework, functionality, the API name, the exact call syntax, its arguments, environment requirements, example code, a performance number, and a description. GPT-4 then generates synthetic natural-language instructions for each one, so the model learns to go from a plain request, “I want to classify an image,” to the one real API call that does it.

Gorilla adds one more piece: retrieval-aware training. During finetuning, the model sometimes sees the correct API documentation alongside the instruction, and sometimes sees none, or the wrong one, on purpose. That makes the model robust to exactly the failure mode Chapter 0 named for RAG: a retriever that returns something irrelevant should not drag the model into hallucinating a call that fits the bad document instead of answering from what it already knows.

The hallucination problem, made visible

The lecture’s own example shows why this matters. Ask three systems the same question: “Help me find an API to convert spoken language to text using Torch Hub.”

Both large general models produce code that looks entirely plausible, correct Python syntax, a sensible-sounding repository name, and both are wrong in different ways: one invents a repository that does not exist, the other reaches for the wrong library altogether. Gorilla, trained specifically on real API records with their exact call syntax, gets it right. Plausible is not the same as real, and the next question is how you even measure that difference automatically.

AST subtree matching: checking a call’s shape, not its words

You cannot grade a generated API call with simple text matching: two calls can use different variable names and still be identical in meaning, or look nearly identical and call a completely different, hallucinated API. Gorilla’s fix is AST subtree matching. An AST (abstract syntax tree) is the call’s structure, not its literal text: torch.hub.load('pytorch/vision', 'densenet121', pretrained=True) becomes a small tree, torch under it hub, under that load, with two argument branches, the repository string and the model name, and a keyword branch, pretrained=True. Checking whether a generated call is real becomes checking whether its AST appears as a subtree inside a reference tree built from the whole real API documentation. A hallucinated repository name or a flipped pretrained=False breaks the subtree match immediately, even though the surrounding code still looks perfectly normal.

With this check in place as a second axis alongside ordinary accuracy, Gorilla’s own results chart shows the expected shape: giving the model a retriever (so it reads real documentation before answering) raises accuracy and lowers the hallucination rate at the same time, across all three API hubs the lecture reports, Torch Hub, Hugging Face, and TensorFlow Hub. Accuracy and hallucination are not independent measurements by luck; a call that is not hallucinated is, by definition, one whose structure matches something real, which is most of what makes it likely to be correct.

ToolLLM: the same idea, at API-marketplace scale

ToolLLM pushes the same approach further: 16,000-plus real APIs, crawled from RapidAPI, an online marketplace of API services, with the same instruction-generation-and-finetune recipe applied to LLaMA, producing ToolLLaMA. Its own API retriever recommends likely tools out of that enormous pool for each instruction, because no single prompt could list 16,000 tool schemas and still fit in a context window. The summary table the lecture closes this section with makes the whole family’s shared structure explicit: Toolformer calls are text, learned by self-supervision, at a scale of 5 tools; ToolkenGPT calls are one embedding, learned from ground-truth or synthetic calls, at 234 tools; ToolLLM calls are a ReAct-style trace (reason, then act, Lecture 1), learned from ChatGPT-generated traces, at 16,000 APIs; Gorilla calls are text and API code, learned from GPT-4 self-instruct data, at 1,645 APIs. Four different training recipes, the same underlying bet: the more tools a model must juggle, the more its biggest risk shifts from choosing the wrong one to confidently inventing one that was never real.

Look back at the last four chapters and notice the one thing Toolformer, ToolkenGPT, ToolLLM and Gorilla all share: every one of them changes the model to fit the tools. New API, new finetuning run, new toolken, new synthetic instructions. That worked up to a few thousand tools. It cannot be how the one million APIs on the real internet get connected to language models, because nobody can afford to retrain a model for each one, and a brand-new API appears faster than any training run could keep up.

The lecture’s turn here is simple to say and large in consequence: instead of changing the model to fit the tools, change how the tools are described so that any model can use them. Describe a tool once, in a format every model already understands from its training, a schema, and the model needs no retraining at all to use a tool it has never seen before. The slide calls this the shift from REST and function calling and MCP being “a schema at runtime instead of fine-tune per API.”

Three generations, one direction

The lecture lays three standard interfaces side by side, in the order they were adopted, and each one is facing a different audience. REST is human-facing: endpoints and documentation meant for a person to read, authentication handled per service, and the model must read the docs and hand-write a request exactly as a person would. Function calling is model-facing: a JSON schema per tool, the model emits a name and arguments in a fixed format, and a harness validates and dispatches the call. MCP is ecosystem-facing: one protocol, usable by any client, where servers expose their tools and the client discovers them at runtime instead of reading them out of a prompt written in advance. Each generation removes one more piece of hand-work from the step before it; Chapters 6 and 7 take function calling and MCP apart in the same order.

REST: tools over plain HTTP

REST (Representational State Transfer) is the interface the ordinary web already runs on, and it is the oldest way a model has been pointed at a tool: a plain HTTP request with four verbs for four meanings. GET reads, POST creates, PUT updates, DELETE removes.

Nothing here is specific to language models; this is the same request a browser or a curl command could send. That is exactly the limitation the lecture names: the model must already know the URL, the parameter names, and the shape of the response. None of that is written down anywhere a model reads automatically. A person has to read the provider’s documentation, by hand, and either bake that knowledge into a prompt or write wrapper code around it. Every new API means another round of a human reading docs.

It is worth sitting with why this shift is such a large deal for the rest of the lecture. Every method in Chapters 2 through 4 paid a real, recurring cost to add one tool: Toolformer ran a finetuning pass, ToolkenGPT trained a fresh embedding, Gorilla and ToolLLM both needed a curated dataset of real calls before a model could use them at all. REST, by contrast, costs nothing to add a tool; a provider stands up an endpoint and anyone, human or model, can read its documentation and start calling it. The cost REST leaves behind instead is borne every single time a model tries to use that tool: someone, a person or a prompt engineer, has to translate the documentation into something the model can act on, over and over, for every tool, for every model. The next two chapters are the two ways that translation step itself got standardised and finally stopped being repeated by hand.

REST assumes a person reads the documentation. Function calling removes the person: describe the tool as a JSON schema (a structured, machine-readable description of a function’s name, purpose, and typed arguments) that the model reads directly, in its own context, at the moment it needs it.

Look at the schema field by field, because every one of its parts does a job function calling needs. name is the identifier the model writes back verbatim when it decides to call this tool, so it must be unique and stable; rename it and every prompt that ever referenced the old name breaks silently. description is prose, the same kind of plain sentence this lesson insists on everywhere else, and it is doing real work: it is the only thing telling the model when this tool is the right one to reach for, among however many others are listed alongside it. parameters is a JSON Schema, the same format a web API might use to validate a request body, reused here so that a harness can check the model’s arguments mechanically before anything executes: wrong type, missing required field, out-of-range value, all catchable without ever running the tool.

Notice what changed and what did not, next to Chapter 1’s loop. The model still writes a call and a harness still executes it and hands the result back; that four-step loop is untouched. What changed is where the tool’s shape comes from. With REST, that shape lived in a document somewhere, read once by a human, baked into a prompt by hand. With function calling, the shape is a schema, sitting in your codebase as data, handed to the model fresh every time, and the model’s output is now a structured object, {"tool_call": {"name": ..., "arguments": {...}}}, not a string of code someone has to parse out of free text. A harness can validate that object against the schema before it ever runs, catching a missing required field or a wrong type before anything executes.

The limitation function calling leaves standing

The lecture states it in one line: a tool written for one API has to be rewritten for the next. Every model provider has its own function-calling format, its own field names, its own conventions for how a multi-tool choice is expressed. Write a weather tool schema for one model’s function-calling API, and switching to a different model, or a different host application, often means rewriting that same schema in a slightly different shape. Function calling fixed “does the model need to read prose documentation”; it did not fix “does this tool definition work everywhere.”

That is exactly the gap the next chapter closes. Hold the shape, name plus typed arguments plus a human-readable description, in your mind; MCP keeps that shape and adds the one thing missing: a single protocol any client can speak, so a tool defined once no longer needs rewriting for the next model that comes along.

Notice, too, how much smaller this chapter is than Chapters 2 through 4. That is not an accident of how it is written; it reflects the real engineering effort each generation actually costs. Toolformer needed a whole finetuning run and a loss-filtering pipeline to teach one model five tools. Function calling needs a JSON file. The schema in the device above is a few lines a human can read in seconds, and it is the entire integration cost: no training, no synthetic data generation, no API retriever. That collapse in effort, from a research pipeline to a text file, is the real reason function calling and its successor spread to nearly every production model within a couple of years of being introduced.

The Model Context Protocol (MCP) is the lecture’s third generation of standard interface, and the one this chapter stays with, quoting its own specification (the Linux Foundation, version 2025-06-18) exactly where it matters: the message names, the fields, and the transports are not this lesson’s paraphrase, they are what any compliant MCP server and client must actually send.

Three roles, one protocol

MCP names three participants. A host is the LLM application that initiates connections, Claude Desktop, an IDE, a custom agent. A client is a connector inside that host, one per server it talks to. A server is a separate process that exposes capabilities: tools (functions the model can execute), resources (context and data for the model or user to read), and prompts (templated messages for users). One host can hold several clients at once, each wired to its own dedicated server, a local filesystem server, a database server, a remote issue tracker; the slide draws exactly this fan-out, one host, many clients, many servers.

Every message between them is JSON-RPC 2.0 (a lightweight remote-procedure-call format built on JSON), and the specification is explicit that the connection is stateful: client and server first agree on what they both support, then operate inside that agreement, which is the lifecycle this chapter walks next.

Discover, then call

A server that supports tools declares it in its capabilities ({"capabilities": {"tools": {"listChanged": true}}}), and a client discovers what is actually on offer with tools/list, then invokes one with tools/call. These are, word for word, the specification’s own messages.

A client sends {"jsonrpc": "2.0", "id": 1, "method": "tools/list"}; the server answers with a tools array, and the specification fixes the fields a tool entry carries: name (its unique identifier), an optional title (a human-readable display name), description, inputSchema (the JSON Schema for its arguments, the same shape Chapter 6 showed), an optional outputSchema, and optional annotations describing the tool’s behaviour, which a client must treat as untrusted unless the server itself is trusted. To invoke one, the client sends tools/call with params: {"name": ..., "arguments": {...}}; the server’s result carries a content array (typed blocks: text, image, audio, a resource_link, or an embedded resource) and a boolean isError. The specification is careful to separate two kinds of failure: a protocol error, an ordinary JSON-RPC error object, for an unknown tool or invalid arguments, and a tool execution error, a normal result with isError: true, for when the tool itself ran but failed, an API rate limit, bad input data. A client that only checks for the first kind will miss a tool that silently reports failure inside a successful-looking response.

The lifecycle: initialize, then everything else

Before any tools/list can happen, the specification requires a fixed lifecycle: initialization, then operation, then shutdown. The client must send initialize first, with a protocolVersion it supports, its own capabilities (such as roots, filesystem boundaries it can expose, or sampling, letting the server ask the client’s model to do some thinking of its own), and clientInfo (its name and version). The server must answer with its own protocolVersion, capabilities, and serverInfo. Only after the client sends a final notifications/initialized message may normal operation, tools/list, tools/call, and the rest, begin. The client should not send ordinary requests before the server answers initialize; the server should not send ordinary requests before it receives notifications/initialized. Shutdown has no special message at all: for the stdio transport the client closes the server’s input stream (escalating to SIGTERM, then SIGKILL, if it will not exit); for HTTP, closing the connection is the whole story.

Two transports, the same messages riding on top

MCP is explicit that the JSON-RPC messages themselves never change; only how they travel does. The specification names exactly two standard transports. stdio: the client launches the server as a subprocess and the two exchange newline-delimited JSON-RPC messages over standard input and standard output; the server may log to stderr, and must write nothing else to stdout. Streamable HTTP: the server exposes one HTTP endpoint supporting both POST (the client sends a message, and the server may answer with one JSON object or open a text/event-stream, an SSE stream, to send several messages before the final response) and GET (the client may open its own SSE stream to let the server push messages without waiting on a POST). A server using Streamable HTTP may hand back an Mcp-Session-Id header at initialization, which the client must then include on every later request on that session, and the client must send an MCP-Protocol-Version header on every HTTP request after initialization so the server knows exactly which version it is speaking.

One server, every client: the course example

The lecture makes the payoff concrete with a small course learning-management system. Three REST routes, GET /assignments, GET /submissions, POST /grade, become three MCP tools, list_assignments, get_submissions, post_grade, with a library (FastMCP) generating the whole server directly from the API’s existing OpenAPI specification.

python
from fastmcp import FastMCP
import httpx

api = httpx.AsyncClient(
    base_url="https://lms.stanford.edu/api/v1")
spec = (await api.get("/openapi.json")).json()

mcp = FastMCP.from_openapi(
    openapi_spec=spec, client=api, name="Course MCP",
)
# -> list_assignments, get_submissions, post_grade

No tool was hand-written; the REST routes already existed, and the schemas MCP needs came straight out of documentation the API already had to publish. The lecture’s next slide adds a detail every real deployment needs: inbound versus outbound authentication are two separate concerns. Outbound is the server authenticating itself to the underlying REST API it wraps, an LMS_TOKEN attached as a bearer header on every call the server makes onward. Inbound is the server deciding which callers may use it at all, an AGENT_TOKEN checked by a token verifier before any tool runs. Mixing these up, trusting the agent’s own token to also authenticate to the LMS, or skipping inbound checks because the outbound call already succeeded, is a common way an MCP server ends up granting more access than intended.

Every system this lesson has met, Toolformer, ToolkenGPT, Gorilla, an MCP server, eventually faces the same question: is it actually good at calling tools? The lecture covers three benchmarks, each testing a different unit of “good”: one call, a chain of calls, and a whole conversation with a real user in the loop.

BFCL: does one call match a real function, exactly?

The Berkeley Function-Calling Leaderboard (BFCL; Patil et al., 2025) evaluates a model’s ability to invoke functions three ways at once: AST-based offline checking (does the generated call’s structure match the real function, the same subtree-matching idea Chapter 4 introduced), execution (actually run the function for real-world evaluation), and state-based evaluation across multi-step and multi-turn calls. Its test data covers several taxonomies of function calling and several languages, Python, C++, Java, SQL, and it deliberately tests relevance detection: does the model correctly refuse to call anything when none of the offered functions actually fit the question?

The lecture’s own three examples show exactly what BFCL is grading. Asked “What is 2 + 3?” with only add(a, b) on offer, a correct agent calls add(a=2, b=3), one multiple-function choice correctly made. Asked “What is (2+3) and (4+5)?” with add and mult both available, a correct agent issues add(a=2, b=3) and add(a=4, b=5) as parallel functions, two independent calls from one turn, not one call reused twice. And asked “What is 2*3?” with only add(a, b) on offer, the correct response is not a creative workaround, it is an error: “request is multiplication; only add is available”, exactly the relevance detection case, refusing cleanly when nothing offered actually fits.

ToolBench: 16,000 real APIs, a whole collection path

ToolBench (the benchmark behind ToolLLM, Chapter 4) tests a longer unit: crawling 6,000-plus real APIs from RapidAPI and curating instructions that chain them. Its own diagram shows single-tool instructions, intra-category multi-tool instructions (several APIs from one category, say Movies), and inter-category multi-tool instructions (APIs crossing categories entirely), each requiring the model to retrieve the right subset of relevant APIs out of thousands before it ever gets to call one correctly.

τ-bench: a tool, an agent, and a real user, together

Neither BFCL nor ToolBench puts a live, rule-following user in the loop. τ-bench (Yao et al., 2024) does: an agent, a simulated user, and a real backing database interact over a full conversation, in two domains the lecture reports with exact sizes.

In τ-bench’s own trajectory example, the user instruction carries a real-world constraint, “Current time is 2024-05-15 00:00:00 EST. Basic economy cannot be changed. Basic economy cannot be cancelled after 24 hours of booking,” and the user separately states a private preference the agent has to draw out by asking, not by assuming. Getting one tool call right is necessary but nowhere near sufficient here: the agent must also respect the domain’s written policy, read what the user actually wants rather than what seems likely, and take several correctly sequenced tool calls (read before writing, confirm before cancelling) to finish the task at all.

Pass^k: why one success does not mean the agent is reliable

A single successful run of a τ-bench task can still hide an agent that only half the time does the same task correctly. The benchmark’s reliability metric, passˆk, answers a sharper question: if you sample k independent trials from the same task, how often do all of them succeed?

passˆk

The probability that all k sampled trials of a task succeed, not just one of them; the harder, stricter sibling of ordinary pass@k.

E_task

Averaged across the benchmark’s tasks.

C(c, k) / C(n, k)

Out of n total sampled trials for a task, c of which succeeded, the chance that a random draw of k of them are all successes; C(·,·) is the binomial coefficient, the count of ways to choose k items from a set.

pass@k, the more familiar twin, asks the opposite question: the chance that at least one of k succeeds, 1 − E_task[C(n−c, k) / C(n, k)]. The two bound the same underlying reliability from either side.

With 4 of 8 trials succeeding, drawing just k=1 gives 50%, the ordinary single-run success rate. Raise k to 4, demanding all four drawn trials succeed together, and the number falls hard, because every additional trial is one more chance for the agent’s non-determinism (Lecture 2, Chapter 2: the model samples, it does not compute a fixed answer) to land on a bad run. A model that looks reliable at pass@1 can look fragile at passˆk, and τ-bench treats that fragility, not the single lucky success, as the real measurement of whether an agent is trustworthy in production.

Chapter 8’s benchmarks grade whether an agent succeeded. The lecture’s last content slide names how it fails, sorted into the same three units Chapter 8 measured: a single call, a multi-step chain, a whole conversation. The grouping matters because the fix for each unit is different. A single-call failure is usually a schema problem: a clearer description, a stricter type, a better-worded relevance check (Chapter 8’s own BFCL example). A multi-step failure is a planning problem: the agent needs to track what it has already done and what still has to happen before the next call is safe. A conversation failure is a listening problem: the agent has to notice what the user actually said, and what the domain’s written policy actually forbids, rather than filling in the gaps with whatever seems likely.

Put this another way: every method earlier in this lesson removed one specific failure from this very table. Toolformer’s loss filter (Chapter 2) exists to stop a model calling a tool when none was needed, selection’s other half. Gorilla’s retrieval-aware training and AST subtree matching (Chapter 4) exist to catch the hallucinated-tool row before it ever reaches a user. The Model Context Protocol’s required human-in-the-loop consent (Chapter 7) exists because policy and interaction failures are exactly the ones a protocol alone cannot prevent; they need a person watching, not just a better schema.

Sort four short traces into the row they belong to, and check your own eye against the category each one was written to exercise.

Two of these nine deserve one more sentence each, because they are the ones a single benchmark run cannot catch by itself. Non-determinism only shows up across repeated trials of the identical task, which is precisely why τ-bench built passˆk (Chapter 8) instead of grading a single transcript: an agent can pass once by luck and fail the very next attempt with nothing in the world having changed. And recovery failures only show up once an earlier call has already failed; an agent graded only on tasks where every call succeeds on the first try will never be tested on whether it knows how to back off, rather than hammer the same broken call a second and a third time.

A model cannot compute, cannot reach, and never saw everything an agent needs. A tool is the fix: a function the model calls, that runs externally, with a result that returns in plain data. Early systems, Toolformer, ToolkenGPT, ToolLLM, Gorilla, taught the model to know its tools, one training run at a time. Standard interfaces, REST, function calling, and the Model Context Protocol, flip that: describe the tool once, in a schema any model can read at runtime, and the model needs no retraining to use one it has never seen. BFCL, ToolBench and τ-bench grade the result at three different units, one call, a chain, a whole conversation, and passˆk insists an agent prove it can do it again, not just once.

The methods, side by side

The checklist

• One shape for every tool: a name, typed arguments, a typed or free result, nothing else (Chapter 1).
• Teach a model its tools by grading what actually helped, a loss filter (Chapter 2) or a usage check, not by assertion.
• Past a few hundred tools, budget for hallucination, not just accuracy: check a call’s structure against real documentation (Chapter 4).
• Prefer describing a tool over training a model to know it: a schema at runtime beats a finetune per API (Chapter 5, Chapter 6).
• Reach for MCP when more than one client needs the same tool: one server, the lifecycle and transport the specification defines, every compliant client (Chapter 7).
• Grade reliability, not one success: passˆk over pass@1 before trusting an agent unattended (Chapter 8).
• Name the failure by its unit, a call, a chain, a conversation, before you try to fix it (Chapter 9).

The core, as code

The smallest real MCP tool server, using the same FastMCP library Chapter 7’s course example used, built by hand instead of generated from an OpenAPI spec:

python
from fastmcp import FastMCP

mcp = FastMCP("Calculator MCP")

@mcp.tool()
def calculate(expression: str) -> float:
    """Evaluate a basic arithmetic expression, e.g. 17423 * 892."""
    allowed = set("0123456789+-*/(). ")
    if not set(expression) <= allowed:
        raise ValueError("unsupported characters")
    return eval(expression, {"__builtins__": {}})

if __name__ == "__main__":
    mcp.run(transport="stdio")
    # or transport="http", port=8001

Decorating a plain Python function is enough: FastMCP reads its type hints and docstring to build the inputSchema and description that tools/list returns, so the schema Chapter 6 showed by hand here comes from the function signature itself.

Keep going

Three lessons already on this site go deeper into pieces of today’s lecture. MCP: Model Context Protocol builds the protocol this chapter covered from zero, with its own interactive host, client and server. Agents & Tool Use builds a tool-using agent’s loop from scratch in code, the same four-step loop Chapter 1 walked. Tools, Exec-Approvals & Sandboxing takes the human-in-the-loop consent the MCP specification requires (Chapter 7) and builds the hard fence around it: what happens when a tool call is dangerous, not just wrong.

Homework 1 (cs329z/assignment1-harness) was released the day of this lecture: building a tool-using system from scratch, the same loop this lesson walked chapter by chapter. Go back and run the hero’s calculator question once more, now that you know exactly what happens inside the tool bubble; then try Teach or Sparks to test what stuck.