Sandbox Fleets

Picture a program that uses a computer the way you do. It looks at the screen, decides on one small move, a click or a few typed letters, and looks again. Today its job is plain: download the March report and save it in the Downloads folder. It tries, gets it wrong, and tries again. Every try needs a computer of its own, and that computer must be exactly as it was before the first try.

This lesson is about where those computers come from. First, though, we need to see what the program does with them, and why it cares so much about the state they are in.

A program that uses a computer

The program sees the computer the way you see it: as a picture of the whole screen, a screenshot. It answers with one small move on the computer, a click, a key, some typing: an action. Then it takes a fresh screenshot to see what its action did, and decides again.

Think of a person working a computer while describing every move aloud. "I see a browser. I click the address bar. I type the address." A program that looks at a screenshot, decides on one small action, a click or a few typed letters, and looks again until the task is done is a computer-use agent. From here on we just say the agent.

The agent never reaches inside the computer. It does not open files by name or call the computer's own functions. It sees pictures and sends clicks and keys, exactly as you would at a keyboard. We call the computer it works on its desktop: the screen, the programs that are open, and the files on its disk.

One job written in words, such as "save the March report in Downloads", is a task. It is the agent's homework question. The agent gets the words and the first screenshot, and nothing else.

How an agent learns

How does an agent get better at its homework? Something has to mark it. After the agent stops, a small program looks at the desktop and says pass or fail: the checker. It is the examiner with the marking scheme. For our task the checker asks one question: is there a file called report.pdf in the Downloads folder?

The checker's verdict as a number, 1 for pass and 0 for fail, is the reward: the mark on the paper. One attempt at one task, from a clean start to the checker's verdict, is an episode. Training code often calls it a rollout. It is one attempt at a level in a video game: you start fresh, you play, you get a score.

Now the training itself. Think of a puppy learning a trick from treats. It tries things, and when a treat comes, it does more of whatever it just did. Training by trial and error works the same way: the agent tries, the checker scores, and the training nudges the agent toward whatever scored well. That is reinforcement learning.

Read that last sentence again, because the whole lesson hangs on it. The agent learns whatever the reward rewards, nothing more. If the reward is honest, the agent learns the task. If the reward is wrong, the agent learns the wrong thing, and it learns it just as eagerly.

Why so many desktops

One attempt teaches very little. An agent improves by trying the same task many times, and a training run has many tasks. So a run needs thousands of episodes, and every episode needs a desktop to work on.

Count it slowly, with timings made up to show the idea. Say an attempt takes one minute of work, and getting the desktop ready again takes another minute. That is two minutes per attempt, so one desktop manages 30 attempts an hour. A thousand attempts on that one desktop, back to back, take 1,000 × 120 seconds: 120,000 seconds, about 33 hours.

Now put a thousand desktops side by side, each on its own attempt. The same thousand attempts take one round: two minutes. Nothing about the agent changed. Only the number of desktops did.

Real numbers have the same shape. One well-known test set of computer tasks has 369 of them. Its makers run 10 desktops side by side and report getting through one pass of all 369 in under an hour. Training needs many passes over many tasks, so it needs many more desktops than that.

worked exampleOne desktop, attempts back to back (illustrative: 60 s of work per attempt)
  wipe by rebuilding (illustrative 60 s):   60 + 60 = 120 s per attempt
                                            3,600 ÷ 120 = 30 attempts an hour
  1,000 attempts                          = 1,000 × 120 s = 120,000 s = 33.3 hours
  leave it as it was:   60 s per attempt  → 60 an hour; 1,000 in 16.7 hours, every one after the first dirty
  rewind (illustrative 1 s): 61 s         → 59 an hour; 1,000 in 16.9 hours, every one clean

A fleet of 1,000 desktops, one attempt each
  1,000 attempts in one round: 120 s if rebuilding, 61 s if rewinding

The wall at the top of the page: 48 desktops (Little's law, Chapter 9)
  leave:   48 × 60 ÷ 60  = 48 tasks a minute
  rebuild: 48 × 60 ÷ 120 = 24 tasks a minute
  rewind:  48 × 60 ÷ 61  = 47.2 tasks a minute

A real test set: 369 computer tasks; its makers run 10 desktops side by side
  to finish one pass in under an hour

Every attempt starts fresh

Here is the catch. Each attempt must start from the same clean desktop: the same files, the same open programs, the same settings. Making the desktop exactly like its clean start again before the next episode is a wipe, which code calls a reset. It is wiping the whiteboard before the next class.

There are a few ways to wipe. You can rebuild it: throw the desktop away and start a fresh one from scratch. That is slow, because a fresh desktop has to start up from nothing. You can rewind it: return it to a saved moment from when it was ready, like going back to a save point in a game. Or you can skip the wipe and leave the desktop as it was. That is the fastest of all. Let us see what it costs.

Here is the same task tried twice on one desktop. Pick how the desktop is wiped between the two attempts, then run attempt 2.

Look at what happened when you left the desktop as it was. Attempt 1 did the job and saved report.pdf. Attempt 2 clicked the wrong link and gave up after three actions. But report.pdf was still in Downloads from attempt 1, and the checker only asks whether the file is there. So it passed attempt 2: a reward of 1 for an attempt that did nothing.

Rebuilt or rewound, attempt 2 started on an empty Downloads folder. The same weak attempt left nothing there, and the checker failed it. A reward of 0 is the honest mark for an attempt that gave up.

Now picture the left-as-it-was version repeated thousands of times. Every lazy attempt that follows a good one gets full marks. The training does exactly what it should with those marks: it pushes the agent toward the lazy behaviour. You set out to teach the agent to download reports, and you taught it that doing nothing works.

Anything one episode leaves behind that the next one can see is a leftover; engineers call it a leak. It is the last guest's things still in the hotel room when you check in. A task that starts on leftovers starts dirty, and one that starts from the clean state starts clean. A leftover file is the plainest kind of leak. Later chapters meet subtler ones: a browser still signed in, a secret two desktops share, a clock that is hours wrong.

Sandboxes, and a fleet of them

Why must the desktop be a separate, throwaway thing at all? Because the agent may do anything. It can delete files, run a command it read on a web page, fill the disk, or sign in to things. Some of that is the task. Some of it is a mistake. A little of it may be a web page tricking the agent into it.

Think of a child's sandbox: dig freely, and the sand stays in the box. A closed space where a program may do anything at all, and nothing it does reaches the outside, is a sandbox. Each practice desktop is one.

A sandbox has to seal in two directions. It must seal each desktop off from its neighbours, the other desktops running at the same moment on the same machine. And it must seal one episode off from the next one on the same desktop. The leftover report broke the second seal. Chapter 1 shows how easily the first one breaks too.

One machine holds only so many desktops, so a run spreads them over many servers, the real computers in a data centre. All the sandboxes a training run uses at once, spread over many servers, are the fleet. Think of a hotel chain: many buildings, many rooms, one booking desk.

What a fleet must be

So a fleet has three demands. It must be sealed: nothing crosses between desktops or between episodes. It must be fast: a wipe takes a second, not a minute. And it must be light: little memory goes on each desktop beyond the desktop itself, so each server holds many.

The wall at the top of the page showed the first two demands pulling against each other. Rebuilding every desktop kept every task clean, but it halved the wall's work: 24 tasks a minute instead of 48, because at any moment half the desktops were rebuilding. Leaving them as they were kept all 48 busy, but every task after a desktop's first started dirty. Rewinding kept every task clean and still finished 47 tasks a minute.

Rewinding is not magic. It skips the slow part of a rebuild, the start-up from nothing, by keeping a copy of the desktop at the moment it was ready. Chapter 7 opens that copy up and shows why going back to it takes about a second.

The road ahead

The rest of the lesson builds the walls and the wipes from the bottom up. It helps to picture each server as a hotel and each desktop as a guest room. The rooms can share one front desk in one building. A guard can be posted inside every room. Each room can instead be a small cabin on the grounds, with a little front desk of its own. Or each can be a full house. Those are four kinds of wall, from the cheapest to the strongest, and the first half of the lesson builds each one.

Then come the wipes: rebuild, rewind, and a third way that throws away just the changes. Then the counting: how many desktops a fleet needs, and how to tell the agent's failures from the fleet's own. Here is the road, in order:

1. The front desk every program asks: what a kernel is, and why sharing one is the weak spot.
2. Rooms with rules: containers, built from the kernel's own walls and meters.
3. Layers: why containers start fast, and how throwing one layer away wipes them.
4. A guard in every room: a small kernel of each desktop's own, written as a program.
5. A kernel of its own: light virtual machines that boot in about a tenth of a second.
6. The desktop inside the box: a screen with no monitor, and clicks that land where they should.
7. Rebuild or rewind: snapshots, and why restoring one skips the boot.
8. Proving nothing leaks: what survives a wipe, and how to check every start.
9. How many desktops: one line of arithmetic that sizes a fleet.
10. When the fleet fails: keeping the fleet's faults out of the agent's score.
11. Field Guide: the numbers, the checklist and the core loop.

Two practice desktops run on one server. In desktop 17 the agent opens a terminal, a window where you type commands for the computer to run, and pastes a command it found on a web page. The command tries to read every file it can reach. What stops it from reading desktop 18's files, or the server's own?

To answer that, we need to know how a program reaches a file at all. The answer is the same for files, network messages and pixels on a screen, and it is the key to every wall in this lesson.

Nothing touches the hardware directly

Start with what a program can do on its own. A running program, which the kernel calls a process, can add numbers and move data around inside its own memory. That is nearly all. It cannot read the disk, send a message over the network or draw a single pixel by itself. For all of that it has to ask.

It asks one particular program. The one program that runs everything else and is the only one allowed to touch the hardware is the kernel. Every operating system has one; most servers run the Linux kernel. The kernel starts every other program, hands out memory, and owns the disk, the network and the screen.

A hotel helps here, and we will use it to the end of the lesson. The server is the hotel building. Each practice desktop is a guest room. Each program is a guest. The kernel is the front desk. Guests never walk into the back office to fetch things. They ask the front desk for everything, and the front desk decides.

A request is a system call

Asking the kernel for something, a file, a network message, a bit of memory, is a system call. Opening a file is one system call. Reading from it is another. Sending a message is a third. A program that seems to do one simple thing, such as saving report.pdf, makes dozens of them.

The desk checks every request. Who is asking? Is this guest allowed that file? That checking is the first wall between programs: the kernel's own rules about who may touch what. It is a good wall, and every operating system has it.

Two more names. The real computer underneath is the server, as in Chapter 0. The server's own kernel is the host kernel. When this lesson says "the server's kernel", it means the host kernel: the one front desk of the whole building.

Counting the doors

Here is the weak spot. Each kind of request runs a different piece of the kernel's code. Opening a file runs the file code. Sending a message runs the network code. So each kind of request is a door into a different part of the back office.

How many doors are there? People have counted. A trivial Linux program needs 15 kinds of system call. A typical Linux install was counted using 224 kinds. On top of those it used 52 kinds of ioctl (input and output control), a catch-all request for talking to a device. Think of asking the desk to adjust your room's heating: one general request with many possible meanings.

Every kind of request a program can make is a door into the kernel's code; all those doors together are the attack surface. The kernel is millions of lines of code written by people, and people make mistakes. A bug behind any one door is a way in. A program that finds such a bug may be able to make the kernel do things its rules forbid, such as hand over another room's files.

Why agents need walls

An ordinary office program is written by people you trust. An agent is different. It runs commands it found on web pages. It opens files from strangers. It installs things to finish a task. Most of the time that is fine. But a fleet runs thousands of episodes a day, and the web is full of pages that try to trick a program.

So we want each desktop sealed. Keeping everything a desktop does inside it, so it can neither see nor change its neighbours, is isolation: soundproof rooms. And when isolation fails somewhere, we want the damage to stay small. How many desktops one break can reach is the blast radius, like how far a kitchen fire spreads before it is stopped. The lesson on Failure Modes and Isolation uses the same word for whole services; here it counts desktops.

One front desk for every room

Now look at the opening scene again. If desktops 17 and 18 are plain programs on one server, they send their requests to the same host kernel. The kernel's rules may stop desktop 17 from opening desktop 18's files directly. But if the pasted command finds a bug behind any door, it is inside the one kernel that guards every room.

The engineers who built one widely used fleet put the trade plainly. Relying on a single shared kernel, they wrote, means "a fundamental tradeoff between security and code compatibility". The more kinds of request you allow, the more programs run unchanged. The more kinds you allow, the more doors an attacker can try.

Three ways to stop sharing the desk

There are three ways to put more between a room and the shared desk. Here they are in plain words first; the names come after each one.

First, keep the shared desk, but give each room its own view, its own meters and a short list of requests the desk will take from it, a filter. That is a container: a room in the same building. Chapters 2 and 3 build one.

Second, post a guard in every room who answers almost every request itself, so the guest almost never reaches the front desk. That is a guard kernel: a concierge stationed in each room. Chapter 4 opens one up.

Third, give each room its own small front desk on pretend hardware. A pretend computer, made by software with help from the processor, that runs a kernel of its own is a virtual machine, or VM. It is a cabin on the grounds with its own front desk. Chapter 5 builds the lightest kind.

Send one request from desktop 2 through each kind of wall, then find a bug in the kernel it reaches.

With the shared kernel, the request went through the filter and straight into the server's kernel, and a bug there sat under all four desktops. Behind a guard, the request stopped in desktop 2's own guard, and a bug in that guard exposed one desktop. To reach the others, an attacker would also have to break the server's kernel. With a kernel of its own, the request never left desktop 2's virtual machine.

Notice the thin dotted lines in the last two pictures. The guard is a program too, so it still makes a few requests of its own to the server's kernel. The virtual machine is run by a program on the server, and that program makes requests too. The walls do not remove the server's kernel. They shrink the list of doors that lead to it.

Every wall costs something

Read the ladder below from top to bottom. The further a desktop's requests stay from the host kernel, the stronger the wall. And the stronger the wall, the more start-up time or memory it can cost.

A container boots no kernel, because the host kernel is already running. A full virtual machine starts in seconds, because it starts a whole pretend PC and then a whole kernel inside it. A stripped-down virtual machine that pretends to be almost nothing is a microVM. It gets a kernel of its own too, yet it reaches its first program in under 125 milliseconds (thousandths of a second, written ms), with about 3 MB of machinery around it.

The microVM row holds the surprise of the lesson: the strongest kind of wall, at close to a container's price. Chapter 5 shows how a virtual machine gets that cheap. First, Chapters 2 and 3 build the cheapest wall, because every other wall reuses its parts.

worked exampleDoors into a kernel: kinds of request
  a trivial Linux program:              15 kinds of system call
  a typical Linux install (counted by Tsai et al.):
                                        224 kinds of system call + 52 kinds of ioctl = 276
  the program that runs one microVM, as its filter allows:
                                        24 kinds of system call + 30 kinds of ioctl = 54
                                        (22 of the 30 drive the processor's virtual machine support, Chapter 5)
  54 ÷ 276 = 0.196: about one fifth as many doors

Blast radius on a server of 48 desktops (illustrative server)
  the shared kernel broken:             48 desktops exposed
  one desktop's own kernel broken:      1 desktop exposed

The two counts measure different things, a whole system's needs against one program's allowed list, but they show the shape: a microVM's machinery talks to the host kernel through far fewer doors.

You start desktop 17 as a container and open a terminal inside it. You list the running programs and see 25. The server is running about 340. You ask the computer's name and it says desk-17; the server is called server-a. Nothing was copied and no second kernel started. So how does one kernel tell two different stories?

(The counts and names in this chapter are made up to show the idea. The walls and their names are Linux's own.)

The kernel keeps lists

The kernel keeps track of everything with lists. There is a list of every running program. There is the tree of folders and files. There is a list of network cards and the addresses on them. There is the computer's name, and there is a list of users. Normally every program asks for the same lists and gets the same answers.

Every running program gets a number from the kernel, its process ID, or PID. It is like a guest's room key number: the desk uses it to know who is asking. A command that lists the running programs simply reads the kernel's list of PIDs.

Namespaces: one copy of a list per room

Here is the trick. The kernel keeps a separate copy of one list, the running programs, the files, the network, for each room; each copy is a namespace. A program inside a namespace asks for the list and gets its room's copy. It has no way to ask for the real one.

The Linux manual says it in one line: a namespace wraps a global resource so that the programs inside it see their own isolated copy of it. In the hotel, each room gets its own view out of the window and its own phone directory. The building is the same for everyone. What each guest can see is not.

Linux has eight kinds of namespace, one for each list it can copy. Each has a flag, the name a program passes to the kernel to ask for a fresh copy:

One row needs a word. The user who may do anything on a Linux machine is root: the hotel manager's master key. A user namespace can map root inside a room to an ordinary, powerless user outside it. So a program can hold the master key to its own room and still be nobody special in the building.

You rarely make namespaces by hand. docker run, the command that starts a container in Docker, the most common container tool, creates a set of namespaces and control groups for each container behind the scenes. Control groups come in a moment.

Take the walls down one at a time around desktop 17, and watch what its terminal can suddenly see.

With every wall up, desktop 17's agent sees its own 25 programs, its own name, its own files and its own network card, and its root is nobody special. Each wall you took down let a different list through. The programs wall hid the other desktops' browsers. The files wall hid the folder where every desktop keeps its files.

The user wall is the one you least want to lose. Without it, root inside desktop 17 is root on the server: the master key to every room. Together, these walls fix the opening story of Chapter 1, where 40 desktops shared one temporary folder. With a mount namespace each, every desktop has a /tmp of its own.

Seeing less is not using less

Namespaces change what a desktop can see. They do nothing about what it can use. A desktop that sees only its own 25 programs can still open a page that eats all the server's memory. Its neighbours suffer, even though it cannot see them.

So the kernel has a second tool. A group of programs whose memory, processor time and number of programs the kernel meters and caps is a control group, or cgroup. In the hotel, it is each room's meter and breaker. The room may draw power up to a limit, and its breaker trips before the whole building goes dark.

One more word before the knobs. A processor has several cores, each able to run one program at a time: several clerks at the desk. A server with 48 cores runs 48 programs at the very same instant, and the kernel switches the rest in and out between them.

Three knobs, file by file

A control group is a folder, and its settings are small files you write one line into. Three of them matter most for a fleet.

memory.max is a hard ceiling on memory. Its default is the word max, which means no ceiling at all. When memory runs out, the kernel's last resort is to stop a program to get memory back: the out-of-memory killer, a breaker tripping. With a ceiling set, it acts inside the group that hit the ceiling, and nowhere else.

cpu.max holds two numbers, written $MAX $PERIOD. The group may run $MAX microseconds of processor time in every $PERIOD microseconds. (A microsecond, written µs, is a millionth of a second.) The default is max 100000: no cap, checked every 100,000 µs, a tenth of a second. So 50000 100000 is half of one core, and 200000 100000 is two cores' worth.

pids.max is a ceiling on how many programs the group may run at once. Its default is max as well. It stops a script that keeps starting copies of itself before the copies crowd out everyone else.

Here is a server of 32 GB with six desktops at 3 GB each. One of them is about to open a page that never stops asking for memory. Choose the ceiling, then open the page.

With no ceiling, the runaway grew until the server was full: 14 seconds at 1 GB a second, from 3 GB to 17. After that the kernel must stop something to get memory back, and nothing tells it the runaway is to blame. Every desktop on the server is now at risk.

With a 4 GB ceiling, the runaway reached its own fence after 1 second, and the kernel stopped it inside its own group. One episode failed. The other five desktops never noticed. That is the whole job of a control group: a problem stays in the room that caused it.

The request filter

Namespaces and control groups still leave every door of Chapter 1 open. A container's programs can make any system call the kernel offers. The third part of a container closes most of those doors. A list, set once per room, of the requests the front desk will take from it is a seccomp filter, short for secure computing. It is the desk's list of what a room may order. This is the filter you sent a request through in Chapter 1.

The Firecracker paper calls this filter the most important security boundary a container has. It also names the price. You can limit system calls, it says, "at the cost of breaking code which requires the restricted calls". Block too few, and the doors stay open. Block too many, and the desktop's own programs stop working.

The whole recipe

Now we can say exactly what a container is. It is namespaces, plus control groups, plus a seccomp filter, plus one more piece. Making a folder look like the top of the whole file tree for one program is chroot (change root): a room whose door opens only onto its own corridor. All four pieces run on one shared kernel.

Here is a room built by hand from the kernel's own walls, in C. It is a teaching program, not a real container tool, but the calls and the flags are the real ones. Like any program that makes namespaces, it needs root's rights to run.

c/* start a shell inside new namespaces: its own name, programs, files and network */
#define _GNU_SOURCE
#include <sched.h>
#include <unistd.h>
#include <string.h>
int main(void) {
    unshare(CLONE_NEWUTS | CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET);  /* ask for 4 new namespaces */
    sethostname("desk-17", strlen("desk-17"));     /* changes the name in this room only */
    if (fork() == 0)                                /* the first child is program 1 in the new list */
        execl("/bin/sh", "sh", (char *)0);         /* a shell that sees only the new room */
    return 0;
}

Read it line by line. unshare asks the kernel for fresh copies of four lists: the name, the programs, the files and the network. sethostname changes the computer's name, and only inside the new room; the server is still server-a. fork starts a child, and the first child becomes program number 1 in the new list of programs. That is why ps inside a container shows so few programs. execl turns that child into a shell. At no point did a second kernel start.

The meter and the breaker are even simpler: lines written into files.

cgroup filesmemory.max   4G              the desktop's hard memory ceiling (default: max)
cpu.max      200000 100000   2 cores' worth: 200,000 µs in every 100,000 µs (default: max 100000)
pids.max     512             at most 512 programs in the group (default: max)

The values are illustrative; the file names and their defaults are the kernel's.

Each file lives in the desktop's control-group folder. Writing a line into it sets the knob. The desktop's programs are moved into the group when it starts. Container tools such as Docker do all of this for you, every time you start a container.

What a container does not change

Notice what is missing from the recipe: a kernel. Every one of these walls is a rule that the shared kernel enforces. Everything Chapter 1 said about the shared front desk still holds. A bug behind one door of the host kernel is under every container on the server.

These walls are not new or fragile. Namespaces arrived in Linux between versions 2.6.15 and 2.6.26. Control groups arrived in 2.6.24, and their second version, the one these files belong to, became official in Linux 4.5. Every container tool builds on them, and so does every stronger wall in the next three chapters.

worked examplecpu.max is "$MAX $PERIOD": the group may run $MAX µs in every $PERIOD µs
  default "max 100000"   → no cap, counted every 100,000 µs (0.1 s)
  "50000 100000"         → 50,000 ÷ 100,000 = 0.5 of one core   (illustrative setting)
  "200000 100000"        → 200,000 ÷ 100,000 = 2 cores          (illustrative setting)

memory.max on a server of 32 GB with six desktops at 3 GB each (illustrative)
  in use: 6 × 3 GB = 18 GB; free: 32 − 18 = 14 GB
  a page that leaks 1 GB every second (illustrative):
    no limit:      the server is full after 14 ÷ 1 = 14 s; the runaway has grown 3 → 17 GB
    "4G" limit:    the runaway reaches its own fence after (4 − 3) ÷ 1 = 1 s;
                   the kernel stops it inside its group

pids.max "512" (illustrative): a script that keeps starting copies of itself
  stops at 512 programs in that group

Forty-eight desktops start from the same set of files: the same browser, the same fonts, the same empty Downloads folder. The browser and its libraries alone are a few hundred megabytes. Does the server keep 48 copies? And when one agent changes one line of one file, what happens to the copy all the others read?

(The file sizes in this chapter are made up to show the idea. The rules are those of Docker's storage, which most container fleets use.)

The image

Start with the files every desktop needs before any task begins: the operating system's programs, the browser, the fonts, the settings, an empty home folder. A frozen copy of a desktop's files, made once, that every container starts from is an image. It is the room's standard furniture plan: every room in the hotel is furnished from the same plan.

An image is not one big file. An image is a stack of layers, each a folder of files, and none of them can be changed: they are read-only. The bottom layer might hold the operating system, the next the browser, the next the task's test files. Think of furniture bolted to the floor. Every guest uses it, and no guest can carry it out or saw it in half.

Layers are built once and shared by every container started from the image. That is why a server can hold the files for 48 desktops in roughly the space of one. The image is written to disk once, and every desktop reads the same copy.

The writable layer

But a desktop must be able to change things. The agent downloads files, the browser writes its history, settings change. Each container gets one thin, empty layer of its own on top, where all its changes go: the writable layer. In the hotel, it is a clear sheet laid over the furniture. Guests may write on the sheet as much as they like. The furniture underneath stays as it was.

What the agent sees is the layers merged, top one first: the merged view. It is like looking through the sheet at the room. Where the sheet has something written, you see the writing. Where it is blank, you see the furniture beneath it.

Docker's usual way of doing this is called overlay2, and it names the three parts plainly. The image's read-only layers are the lowerdir. The writable layer is the upperdir. The view the agent sees is merged. They are three folders that the kernel joins into one.

Reading, writing and deleting

Reading is simple. To read a file, the kernel looks for it in the top-most layer that has it. If the writable layer has no copy, the read goes straight to the image. Nothing is copied to read.

Writing is where the cost hides. The image is read-only, so a change cannot happen there. The first time a container changes a file from the image, the whole file is copied into its writable layer first: a copy-up. Then the change is made to the copy. It is photocopying a whole book to correct one line.

The whole file, not just the part that changed. overlay2 works on whole files, not on smaller blocks of them, so every copy-up copies the entire file. Say the browser keeps a history file that came with the image at 200 MB. The agent visits one page and the browser adds one row, about 100 bytes. Before it writes those 100 bytes, the kernel copies all 200 MB into the writable layer.

Deleting is the strangest of the three. You cannot remove a file from a read-only layer. So a delete writes a small marker in the writable layer that says "this file is gone": a whiteout. It is a sticky note over the furniture saying "not here". The image still holds the file. The merged view simply hides it.

One copy in memory for everyone

There is a second kind of sharing, in memory. The kernel keeps files it has read recently in memory, so the next read is fast: the page cache. It is the front desk's copy of the hotel guide, shared by every room.

Containers that read the same file from the image share a single page-cache copy of it. Say all 48 desktops start the same browser, 250 MB of files. Without sharing, that is 48 × 250 = 12,000 MB of memory. With one shared copy, it is 250 MB. The server saves 11,750 MB, room for several more desktops.

Make a few changes to one desktop below, then wipe it, and see what is left.

Three things to notice. Visiting a page cost a 200 MB copy for a 100-byte change. Clearing the cache did not remove anything from the image; it added a whiteout. And the wipe undid every change in one move, except the file in the shared folder.

Why a container starts fast

Now the speed. Starting a container copies nothing. It lays a new, empty writable layer on top of the shared image, and making an empty folder takes almost no time. No kernel boots, because the server's kernel is already running.

The programs inside still have to start, of course. A desktop needs its screen, its window manager and its browser, and Chapter 6 opens those up. That part takes real time. But the desktop's files are ready the instant the container exists.

Wiping is throwing the layer away

And now the wipe. The image never changes, because it is read-only. So the clean state of every desktop is simply the image. To wipe a container, delete its writable layer and start its programs again from the image. Every change of the episode, every copy-up, every new file, every whiteout, sits in that one folder, and it all goes at once.

There is nothing to repair and nothing to compare. You do not need to know what the agent did. Whatever it was, it lived in the writable layer, and the writable layer is gone. This is the third way to wipe from Chapter 0: throw away just the changes.

What the wipe does not reach

Not everything a desktop touches lives in its layers. A folder on the server that also shows up inside the desktop, so files put there live on the server, is a shared folder, which engineers call a bind mount. It is the mail slot to the back office: what you put there stays after the room is cleaned. Files in it are in no layer, so dropping the layer leaves them exactly where they were.

Two other things escape this wipe. Programs that are still running keep whatever they hold in memory until they are restarted, so a wipe must restart them too. And anything the agent changed outside the box, such as an account on a real website, is not on the server at all. No wipe of the box reaches it.

Chapter 8's Code Lab builds this overlay in Python, with a leak detector on top of it. Here we keep to the rules and the device.

worked exampleOne row added to a 200 MB file that came with the image (illustrative sizes)
  bytes the agent's browser changed:   about 100 (one row)
  bytes copied up first:               200 MB   (overlay2 copies the entire file)
  48 desktops each visiting one page:  48 × 200 MB = 9,600 MB of writable layers

Reading the same browser files from the image (illustrative: 250 MB)
  48 desktops, each with its own copy:  48 × 250 = 12,000 MB of memory
  one shared page-cache copy:           250 MB;  saved: 12,000 − 250 = 11,750 MB

Starting a container:  copy nothing; lay an empty writable layer on the image; boot no kernel
Wiping a container:    delete the writable layer; the read-only image needs no repair

A container's programs talk straight to the server's kernel, through a filter. Now picture a guard posted inside each room. Every request a guest makes goes to the guard first, and the guard handles it: it reads the file, sends the message, hands out the memory. The guard almost never calls the front desk itself. Who is the guard? Just another program.

Kernel and user space

Start with a split that every computer has. The kernel runs with full rights over the hardware. Everything that is not the kernel runs with limited rights, in what is called user space: the guest floors, as against the back office. Every program you have written ran in user space. That is why it had to ask the kernel for files: it was not allowed to touch the disk itself.

But does every part of a kernel need those special rights? Most of what a kernel does is bookkeeping: tracking open files, handing out memory, keeping the lists of Chapter 2. Bookkeeping can be done by an ordinary program, as long as something below it does the few jobs that truly need the hardware.

A kernel written as a program

That is the idea of this chapter. A kernel written as an ordinary program, that sits between one desktop and the server and answers its requests, is a user-space kernel. We call it the guard. Each desktop gets a guard of its own, like a concierge stationed inside each room.

The best-known one is gVisor. Its kernel part is called the Sentry, and it is written in the Go language. The Sentry carries out Linux's system calls itself. When a desktop's browser asks to open a file, the Sentry checks the request, keeps the bookkeeping and answers. To the browser it looks exactly like Linux.

How the guard catches a request

There is a puzzle here. The desktop's programs think they are talking to the kernel, and their system calls go to the server's kernel by default. How does the guard get them first?

The answer reuses Chapter 2's filter, with a twist. First, one more word. A tap the kernel gives a program, stop and run this handler now, is a signal: a knock on the door. A program can say in advance which of its own pieces of code should run when a given knock comes. That piece of code is the signal's handler.

The guard sets a seccomp filter on the desktop's programs whose answer, for every system call, is "trap" (SECCOMP_RET_TRAP). So when one of those programs makes a call, the server's kernel does not carry it out. Instead it sends the program the signal SIGSYS, and the guard handles that signal. It reads the request, does the work and writes back the answer. The program carries on as if the kernel had replied.

one request, caughtdesktop program:      open("/home/user/notes.txt")          a request for a file
seccomp filter:       SECCOMP_RET_TRAP                       the server's kernel does not open it
server's kernel:      sends SIGSYS to the program            a tap: run the handler now
the guard (Sentry):   reads the request, checks it, asks the Gofer for the file, writes back the answer
desktop program:      carries on as if the kernel had answered

An illustrative trace in words. The two names in capitals are the real ones from the kernel and from gVisor's documentation.

This way of catching requests has a name. The guard's way of catching requests, where a filter turns every request into a signal the guard handles, is what gVisor calls the systrap platform. It needs no special processor support, so it runs on ordinary cloud machines. It has been gVisor's default since mid-2023, when it replaced an older method called ptrace, which is no longer supported. gVisor has a second platform built on the processor's own virtual machine support, called KVM, which Chapter 5 explains.

What reaches the server's kernel

The guard answers the desktop's requests itself. In gVisor's own words, it "never passes through any system call to the host". Files are the one special case: a small helper process beside the guard, called the Gofer, fetches them from the server and hands them over.

The guard itself still runs on the server's kernel, of course. It is a program, so it makes system calls of its own. But it makes a narrower set of them, chosen by the people who wrote it, not by whatever code the agent happened to run. The many kinds of request a desktop can make shrink to the guard's own short list.

What it stops, and what it does not

To get out of a guarded desktop, an attacker would have to break the guard and then the server's kernel, both at the same time. One bug is no longer enough. That is the guard's whole point: two walls in a row instead of one.

It does not stop everything. A trick that learns a secret from how long the processor takes to do things is a side channel, like guessing a safe's combination by listening to the lock. The Spectre family of attacks works this way. The guard only intercepts system calls and memory faults, so a program that never makes a strange request can still time the processor. Chapter 5 says what fleets do about that.

What it costs

Nothing here is free. gVisor's documentation names two kinds of cost. One is structural: the guard needs memory of its own, and every system call now crosses extra layers on its way. The other is in the details: the guard brings its own network code and its own file code, and those behave a little differently from Linux's.

The same documentation says where the cost does not fall. Raw computing runs at native speed, and so does memory access. A program that spends its time thinking pays almost nothing. The cost falls mainly on work that makes many system calls.

The docs give their measurements only as charts, with no numbers in the text. So the model below uses made-up costs and says so. Let a task spend C milliseconds computing and make S system calls. On the server's kernel each call costs c₀. Behind the guard, each call costs c₀ + e, where e is the guard's extra:

T_guard

The task's time behind the guard, in ms.

C

Time spent computing, in ms. The guard does not change it.

S

The number of system calls the task makes.

c₀

What one call costs on the server's kernel (illustrative: 0.5 µs).

e

The guard's extra cost per call (illustrative: 2 µs).

Indexing files: 100 ms + 500,000 × 2.5 µs = 100 + 1,250 = 1,350 ms (against 350 ms on the plain kernel)

Read it as two parts. C never changes, because computing runs at native speed. S × (c₀ + e) grows with every call. So the guarded time divided by the plain time sits near 1 when C is large and S is small. It climbs toward (c₀ + e) ÷ c₀ as calls take over. With the toy values, that ceiling is 2.5 ÷ 0.5 = 5.

Pick a task and compare its two bars, then change the guard's extra cost per call.

Thinking barely moved: 2,001 ms became 2,005. Rendering a page grew by about a tenth. Indexing a folder of files, a task that makes a request for every file it looks at, took 3.86 times as long. Same guard, same cost per call. The difference is all in how often each task asks.

worked exampleIllustrative model: time = computing + calls × (0.5 µs, plus 2 µs more behind the guard)
                     computing    calls     plain kernel             guard                     ratio
  Thinking           2,000 ms     2,000     2,000 + 1 = 2,001 ms     2,000 + 5 = 2,005 ms      1.002
  Rendering a page     400 ms    20,000       400 + 10 = 410 ms        400 + 50 = 450 ms       1.10
  Indexing files       100 ms   500,000       100 + 250 = 350 ms       100 + 1,250 = 1,350 ms  3.86

  the ceiling as calls take over:  (0.5 + 2) ÷ 0.5 = 5

To get out: break the guard AND the server's kernel

Its own wipe tools

The guard brings one more gift. It holds the desktop's whole kernel state as its own ordinary memory, so it can freeze a whole sandbox to a file and bring it back later. Here are the commands, from gVisor's guide:

shellrunsc checkpoint --image-path=/snap/desk-17 desk-17            # save it; the container stops by default
runsc create desk-17b                                            # a new, empty container
runsc restore --image-path=/snap/desk-17 desk-17b                # bring the saved desktop back in it
runsc restore --image-path=/snap/desk-17 --background desk-17b   # let it run while memory is still loading

Container names and paths are illustrative.

Line by line: runsc checkpoint saves the desktop into the folder named by --image-path. By default the container stops after the save; --leave-running keeps it going. To bring it back, you make a new, empty container with runsc create and restore into it with runsc restore. With --background, the desktop's programs start as soon as the kernel state is loaded, while its memory is still arriving. And --exclude-committed-zero-pages leaves empty memory out of the file, which makes the file smaller.

By default the guard also lays the desktop's root files under a memory-backed overlay (--overlay2=root:self), so writes never reach the image. That is Chapter 3's writable layer again, built into the guard. Chapter 7 puts all these save-and-restore tools side by side.

A full virtual machine pretends to be a whole PC: a start-up chip, slots for expansion cards, USB ports, a graphics card. Starting one takes seconds. Now imagine one that pretends to be almost nothing: a disk, a network card, a serial line and a keyboard controller with one job. How fast could it start, and how little could it cost?

Pretend hardware, real processor

A virtual machine needs a kernel of its own, and a kernel expects hardware: a processor, memory, a disk. Pretending to be a whole processor in software would be painfully slow. So modern processors help. They can run a second kernel in a fenced-off mode, where it runs at nearly full speed but cannot touch the real hardware directly.

The part of Linux that switches the processor in and out of that fenced-off mode is KVM (kernel-based virtual machine). In the hotel, it is the land the cabins stand on: the server's kernel owns it and lends plots of it out.

The kernel inside the virtual machine is the guest; the server's kernel is the host, as before. The guest has its own system calls, its own files, its own lists. A request from the desktop's browser goes to the guest kernel and never reaches the host. That is the "own kernel" picture from Chapter 1.

The fence has one gap, on purpose. When the guest touches a device, say to write to its disk, the processor steps out of the fenced-off mode. Then something on the host must answer for the pretend disk.

The program that answers

That something is an ordinary program. An ordinary program that creates the pretend devices and answers the guest when it uses them is the virtual machine monitor, or VMM. In the hotel, it is the groundskeeper, who brings water and power to each cabin through a few fixed pipes.

A pretend disk or network card made by the VMM is an emulated device. Every emulated device is code that answers the guest. And here Chapter 1 comes back: every pretend device is a door. A guest that finds a bug in the code behind a pretend USB port has found a way out of its cabin.

A full PC is a lot to pretend

The common VMM, QEMU, pretends to be a whole PC. It can show a guest a start-up chip, expansion slots, USB, graphics cards and much more. That makes it wonderfully general: it can boot almost any operating system. It also makes it big. QEMU is more than 1.4 million lines of code, and it can need up to 270 kinds of system call on the host.

For a fleet of practice desktops, nearly all of that is waste. A Linux desktop in a box needs a disk, a network card and a way to print its start-up messages. Everything else is attack surface and start-up time.

Firecracker: keep KVM, replace the rest

Engineers who run a huge fleet of small, short-lived programs, the Lambda service, reached the same conclusion. Their VMM, Firecracker, kept KVM and replaced QEMU with a new, minimal VMM, device model and API. It is about 50,000 lines of Rust, a programming language: 96% fewer lines than QEMU.

It offers only a few emulated devices. A simple, fast style of pretend device made for virtual machines is virtio, and Firecracker offers virtio network and disk, a serial port, and part of a keyboard controller. It has no start-up chip (no BIOS), no expansion slots (no PCI), no USB and no graphics card. It cannot move a running VM to another server, and it cannot boot Windows.

A virtual machine that pretends to be almost nothing, a disk, a network card, a serial line, is the microVM from Chapter 1. For practice desktops this means two things. The screen is drawn in software, which Chapter 6's virtual screen does anyway. And a Windows desktop needs a full VM.

One process per microVM, locked in a jail

Each microVM is one Firecracker process on the server. You configure it through a small web-style API on a local socket, and killing the process stops the VM. That makes a fleet simple to manage: one desktop, one process, one socket.

The process itself is locked down with the walls of Chapter 2. A small helper that starts each Firecracker process already locked in its own folder, its own namespaces and its own short list of allowed requests is the jailer: the gate around the groundskeeper's shed. It uses chroot, its own PID and network namespaces, dropped rights, and a seccomp filter that allows 24 system calls and 30 ioctls. Of those 30, 22 are the ones KVM needs.

Why so careful? The project's threat model treats every vCPU thread, a pretend processor core given to a virtual machine, as running hostile code from the moment it starts. The design assumes the guest is an attacker. For an agent fleet, that is exactly the right assumption.

Two practical facts. Firecracker needs read and write access to /dev/kvm, the file through which programs reach KVM. And its defaults are small: 1 vCPU and 128 MiB of memory. (A MiB is 1,048,576 bytes, a little more than a megabyte.) The number of vCPUs may be 1 or an even number up to 32.

The numbers, with their roles

Starting a machine's kernel from nothing is a boot; the first program the kernel starts is init. It is unlocking the cabin and switching on the first light. Boot time is the number everyone quotes, so it pays to know which number is which.

The promise, from the paper that introduced Firecracker: memory overhead under 5 MB per VM, a boot to application code in under 125 ms, and up to 150 microVMs created per second on one host.

The ceiling the project enforces in its own tests: at most 125 ms from the start call, named InstanceStart, to the guest's first program. At most 5 MiB of overhead at 1 vCPU and 128 MiB. And the process ready to take calls within 8 ms of processor time.

The measurement: about 3 MB of overhead per VM, against about 13 MB for another light VMM, Cloud Hypervisor, and about 131 MB for QEMU. The overhead was the same at every VM size. One more real number: a production kernel with a fuller set of programs boots in as little as 150 ms. That does not contradict the 125. It is a different, fuller system.

The purple bar below is a minimal microVM's boot, from the VM program starting to the guest's first program. Switch on the extras that a stock setup brings with it.

Why a boot can be slow

The paper timed each extra. A stock Ubuntu 18.04 kernel adds about 900 ms, because it probes for old devices and loads drivers that a microVM does not have. Unpacking a compressed kernel at boot adds about 40 ms. A network card set up in advance adds about 20 ms. Writing start-up messages to the serial console costs up to 70 ms.

The fix is a minimal kernel. The paper's minimal kernel is 4.0 MB with no modules (the add-on pieces a kernel can load), against Ubuntu's 6.7 MB plus 44 MB of modules. Turn off what the microVM cannot use, and the boot falls back near its ceiling.

A thousand at once

A fleet does not boot one desktop; it boots hundreds. The time 99 of every 100 boots beat is the 99th percentile, written p99. In a race of 100 runners, it is roughly the time of the second-slowest. It matters more than the average, because a fleet has to wait for its slow boots too.

Booting 1,000 microVMs 50 at a time, pre-configured Firecracker's p99 was 146 ms; Cloud Hypervisor's was 158 ms. Booting 100 at a time, Firecracker's p99 rose only to 153 ms. At the promised 150 creations a second, 1,000 microVMs take 1,000 ÷ 150 = 6.7 seconds to create on one host.

Memory at fleet scale

Overhead is memory the server cannot give to desktops. Against one desktop, it looks small either way. A desktop VM in one well-known benchmark gets 4 GB of memory. Against that, 3 MB is 0.07% and 131 MB is 3.2%.

Across a fleet it adds up. A thousand desktops under Firecracker spend 3 GB on machinery. Under QEMU they spend 131 GB, which is 128 GB more: room for 31 more 4 GB desktops. Pick the program and the fleet size below.

Here are the machine settings a fleet sets for each microVM, with Firecracker's own field names. Only the values are made up:

jsonMachineConfiguration (Firecracker's API, illustrative values)
{
  "vcpu_count": 2,             one or an even number, up to 32 (default 1)
  "smt": false,                pretend hyper-threading off (default false)
  "track_dirty_pages": true,   record written pages, needed for diff snapshots (default false; Chapter 7)
  "huge_pages": "None"         or "Transparent" or "2M" (Chapter 7)
}
then: point the VM at a kernel and a disk image, and send InstanceStart:
      at most 125 ms later the guest's first program runs

Field by field: vcpu_count is how many pretend cores the desktop gets. smt says whether those cores pretend to run two threads each, the trick called hyper-threading. track_dirty_pages makes the VMM keep a record of which parts of memory the guest wrote, which Chapter 7 needs for small snapshots. huge_pages chooses the size of the blocks the guest's memory is handed out in. The memory size itself defaults to 128 MiB.

Side channels

Chapter 4 left one question open: timing tricks. A virtual machine alone does not stop them, because guests still share the real processor. The paper's hosts close them by turning off simultaneous multithreading (hyper-threading, where one core runs two threads at once), turning off swap and memory-page merging, and not sharing files between VMs. The paper's own test host ran with hyper-threading off.

worked exampleBoot to the guest's first program
  minimal microVM:                      at most 125 ms (enforced ceiling)
  production kernel and user space:     as little as 150 ms
  stock Ubuntu 18.04 kernel:            125 + 900 = 1,025 ms  →  1,025 ÷ 125 = 8.2 × as long
  every extra switched on:              125 + 900 + 40 + 70 + 20 = 1,155 ms  (the sum is illustrative)

A thousand at once
  1,000 microVMs at 150 a second:       1,000 ÷ 150 = 6.7 s on one host
  1,000 microVMs, 50 at a time:         1,000 ÷ 50 = 20 waves; p99 146 ms (100 at a time: 153 ms)

Memory around each VM
  Firecracker 3 MB × 1,000 = 3,000 MB = 3 GB
  Cloud Hypervisor 13 MB × 1,000 = 13 GB;   QEMU 131 MB × 1,000 = 131 GB
  against one 4 GB (4,096 MB) desktop:  3 ÷ 4,096 = 0.07 %;   131 ÷ 4,096 = 3.2 %
  QEMU minus Firecracker at 1,000:      128,000 MB ÷ 4,096 MB = 31 more 4 GB desktops' worth

Code size: 1 − 50,000 ÷ 1,400,000 = 0.964 → 96 % fewer lines

Inside the box there is no monitor, no mouse and no keyboard. Yet the agent needs a screen to look at and a mouse to click with. So the box runs a screen that exists only in memory, something that takes its picture, and a pretend mouse and keyboard. Here is what starts, in order, when one desktop wakes up.

A screen with no monitor

On a Linux desktop, windows do not draw themselves. The program that draws windows on a Linux desktop is the X server; each screen it draws is a display, numbered :0, :1 and so on. Think of a painter with numbered canvases: programs say what they want, and the painter puts it on canvas number 1.

The block of memory that holds the screen's pixels is the framebuffer: the canvas itself. On an ordinary computer the graphics card reads the framebuffer many times a second and lights up the monitor. Inside a box there is no graphics card and no monitor.

So the box uses an X server with no monitor that keeps the screen as a picture in memory: Xvfb (X virtual framebuffer). Its manual says it runs on machines with no display hardware and no input devices, and that it emulates a framebuffer in virtual memory. It is painting on a canvas nobody hangs up. The painting is complete; only the programs that ask for a copy ever see it.

Its default is a trap

Start Xvfb with no options and you get a surprise: its screen 0 is 1280×1024 at 8 bits of colour. How many bits each pixel gets for its colour is the colour depth. With 8 bits there are 256 possible colours; with 24 bits, about 16.7 million. It is a box of 256 crayons against millions.

At 8 bits, smooth shading turns into visible bands, and small icons lose detail. A model trained on normal screenshots then sees an odd, banded desktop. So always pass the size and depth yourself, with -screen 0 WxHxD: width, height and depth.

A reference desktop, line by line

One model maker publishes a reference desktop container for its computer-use tool, built on Ubuntu 22.04. Its start script is a good model of the whole stack. Here is how it starts the screen:

shellDISPLAY_NUM=1   WIDTH=1024   HEIGHT=768   DPI=96      # $DISPLAY is :1
RES_AND_DEPTH=${WIDTH}x${HEIGHT}x24
Xvfb $DISPLAY -ac -screen 0 $RES_AND_DEPTH -retro -dpi $DPI -nolisten tcp -nolisten unix &

Line by line. $DISPLAY is :1, display number 1. -ac turns off access control, so any program in the box may draw. -screen 0 $RES_AND_DEPTH sets screen 0 to 1024x768x24: 1024 by 768 pixels at 24-bit colour. -retro draws the classic patterned background. -dpi 96 tells programs how many dots make an inch, so text comes out the expected size. The two -nolisten flags stop the screen from taking connections from outside the box. And & runs it in the background, so the script can go on.

Then the rest of the desktop starts. mutter is the window manager, the program that puts frames, title bars and a panel around windows: the picture frames. tint2 is the panel along the bottom. xdotool moves the pointer and types for the agent. x11vnc listens on port 5900, a numbered door on a machine where a program listens for connections, like a phone extension. noVNC listens on port 6080 and shows the screen inside a web page. Together they offer VNC, a way for a person to watch and drive a remote screen: a window into the room. A web interface uses ports 8501 and 8080.

Say this plainly, because it is often confused: VNC is for the human watching. The agent does not look through VNC. Its screenshots come from the tool code, which reads the framebuffer itself. You can change the screen size with WIDTH and HEIGHT when you start the container.

The box-side loop

Now the loop that runs inside every episode. Take a screenshot of the framebuffer. Shrink it if the model needs a smaller picture. Send it to the model. Get back an action. Replay the action with the pretend mouse or keyboard. Repeat until the agent says it is done.

The same model maker's docs describe the virtual display as an X11 server using Xvfb. They recommend a screen of 1024×768 or 1280×720 for desktop tasks, and 1280×800 or 1366×768 for web apps.

What a screenshot costs the model

A model does not read pixels one by one. The unit a model reads is a token; for these models one image token covers a 28 by 28 pixel square, like one tile in a mosaic. The docs give the rule for the newer models: tokens = ⌈width ÷ 28⌉ × ⌈height ÷ 28⌉. The ⌈ ⌉ brackets mean round up, because a partly covered square still costs a whole token.

So a 1024×768 screenshot is ⌈1024 ÷ 28⌉ = 37 squares across and ⌈768 ÷ 28⌉ = 28 down: 37 × 28 = 1,036 tokens. A 1920×1080 screenshot is 69 × 39 = 2,691. Over 50 steps of one episode, a made-up but ordinary length, the 1024×768 screen costs 51,800 tokens of pictures.

There are limits. The newer models accept at most 2,576 pixels on the long edge and 4,784 tokens. Earlier models accept at most 1,568 pixels on the long edge and about 1.15 megapixels (million pixels). The API does not shrink an image for you, and an oversized one is rejected. So the harness, the code around the model that runs the loop, must shrink it first.

Pick the desktop's screen and the model's image limit below, then ask the model to click Save.

Clicks come back in screenshot pixels

Here is the slip that looks exactly like a dumb agent. The model answers in the pixels of the picture it saw. If you shrank the picture, you must scale its coordinates back up before clicking. The docs say so in those words.

Work it through. A 1920×1080 screen is 2,073,600 pixels, over the earlier models' 1.15 megapixels. The long-edge rule allows 1,568 ÷ 1,920 = 0.817 of the size. The megapixel rule allows √(1,150,000 ÷ 2,073,600) = 0.745. Both rules must hold, so the smaller one wins, and the screenshot shrinks to 1429×804.

The Save button's centre is at 960, 540 on the screen. In the shrunk picture, the model sees it at 960 × 1429 ÷ 1920 = 715 and 540 × 804 ÷ 1080 = 402. Scaled back up, 715 × 1920 ÷ 1429 = 961 and 402 × 1080 ÷ 804 = 540: on the button. Not scaled, the click lands at 715, 402, which is 245 pixels left of the button and 138 pixels above it.

Browser-only boxes

When a task lives entirely in a browser, the box can be just a browser. Chrome started with --remote-debugging-port=9222 opens a local server for remote control, and the address to connect to is listed as webSocketDebuggerUrl at /json/version. Running a browser with no window at all is headless, and the flag for it is --headless.

A set of messages that lets another program drive Chrome, take a screenshot, move the mouse, type, is the Chrome DevTools Protocol: a remote control for the browser. Here are the messages a harness sends for one screenshot, one click and some typing:

json{"id": 1, "method": "Page.captureScreenshot", "params": {"format": "png"}}
{"id": 2, "method": "Input.dispatchMouseEvent",
 "params": {"type": "mousePressed", "x": 960, "y": 540, "button": "left", "clickCount": 1}}
{"id": 3, "method": "Input.dispatchMouseEvent",
 "params": {"type": "mouseReleased", "x": 960, "y": 540, "button": "left", "clickCount": 1}}
{"id": 4, "method": "Input.insertText", "params": {"text": "March report"}}

The field names are the protocol's own; the values are illustrative.

Message by message. Page.captureScreenshot returns the picture as base64 text in a field named data, PNG by default; jpeg and webp are the other formats, and quality applies to JPEG. In Input.dispatchMouseEvent, x and y are in the page's CSS pixels, measured from the top-left of the visible page. A click is two messages, a press and a release. Input.insertText types a whole string at once. And Page.setDeviceMetricsOverride fixes the page's size, so every episode's screenshots match.

A cheaper wall inside one browser

Playwright, a popular tool for driving browsers, offers an even lighter box. A private, empty browser profile with its own cookies and storage, made in a moment inside a running browser, is a browser context: a fresh incognito window. In Playwright's docs, contexts are fast and cheap to create and completely isolated, even within one browser. Their default viewport, the visible page area, is 1280×720.

The trade is plain. Contexts wipe the browser's state between episodes: cookies, storage, history. But all of them share one browser program and one kernel. That is the thinnest wall on the ladder, fine for trusted web tasks and nothing more.

Phones

Some agents practise on phones. A program that pretends to be a phone is an emulator; one saved phone setup is an AVD (Android virtual device). By default, the Android emulator saves a Quick Boot snapshot of the whole AVD when it closes, and loads it when it starts. In Android's own words, a Quick Boot can start up to 10 times faster than a cold boot.

Any change to the emulator, the system image or the AVD's settings makes old snapshots unusable. For a fleet, a few flags matter: -no-snapshot-load forces a full boot, -no-snapshot-save keeps the saved state from changing, -no-window runs it with no window, and -read-only and -wipe-data control what the phone keeps from one run to the next.

A benchmark's defaults

Finally, an expert view: one well-known benchmark's environment class, with its defaults.

pythonenv = DesktopEnv(
    provider_name="vmware",          # where the desktop VM runs (12 providers exist; Chapter 7)
    snapshot_name="init_state",      # the saved moment every task rewinds to
    action_space="pyautogui",        # actions arrive as pyautogui calls
    screen_size=(1920, 1080),        # from SCREEN_WIDTH and SCREEN_HEIGHT
    headless=False,
    require_a11y_tree=True,          # also fetch the accessibility tree with each observation
    os_type="Ubuntu",
)

It defaults to VMware (a commercial virtual machine program), to a snapshot named init_state, and to actions written as calls to pyautogui (a Python library that moves the mouse and presses keys). Its screen defaults to 1920×1080, read from SCREEN_WIDTH and SCREEN_HEIGHT. It also fetches the accessibility tree: the list of on-screen elements with their names and positions that the system keeps for screen readers. Notice the screen: 1920×1080 is over the earlier models' limit, so a harness on this default must shrink every screenshot and scale every click back.

worked exampleFramebuffer size: width × height × bytes per pixel (24-bit colour is stored in 4 bytes)
  1024 × 768 × 4   = 3,145,728 bytes = 3.0 MiB
  1280 × 1024 × 1  = 1,310,720 bytes = 1.25 MiB   (Xvfb's 8-bit default: 1 byte per pixel, 256 colours)
  1920 × 1080 × 4  = 8,294,400 bytes = 7.9 MiB

Image tokens, newer models: ⌈w ÷ 28⌉ × ⌈h ÷ 28⌉, at most 4,784
  1024 × 768:   37 × 28 = 1,036      1280 × 720:  46 × 26 = 1,196      1280 × 800:  46 × 29 = 1,334
  1366 × 768:   49 × 28 = 1,372      1920 × 1080: 69 × 39 = 2,691      1280 × 1024: 46 × 37 = 1,702
  50 steps of 1024 × 768 screenshots: 50 × 1,036 = 51,800 tokens  (50 steps illustrative)

Earlier models: at most 1,568 px on the long edge and about 1.15 megapixels
  1920 × 1080 = 2,073,600 px: long-edge scale 1,568 ÷ 1,920 = 0.817;
                              megapixel scale √(1,150,000 ÷ 2,073,600) = 0.745
    the smaller wins: 0.745 → 1429 × 804
  1280 × 1024 = 1,310,720 px: long edge fits; megapixel scale √(1,150,000 ÷ 1,310,720) = 0.937 → 1198 × 959

Scaling a click back (screen 1920 × 1080, screenshot 1429 × 804)
  the Save button's centre on screen:    960, 540
  where the model sees it:               960 × 1429 ÷ 1920 = 714.5 → 715;   540 × 804 ÷ 1080 = 402
  scaled back:   715 × 1920 ÷ 1429 = 960.7 → 961;   402 × 1080 ÷ 804 = 540   (on the button)
  not scaled:    clicks at 715, 402: 245 px left of and 138 px above the button

Getting a practice desktop ready takes a while: the machine boots, the screen starts, the browser opens, and the task's setup signs in to a test account. Then the agent works for a minute. Then all of it must be undone. What if you could save the desktop at the exact moment it was ready, and simply go back to that moment, like a save point in a video game?

Three ways to wipe

We now have everything we need to name the three ways to wipe from Chapter 0 properly. To rebuild, throw the box away and start a fresh one from the image: boot it, start the desktop, run the task's setup. To throw away the changes, drop the writable layer (Chapter 3) and restart the desktop's programs. To rewind, go back to a saved moment.

A saved copy of a whole running machine, its memory, its processor state and its devices, is a snapshot, like a save point in a video game. Starting a machine from a snapshot instead of booting it is a restore: loading the save. The restored machine wakes up exactly where the snapshot left it, programs open, screen drawn, in the middle of whatever it was doing.

So a snapshot of a ready desktop skips everything a rebuild repeats: the boot, the desktop's start-up and the task's setup. That is why the Rewind button at the top of the page kept the wall busy.

How real benchmark code wipes

Real code shows how different these can be. One well-known benchmark supports a dozen places to run its desktop VMs, from a VM program on your own computer to several clouds. Three of them wipe in three different ways.

On VMware, a commercial virtual machine program, it reverts the VM to a named snapshot with vmrun revertToSnapshot <vmx> <name>, and then waits a fixed 3 seconds (WAIT_TIME = 3). That is a true rewind, with a pause for safety.

Its Docker option is not a container desktop at all. It runs a full QEMU/KVM virtual machine inside a container, from a read-only disk image, /System.qcow2, with 4 GB of memory and 4 cores. It has no snapshot: asking it to save raises NotImplementedError("Snapshots not available for Docker provider"). To "revert", it stops and removes the container and boots a fresh VM. Then it asks for a screenshot once a second, for up to 300 seconds, until the desktop answers. That is a rebuild.

On AWS, Amazon's cloud, saving a snapshot makes a machine image: a saved disk a cloud can start new machines from, like a mould for casting new rooms. Reverting launches a brand-new machine from it (a t3.large if none was recorded) and waits up to 15 s × 10 tries = 150 s for it to run. That is a rebuild too, of a whole cloud machine.

One more detail from the same code. Its reset() skips the revert when the desktop was not used since the last one, because on a cloud a needless revert is, in the code's own words, "costly and time-consuming".

No source gives one clean number per way of wiping. So the race below uses made-up wipe times, in the order the sources support, with each real anchor marked on its bar. Pick a way to wipe and a task length.

With one-minute tasks, a 1-second restore keeps a desktop working 98% of the time, and it finishes 59 tasks an hour. A one-minute reboot keeps it working half the time, 30 tasks an hour. Slide the task length down to 10 seconds and the gap grows: short tasks make the wipe matter even more.

What is inside a microVM snapshot

Now open a snapshot up. Memory is handed out in small blocks called pages, like the pages of a notebook. A Firecracker snapshot is two files: the guest's memory, page by page, and the microVM's state, its processor and its devices. The disks are separate, and you keep those yourself.

Why restoring is fast

Telling the kernel that a stretch of memory is a file, so reading that memory reads the file, is mapping the file. It is a book on a shelf you read in place, instead of copying it out first. On restore, Firecracker maps the memory file privately (MAP_PRIVATE). Nothing is read up front.

A page is read from the file only when the guest first touches it. That first touch is a short pause, called a page fault, while the kernel fetches the page. And any write goes to a private copy. The first time a page is written, the kernel makes a private copy of it and writes to the copy; the file never changes: copy-on-write. It is writing in pencil on a photocopy, never in the library's book.

You have seen this idea before. It is Chapter 3's copy-up again, per page instead of per file. The price is that the memory file must be kept for as long as any restored VM uses it, because the VM still reads its untouched pages straight from that file.

Why saving is slow

Saving runs the other way. Taking a full snapshot pulls in every page of the guest's memory and writes it all to the file. Firecracker's docs say it directly: restoring is the direction built for speed, and taking a snapshot costs extra work to write the memory out.

So the recipe is simple. Take one snapshot of a ready desktop, once. Restore it thousands of times. Each square below is 8 MB of a 4 GB desktop's memory.

The task touched 600 MB of the 4,096: only 15% of the file was read. It wrote 160 MB, and only those pages were copied. The rewind threw the copies away and mapped the same, unchanged file again, ready for the next task.

The API, line by line

Here are Firecracker's snapshot calls in order. The paths and values are illustrative; the fields are the API's own.

firecracker apiPATCH /vm              {"state": "Paused"}
PUT   /snapshot/create  {"snapshot_type": "Full",
                         "snapshot_path": "/snap/desk.state",
                         "mem_file_path": "/snap/desk.mem"}
... later, in a NEW Firecracker process, before it boots ...
PUT   /snapshot/load    {"snapshot_path": "/snap/desk.state",
                         "mem_backend": {"backend_path": "/snap/desk.mem", "backend_type": "File"},
                         "track_dirty_pages": false,
                         "resume_vm": true,
                         "clock_realtime": true}
PATCH /vm              {"state": "Resumed"}     (not needed when resume_vm is true)

Line by line. Pause first, so the memory stops changing while it is saved. /snapshot/create writes the state file and the memory file; Full is the default snapshot_type, and Diff needs dirty-page tracking (below). By default sync_snapshot_files is true, so the files are fully written to disk before the call returns. The load must happen in a new Firecracker process before its VM boots: pause, resume and create work only after boot, load only before it. backend_type File lets the host kernel read pages from the file; Uffd hands page faults to a handler of your own. clock_realtime moves the guest's clock forward to now, and Chapter 8 explains why that matters. Resuming lets the guest's processors run again.

Smaller snapshots and faster faults

A full snapshot saves every page; a diff snapshot saves only the pages written since the last one, like a full backup against an incremental one. A page written since the last snapshot is dirty.

Diff snapshots need dirty-page tracking, the track_dirty_pages setting from Chapter 5, and that tracking costs processor time. Firecracker still calls diff snapshots a developer preview, not a finished feature.

There is also a choice of who serves the pages on restore. Normally the host kernel reads them from the file. Instead, a page-fault handler in user space can serve them over a socket (backend_type: "Uffd"). A fleet might do that to fetch pages from somewhere smarter than a local file. Explicit 2 MB huge pages require it.

Rules for restoring

Two rules from the docs. Restore only on hardware and software identical to where the snapshot was taken. And on hosts still using control groups version 1, restores are slow; Firecracker strongly recommends version 2, the version whose files Chapter 2 wrote.

The other boxes' rewinds

Every kind of box has some way back to a clean start. Here they are side by side:

And here is the benchmark's VMware revert, as its code does it:

shellvmrun revertToSnapshot /vms/ubuntu.vmx init_state
sleep 3                      # WAIT_TIME = 3: "Wait for the VM to revert"

The .vmx path is illustrative.

worked exampleWipe times, ILLUSTRATIVE, ordered as the sources support        share of a 60 s task cycle spent working
  launch a new cloud machine from an image:   90 s   [the code waits up to 15 s × 10 = 150 s]   60 ÷ 150 = 40 %
  reboot a full VM from its disk:             60 s   [the code polls for up to 300 s]            60 ÷ 120 = 50 %
  revert a full VM to its snapshot:            6 s   [then a fixed 3 s wait]                     60 ÷ 66  = 91 %
  boot a fresh microVM, start the desktop:     5 s   [the boot itself is at most 125 ms]         60 ÷ 65  = 92 %
  drop a container's layer, start the desktop: 4 s   [the image never changes]                   60 ÷ 64  = 94 %
  restore a microVM snapshot:                  1 s   [no published time]                         60 ÷ 61  = 98 %
  per desktop per hour, 60 s tasks:  3,600 ÷ 120 = 30 (reboot)   against   3,600 ÷ 61 = 59 (restore)

A 4,096 MB desktop (the benchmark VM's memory); how much a task touches and writes is illustrative
  take a full snapshot:   write all 4,096 MB, every page
  restore and run:        the task touches 600 MB → 600 ÷ 4,096 = 15 % of the file is read
                          the task writes 160 MB  → 160 MB of private copies; the file unchanged
  rewind:                 drop the 160 MB of copies; map the same file again

You restore sixteen desktops from the same snapshot at the same moment. Each one opens a test website and asks for a fresh, random session number. All sixteen get the same number.

(That scene is imagined. The mechanism behind it is real, and Firecracker's own docs warn about it.)

Why clones match

Each machine restored from the same snapshot is a clone of it: cabins built from one save point. A clone resumes with exactly the memory the desktop had when it was saved. Byte for byte, every clone starts the same.

That includes things that are supposed to be unique. A random-number generator does not pull randomness from nowhere. The unpredictable starting point a random-number generator stirs from is its seed; the unpredictability itself is entropy. Think of shuffling a deck from a starting order: the same starting order and the same shuffle give the same deck every time.

A program that seeded its generator before the snapshot carries that seed into every clone, so every clone draws the same "random" numbers next. The same goes for every ID or secret a program had already made: a session ID, a login token, a key. In the hotel, cabins built from one save point all have the same safe combination.

Firecracker's docs put it bluntly: without a way to make unique things stay unique, resuming from the same state more than once is insecure.

What VMGenID fixes, and what it does not

The guest kernel can at least be told that it was just copied. A number the virtual hardware changes on every restore, so the guest kernel can tell it is a clone and re-stir its own randomness, is VMGenID (VM generation ID). It is a sign on the door saying "you were just copied". Linux supports it from version 5.18.

But it re-stirs only the kernel's own entropy. In the docs' words, unique identifiers, cached random numbers and secrets "will still be replicated" across clones made from one snapshot. The kernel's fresh random numbers differ; the session ID the browser made before the snapshot does not. Changing the lock on the front door does not change the combination of the safe.

Clones share an address

The number other machines use to reach this one is its IP address: a room's phone number. Every clone resumes with the same network device name and the same IP address, because both were in the saved state. Sixteen desktops answering to one number confuse every server they talk to.

The fix uses Chapter 2's namespace again. Start each clone in its own network namespace, which the jailer's --netns flag does, so each clone's identical address lives in a separate copy of the network. Then put a translator at the border that gives each inside address its own outside one: NAT (network address translation), a switchboard giving each room its own outside line.

Clones live in the past

The time of day a machine believes it is, is its wall clock: the clock on the room's wall. A restored guest's wall clock carries on from the moment the snapshot was taken. If the snapshot was taken at nine in the morning and restored after lunch, the desktop believes it is still nine.

Anything that checks dates may then refuse it: a login that expires, a security certificate checked against today's date. Setting clock_realtime: true on load moves the clock to now. It needs a host kernel with kvm-clock support, Linux 5.16 or later.

Restore six clones below, then switch on the fixes one at a time.

The sharpest moment is the one with only VMGenID on. The kernel's random values now differ, and everything else is still shared. Each repeated value needs its own fix, and each fix is one fresh input that arrives after the restore: a new kernel seed, IDs made after the restore, a separate network, the real time.

That gives a rule for when to take the snapshot: before anything unique exists. Snapshot the desktop when it is ready but before the task's setup signs in or makes an ID. Then run the setup after each restore, in each clone, so each clone makes its own.

The snapshot file is code

Firecracker checks a snapshot file only with a short number computed from a file that changes if the file is damaged by accident: a checksum, like a tally at the bottom of a page. Firecracker uses a 64-bit CRC on the state file, a common kind of checksum. It treats snapshot files as trusted input that the people running it must protect.

A checksum catches accidents, not tampering: anyone who can change the file can recompute the tally. So store snapshots like code: read-only, with access controlled, and never writable by the desktops they restore.

What survives each wipe

Put the chapters together. Leaving the desktop as it was keeps everything (Chapter 0). Dropping the writable layer keeps anything in a shared folder, and anything outside the box (Chapter 3). Rewinding to a snapshot copies whatever unique thing was saved in it into every clone (this chapter). Rebuilding keeps nothing on the box, but an account the agent changed on a real website still remembers.

Every wipe has a blind spot. So a fleet does not trust its wipe. It checks.

Check every start

Here is one way to check, run at the start of every episode before the agent moves. It is not a named standard; it is plain engineering that catches the leaks above.

First, a canary: a small file with a unique mark, planted on purpose in every episode, so the next episode can look for it. It is a coin left on the windowsill to see whether housekeeping cleaned. If any canary is found at a start, the wipe failed.

Second, a fingerprint. A short number computed from data, which changes completely if any byte changes, is a hash; the hash of a whole file tree is its fingerprint. Fingerprint the files that should match the image, their sorted paths and their contents, and compare with the image's own fingerprint. If they differ, list the paths that differ.

And fail closed. An episode whose start fails either check does not run. Its desktop is rebuilt, and its server is set aside to be looked at (Chapter 10). A wrong reward costs far more than one lost episode.

pythonimport hashlib, secrets

def plant_canary(desk, episode):
    mark = f"canary-ep{episode}-{secrets.token_hex(4)}"     # unique every episode
    desk.write("/home/user/.canary", mark)
    return mark

def fingerprint(files):                                       # files: {path: bytes}
    h = hashlib.sha256()
    for path in sorted(files):                                # same order every time
        h.update(path.encode()); h.update(b"\0"); h.update(files[path]); h.update(b"\0")
    return h.hexdigest()

def clean_start(desk, image_print):
    if desk.exists("/home/user/.canary"):
        return False, "a canary from an earlier episode survived the wipe"
    if fingerprint(desk.read_tree(desk.watched_paths)) != image_print:
        return False, "the files differ from the image"
    return True, "clean"

An illustrative sketch: desk and its methods are made-up names for whatever your fleet uses to reach a desktop's files.

Block by block. The canary carries a random part, so two episodes never plant the same mark. The fingerprint sorts the paths first, so the same tree always gives the same hash, and it puts a zero byte between path and contents so that two different trees cannot run together into the same bytes. clean_start runs before the agent's first screenshot. A failed check stops the episode before it can earn a wrong reward.

worked example16 clones of one snapshot, nothing fixed (illustrative app and values)
  session IDs made before the snapshot:     1 distinct value among 16 desktops
  with VMGenID (Linux 5.18+):               the kernel's fresh random numbers: 16 distinct
                                            the app's IDs made before the snapshot: still 1
  network:                                  16 clones at one address inside
                                            16 network namespaces + NAT: 16 distinct outside addresses
  clock:                                    saved at 09:00:00, restored at 14:32:10
                                            → 5 h 32 min 10 s behind, until clock_realtime moves it to now

Checking every start (one way to check)
  canary:       episode 41 plants /home/user/.canary = "canary-ep41"; episode 42 must find no canary
  fingerprint:  hash(the start's files) must equal hash(the image's files)
  the Code Lab's toy fleet, 12 episodes on one desktop:
                leave it as it was:          96 leaked cells flagged
                drop the writable layer:     15 leaked cells, all 15 in the shared host folder

The trainer wants 10,000 episodes an hour. Each one takes about 90 seconds, and the wipe after it takes 1 more. How many desktops must be running at once? Not a guess: one line of arithmetic answers it, and the same line works for a hotel's rooms, a bank's tellers or a fleet of practice desktops.

Three numbers

How many episodes start each second is the arrival rate, written λ (lambda): the pace of guests checking in. How long each episode keeps a desktop, including the wipe after it, is the hold time, W: how long a guest stays, plus the time to clean the room after. And the number this chapter is after, the average number of desktops busy at once, is L.

A hotel makes the idea concrete before any letters do. If 10 guests check in each hour and each one stays 3 hours, how many rooms are occupied on average? Not 10, and not 3. At any moment, a guest who checked in up to 3 hours ago is still there, so the hotel is carrying roughly 3 hours' worth of check-ins at once: 10 × 3 = 30 rooms.

The same area, counted two ways

That arithmetic is not a coincidence, and the reason it always works is worth seeing once. Picture a long stretch of time, T seconds, and draw every episode as a horizontal bar on it: the bar starts the moment the episode takes a desktop and ends the moment that desktop is clean again, W seconds later.

Over that stretch, about λT episodes start, each one's bar W seconds long. Add up the length of every bar and you get λT × W, the total bar-length drawn on the picture.

Now count the very same picture a different way. At any one instant, the number of bars crossing that instant is exactly the number of desktops busy right then. Add that count up over the whole stretch of T seconds and you get L × T, because L is defined as the average number busy at once.

Both sums measured the same drawing, the same ink on the same page, so they must be equal: L × T = λT × W. The T on both sides cancels, leaving Little's law: the average number busy at once, L, equals the arrival rate times the hold time. Nothing in that argument cared how the episodes arrived in time, in a steady trickle or in bursts, or how long any one of them ran. The bars could be any lengths at all; the two counts of the same picture still match. That is why the law holds for any pattern of arrivals and any spread of episode lengths.

L

Desktops busy at once, on average.

λ

Episodes started per second: the arrival rate.

W

Seconds each episode holds a desktop: the episode plus the wipe. The hold time.

10,000 ÷ 3,600 = 2.78 a second × 91 s = 253 desktops busy; × 1.2 headroom = 304 to run.

This is the same law the async-work lesson uses for queues of jobs instead of queues of desktops: Async Work and Long-Running Tasks is worth a visit if job queues are the shape you think in.

The wipe is in the bill

Slide the wipe over to Reboot and watch the fleet grow, because W is not just the episode, it is the episode plus whatever comes after it. At 10,000 episodes an hour with 90-second episodes, a 60-second reboot needs 2.78 × 150 = 417 desktops busy at once, 500 with the usual headroom. A 6-second revert needs 2.78 × 96 = 267, 320 with headroom. A 1-second restore needs 2.78 × 91 = 253, 304 with headroom.

Chapter 7 showed restoring a snapshot is far cheaper than rebooting a machine. Here is the same fact in a different currency: the slow wipe does not just waste a desktop's own time, it forces the fleet to buy more desktops to make up for it. Every second spent being wiped is a second that still has to be paid for in hardware.

Headroom

Extra desktops beyond the average, kept running for the bursts that arrive faster than the average predicts, is headroom: spare rooms kept ready for a busy weekend. Little's law gives you the average only. Real arrivals bunch up, a slow afternoon followed by a rush, and a fleet sized at exactly L queues episodes during every one of those rushes while running under capacity the rest of the time. A 20% headroom is a choice this lesson makes, not a law; different training setups pick their own margin based on how bursty their arrivals actually are.

Warm pools

A desktop that is created only the moment it is asked for is a cold start: nobody made up the room ahead of time, so the guest waits while it happens. Desktops created ahead of time and sitting ready is a warm pool: rooms made up before any guest arrives. The share of requests a pool can hand a desktop to immediately, with no wait, is the hit rate.

The Firecracker paper sizes its own pool of pre-booted microVMs with the same law that just gave you the fleet size. If microVMs are created at 8 a second and each one takes 125 milliseconds to finish booting, then by Little's law the average number in creation at once is 8 × 0.125 = 1. So the paper keeps one pooled microVM ready for roughly every 8 it creates a second.

The average is not enough

A pool holding exactly the average, 1 microVM, sounds like enough. It is not. Requests arrive in clumps, and a pool of 1 is empty for about half of them.

An old formula from telephone exchanges, built for exactly this problem, gives the chance that every line into the switchboard is busy: Erlang's loss formula. Treat the microVMs in creation as telephone lines, each one lasting the creation time, with calls arriving at the creation rate. Call the average number busy a (here a = λC = 8 × 0.125 = 1) and the pool size P. The chance all P lines are busy, so the pool is empty, is:

B(P, a)

The chance a request finds the pool empty: a cold start.

a^P ÷ P!

The weight of "all P pool slots are busy."

denominator

The same weight added up for every count from 0 busy up to P busy, so the chances sum to 1.

The hit rate is 1 − B. With a = 1: a pool of 1 line gives 1 − (1÷2) = 50%. A pool of 2 gives 1 − (0.5÷2.5) = 80%. A pool of 3 gives 1 − ((1÷6)÷(8÷3)) = 93.75%. This holds whatever the spread of creation times looks like, fast and slow mixed in; only the average matters.

Limits a fleet runs into

Three ceilings show up once a fleet gets large. Firecracker can create up to 150 microVMs a second on one host, so refilling a pool of 1,000 empty slots on one host takes 1,000 ÷ 150 = 6.7 seconds: not instant, but not a bottleneck either. The paper's production fleet recycles every slot within 12 hours, whatever happens to it in the meantime; spread across a 1,000-desktop fleet run the same way, that is one recycle roughly every 43.2 seconds across the whole fleet.

Memory is the ceiling that bites hardest. The paper's evaluation host had 384 GB. At 4 GB per desktop, Chapter 2's limit for the memory-heavy example, that host holds 96 desktops before any oversubscription: promising more memory or processor time than the server actually has, betting that not every desktop uses its full share at the same moment, the way an airline sells more seats than the plane holds. The paper's production fleet runs up to 10 times oversubscribed on average, and has been tested past 20 times. That bet only pays off when sandboxes sit partly idle; agent desktops that are genuinely working most of the time, taking screenshots and clicking, leave far less slack to borrow against.

Monday's training report says the agent's success rate fell from 40% to 38% overnight. Did the agent actually get worse? Or did 50 desktops fail to start, run out of memory or lose their connection, and get scored as the agent's failures when the agent never got a fair turn at all?

Two kinds of failure

An episode can end badly in two very different ways. The agent tried the task and did not finish it: an agent failure, like a student who sat the exam and failed it. Or the box never gave the agent a fair chance at all: a fleet fault, like the exam room flooding before anyone could pick up a pen.

Fleet faults are the failures earlier chapters already named, now gathered under one word. The desktop never finished starting. It never answered when asked for a screenshot. It ran out of memory, Chapter 2's fence closing on it. Its network connection dropped. Its start-of-episode check, Chapter 8's canary or fingerprint, failed and the episode was refused before the agent ever saw a screen.

Why mixing them hurts

A fleet fault scored as a 0, a plain failure, quietly lowers the measured success rate, and training treats that 0 the same as a real mistake: it nudges the agent away from whatever it was doing, even though the agent did nothing wrong.

Work through 1,000 episodes with a true success rate of 40%, where 5% of episodes are hit by a fleet fault at random, striking easy and hard tasks alike. Of those 50 faulted episodes, 40% of them, 20 episodes, would have succeeded if the box had worked. Score every fault as a failure and the dashboard reads (400 − 20) ÷ 1,000 = 38.0%, two points below the truth. Score only the 950 episodes the fleet actually ran to completion and it reads 380 ÷ 950 = 40.0%, the honest number.

Turn the knobs below and watch both readings move.

Watch the sharpest moment: at 5%, turning the toggle off slides the 50 red dots out of the grid into the retry queue, and the bar above climbs back to 40.0%, right where it started. The agent never changed. Only the arithmetic did.

Where fleet faults hide in real code

This is not a hypothetical shape. One well-known benchmark's reset logic retries a failed reset up to a fixed number of times before giving up. Its Docker-hosted desktop VM is declared ready only once a request for a screenshot finally answers, polled once a second for up to 300 seconds. Its AWS launch path waits up to 15 seconds, ten tries, 150 seconds total, for a freshly launched machine to come up.

A desktop that never answers holds its slot for the whole budget it was given, not for the instant it actually needed. If, say, 2% of wipes hang for the full 300-second budget before being given up on, that adds 0.02 × 300 = 6 seconds to the average hold time across the whole fleet, and Chapter 9's Little's law turns that extra 6 seconds straight into extra desktops the fleet must run.

The trace

One record per episode, saying which desktop ran it, how it was wiped, how long that took and how it ended, is the episode's trace: a flight's logbook entry, written whether the flight went well or not.

json, one episode's trace (illustrative field names){
  "episode_id": "ep-2026-10-11-004217",
  "task_id": "save-march-report",
  "desktop_id": "desk-17",
  "server": "host-b07",
  "box": "microvm",
  "snapshot_id": "ready-v12",
  "wipe": "restore",
  "wipe_ms": 940,
  "ready_ms": 310,
  "start_check": "clean",
  "steps": 23,
  "exit": "agent_done",
  "reward": 1
}

Every field earns its place. episode_id and task_id say which attempt at which job this was. desktop_id, server and box say exactly which machine ran it, down to whether it was a container, a guard kernel or a microVM, so a fault can be traced back to one box or one server. snapshot_id and wipe say how the desktop was made clean, and wipe_ms plus ready_ms how long each step took, the numbers Chapter 9's fleet-sizing arithmetic feeds on. start_check is Chapter 8's canary-and-fingerprint verdict. steps, exit and reward are the episode's own outcome.

The rule that keeps the score honest fits in one line: any exit starting with fleet_ never becomes a reward. The field names agent_done, agent_gave_up and max_steps are agent outcomes, and they alone get scored. fleet_boot_timeout, fleet_out_of_memory, fleet_lost_connection and fleet_dirty_start are fleet faults, never scored, and sent to be retried somewhere else.

Retry and quarantine

Running a faulted episode again on another desktop is a retry: moving the flooded exam to another room instead of failing everyone who was supposed to sit it there. Cap how many times one task can be retried, and keep a running list of tasks that fault every single time for a person to look at by hand (the async-work lesson calls that kind of list a dead-letter list).

Taking a server out of the fleet until someone looks at it is quarantine: closing a flooded floor rather than one flooded room. When several desktops on one server all fault together, that server is almost always the real cause, and the right unit to pull out is the server, not each desktop one at a time. The failure-modes lesson has the fuller picture of blast radius and cascades at a larger scale: Failure Modes and Isolation.

The bias trap

Dropping fleet faults from the score, instead of scoring them as 0, is only fair when faults strike tasks at random: easy and hard ones alike. If faults strike hard tasks more often, say because memory-heavy tasks hit the 4 GB fence more often than light ones, then dropping those episodes removes exactly the attempts the agent was going to fail anyway. If, in this chapter's numbers, all 50 faults happened to land on tasks the agent would have failed regardless, the score reads 400 ÷ 950 = 42.1%, higher than the true 40%, for the opposite reason: not too low this time, but too high.

The fix is the same either way: do not drop faulted episodes and move on. Retry them elsewhere, and keep a count of how many episodes needed a retry. A high retry count is itself a signal worth watching, not noise to throw away.

Recycle on a schedule

Replacing a long-running desktop or server with a fresh one on a fixed schedule, whether or not anything has gone visibly wrong with it yet, is recycling: a deep clean every night, not only after a guest complains. The Firecracker paper's own production slots live at most 12 hours before being recycled this way. Recycling bounds the kind of slow failure that never trips an alarm on its own: memory that leaks a little more every hour, a disk that slowly fills, a process that gets a little slower with every episode until, eventually, it starts timing out.

A fleet runs hundreds of practice desktops at once and wipes each between episodes, because the checker reads the desktop and a leftover is a wrong reward. Desktops that share a kernel share its doors; namespaces and control groups make rooms in one building, a guard kernel answers each room's requests itself, and a microVM gives each desktop a kernel of its own for about 3 MB and a tenth of a second.

A snapshot of a ready desktop turns the wipe from a rebuild into a rewind, and a check at every start proves it worked. Little's law sizes the fleet with the wipe included, and a trace on every episode keeps the fleet's own faults out of the agent's score.

The numbers

The checklist

1. Choose the wall by what runs inside: a shared kernel only for code you trust; a guard kernel or a microVM for agents that run code pulled off the open web.
2. Give every desktop its own PID, mount, network, UTS and user namespaces.
3. Set memory.max, cpu.max and pids.max on every desktop's control group.
4. Keep the image read-only, and mount no writable folder from the server into the desktop.
5. Start Xvfb with an explicit size and 24-bit colour depth; never rely on its 1280×1024×8 default.
6. Size screenshots to the model's own limit, and always scale clicks back before sending them.
7. Snapshot the desktop the moment it is ready, before any sign-in, ID or secret is made.
8. Restore only on hardware and software identical to where the snapshot was taken; keep one snapshot per kind of server.
9. Run restores on hosts using control groups version 2.
10. Give each clone its own network namespace and NAT; set clock_realtime on load.
11. Use a guest kernel with VMGenID, Linux 5.18 or later, and make every program's IDs and secrets fresh after the restore, not before.
12. Store snapshot files the way you store code: read-only and access-controlled, since Firecracker checks them only with a checksum.
13. Check every start: no canary from a previous episode, and a fingerprint equal to the image's own. Fail closed.
14. Size the fleet with the wipe included: L = λ × (episode + wipe), plus a margin of headroom.
15. Keep a warm pool larger than λ × creation time: the average alone is only half the hits.
16. Write a trace for every episode: box, snapshot, wipe, wipe time, start check, exit.
17. Never score a fleet fault as a failure; retry it elsewhere, cap the retries, and quarantine servers whose desktops fault together.
18. Recycle desktops, and the servers under them, on a fixed schedule.

A fleet worker's core loop

An illustrative fleet worker; the names below are ours, not any one project's:

pythondef run_episode(task, pool, policy, trace):
    desk = pool.take()                         # a warm desktop, just restored from the ready snapshot
    rec = {"task_id": task.id, "desktop_id": desk.id, "server": desk.server,
           "box": desk.box, "snapshot_id": desk.snapshot_id, "wipe": "restore",
           "wipe_ms": desk.wipe_ms, "ready_ms": desk.ready_ms}
    try:
        ok, why = desk.clean_start()            # no canary, fingerprint equals the image's
        rec["start_check"] = "clean" if ok else why
        if not ok:
            raise FleetFault("dirty_start")
        desk.plant_canary()
        shot = desk.screenshot()                # sized to the model's limit
        for step in range(task.max_steps):
            action = policy(task.instruction, shot)
            if action.kind == "done":
                break
            shot = desk.act(action)             # click scaled back to the screen, then a fresh screenshot
        rec["steps"] = step + 1
        rec["reward"] = task.checker(desk)      # read the desktop's final state
        rec["exit"] = "agent_done" if action.kind == "done" else "max_steps"
    except FleetFault as fault:                 # boot timeout, out of memory, lost connection, dirty start
        rec["exit"] = "fleet_" + fault.kind     # never a reward
        pool.retry_elsewhere(task)
    finally:
        trace.write(rec)
        pool.give_back(desk)                    # rewind to the snapshot; quarantine the server if it keeps faulting
    return rec

Every chapter of this lesson lives somewhere in that loop. Chapter 7's rewind is pool.take and pool.give_back. Chapter 8's checks are clean_start and plant_canary. Chapter 6's screen and clicks are desk.screenshot and desk.act. Chapter 10's rule lives in the except block and the trace it writes. And Chapter 9 decides how many copies of this exact loop run at once.

Where this lesson sits

• Before it, on the site now: Tools, Exec-Approvals and Sandboxing (the sandbox from an agent harness's own side) and Failure Modes and Isolation (blast radius and cascades at a larger scale).
• Right after it: Async Work and Long-Running Tasks, which uses this chapter's own Little's law for queues of jobs instead of queues of desktops.
• Further along: Large-Scale Distributed Training, the trainer this fleet feeds episodes to.
• Coming soon in this path: Agent Environments (the task, the checker and the reset, told from the environment's own side), OSWorld (the benchmark whose provider code Chapter 7 read directly), Computer-Use Agents (the loop this fleet exists to host), GUI Grounding (where, exactly, the click should land), Agentic RL and UI-TARS-2 (training that needs fleets built exactly like this one), and Prompt Injection (why these walls matter most when a web page itself tries to give the agent orders).

Frontier, briefly

Diff snapshots and dirty-page tracking are still a developer preview in Firecracker's own docs, not a finished feature. A user-space page-fault handler can serve restores over a socket instead of a local file, and is required for explicit 2 MB huge pages. gVisor's guard kernel can also run on KVM, not only on its default systrap platform. And at least one benchmark, OSWorld, already runs the same tasks across a dozen different providers, from a VM on your own laptop to several different clouds, which is a sign of how much the box underneath can vary while the task on top stays the same.

Go back to the wall at the top of the page and pick Rewind it: 47 tasks a minute, every one starting clean. You now know what each tile on that wall is made of, how it comes back from a saved moment, and how many of them a training run actually needs. Then open Teach and explain the clone leak to someone, or Present it in your own words.