How a Robot Sees, Thinks, and Acts

Chapter 1: See → Think → Act — The Cycle That Never Stops

Every autonomous robot runs the same loop, forever, dozens of times a second. It is worth burning into memory because every algorithm you will ever meet in robotics lives at one specific point on this loop.

The loop has three beats — See, Think, Act — wrapped around the real world:

• See (Sensing). Raw, noisy measurements arrive from the robot's sensors: camera pixels, LiDAR range returns, wheel-encoder ticks, IMU accelerations. This is raw data — high-volume, low-meaning.
• Think (Perception + Localization + Planning). The robot turns raw data into meaning: it extracts information (this blob is a stop sign), figures out where it is (my pose is here on the map), and decides what to do (follow this trajectory). Out comes a plan.
• Act (Actuation). The plan becomes actuator commands: motor torques, steering angles, wheel velocities. The robot physically moves.
• The world responds. Acting changes the world — and the robot's place in it. So the next "See" measures a different world, and the loop turns again.

Stanford draws this exactly — the "see-think-act" cycle from Lecture 1. Let's make it turn, and watch the data morph as a packet rides around the ring.

Notice that nothing waits for anything. The loop never "finishes." There is no end state where the robot is done thinking and starts acting — it is always doing all three, with fresh data, because the world never stops changing. A robot that pauses to think while moving at 30 mph is a robot that has already crashed.

Where does each lesson live on this loop?

Here is the payoff of memorizing the loop: it is a coat-rack for the entire field.

Chapter 2: The Autonomy Stack — Four Modules

"See-think-act" is the rhythm. The autonomy stack is the machinery that produces it. Stanford splits the work into four modules of roughly equal weight — and this series devotes its lessons to each in turn.

The interfaces — the type of data on each arrow — are the contract that makes a robot buildable by a team. Click each module below to see exactly what it consumes and produces, and which lessons cover it.

That outer teal frame is the Executive. Notice it touches everything — it doesn't sit in the data flow, it sits around it, deciding which mode each module runs in. We'll formalize the Executive as a finite state machine in Chapter 5. First, two prerequisites every layer assumes: what "state" means (Chapter 3), and how a robot represents the world as a map (Chapter 4).

Chapter 3: What Is "State"? — Configuration & Pose

Every module on the stack passes around one quantity above all others: the robot's state. Before we can plan, localize, or control, we have to agree on what numbers describe "where the robot is and how it's oriented." Those numbers are the robot's configuration.

Stanford's Lecture 1 defines it precisely. The generalized coordinates of a robot are the smallest set of numbers that completely specify its configuration. We collect them in a vector written ξ (the Greek letter xi):

Here n is the number of generalized coordinates, ℝⁿ means "an ordered list of n real numbers," and the small ⊤ (transpose) just means we treat it as a column. For a wheeled robot rolling on a flat floor, three numbers are enough:

where (x, y) is the position of the robot on the floor and θ (theta) is its heading — the direction it faces, measured as an angle. The pair (position + orientation) has a special name: the robot's pose. A robot that knows its pose knows everything it needs to know about where it is.

Let's make the pose vector concrete. Drag the robot to set (x, y); rotate it to set θ. The readout shows the live configuration vector and the robot's heading vector — the unit vector pointing where it faces:

With the constraint on, try to drag the robot sideways to its heading — it resists, sliding only along the barred line's perpendicular. That is the no-slip constraint: a rolling wheel cannot translate along its axle. Written as an equation, the sideways velocity must be zero:

Here ξ̇ (xi-dot) means "the rate of change of ξ" — the velocity. Don't worry about deriving this; Lesson 2 builds it from scratch and shows how it produces the famous unicycle and differential-drive models. The point for now: state is three numbers, and the robot's geometry restricts how those numbers can change. That restriction is what makes robot motion planning interesting rather than trivial.

Chapter 4: How a Robot Sees the World — Map Representations

A robot that knows its pose still needs to know what's around it: where the walls are, which rooms connect, where the kitchen is. That knowledge is a map. But "map" is not one thing — the same physical environment can be stored four very different ways, each good at a different job. Choosing the wrong representation makes a problem hard that the right one makes trivial.

Here is one apartment, rendered four ways. Flip between them and notice they describe the same world through different lenses:

Worked example: updating one occupancy cell by hand

How does a single grid cell decide it's occupied? Not all at once — it accumulates evidence across measurements. The trick (full story in Lesson 12) is to track belief in log-odds ℓ, because then each new measurement just adds a number instead of multiplying probabilities:

Start at total ignorance: P = 0.5, so ℓ₀ = log(0.5/0.5) = 0. A LiDAR "hit" on this cell contributes ℓ_hit = +0.85; a "miss" (beam passed through) contributes ℓ_miss = −0.4. Watch evidence pile up:

Two hits push the cell to 85% sure it's occupied; one stray miss nudges it back to 79%. The belief is robust to single bad readings because evidence accumulates. That is why we store probabilities, not booleans — sensors are noisy, and a probability lets a cell remember how sure it is. (We add in log-odds because logarithms turn Bayes' multiplication into addition — the same trick reappears in every filter in Lessons 13–14.)

Chapter 5: The Executive — Finite State Machines

We have Perception, Localization, and Planning–Control. What decides which of these the robot should be doing right now? When a TurtleBot bumps a wall, who says "stop driving forward, start backing up"? That is the job of the Executive, and the oldest, clearest tool for the job is the Finite State Machine (FSM).

An FSM is defined by a small, fixed set of pieces. Every symbol here matters, so we define each before using it:

• A finite set of states S — the distinct "modes" the system can be in (e.g. waiting, raising, raised, lowering).
• A set of inputs I, the input vocabulary — what the machine can sense each tick (e.g. car arrived, gate at top).
• A set of outputs O, the output vocabulary — what it can command (e.g. raise, lower, hold).
• A next-state function n(i, s) → s′ — given the current input i and current state s, which state do we move to?
• An output function o(i, s) → o — given input and state, what do we command?
• An initial state s₀ — where the machine starts.

That's the whole model: (S, I, O, n, o, s₀). Each tick the executive reads an input, applies n and o, emits a command, and updates its state. Written as a loop, it's almost embarrassingly simple:

We draw FSMs as graphs: a circle for each state, an arrow for each transition. We label each arrow with the input that triggers it and the output it produces, written input / output. The current state glows; when an input fires a transition, that arrow flashes and the glow jumps. That picture is the next-state function n and the output function o, drawn.

This is exactly how your TurtleBot's navigator.py works, and how production drone autopilots like PX4 are structured. In ROS, the SMACH library builds these for you, with visualization. But you don't need a library — a class with a state variable and an if/elif block per state is a complete FSM. We'll build one by hand next chapter.

Chapter 6: The Parking Gate — An FSM by Hand

Stanford's canonical FSM example is a parking-gate arm — small enough to trace by hand, real enough to matter. Let's specify it completely, then trace a car through it, then run it live, then build it in code.

The problem. A gate arm guards a parking lot. It can be in four states. A sensor reports whether a car is waiting; another reports whether a car just passed; position sensors report whether the arm is fully up or fully down. The arm can raise, lower, or hold (nop). Desired behavior: when a car arrives, raise the arm; keep it up until the car drives through; then lower it.

The transition table (the FSM, written out)

This table is the next-state function n and output function o. Each row is one tick: given the state and input, here is the commanded output and the next state.

Trace it in words. We start down, holding (tick 0). A car arrives — tick 1 fires n(car waiting, down) = raising and commands raise. The arm rises (ticks 2–3) until the top sensor trips, landing in up. It waits, holding, until the car passes — tick 5 fires n(car passed, up) = lowering and commands lower. The arm descends (6–7) and returns to down. One clean cycle. Notice that raise and lower each fire exactly once.

Now press the sensor buttons yourself and watch the arm move and the graph light up. Try the fault inject toggle — a stuck sensor — to see the FSM hang in a state, which motivates the robustness ideas in Chapter 7.

Building it in code

The FSM is a class with a state variable and one if/elif block per state — the textbook OOP pattern. Verbose first, with every branch matching a table row:

python
class ParkingGateFSM:
    def __init__(self):
        self.state = 'down'          # s0: the initial state

    def step(self, car, gate):
        # one tick: read inputs (car, gate sensors) -> (output, next state)
        out = 'hold'               # default output: do nothing
        if self.state == 'down':
            if car == 'waiting':
                out, self.state = 'raise', 'raising'
        elif self.state == 'raising':
            out = 'raise'
            if gate == 'top':
                out, self.state = 'hold', 'up'
        elif self.state == 'up':
            if car == 'passed':
                out, self.state = 'lower', 'lowering'
        elif self.state == 'lowering':
            out = 'lower'
            if gate == 'bottom':
                out, self.state = 'hold', 'down'
        return out

The compact, table-driven form scales better — the transitions live in a dictionary, so adding a state is data, not code:

python
# (state, input) -> (output, next_state); missing key = hold + stay
TABLE = {
    ('down',    'waiting'): ('raise', 'raising'),
    ('raising', 'top'):     ('hold',  'up'),
    ('up',      'passed'):  ('lower', 'lowering'),
    ('lowering','bottom'):  ('hold',  'down'),
}
def step(state, sym):
    out, nxt = TABLE.get((state, sym), ('hold', state))
    return out, nxt

In the Code Lab below, you'll implement the step method and run a scripted car through the gate, animating the arm as the FSM commands it — and your trace must match the table above. This is the executive layer, in 15 lines.

Chapter 7: Taming Complexity — Reduction, Hierarchy, Composition

The parking gate had four states. A real self-driving stack has hundreds — and adding one state can spawn an exponential number of transitions. Stanford warns that designing these "extremely complex state machines... is still one of the most time-consuming, difficult tasks faced by companies." Three tools keep FSMs tractable.

1. State reduction

Two states are equivalent if (a) they produce the same output, and (b) for every input they transition to the same-or-equivalent states. Equivalent states can be merged with no change in behavior. There's a polynomial-time algorithm: put all states in one set, partition by output behavior, then repeatedly split subsets whose members disagree on where they transition, until nothing splits further. A classic sequence-detector for "010 or 110" collapses from 7 states to 4 this way — same behavior, fewer boxes.

2. Hierarchy (super-states)

Even when states aren't equivalent, closely related ones can be grouped into a super-state, with generalized transitions between super-states. Your TurtleBot's "navigate" super-state might contain follow-path, avoid-obstacle, and recover sub-states; from the outside, the rest of the system just sees "navigating." This is how ROS's SMACH library lets you nest machines. Hierarchy turns an unreadable flat graph of 200 states into a handful of nested machines of a dozen each.

3. Composition

Build big machines by wiring small ones together. Three standard patterns:

In SMACH these are smach.Sequence (cascade), smach.Concurrence (parallel), and smach.Iterator (feedback). The takeaway: complex behavior is composed from simple, individually-verifiable machines — exactly like functions in code.

Chapter 8: Showcase — A TurtleBot Runs the Full Stack

Time to put every piece together. Below is a TurtleBot in a gridworld it has never seen. It must reach the goal flag. It does not know where the obstacles are — it discovers them by sensing, builds an occupancy grid, plans a path, drives along it, and an FSM executive coordinates the whole thing. Every tick is one full see-think-act cycle, and the panel on the right shows what each module of the stack is doing right now.

Watch the discovered grid fill in as the robot moves: grey is unknown, dark is sensed-free, red is sensed-occupied. The dotted line is the current plan; when the robot senses a new obstacle on its path, the Planning card flips to REPLAN, a fresh search floods the grid, and a new path appears. The robot never sees the whole map — it acts on its belief, updates the belief by sensing, and re-plans. That is autonomy. That is the loop from Chapter 1, fully realized.

Now break it on purpose: click cells to build a wall between the robot and the goal. The next REPLAN finds no path, the executive transitions to BLOCKED, and the robot stops cleanly instead of driving into the wall. Handling failure gracefully — "how do I get my TurtleBot to back off after a collision?" — is exactly what the executive layer is for.

Chapter 9: The Map of the Series — Connections & Cheat-Sheet

You now hold the skeleton the whole course hangs on. Here is where each of the next 14 lessons sits on the autonomy stack — your roadmap.

Cheat-sheet

Related lessons on Engineermaxxing

These existing lessons go deeper on topics we touched:

• Robotics Simulation — kinematics, dynamics, and the simulator that powers gridworlds like the showcase.
• The Bayes Filter — the belief-update math behind the occupancy-grid log-odds and the localization jitter.
• Markov Decision Processes — when behavior coordination outgrows hand-built FSMs and becomes a decision problem.
• Extended Kalman Filter & UKF — the localization engines you'll meet in Lesson 13.