Reflexion: Verbal Reinforcement Learning

Chapter 1: Self-Reflection

After a failed attempt, Reflexion asks the agent to do something humans do naturally: think about what went wrong. Given the task, the trajectory (every action and observation from the failed attempt), and the final outcome, the agent produces a verbal reflection — a free-text diagnosis of the failure and a strategy for the next attempt.

What makes a good reflection? The paper doesn't specify a fixed format — it uses few-shot examples of (trajectory, outcome) → reflection pairs to teach the LLM what a useful critique looks like. Good reflections tend to:

• Identify the specific action that caused failure ("I searched for 'climate change causes' too broadly instead of 'CO2 warming mechanism evidence'").
• Propose a concrete alternative ("Next time, search for the specific mechanism, not the general topic").
• Avoid generic statements ("I should try harder") that carry no actionable information.

The reflection is written by the same LLM that performed the task — there is no separate critic model. This works because LLMs have excellent meta-cognitive capabilities: they've seen enormous amounts of human writing about mistakes, corrections, and strategies. Given a failure, they can identify plausible explanations and better approaches.

python (pseudocode)
def generate_reflection(task, trajectory, outcome, llm):
    # Build the reflection prompt
    prompt = REFLECT_TEMPLATE.format(
        task=task,
        trajectory=format_traj(trajectory),   # all thought/action/obs
        outcome=outcome,                          # e.g., "FAILED: wrong answer"
        examples=FEW_SHOT_REFLECTIONS            # 1-3 examples of good critiques
    )
    # LLM generates verbal critique
    reflection = llm.generate(prompt)
    return reflection  # e.g., "I searched too broadly. Next time focus on..."

Chapter 2: The Verbal RL Loop — Interactive

Reflexion wraps any base agent (like ReAct) in an outer loop. The inner loop is the agent solving the task — taking actions, observing outcomes, thinking through steps. The outer loop handles failure: generate a reflection, store it in memory, reset the environment, and try again.

Watch the full loop unfold below. Each trial shows what the agent does differently because of its accumulated reflections. You can control how many trials run and see the memory fill up.

python
def reflexion_agent(task, env, llm, max_trials=4):
    memory = []  # persistent verbal memory across trials

    for trial in range(max_trials):
        # Build context: task + all past reflections
        context = build_context(task, memory, FEW_SHOT)

        # Run inner agent (e.g. ReAct) to completion
        trajectory, success = run_agent(llm, env, context)

        if success:
            return trajectory  # done

        # Generate verbal reflection from failed trajectory
        reflection = reflect(llm, task, trajectory)
        memory.append(reflection)  # accumulate in memory
        env.reset()               # reset environment, not memory

    return "Max trials reached"

Chapter 3: Memory

Reflexion uses three types of memory, which the paper calls short-term, long-term, and episodic. Understanding the distinction helps clarify both how Reflexion works and where it can fail.

Memory Limits and What to Do About Them

The context window is finite. With long reflections and many trials, the episodic memory can overflow. Reflexion handles this with a simple priority: keep the most recent reflections, drop older ones if needed. Alternatively, a summarization step can compress multiple reflections into one.

The episodic memory architecture also explains why Reflexion scales gracefully: storing a reflection costs only tokens, not compute. Adding a third trial's reflection is as cheap as adding a sentence to a document. Compare this to RL, where each additional episode requires a full forward+backward pass through the model.

Chapter 4: vs Fine-Tuning

The natural competitor for Reflexion is fine-tuning: update the model's weights using the failed trajectory as a negative training signal (or the successful trajectory as a positive one). This is the standard RL approach. Reflexion deliberately avoids this — why?

But Reflexion has its own failure modes compared to fine-tuning:

• Context window limit: After enough trials, the reflections fill the context. Fine-tuning encodes knowledge in weights, which don't have this limit.
• Cross-task transfer: Reflexion's reflections are task-specific. Fine-tuning on 1000 similar tasks builds general knowledge that transfers. Reflexion restarts from scratch on a new task.
• Hallucinated reflections: The LLM might generate an incorrect diagnosis of why it failed. Fine-tuning uses actual gradient signals — the loss function is an honest measure of error.

Chapter 5: Applications

Reflexion was evaluated on three very different task families. The diversity is the point: verbal self-reflection is a general mechanism that works wherever (1) there is a clear success/failure signal and (2) the failure modes can be diagnosed in words.

Sequential Decision Making: AlfWorld

AlfWorld is a text-based household environment. 134 test tasks from 6 categories: pick up, put in, cool, heat, clean, examine. The agent must navigate rooms, interact with objects, and achieve multi-step goals in natural language. Failure modes include wrong room, wrong object, wrong sequence.

Reflexion's reflections on AlfWorld failures look like: "I picked up the pan before heating it, but the task requires heating a pan already in the microwave. I should first identify which pan is already placed, then operate the appliance." This kind of sequential reasoning correction transfers directly to the next attempt.

Reasoning: HotpotQA

HotpotQA requires multi-hop reasoning over Wikipedia. Failure mode: searching too broadly or in the wrong direction. Reflexion reflects: "My search for 'French revolution leaders' was too broad. I need the specific person who was executed on Thermidor 9 — searching 'Robespierre execution date' directly would have worked." The next trial uses a targeted search strategy.

Programming: HumanEval + LeetCode

Code generation is perhaps the most natural fit. Unit tests provide a precise, binary signal. When a test fails, the failure message contains the exact input, expected output, and actual output. A reflection might say: "My solution fails for empty lists because I don't check for the edge case before calling max(). I need to add an early return for len(nums) == 0." The next attempt includes this check.

Chapter 6: Results

Across all three task families, Reflexion shows consistent improvement over the base agent (ReAct or simple prompting) with only 1-4 additional trials.

The AlfWorld improvement is the most striking. A +26% absolute gain from verbal self-reflection alone, across 134 household tasks. The paper attributes this to AlfWorld's structured failure modes: the same types of mistakes (going to the wrong room, not recognizing synonyms for objects) recur across tasks. Reflexion identifies these patterns and eliminates them.

Failure analysis from the paper: tasks where Reflexion fails tend to involve hallucinated reflections ("I think I failed because X" where X is wrong), multi-step compounding errors where no single reflection captures the root cause, and tasks requiring genuine knowledge the model simply doesn't have. Verbal reflection can correct strategy; it cannot supply missing knowledge.

Chapter 7: Connections

Reflexion sits at the intersection of RL theory, agentic AI, and cognitive science. Its descendants and relatives reveal how the field has been building toward language-native learning.