The Complete Guide to Offline RL

What You'll Master

RL Recap & Value Functions

Before we dive into offline RL, we need the machinery of value functions and temporal difference learning. These are the building blocks every offline method uses to extract signal from a fixed dataset.

Fitting Value Functions from Data

Given a dataset of transitions (s, a, r, s'), how do we learn V or Q?

Online RL Algorithm Summary

Interactive: hover to see V(s) and Q(s,a) for a simple gridworld. Red = low value, green = high value.

Why Offline RL?

You're an engineer at a hospital. You have five years of patient treatment records — thousands of trajectories of diagnoses, treatments, and outcomes. You want to learn an improved treatment policy. But you can't just "try random actions" on real patients to explore. Online RL would require experimenting with people's lives.

This is the core motivation for offline reinforcement learning (also called batch RL): learning a policy from a fixed dataset of previously collected experience, without any further interaction with the environment.

Where Offline Data Comes From

Online vs. Offline: The Fundamental Difference

In online RL, the agent collects data, updates its policy, then collects more data with the improved policy. The data distribution improves over time because the agent explores more promising regions.

In offline RL, the dataset is fixed. The policy was collected by π_β — which might be suboptimal, might cover only a small part of state space, might have dangerous gaps. You must learn the best policy you can from whatever data you got.

Left: Online RL iterates between data collection and policy improvement. Right: Offline RL learns from a fixed dataset only.

Data Stitching

Why not just do imitation learning on the best trajectories? After all, if the dataset contains some good trajectories, can't we just clone those?

The answer reveals the deepest insight of offline RL: data stitching. The optimal policy might never appear as a complete trajectory in the data, but it might be constructible from pieces of different trajectories.

TD Learning Enables Stitching, MC Does Not

The ability to stitch comes from how we estimate value.

Monte Carlo estimates V(s₃) by averaging the returns of trajectories passing through s₃. Trajectory A gives return 3 from s₃. Trajectory B gives return 10 from s₃. MC average: V(s₃) ≈ 6.5. But crucially, MC assigns this value to s₃ independently of how you arrived there — and it can't propagate this back to s₁ because no single trajectory achieves return 10 from s₁.

Temporal Difference propagates value through the Bellman equation: V(s₁) = r + γ V(s₃). Since V(s₃) is high (because from s₃ you can reach s₉), TD assigns high value to s₁ — even though no trajectory starting at s₁ ever reached s₉. TD stitches across trajectories via the Bellman backup.

Data stitching: TD combines segments from different trajectories (blue and orange) to find the optimal path (green).

Distribution Shift Deep Dive

Here's the fundamental problem. You have a dataset collected by π_β. You want to learn a policy π_θ that's better than π_β. To improve, you need to evaluate actions that π_β never took. But your Q-function was only trained on actions that π_β did take. You're asking the Q-function to extrapolate to regions it's never seen.

The Off-Policy Actor-Critic on Static Data

Consider running a standard off-policy actor-critic (like SAC) on a fixed dataset. The algorithm has two components:

The actor finds actions that maximize Q. The critic bootstraps from Q(s', a') where a' comes from the actor. But a' is the action the learned policy would take — not the action in the dataset. If π_θ ≠ π_β, then a' is likely OOD.

The Overestimation Feedback Loop

Here's where it gets catastrophic. It's not just that Q is wrong for OOD actions — the error compounds:

Q-value overestimation: blue = true Q values (in-distribution), red = estimated Q on OOD actions. Click "Run" to watch the feedback loop explode.

The Core Challenge of Offline RL

Every offline RL algorithm is, at its core, a different answer to one question: how do we prevent the policy from exploiting unreliable Q-estimates on OOD actions?

Three families of solutions:

Filtered Behavior Cloning

The simplest offline RL algorithm. So simple it barely counts as RL — but it's a strong baseline that many complex methods fail to beat.

The idea: rank all trajectories in D by their total return. Keep only the top k%. Do imitation learning on this filtered subset.

Filtered BC: slide k% to see how filtering threshold affects learned behavior. Only shaded trajectories are imitated.

Advantage-Weighted Regression

Filtered BC is binary: keep or discard. But some trajectories are slightly good, some are very good. Shouldn't better trajectories get more weight? AWR makes the weighting continuous.

The Core Idea

Instead of hard filtering, weight each transition (s, a) by exp(A(s,a) / α) where A is the advantage and α is a temperature:

Why Exponential Weighting?

Computing the Advantage

AWR (original version) uses Monte Carlo returns for the advantage:

AWR weighting: adjust temperature α to see how exponential weights concentrate on high-advantage actions.

AWAC: Advantage-Weighted Actor-Critic

AWR's fatal flaw: MC returns prevent stitching. AWAC fixes this by replacing MC with TD-learned Q-values.

The AWAC Objective

The Q-function Update

Here's where AWAC gets subtle. The Q-function is updated with standard TD:

The next-state action a' comes from the learned policy π_θ, not the behavior policy. This means Q evaluates π_θ (the thing we're improving), not π_β (the thing that collected data).

AWAC combines TD-learned Q-values (enabling stitching) with constrained policy updates (preventing OOD exploitation).

Implicit Q-Learning (IQL)

AWAC still has a weakness: it queries Q(s', a') where a' comes from the learned policy. If π_θ occasionally picks OOD actions, the Q-function gets corrupted. Can we build a method that never evaluates Q on actions outside the dataset?

The Three-Step IQL Update

IQL separates the update into three decoupled steps, each using only in-distribution quantities:

The Magic: Expectile Regression

Expectile regression: adjust τ to see how V(s) shifts from the mean toward the maximum of in-data Q-values.

python
import torch
import torch.nn as nn

class IQL:
    def __init__(self, q_net, v_net, policy, tau=0.7, beta=3.0, gamma=0.99):
        self.q = q_net       # Q(s, a) -> scalar
        self.v = v_net       # V(s) -> scalar
        self.pi = policy     # pi(a|s) -> distribution
        self.tau = tau
        self.beta = beta
        self.gamma = gamma

    def expectile_loss(self, diff):
        # Asymmetric squared loss
        weight = torch.where(diff > 0, self.tau, 1 - self.tau)
        return (weight * diff**2).mean()

    def update_v(self, s, a):
        # Step 1: V toward high-Q actions via expectile regression
        with torch.no_grad():
            q_target = self.q_target(s, a)
        v_pred = self.v(s)
        loss = self.expectile_loss(q_target - v_pred)
        return loss

    def update_q(self, s, a, r, s_next):
        # Step 2: Standard TD using V (NO max, NO policy query)
        with torch.no_grad():
            target = r + self.gamma * self.v(s_next)
        q_pred = self.q(s, a)
        loss = ((q_pred - target)**2).mean()
        return loss

    def update_policy(self, s, a):
        # Step 3: AWR-style extraction
        with torch.no_grad():
            adv = self.q_target(s, a) - self.v(s)
            weights = torch.exp(adv / self.beta)
            weights = weights / weights.sum()  # normalize
        log_probs = self.pi.log_prob(s, a)
        loss = -(weights * log_probs).sum()
        return loss

Summary & Connections

The Progression of Ideas

The Three Solutions to Distribution Shift

Method comparison: each approach's tradeoff between stitching ability (performance ceiling) and safety (resistance to OOD errors).