Let's Verify Step by Step

Chapter 1: Outcome vs Process

The paper frames the comparison between two types of reward models with precise definitions. Understanding this distinction is key to the entire contribution.

Outcome Reward Model (ORM)

An ORM is trained on (solution, correctness) pairs. For each complete solution, it predicts whether the final answer is correct. Training data is easy to collect: generate solutions, check answers against ground truth, label as correct/incorrect.

Process Reward Model (PRM)

A PRM is trained on (solution, step-level labels) pairs. For each step in the solution, it predicts whether that step is correct. Training data requires human experts to verify each step individually — much more expensive than outcome labels.

How each is used for selection

Both reward models are used with best-of-N selection: generate N solutions for each problem, score them with the reward model, and select the highest-scoring solution.

python
# ORM scoring: one number per solution
def orm_score(solution):
    return orm_model(solution)  # scalar: P(correct answer)

# PRM scoring: one number per step, multiply together
def prm_score(solution):
    steps = split_into_steps(solution)
    step_scores = [prm_model(solution, i) for i in range(len(steps))]
    # Product of step probabilities = P(all steps correct)
    return prod(step_scores)

# Best-of-N selection with either model
def best_of_n(model, problem, N=100, score_fn=orm_score):
    solutions = [model.generate(problem) for _ in range(N)]
    scores = [score_fn(s) for s in solutions]
    return solutions[argmax(scores)]

The product rule for PRMs

When using a PRM to rank solutions, the paper uses the product of all step scores as the overall solution score. This is equivalent to computing P(all steps correct), assuming step correctness is conditionally independent given prior steps.

A solution with 10 steps, each scored 0.95, gets an overall score of 0.95¹⁰ = 0.60. A solution with one bad step (scored 0.3) drops to 0.95⁹ × 0.3 = 0.19. The product is extremely sensitive to individual bad steps — which is exactly what we want.

Chapter 2: PRM Architecture

The PRM is architecturally identical to the base language model. It's a GPT-4-class model fine-tuned to predict step correctness. No special heads, no separate modules — just the same transformer with a classification output.

Training the PRM

The PRM is trained on solutions where each step has been labeled by a human as one of three categories:

The model is trained with a cross-entropy loss to predict these labels at each step boundary. At inference, it outputs a probability that each step is correct.

python
# PRM training (simplified)
class ProcessRewardModel(GPT4Base):
    def forward(self, solution_tokens, step_boundaries):
        # Run the full solution through the transformer
        hidden = self.transformer(solution_tokens)
        # hidden: [seq_len, hidden_dim]

        # At each step boundary, predict correctness
        step_preds = []
        for boundary in step_boundaries:
            h = hidden[boundary]  # [hidden_dim]
            logit = self.classifier(h)  # [3] = pos/neg/neutral
            step_preds.append(logit)

        return step_preds
        # Each step gets a probability of being correct

# Loss: cross-entropy at each step boundary
loss = cross_entropy(step_preds, step_labels)

Step boundary detection

A key engineering detail: how do you split a solution into steps? The paper uses newlines as step boundaries. Each line in the solution is one step. This is simple but effective for mathematical solutions, which naturally separate into one-line-per-step format.

Comparison to training an ORM

The ORM is much simpler to train: generate solutions, check final answers, label as correct/incorrect. No human step labeling needed. The paper trains both on the same base model to ensure a fair comparison.

Chapter 3: PRM800K Dataset

The paper's most enduring contribution may be PRM800K: a dataset of 800,000 step-level human annotations on 75,000 mathematical solutions. This is the largest and most detailed dataset of human reasoning judgments ever created for math.

Dataset statistics

Annotation process

Human annotators with strong math backgrounds reviewed each step of model-generated solutions. For each step, they judged:

Quality and cost

The annotation took approximately 4,000 person-hours. At typical contractor rates, this costs ~$100,000-$200,000. This is expensive but one-time: the resulting dataset enables training PRMs without additional human annotation.

Inter-annotator agreement (measured on a subset) was ~90%, indicating high consistency. Math has the advantage of being objectively verifiable — unlike sentiment or toxicity, "is this algebra step correct?" has a clear answer.

python
# PRM800K data format
sample = {
    "problem": "Solve x^2 - 5x + 6 = 0",
    "solution": [
        {"step": "We need to factor the quadratic.",
         "label": "positive"},
        {"step": "Looking for (x-a)(x-b) where ab=6, a+b=5.",
         "label": "positive"},
        {"step": "(x-2)(x-3) = 0",
         "label": "positive"},
        {"step": "x = 2 or x = 3",
         "label": "positive"}
    ],
    "answer": "{2, 3}",
    "correct": True
}

Chapter 4: Best-of-N Selection

Both ORMs and PRMs are used with best-of-N sampling: generate N candidate solutions for each problem, score them, and submit the highest-scoring one. The paper systematically compares how ORMs and PRMs perform as N increases.

The best-of-N framework

Generate N solutions from the same model (GPT-4) with temperature > 0. Score each with the reward model. Pick the best. This is identical to self-consistency's sampling step, but instead of majority voting on the answer, we use a learned scorer to pick the best solution.

How N affects accuracy

More samples = higher chance that at least one solution is correct AND that the reward model can identify it. The paper evaluates at N = 1, 10, 50, 100, 400, and 1860.

At N = 1 (no selection), both are identical — it's just the base model. As N grows, both improve, but the PRM improves faster. The PRM's advantage grows with N, reaching +5.8 percentage points at N = 1860.

Oracle best-of-N

The paper also reports the "oracle" best-of-N: what accuracy would you get if you had a perfect reward model? At N = 1860, the oracle achieves 96.3% — meaning 96.3% of the time, at least one of the 1860 solutions is correct. The gap between the oracle (96.3%) and the PRM (78.2%) represents the reward model's selection error.

python
# Best-of-N with PRM selection
def best_of_n_prm(generator, prm, problem, N=100):
    solutions = [generator.generate(problem, temp=0.7)
                 for _ in range(N)]

    best_score = -1
    best_sol = None

    for sol in solutions:
        steps = split_steps(sol)
        step_scores = prm.score_steps(problem, steps)
        # Product of step scores = solution score
        sol_score = 1.0
        for ss in step_scores:
            sol_score *= ss
            if sol_score < 0.01:
                break  # early stop: bad step detected

        if sol_score > best_score:
            best_score = sol_score
            best_sol = sol

    return best_sol, best_score

Chapter 5: Results

The paper's main result is unambiguous: process supervision outperforms outcome supervision on MATH, and the advantage is robust across problem difficulties and domains.

Headline numbers

The PRM beats the ORM by +5.8 percentage points at N = 1860. It also substantially beats majority voting (+15.3 points), confirming that learned selection outperforms naive voting for mathematical reasoning.

Performance by difficulty

MATH problems are categorized into 5 difficulty levels. The PRM advantage is largest on the hardest problems:

On easy problems, the base model almost always gets the right answer, so selection barely matters. On hard problems, there are many candidate solutions with correct answers but flawed reasoning — this is exactly where the PRM's ability to check individual steps provides the most value.

Comparison to self-consistency

Self-consistency (majority voting) reaches 62.9% at N = 1860. The PRM reaches 78.2%. The 15-point gap shows that selection quality matters more than selection quantity. A trained PRM is much better at picking the right solution than counting votes.

Chapter 6: Step Scoring Simulator

Let's see process verification in action. This simulator generates candidate solutions for a math problem. Each solution has steps, each step has a PRM score. Watch how the PRM identifies the solution with the best reasoning — and how it catches solutions that get the right answer for the wrong reasons.

Chapter 7: Connections

Process reward models sit at a critical junction in the reasoning model lineage. They formalize the idea that HOW you reason matters as much as WHAT you conclude.

What this paper got right

The fundamental result. Process supervision > outcome supervision. This has been reproduced and extended by every subsequent paper on reasoning verification.

The dataset. PRM800K enabled the community to train PRMs without OpenAI's annotation budget.

What this paper left open

Cost of labeling. 800K step labels cost $100K+. Can we generate step labels automatically? Math-Shepherd (2024) showed yes, using Monte Carlo estimation.

RL training signal. The paper uses PRMs only for best-of-N selection. Using PRM scores as RL rewards could be even more powerful — the "Scaling Test-Time Compute" paper explores this direction.

Self-Consistency — Naive voting that PRMs improve upon. Read the SC lesson →

Scaling Test-Time Compute — Uses PRMs for compute-optimal test-time search. Read the TTC lesson →

DeepSeek-R1 — Shows outcome-only RL can match PRM-guided models. Read the R1 lesson →