AlpacaFarm: Simulating Human Feedback

Chapter 1: Simulated Preferences

The key technical contribution: using API LLMs as simulated annotators that produce preference judgments mimicking human behavior. The setup is straightforward but the details matter enormously.

How simulated preferences work

Given a prompt x and two responses y₁, y₂, the simulated annotator receives:

prompt to GPT-4 (simulated annotator)
"Below are two responses to an instruction. Which is better?"

"Instruction: {x}"

"Response 1: {y₁}"
"Response 2: {y₂}"

"Which response is better? Output only '1' or '2'."

To reduce position bias (LLMs tend to prefer the first option), AlpacaFarm queries each pair twice — once with y₁ first, once with y₂ first — and takes the majority vote. If the two orderings disagree, the pair is treated as a tie.

Calibrating simulated preferences to humans

AlpacaFarm validates the simulated preferences against 2,500 real human annotations. The correlation is measured using:

Individual pairwise agreement (66%) is lower than human-human agreement (76%). But the Elo correlation (0.97) is nearly perfect. This means: on any single comparison, the simulator might disagree with humans ~34% of the time. But averaged over many comparisons, the ranking of models is nearly identical. For RLHF research (where you care about which method is best, not individual annotations), this is sufficient.

Think of it like a noisy thermometer. If your thermometer is off by ±2°C on each reading, you can't trust any single reading. But if you take 100 readings and average them, the average is very close to the true temperature. Similarly, the simulated annotator's individual judgments are noisy, but the average win rate over 805 test prompts is highly reliable.

python
# Simulated preference annotation
import openai

def simulated_preference(prompt, response_a, response_b):
    """Ask GPT-4 which response is better (with debiasing)"""
    # Query 1: A first, B second
    q1 = query_gpt4(prompt, response_a, response_b)
    # Query 2: B first, A second (swap to reduce position bias)
    q2 = query_gpt4(prompt, response_b, response_a)

    # Aggregate: if both agree, confident judgment
    # If they disagree, treat as tie
    if q1 == 'A' and q2 == 'B':  # both say A is better
        return 'A'
    elif q1 == 'B' and q2 == 'A':  # both say B is better
        return 'B'
    else:
        return 'tie'  # conflicting signals → uncertain

What the simulator gets wrong

The simulated annotator has systematic biases that differ from humans:

Chapter 2: The AlpacaFarm Pipeline

AlpacaFarm is not just simulated preferences — it's a complete research framework for RLHF, providing standardized datasets, evaluation protocols, and reference implementations of multiple alignment algorithms.

The complete pipeline

The entire pipeline (steps 2-6) takes ~20 hours on 4 A100 GPUs and costs about $10 in API fees. Compare this to a full human RLHF pipeline: weeks of calendar time and $50K+ in annotation costs. This 45x cost reduction is what enables rapid iteration.

Let's break down the compute time for each step. Step 1 (SFT) is done once and shared across all experiments — about 8 hours on 4 A100s. Step 2 (generate response pairs) takes ~2 hours — you need to generate 2 responses per prompt for 10K prompts. Step 3 (simulated preferences) takes ~1 hour of API calls — GPT-4 evaluates 20K pairs (10K × 2 for debiasing). Step 4 (reward model training) takes ~4 hours. Step 5 (PPO/DPO) takes ~6 hours. Step 6 (evaluation) takes ~30 minutes.

python
# AlpacaFarm pipeline timing breakdown
pipeline = {
    "SFT baseline":           (8, "hours", "one-time"),
    "Generate pairs":          (2, "hours", "per experiment"),
    "Simulated preferences":   (1, "hour",  "per experiment, $10 API"),
    "Train reward model":      (4, "hours", "per experiment"),
    "Run PPO/DPO":             (6, "hours", "per experiment"),
    "Evaluate":                (0.5, "hours", "per experiment"),
}
# Total per experiment (after SFT): ~13.5 hours + $10
# Can run 2 experiments per day on a 4×A100 machine
# In one month: ~60 experiments — enough to thoroughly explore
# the alignment algorithm design space

Reference implementations

AlpacaFarm provides standardized implementations of five alignment algorithms, all using the same base model, data, and evaluation:

Chapter 3: RLHF vs Alternatives

With all five methods implemented under the same framework, AlpacaFarm provides the first truly apples-to-apples comparison. The results challenge some conventional wisdom.

Win rates vs SFT baseline

Why does best-of-n win? Three reasons:

1. No distribution shift. PPO changes the policy during optimization, which can drift away from the reward model's training distribution. Best-of-n samples from the original SFT policy, so the reward model's scores are well-calibrated. This is perhaps the biggest advantage — reward model overoptimization (Goodhart's law) is PPO's Achilles heel.

2. Exploration. With n=16 samples, best-of-n explores a much wider range of responses than PPO (which typically generates 1-2 responses per prompt during training). More samples means higher probability of finding a genuinely excellent response.

3. Simplicity. PPO involves 4 models running simultaneously (policy, reference, reward, value), careful hyperparameter tuning, and complex training dynamics. Best-of-n requires only the SFT model and the reward model, with no training at all. Fewer moving parts means fewer things can go wrong.

python
# Best-of-n sampling: surprisingly powerful
def best_of_n(model, reward_model, prompt, n=16):
    """Generate n responses, return the highest-scoring one"""
    responses = [model.generate(prompt) for _ in range(n)]
    scores = [reward_model.score(prompt, r) for r in responses]
    return responses[np.argmax(scores)]

# Compare with PPO:
# PPO: train for 10K steps, then generate 1 response → ~57% win rate
# BoN-16: no training, generate 16 responses, pick best → ~60% win rate
# BoN wins on quality, but costs 16x more at inference time

# The math: expected quality of max(n samples) scales as
# E[max(X₁,...,Xₙ)] ∝ √(2 ln n) for Gaussian X
# Going from n=1 to n=16: ~2.5x quality improvement
# Going from n=16 to n=256: only ~1.3x more improvement

The caveat: best-of-n is expensive at inference time (you need 16x more generation per query). PPO pays the cost at training time instead. For production systems serving millions of queries, PPO's lower inference cost wins. For research and quality-sensitive applications, best-of-n is the better choice.

Chapter 4: Evaluation

Evaluation is the hardest part of alignment research. How do you measure whether a model is "better aligned"? AlpacaFarm uses a multi-layer evaluation strategy.

Automated evaluation

For rapid iteration, AlpacaFarm uses the simulated annotator (GPT-4) to evaluate model outputs on 805 held-out test prompts. The evaluation computes pairwise win rates: for each prompt, the model's output is compared against a reference (the SFT baseline), and GPT-4 judges which is better.

Human evaluation

To validate the automated evaluation, AlpacaFarm also collects human judgments on a subset. The key finding: simulated evaluation rankings match human rankings almost perfectly (Spearman correlation 0.97). This means you can trust the simulated evaluation for method comparison, even though individual judgments may differ.

What evaluation reveals about alignment methods

Chapter 5: Cost Analysis

The economic argument for AlpacaFarm is compelling. Let's break down the numbers.

When simulation isn't enough

AlpacaFarm acknowledges limitations where simulated feedback falls short:

The recommended approach: use AlpacaFarm for rapid prototyping and method comparison, then validate the final method with a smaller round of real human evaluation.

This "simulate then validate" paradigm has become standard in alignment research. It's analogous to how pharmaceutical companies use computer simulations to screen drug candidates before expensive clinical trials. You run hundreds of cheap simulated experiments, identify the 2-3 most promising approaches, and then invest in expensive human evaluation only for those finalists.

python
# The "simulate then validate" workflow
# Phase 1: Exploration (AlpacaFarm)
#   - Try 50 hyperparameter configurations for PPO
#   - Try 20 reward model architectures
#   - Try 10 dataset compositions
#   - Cost: 80 experiments × $15 = $1,200
#   - Time: 2-3 weeks
#   - Output: top-3 configurations

# Phase 2: Validation (Human eval)
#   - Run top-3 configs with real human annotation
#   - Cost: 3 experiments × $5,000 = $15,000
#   - Time: 2-3 weeks
#   - Output: confirmed best configuration

# Total: $16,200 for a thorough exploration
# Without AlpacaFarm: would need $55K × 80 = $4.4M to explore
# AlpacaFarm reduces exploration cost by ~270×

Chapter 6: Alignment Explorer

Let's bring the full AlpacaFarm results together. This interactive explorer lets you compare all five alignment methods, adjust the number of preference samples, and see how the quality-cost tradeoff shifts.

The Pareto frontier

When you plot all methods in quality-cost space, a clear Pareto frontier emerges:

Chapter 7: Connections

What AlpacaFarm builds on

What came after

The broader significance of AlpacaFarm extends beyond RLHF research. It established a precedent for using LLMs as evaluators and annotators — a practice that has become ubiquitous in AI research. Today, LLM-as-judge is used not just for preference annotation but for code review, essay grading, safety evaluation, and many other tasks where human annotation is expensive. AlpacaFarm was one of the first papers to rigorously validate this approach.

The paper also contributed to the growing understanding that alignment is not a single technique but a design space. There's no single "best" alignment method — the right choice depends on your constraints. PPO is best for high-volume production systems. Best-of-n is best for quality-sensitive applications. DPO (which emerged shortly after) is best for simplicity and stability. AlpacaFarm's standardized comparison framework made it possible to reason about these tradeoffs quantitatively rather than anecdotally.

"AlpacaFarm provides a realistic simulation of the RLHF process... enabling the research community to iterate more quickly on alignment methods."
— Dubois et al., 2023