Forget benchmark percentages. The real question: how long would a task take a human expert? RE-Bench measures AI capability as a time horizon — and it doubles every 7 months.
Imagine someone tells you: "GPT-4 scores 87% on SWE-bench." What does that mean? Can it write a web app? Fix a production bug? Refactor a codebase? You have no idea. The number is meaningless without a human reference frame.
This is the fundamental problem with AI benchmarks. They tell you about relative ordering between models, but nothing about absolute capability in human terms. Benchmarks saturate within months. GLUE was solved in two years. SuperGLUE lasted barely longer. MMLU went from 50% to 90% in under three years. Each time, a new, harder benchmark appears — and the cycle repeats.
METR (Model Evaluation & Threat Research) noticed this gap and asked a simple question: what if we measured AI capability not in percentage points, but in human time?
Each benchmark saturates, then gets replaced. Click through the timeline to see the pattern.
The pattern is always the same: a new benchmark appears, models struggle, rapid improvement follows, the benchmark saturates, everyone moves on. But each benchmark measures something different, so you cannot draw a single trend line through them. You cannot answer the question that actually matters: how capable are AI systems right now, in terms a human can understand?
Here is the core idea, in one sentence:
This is inspired by human psychometrics. When you take an IQ test, the score is calibrated against human performance — it tells you where you stand relative to people, not just how many questions you got right. METR does the same thing for AI: calibrate against timed human experts.
The key metric is the 50%-task-completion time horizon: the duration of tasks that the AI can complete with 50% success rate. If an AI's time horizon is 110 minutes, it means:
Why 50%? It is a natural midpoint. Below 50%, the model is unreliable. Above 50%, it is fairly dependable. The 50% threshold gives you the frontier of what the model can barely do.
The beauty of this framing: it turns AI progress into a single, interpretable number that grows over time. And that number, it turns out, follows a strikingly clean exponential trend.
Let us formalize how the time horizon is actually computed. You need three ingredients: a set of tasks, human completion times for those tasks, and AI success rates on those tasks.
Each AI attempt on a task is converted to a binary outcome: success or failure. Some tasks are naturally binary ("did the shell script run correctly?"). Continuously scored tasks use a threshold based on human performance — if a human working for 7-9 hours achieves a certain score, the AI must match it to count as "success."
Each task has a human time rating ttask, defined as the geometric mean time of successful human baselines. This is how long a skilled domain expert takes to complete the task. Tasks range from 1 second (trivial shell commands) to 8+ hours (complex ML research engineering).
Now we fit a model that predicts success probability as a function of task length:
where σ is the sigmoid function, hagent is the agent's time horizon (what we are solving for), and βagent controls how steeply success drops off with task length. When ttask = hagent, the argument to σ is zero, so psuccess = 0.5. That is exactly the definition: hagent is the task duration at which success probability is 50%.
Drag the time horizon slider to see how the success curve shifts. At hagent, success is exactly 50%.
This approach is inspired by Item Response Theory (IRT) from psychometrics — the same framework used to calibrate standardized tests like the GRE. In IRT, each test item has a difficulty parameter and each test-taker has an ability parameter. Here, task duration replaces item difficulty, and the time horizon replaces test-taker ability.
The task suite must cover a wide range of difficulty. Too easy and you cannot distinguish frontier models. Too hard and older models score zero across the board. METR combines three datasets to span nearly the entire difficulty spectrum:
These are the shortest tasks — single-step operations that a software developer does hundreds of times a day. They take 1 second to 30 seconds for a human:
These tasks are crucial because they let the researchers measure pre-2023 models like GPT-2 and GPT-3, which cannot do anything more complex.
The middle range — real software tasks from 1 minute to 30 hours:
The hardest tasks — all rated at 8 hours. These are genuine ML research engineering problems: optimize a neural network for efficiency, implement a complex algorithm from a paper description, debug a distributed training pipeline.
170 tasks spanning 5 orders of magnitude in human time. Each dot is a task. Hover to see examples.
In total: 170 tasks, all automatically scored (binary or threshold-based), covering software engineering, ML research, and cybersecurity domains.
This is the headline result. When you plot each model's 50% time horizon against its release date, the points fall on a clean exponential line.
Each point is a frontier model. The line is an exponential fit. Drag the extrapolation slider to see future predictions.
The actual data points from the paper, in chronological order:
The regression is done in log-space: log(time horizon) = α + β · release_date. The fit gives a doubling time of 207 days (95% CI: 166-240 days) — roughly 7 months. The confidence interval comes from a three-level hierarchical bootstrap over task families, tasks within families, and individual runs.
One notable outlier: o3 sits above the trend line (p = 0.006), suggesting the trend may have accelerated in late 2024 and early 2025. The 2024-2025 only trend shows a doubling time closer to 5 months.
A number going up is interesting. Understanding why it goes up is useful. METR manually inspected 31 failed runs from GPT-4 (November 2023) and 32 failed runs from o1 (September 2024) to categorize what went wrong.
METR also scored tasks on 16 "messiness" factors — things like resource constraints, dynamic environments, lack of clear feedback, and requirement for high reliability. These factors represent the gap between benchmarks and real-world work.
Models perform worse on messier tasks, as expected. But the key finding: the growth trend is similar for both clean and messy tasks. The time horizon doubles at roughly the same rate regardless of messiness. This is cautiously encouraging for external validity — the trend does not appear to be driven solely by improvements on artificially clean benchmark tasks.
The whole methodology rests on one thing: accurate human timing. If your human baselines are wrong, your time horizons are wrong. METR invested heavily here.
Not random Mechanical Turk workers. These are skilled professionals in software engineering, ML, and cybersecurity. Most attended world top-100 universities. Average of ~5 years of relevant experience. They are domain experts but do not have task-specific context — they know the field but have not seen the specific codebase or problem before. Think: a strong senior engineer dropped into an unfamiliar project.
Over 800 human baselines totaling 2,529 hours of work. That is over 100 person-days of skilled engineers solving problems while being timed. Broken down:
Humans vary enormously. On the same task, one expert might take 30 minutes and another 3 hours. METR uses the geometric mean of successful completion times as the task's duration rating. The geometric mean is more robust to outliers than the arithmetic mean — it downweights the occasional expert who takes 10x longer because they went down a rabbit hole.
Simulated distribution of human completion times for a 1-hour task. The geometric mean (orange) is the task's official duration.
Let us look at the full model lineup. METR evaluated 12 frontier models (and 4 near-frontier) released between 2019 and 2025, running 8 attempts per model-task pair.
When you simply average success rate across all 170 tasks (weighted by inverse square root of task family size to reduce the influence of large families):
Success rate drops exponentially with task duration (R² = 0.83). Toggle between models.
There is a strong negative correlation between task length and model success rate. The regression y = -0.07x + 0.66 (where x is log human time) gives R² = 0.83. In plain terms: for every 10x increase in task duration, success rate drops about 7 percentage points. This is remarkably consistent across models — the curve just shifts right as models improve.
The 80% time horizon (tasks the model completes 80% of the time) is 4-6x shorter than the 50% horizon. For o3: 50% horizon is ~110 minutes, but 80% horizon is only ~20 minutes. This gap reveals a key limitation: even frontier models that sometimes succeed at difficult tasks cannot reliably perform tasks of moderate length. Reliability lags capability significantly.
If the doubling trend continues, where does it lead? The paper extrapolates toward a critical threshold: one month (167 working hours) of human time.
One month is approximately the onboarding period for a new hire. An AI that can do one-month tasks can:
At the 2019-2025 doubling rate (7 months), the 50% time horizon reaches 167 hours between mid-2028 and mid-2031 (80% confidence interval, central estimate mid-2029). If the accelerated 2024-2025 trend continues (doubling every ~5 months), it could arrive as early as late 2026 or 2027.
The horizontal line is 167 working hours (1 month). Toggle between the full trend and the accelerated 2024-2025 trend.
The paper is careful to flag several reasons the extrapolation might be too aggressive:
The paper discusses implications for AI safety directly. An AI with a month-long time horizon could potentially: develop novel malware, conduct sophisticated social engineering campaigns, or autonomously discover and exploit zero-day vulnerabilities. METR argues this makes capability measurement critical for safety — you cannot set appropriate guardrails if you do not know where the frontier is.
RE-Bench / METR's time horizon paper does something benchmarks rarely do: it gives us a common currency for measuring AI progress over time. Instead of comparing apples (GLUE scores) to oranges (SWE-bench percentages), we can now say "AI capability doubles every 7 months" — a single number that spans from GPT-2 to o3 and beyond.