Can frontier AI models do the actual work of industry professionals? A benchmark of 1,320 tasks from 44 occupations across 9 GDP sectors, graded by experts with 14 years average experience. The answer: they are getting close.
You want to know whether AI can do economically valuable work. Not solve math puzzles. Not ace multiple-choice exams. Not write hello-world programs. You want to know: can a model write a legal brief that a partner at an AmLaw 100 firm would accept? Can it build a financial forecast that a 20-year Wall Street veteran would trust? Can it produce a marketing deck that a retail executive would actually present to her board?
Every existing AI benchmark fails this test. MMLU measures academic knowledge. HumanEval measures coding. GPQA measures graduate-level reasoning. These are all academic proxies for capability. They tell you the model is smart. They do not tell you the model is useful.
The problem runs deeper than task selection. Consider what makes professional work hard:
Historical precedent makes this urgent. Electricity took 40 years to reshape manufacturing. Computers took decades to show up in productivity statistics (the Solow Paradox). We cannot wait for lagging indicators like adoption rates or GDP growth to tell us whether AI is economically transformative. We need a leading indicator: directly measure what models can do on real work, right now.
GDPval's design principle is deceptively simple: map tasks to actual GDP-contributing occupations, then have the professionals who do those jobs grade the outputs.
This is not "find some hard tasks and see if the model can do them." It is a systematic, top-down approach to covering the U.S. economy:
The brilliance of this design is in the grading. Most benchmarks grade against a fixed correct answer, which works for math but fails for subjective work. GDPval uses pairwise expert comparison: show a grader two unlabeled deliverables (one from a model, one from a human expert) and ask which is better. This mirrors how real work is evaluated — by peers and supervisors using professional judgment.
The five-step process from GDP sectors to graded model outputs. Click each stage to see details.
GDPval covers 1,320 tasks in the full set (at least 30 per occupation) and open-sources a 220-task gold subset (5 per occupation). The tasks span 44 occupations across 9 sectors. Let's look at what this actually covers.
| Sector | % GDP | Top Occupations (sample) |
|---|---|---|
| Real Estate & Leasing | 13.8% | Property Managers, RE Agents, RE Brokers |
| Government | 11.3% | Compliance Officers, Admin Managers, Social Workers |
| Manufacturing | 10.0% | Production Supervisors, Industrial Engineers, Mechanical Engineers |
| Prof./Scientific/Tech | 8.1% | Software Developers ($239B), Lawyers ($137B), Accountants ($135B) |
| Healthcare | 7.6% | Registered Nurses ($323B), Medical Managers, Nurse Practitioners |
| Finance & Insurance | 7.4% | Financial Managers ($148B), Customer Service Reps, Securities Agents |
| Retail Trade | 6.3% | General/Operations Managers ($477B), Retail Supervisors, Pharmacists |
| Wholesale Trade | 5.8% | Sales Reps ($103B), Sales Managers ($97B) |
| Information | 5.4% | Producers/Directors, Editors, Journalists |
Notice the compensation numbers. The Professional/Scientific/Technical sector alone has software developers earning $239 billion in aggregate compensation. These are not toy tasks — they represent the most economically valuable knowledge work in the United States.
Each task has two components: a request (the assignment, often with reference files like spreadsheets, PDFs, images, or audio) and a deliverable (the work product). Experts classified each task against O*NET occupational tasks to ensure coverage. The dataset covers 208 unique O*NET tasks, 25 skills, and 26 work activities.
| Metric | Mean | Median | Max |
|---|---|---|---|
| Quality rating (1-5) | 4.47 | 4.50 | 5.00 |
| Difficulty (1-5) | 3.32 | 3.00 | 5.00 |
| Representativeness (1-5) | 4.50 | 5.00 | 5.00 |
| Time to complete (hours) | 9.49 | 5.00 | 100 |
| Dollar value | $398 | $175 | $4,114 |
Each task went through a multi-stage quality control pipeline: automated model screening, then at minimum three human expert reviews (averaging five reviews per task). Experts iteratively revised tasks between review stages. The result: 89% of tasks were rated as well-specified by occupational graders.
The 9 GDP sectors and their relative economic weight. Hover over sectors to see occupation count and compensation.
Grading is the hardest part of GDPval. You cannot auto-grade a legal brief the way you auto-grade a math problem. The paper uses two grading approaches, and the tension between them reveals something important about evaluating real work.
Experts in the relevant occupation are shown a request, reference files, and two or more unlabeled deliverables. They rank them and provide detailed justifications. The comparison is blinded — no model names, no labels. (Though stylistic tells remain: OpenAI models favor em dashes, Claude uses first-person phrasing, Grok occasionally refers to itself.)
On average, grading a single comparison takes over an hour. The grader must read the request, examine reference files, carefully review each deliverable, and write a justification. This is expensive: the average review costs $86 (based on median occupational wages times review time of 109 minutes).
For the open-sourced gold subset, the team trained a grading model to mimic expert pairwise comparisons. It achieves 66% agreement with human experts — only 5% below human inter-rater agreement of 71%.
The human baseline deliverables were created by the same pool of industry experts. Average completion time: 404 minutes (~6.7 hours). Average cost per task: $361 (based on BLS median hourly wages). These are experienced professionals at top firms — Goldman Sachs, Apple, JPMorgan, Lockheed Martin, BBC News, and dozens more.
Human experts agree with each other 71% of the time. The automated grader agrees with humans 66% of the time. This 5% gap is small given the inherent subjectivity of professional work.
Here is the headline result that makes GDPval matter: frontier models are approaching parity with industry experts on real-world economic tasks.
The paper evaluated seven models using blind pairwise comparison by professional experts: GPT-4o, o4-mini, o3, GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, and Grok 4. The results:
| Model | Win Rate | Win + Tie Rate | Key Strength |
|---|---|---|---|
| Claude Opus 4.1 | ~32% | 47.6% | Aesthetics (formatting, layout, PDFs, slides) |
| GPT-5 (high) | ~30% | ~45% | Accuracy (instructions, calculations, text) |
| Grok 4 | ~27% | ~42% | Fewer accuracy errors |
| Gemini 2.5 Pro | ~25% | ~40% | Broad capability |
| o3 | ~24% | ~38% | Reasoning-heavy tasks |
| o4-mini | ~20% | ~34% | Speed/cost efficiency |
| GPT-4o | ~12.5% | ~25% | Baseline comparison |
Just as striking: OpenAI frontier model performance improved roughly linearly over time. Not exponential. From GPT-4o to o4-mini to o3 to GPT-5, each generation gained roughly the same increment. This is a crucial finding — it means we can somewhat predict when models will reach full expert parity on GDPval tasks.
The paper built a clustering pipeline to analyze why experts preferred or rejected deliverables. The top failure modes across all models:
When GPT-5 failures were categorized by severity: most were "acceptable but subpar" (the model produced usable work, just not as good as the human). About 29% were "bad" or "catastrophic," with only ~3% truly catastrophic (harmful or dangerously wrong).
Drag the slider to see how OpenAI frontier model performance improved roughly linearly. The dashed line at 50% represents expert parity.
Even when a model doesn't beat the human expert, it might still be economically useful. The question is not "is the model as good as a human?" but "is model + human oversight cheaper and faster than a human alone?"
For the gold subset, the paper computes three key quantities per task:
The paper analyzes increasingly realistic workflows:
| Model | Win Rate | Speed Improvement | Cost Improvement | ||||
|---|---|---|---|---|---|---|---|
| Naive | Try 1x | Try Nx | Naive | Try 1x | Try Nx | ||
| GPT-4o | 12.5% | 327x | 0.87x | 0.46x | 5172x | 0.90x | 0.53x |
| o4-mini | 29.1% | 186x | 1.02x | 1.06x | 1265x | 1.06x | 1.22x |
| o3 | 35.2% | 161x | 1.08x | 1.28x | 480x | 1.13x | 1.47x |
| GPT-5 | 39.0% | 90x | 1.12x | 1.39x | 474x | 1.18x | 1.63x |
Notice the critical threshold: GPT-4o with 12.5% win rate actually makes workers slower (0.87x) and more expensive (0.90x) in the "try once" scenario. The review overhead eats any savings. But once win rate crosses roughly 25-30%, the economics flip positive. Models become genuinely helpful even with human oversight.
Adjust the model win rate to see how the "try N times" economics change. The break-even point is where the improvement ratio crosses 1.0x.
Three levers improve model performance on GDPval: more reasoning, better prompts, and improved scaffolding. The paper tests each one.
Running o3 and GPT-5 at low, medium, and high reasoning effort shows a clear trend: more thinking time improves quality. This is not surprising given the nature of the tasks — a 7-hour professional task benefits from deeper reasoning about what the client wants, how to structure the deliverable, and whether the output actually addresses the prompt.
Many GPT-5 failures were obvious formatting errors — black squares in PDFs, broken slide layouts, non-standard Unicode characters. The team created a general prompt that instructs the model to:
Results were dramatic. Black-square artifacts: eliminated entirely (from affecting over half of PDFs to zero). Egregious formatting errors in PowerPoints: reduced from 86% to 64%. Models using their multi-modal capabilities to inspect deliverables: jumped from 15% to 97%. Overall win rate: +5 percentage points.
The team also enabled GET requests in the container, performed best-of-4 sampling with a GPT-5 judge picking the best output, and combined these with the prompt improvements. Each layer stacked: reasoning effort + prompt engineering + scaffolding all contributed independently.
In a separate experiment, they created under-contextualized versions of GDPval tasks — prompts that were only 42% the token length of the originals, omitting details about where to find data in reference files, how to approach the problem, and formatting expectations. Model performance degraded significantly. This reveals that much of professional skill is knowing how to figure out context — understanding what a client actually wants when the brief is vague.
Toggle each lever on/off to see its estimated effect on model performance. All three stack.
The aggregate win rates mask enormous variation across sectors and occupations. Some sectors are nearly solved; others remain stubbornly hard.
Three sectors stand out where the strongest models approach expert-level performance:
Some sectors show low win rates across all models:
Performance varies dramatically by file format:
Win rates are highest for shorter tasks (0-2 hours) and decline steadily as completion time increases. This makes intuitive sense: longer tasks require more planning, more context management, and more sustained coherence — all areas where models still struggle relative to humans.
Relative model performance across sectors. Warmer colors indicate higher win rates. Hover to see details.
"Approaching expert quality" sounds like a headline. But what does it actually mean for the economy and for workers?
Nearly half the time, a model matches or beats a 14-year veteran. But this is an average across 220 tasks in 44 occupations. In some occupations it is much higher; in others, much lower. And even a 47.6% rate means the model still loses more than half the time. This is not "AI replaces experts" — it is "AI assists experts in a meaningful fraction of their work."
The cost/speed analysis from Chapter 5 is the real economic story. Even with review overhead and human fallback:
GDPval finds that OpenAI model performance improves roughly linearly over time, not exponentially. This is significant: linear improvement means full expert parity is predictable but not imminent. If models gain ~5-8 percentage points per generation and a generation takes 6-12 months, reaching 50%+ average win rate (true parity) is likely several years away for the hardest tasks.
But this also means the improvement is steady and reliable. Unlike exponential curves that might plateau suddenly, a linear trend suggests consistent engineering progress. Every generation makes AI slightly more useful for a slightly broader set of tasks.
GDPval only covers self-contained, digital knowledge work tasks. It does not test:
These limitations mean GDPval likely overestimates AI capability for the messiest, most interactive real-world work, while accurately measuring capability on well-specified digital deliverables.
GDPval sits at the intersection of AI evaluation and labor economics. Here are the key threads that connect it to the broader research landscape.
The open-sourced gold subset of 220 tasks with an automated grader at evals.openai.com means anyone can run their model against real economic tasks. As models improve, the benchmark does not saturate — you simply compare against increasingly strong baselines. The pairwise comparison design ensures there is always room to measure improvement.