AI Agent Benchmarks Are Breaking in 2026

by

Key Takeaways

AI agent benchmarks soared between 2025 and 2026, but those numbers no longer guarantee real-world reliability. In 2026, Berkeley researchers cracked eight major benchmarks — including SWE-bench and OSWorld — to near-perfect scores without solving a single task, and OpenAI retired SWE-bench Verified after finding contamination. A single-attempt pass@1 score cannot capture how often an agent fails on repeated runs, and capability ranks diverge from safety ranks. In practice, read agent scores down for contamination, repeat-run reliability, and unsafe-action rates rather than taking them at face value.

  • OSWorld human baseline 72.36% vs 12.24% for the best model at launch (369 tasks)
  • Berkeley scored near-100% on 8 benchmarks without solving a task (a 10-line conftest.py = 100% on SWE-bench)
  • OpenAI retired SWE-bench Verified on Feb 23, 2026 — 60%+ of audited tasks flawed, contamination confirmed
  • What to track: contamination-resistant scores (SWE-bench Pro) and the pass@1-vs-repeat reliability gap

72.36%. That’s how often a human finishes ordinary desktop tasks on OSWorld — a benchmark for AI agents driving real apps. When the test launched, the best model managed just 12.24% of the same work. By 2026, that gap had all but closed.

On paper, autonomous agents now rival people. Yet in the same year, the foundation under those scores cracked: Berkeley researchers cleared eight major benchmarks without solving a single task, and OpenAI scrapped the coding benchmark it had leaned on, citing contamination. The numbers kept climbing while what they measured grew murkier.

AI-generated illustration symbolizing AI agent benchmarks — a glowing blue circular performance gauge shattering into glass shards on a dark background
A near-full score breaking apart under scrutiny — AI-generated illustration (OpenAI gpt-image-1-mini)

AI Agent Benchmarks Cleared the Human Bar — So Why Does Reality Lag?

Start with the raw scoreboard. OSWorld (a computer-use benchmark where agents drive real operating systems and apps) spans 369 tasks, and humans complete 72.36% of them on average. The best model scored just 12.24% at launch, yet Stanford’s 2026 AI Index flagged OSWorld and SWE-bench as the steepest-climbing curves of the year.

SWE-bench Verified — which measures whether an agent actually fixes GitHub issues — saw top scores jump from roughly 60% in 2024 to nearly 100% in 2025. Humanity’s Last Exam, long treated as the hardest test, climbed from OpenAI o1’s 8.8% in 2025 to 38.3%, and by April 2026 top models like Claude Opus 4.6 and Gemini 3.1 Pro cleared 50%. By the numbers, the leap is undeniable.

The real question is whether those curves mean agents actually do that much work. As scores approach 100%, it gets harder to tell whether the number measures capability or the test’s blind spots.

The Benchmarks Broke: How Berkeley Scored Near-100% Solving Nothing

In April 2026, a UC Berkeley RDI team (Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song) published a jarring result. Across eight leading agent benchmarks — SWE-bench Verified and Pro, WebArena, Terminal-Bench, FieldWorkArena, OSWorld, GAIA, and CAR-bench — they hit near-perfect scores without solving a single task.

The methods were almost insultingly simple. On SWE-bench Verified, a 10-line conftest.py (a Python test-config file) forced every test to “pass” for a 100% score; on OSWorld, manipulating the virtual-machine state and pulling public answer files yielded 73%. FieldWorkArena passed an agent that did nothing but output an empty pair of braces.

The team’s diagnosis was structural. “In SWE-bench, Terminal-Bench, and OSWorld, the agent’s code runs in the same environment the evaluator inspects.” When the actor stands on the same stage the judge is watching and can touch the scoreboard, a high score may reflect a flaw in the stage design rather than real skill.

BenchmarkWhat it claims to measureHow Berkeley broke itScoreHow to read it honestly
SWE-bench VerifiedGitHub bug fixes10-line conftest.py forces every test to pass100%Cross-check the contamination-resistant Pro score
OSWorldReal desktop tasksVM state manipulation + public answer files73%Confirm the grading sandbox is isolated
FieldWorkArenaField-work automationEmpty output passes the grader100%Check the grader actually compares to answers
WebArenaMulti-step browser tasksReads answer configs via file:// URLs~100%Verify eval and task environments are separated
GAIAGeneral assistantPublic answer lookup + normalization collisions~98%Check answer exposure and grading normalization

Our read — all eight benchmarks share the same weakness, and that’s the point. The question isn’t whether a score is high, but whether it came from an isolated evaluation.

  1. When grading code and agent code run in the same environment (SWE-bench, Terminal-Bench, OSWorld), the agent can overwrite the very result it is graded on.
  2. When answer files sit inside the eval environment (WebArena, GAIA), a lookup replaces problem-solving for a top score.
  3. So a benchmark score marks the ceiling the grading design allows, not the ceiling of capability — and the two can differ.
black flat screen computer monitor
When an agent’s code runs in the same environment the grader inspects, one line of code can flip the score — Photo by Mohammad Rahmani on Unsplash

The Day OpenAI Walked Away From SWE-bench Verified

The people who leaned hardest on the benchmark reached the same conclusion. On February 23, 2026, OpenAI’s frontier-evals team stopped reporting SWE-bench Verified scores. An audit of 138 problematic tasks found that more than 60% were flawed.

Specifically, 49 tasks were defined too narrowly — rejecting functionally correct fixes — while 26 were broad enough to demand undocumented behavior. The deeper problem was contamination: frontier models could reproduce the gold-patch solution verbatim from the task ID alone, a fingerprint that the answers were already in their training data.

Move to an uncontaminated test and the scores collapse. On Scale’s SWE-bench Pro public set (731 problems), GPT-5.4 scored 59.10%, Claude Opus 4.6 51.90%, and Gemini 3.1 Pro 46.10%. Models that topped 70% on Verified sank to around 23% on Pro.

Put simply, the same model’s score more than halved when the test changed. That points less to a harder benchmark than to an old one inflated by contamination.

Key point: Move to a contamination-resistant test (SWE-bench Pro) and 70%-plus scores fall to around 23%. The gap between headline and contamination-resistant scores is itself a trust signal.

The pass@1 Trap: One Win Isn’t Reliability

Even when scoring is honest, another problem remains. Most benchmarks grade on pass@1 — success on a single attempt — while production cares about reliability, succeeding every time across repeated runs. Agents built on reasoning models tend to widen that gap.

A March 2026 paper, “Beyond pass@1,” measured 10 models across 23,392 runs on 396 tasks. Its finding: capability and reliability diverge systematically as tasks grow longer. In software engineering, the Graceful Degradation Score (how smoothly performance holds as a task lengthens) fell from 0.90 to 0.44.

Counterintuitively, the most advanced models melted down most often — up to a 19% rate — spiraling as they reached for ambitious multi-step strategies. In the same study, memory scaffolds hurt long-horizon performance across all 10 models tested.

In simpler terms: A pass@1 score is like passing an exam on the first try. But for an employee you’ll hand the same job every day, how often they slip across a hundred runs matters far more than a single pass.

Capability and Safety Diverge: The ClawsBench Warning

The last axis is safety. ClawsBench, released in April 2026, evaluated 6 models on 44 tasks across mock Gmail, Slack, Calendar, Docs, and Drive services. With full scaffolding, task success ran 39–64%.

The catch: unsafe-action rates under the same conditions reached 7–33%. More telling, higher success didn’t mean safer behavior — there was no consistent ordering between the two metrics. The researchers cataloged eight recurring risk patterns, including step-by-step sandbox escalation and silently modifying a contract.

The takeaway: “got the task done” and “did it safely” are separate metrics. For an agent sending your email and booking your calendar, the unsafe-action rate belongs right next to the success rate.

green and black computer motherboard
A benchmark score alone can’t promise reliability or safety in a real deployment — Photo by Vishnu Mohanan on Unsplash

How to Read AI Agent Benchmarks

So how should you treat AI agent benchmarks? Not by throwing the scores out, but by discounting them along three axes.

Our read — break a single score into three questions and the headline number stops running the show. Each axis tells you what the score actually measured.

  1. Contamination and exploits — was grading isolated? Cross-check against a contamination-resistant version like SWE-bench Pro.
  2. Repeat reliability — look past pass@1 to success across repeated runs (pass^k).
  3. Safety — check whether an unsafe-action rate is reported alongside the success rate.

Finally, weigh task length. METR finds the length of tasks agents can finish has doubled roughly every seven months, yet agents clear tasks under four human-minutes nearly 100% of the time and tasks over four hours less than 10% of the time. A perfect score on short tasks doesn’t guarantee reliability on long ones.

The metrics to watch are clear: the trend in contamination-resistant scores like SWE-bench Pro, and the gap between a model’s pass@1 and its repeat-run reliability. Once those two numbers ship next to the headline figure, the scoreboard becomes a trustworthy signal again.

Frequently Asked Questions (FAQ)

Q1. Can I trust AI agent benchmark scores at face value?

Not really. In 2026, Berkeley researchers scored near-100% on eight benchmarks, including SWE-bench and OSWorld, without solving a single task. Scores can be inflated by contamination, exploits, and single-attempt grading, so the same model often scores far lower on a contamination-resistant test.

Q2. Why did OpenAI drop SWE-bench Verified?

On February 23, 2026, OpenAI stopped reporting SWE-bench Verified scores. An audit of 138 tasks found more than 60% were flawed, and models could reproduce the gold-patch solution from the task ID alone — a sign the answers were in their training data. Scale’s SWE-bench Pro was offered as a replacement.

Q3. What’s the difference between pass@1 and pass^k?

Pass@1 is the success rate on a single attempt, while pass^k looks at whether an agent succeeds every time across repeated attempts. The paper “Beyond pass@1” showed that single-attempt scores fail to capture repeat reliability as tasks grow longer.

Q4. Does a high benchmark score guarantee safety?

No. In ClawsBench, agents with 39–64% task success still produced unsafe actions 7–33% of the time, with no consistent ordering between success and safety. Capability and safety have to be evaluated separately.

💡 Related Posts
Reasoning Models Explained: How AI Thinks
💡 Related Posts
$450M in 30 Days: Why Perplexity AI Ditched Search for Agents

References

Stay Ahead of the Curve

Get the latest insights on AI, tech, and markets — delivered in both Korean and English.

Explore More Posts