Separating Signal from Noise in Coding Evaluations

engineering blog · source date 2026-07-08 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog

Coding benchmarks are the main public evidence for agent capability, but nobody audits whether their tasks are actually solvable and correctly graded.
Broken tasks and contamination push scores in both directions, and the resulting numbers feed both safety cases and research prioritization.
OpenAI had previously recommended SWE-Bench Pro as a successor benchmark.

An audit of design and contamination flaws estimates roughly 30% of SWE-Bench Pro tasks are broken: 200 tasks (27.4%) flagged by an automated pipeline, 249 (34.1%) by human annotation.
The dominant failure mode is overly strict hidden tests that reject correct solutions.
OpenAI, co-author of SWE-bench Verified, retracts its own earlier recommendation to adopt SWE-Bench Pro.
The conclusion is a call for deliberately curated coding benchmarks over scale-first task harvesting.

This is the highest-impact benchmark-validity event of the month: a frontier lab publicly discrediting the benchmark it recommended reshapes how coding capability gets reported.
It was independently corroborated by a Faros audit, which matters more than the retraction itself.
Practical read: before citing a coding benchmark number, ask who validated the task set and what fraction of failures are grader artifacts rather than model failures.
Caveat: an OpenAI-authored blog rather than a peer-reviewed study, so the definition of "broken" reflects OpenAI's own pipeline.

Comments

No comments yet.