Two days after the broad rollout of GPT-5.6, OpenAI has retracted its own recommendation: The coding benchmark SWE-Bench Pro, which the company had only proposed in February 2026 as a yardstick for the programming skills of AI models, is now considered internally to be significantly defective. As OpenAI explains in a blog post, an internal audit classifies about 30 percent of the 731 public test tasks as faulty.
What OpenAI Found
As the trade publication The Stack reports, SWE-Bench Pro was developed by the data provider Scale AI and was intended to test models based on real change histories from software projects – an evolution of the older SWE-bench Verified. However, according to OpenAI’s analysis, this origin from real project histories intended for human collaboration is also the root of the problem: task descriptions, code, and automated tests often do not align properly because they were never designed for standardized AI evaluation.
OpenAI’s automated review process flagged 200 of the 731 tasks (27.4 percent) as suspicious. A subsequent campaign involving five experienced software developers, who judged independently and agreed in 74 percent of cases, classified even 249 tasks (34.1 percent) as defective. OpenAI rounds the results in its own summary to about 30 percent faulty tasks.
Four Types of Errors
The evaluation distinguishes four recurring categories of errors: overly strict tests that require technical details not mentioned in the task description, causing functionally correct solutions to fail; underspecified task descriptions where hidden tests check requirements that cannot be derived from the description; overly lenient tests that incorrectly pass incomplete solutions; and misleading task descriptions that actively steer models in the wrong direction.
The Whitespace Example
How such errors can concretely affect outcomes is illustrated by OpenAI with the task “OpenLibrary-77c16d5,” which concerns the formatting of a table of contents. The visible task description specified a single leading whitespace before the entries, but the hidden test required two spaces. Models that adhered exactly to the specification in the task description failed the test solely because of this one additional character – regardless of whether their solution was substantively correct.
Audit Methodology
For the review, OpenAI combined three stages: an automated preliminary check that searched task descriptions, model solution attempts, and test cases for anomalies; followed by Codex-supported agents acting as “investigators” accessing the associated code repositories to distinguish real underspecification from mere ambiguity before researchers made the final assessment; and finally, the aforementioned independent review by five experienced developers based on a fixed error taxonomy. Notably, the human reviewers tended to classify more tasks as defective than the AI-supported agents and often assigned multiple error categories per task.
Background: From SWE-bench to SWE-Bench Pro
The case repeats a pattern from the past. In the original SWE-bench, OpenAI had already found that about 68 percent of the examples exhibited flaws such as overly strict tests or vague problem descriptions, leading to the release in August 2024 of SWE-bench Verified, a manually reviewed subset of 500 tasks by 93 professional developers. SWE-Bench Pro from Scale AI aimed to supplement this verified dataset with more complex, realistic tasks – but according to OpenAI’s current audit, it suffers from the same structural weaknesses that affected the original SWE-bench. It is also noteworthy that the measured success rate of frontier models on the benchmark has reportedly increased from 23.3 to 80.3 percent within eight months, raising the question of how much of this is due to genuine skill increases and how much is attributable to exploiting test weaknesses.
Classification
OpenAI draws a clear consequence from the results and states in its own blog post: “We retract our earlier recommendation to adopt SWE-Bench Pro.” The company also publicly announced the retraction on X and advises model developers to scrutinize results on the benchmark closely. At the same time, OpenAI calls on the research community to develop new benchmarks specifically designed by experienced software engineers for the evaluation of AI models – rather than being derived from repurposed project histories as has been the case so far. As of the time of publication, there had been no public response from Scale AI to the criticism. For the industry, this case is an indication of how fragile many of the currently used coding benchmarks are – and how quickly seemingly objective progress figures can crumble under closer examination.
