Statistician Edgar Dobriban of the University of Pennsylvania has used the language model GPT-5.6 Sol Pro to disprove an assumption about the Benjamini-Hochberg procedure that had been considered settled for twenty years. The model found a mathematical counterexample in about ninety minutes, a task on which its predecessor GPT-5.5 had previously spent more than twenty hours without result.
Procedure controls error rates across tens of thousands of studies
The Benjamini-Hochberg procedure comes from a 1995 paper by statisticians Yoav Benjamini and Yosef Hochberg and specifies how many of many simultaneously tested hypotheses may be falsely counted as significant. The so-called false discovery rate caps this share at a predetermined level, usually five or ten percent in research settings. Since its publication, the procedure has become a standard tool in genomics, astronomy, economics, and neuroscience, wherever researchers evaluate thousands of tests at once. Stanford statistician Emmanuel Candès has called the method one of the two most important developments in statistics after 1950.
Whether the guarantee also holds for two-sided tests with correlated measurements remained an open question since the method’s introduction — a case that occurs frequently in practice, for instance when neighboring genes or measuring stations produce similar values. Experts assumed for roughly twenty years that the error rate held even under correlation, even though a complete mathematical proof was missing. This gap between practice and proof went largely unnoticed because the procedure performed reliably in simulations.
AI model finds a counterexample in ninety minutes
Dobriban gave GPT-5.6 Sol Pro the mathematical definition of the procedure along with the open question of whether the guarantee holds for correlated, two-sided Gaussian tests. According to the associated preprint, the model constructed a factor model in which the false discovery rate exceeds the specified significance level of 0.01, asymptotically surpassing 0.0104. To back up the result, the system produced a numerical certificate using interval arithmetic, drawing on the Arb and python-flint libraries, which can be checked independently.
The predecessor GPT-5.5 reportedly worked on the same task for more than twenty hours across several parallel approaches without reaching a usable result. Dobriban then checked the proof delivered by GPT-5.6 Sol Pro by hand before publishing it as a preprint. On the platform X, he wrote that AI had helped resolve an important open question in statistics, pointing to a detailed thread with further explanation. Code, counterexample, and the verification certificate are also available on GitHub.
A summer of AI-assisted mathematical proofs
The case joins a string of similar reports from recent weeks. On July 11, OpenAI announced that the more powerful model variant GPT-5.6 Sol Ultra, running 64 parallel sub-agents, had produced a proof of the Cycle Double Cover Conjecture, a graph theory problem open since 1973. Mathematician Thomas Bloom of the University of Manchester assessed that proof as short and elementary but criticized that it drew on ideas and strategies from the literature without proper citation.
In the current case, Dobriban himself rates the practical impact as limited: the demonstrated deviation of roughly 0.04 percentage points mainly concerns theory, and existing analyses that rely on the procedure do not need to be recalculated. Still, it remains notable for statistics and computer science that a language model produced a complete, checkable proof here rather than merely suggesting a plausible result.
What matters next is whether professional societies clarify under which conditions the procedure’s original guarantee still holds, or whether an adjusted version follows. The episode also shows that language models are now being taken seriously as tools for open mathematical questions, provided their results are backed, as here, by independently verifiable certificates and human review. A formal peer review of the counterexample has not yet taken place.
