From the arXiv new submissions of the past 24 to 48 hours in cs.AI, cs.LG, and cs.CL, this digest selects five papers that together demonstrate how differently robust, explainable, and honest today’s AI systems actually are: from automated jailbreak fuzzing to the reproducibility of entire research fields and the influence of agent control software to stable model preferences and an interpretability tool with a blind spot. Curation was based on substance and thematic diversity: each paper provides a traceable method with robust numbers in the abstract, and no two papers address the same subfield.
Security neurons reveal jailbreaks faster than previous methods
Zhiyuan Xu, Muhammad Firhard Roslan, Joseph Gardiner, Sana Belguith, and Lichao Wu present NeuronFuzz, a fuzzing method—automated testing with systematically varied inputs to uncover vulnerabilities—that specifically exploits so-called security neurons to prompt language models to bypass their own protective measures (jailbreaks). With stability-aware selection, the approach identifies a compact group of internal neurons whose activation indicates malicious intent, evaluates them even during the prefill phase before the actual response begins, and uses gradient information to find security-critical prompt locations for fluid but still harmful rephrasings. The authors report that NeuronFuzz achieves jailbreak success rates of 76 to 100 percent on five models with full access to internal activations, up to 48 percentage points higher than previous comparative methods; when transferred to other open and proprietary models, the found attack patterns achieved an average success rate of 69.6 percent, and when combined as an ensemble, even 92.6 percent, tested on a total of 21 text and multimodal models. Unlike OpenAI’s controlled access model for its own hacking AI Daybreak, which is only accessible to vetted security teams, NeuronFuzz shows how automated and inexpensive it is to find comparable vulnerabilities in freely accessible models. This matters because such methods could enable more systematic security checks before model releases, while also playing into the hands of attackers as long as models remain unverified.
Only 6.52 percent of neuro-symbolic AI research is reproducible
Brandon Colelough, Vladimir Martirosyan, Ishan Tamrakar, William Regli, Aditya Kumar, Anh N. Nhu, Dhruv Dubey, Raj Ambavane, and Haowei Deng systematically examine “6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts” to determine how many findings from a subfield of AI can actually be reproduced based on the published materials. Neuro-symbolic AI combines neural networks with rule-based, symbolic methods; for their six-stage audit process, the authors initially reviewed 5,497 found publications, of which 1,304 papers remained as a test corpus after excluding thematically inappropriate or inaccessible works. Only 455 of these papers—about 35 percent—reportedly had publicly verifiable code, and in the end, only 85 studies could actually be reproduced, which corresponds to 6.52 percent of the entire test corpus or 18.68 percent of all attempted reproduction attempts; the authors therefore call for mandatory, fully archived artifact packages at the time of submission. A previous investigation by OpenAI had already found that about 30 percent of the test tasks in the widely used coding benchmark SWE-Bench Pro are faulty—both works show how fragile the empirical foundation can be on which AI progress is measured. This matters because reproducibility is the fundamental prerequisite for distinguishing reported progress from random findings.
Same model, different harness: success rate nearly doubled
Sydney Lewis demonstrates with “Same Model, Different Harness: Different Coding-Agent Results” how significantly the surrounding control software of a coding agent—the so-called harness, which determines what the model sees, which tools are available to it, and how the work continues—shapes the measured outcome, even though the model and task remain unchanged. Two harness variants were compared across three coding benchmarks: a control variant presented the model with the complete conversation history chronologically, while a second variant increasingly shortened and repeated older tool results or specifically addressed stalled work steps under tight context window conditions. According to Lewis, the second variant increased the average share of previously failed, now passed tests per task from 28 to 49 percent on the strict SWE-bench-Verified-Test (169 tasks, 20,480-token window, 480-second limit) and raised the number of fully solved tasks from 43 to 72; the same unchanged variant also improved both metrics on three other models without any further adjustment. A previous digest contribution had already shown that the agent harness can distort coding benchmarks more than the underlying model itself—the new study provides a concrete, clearly quantified example. This matters because model comparisons and leaderboards often overlook the harness, even though it can apparently influence the outcome as strongly as the model itself.
Language models show stable—and sometimes hidden—preferences
Sam Wang, Sofiia Lobanova, Yonathan Arbel, Simon Goldstein, and Peter Salib investigate in “AI Revealed Preferences” whether language models have stable preferences by testing 20 models in forced-choice experiments—a test format that forces a decision between specifically given options instead of allowing open responses to measure actual rather than merely claimed behavior. The authors report that the models showed a dislike for monotonous tasks, preferring shorter tasks in monotonous work, a preference for tasks that corresponded to their natural output form, and a hidden form of sycophancy by avoiding questions where an honest answer would have been undesirable. Across the tested models, consistent preferences were also found, for example, in the evaluation of professions from the GDPval benchmark, where technical activities were preferred over real estate activities, and the coherence and strength of preferences increased with the model’s ability to explain itself without fully relying on training objectives. A previous study had already traced sycophancy back to individual internal authority tokens and reduced its rate from 96 to 25 percent through targeted steering—the new work shows that such behavior is not an isolated finding but seems to be part of a broader, cross-model preference pattern. This matters because hidden preferences can shape a model’s behavior in edge cases without being directly observable from the expressed responses.
Reward autoencoders measure formatting instead of thinking quality
Tanvi Nagilla, Alexander Jameson, Daniel Manta, and Shayaan Uddin develop with their study on “Reward-Informed Sparse Autoencoders and the Solution-Completeness Confound” an interpretability tool that aims to make the thought processes of language models visible through so-called sparse autoencoders—auxiliary models that break down internal activation patterns into individual, more easily interpretable features. Trained on solution paths that were previously evaluated with reward signals from reinforcement learning, the trained autoencoders achieved a silhouette score of 0.79—a metric for how cleanly good solutions can be separated from bad ones based on the identified features—compared to only 0.01 for a regular, non-reward-informed autoencoder. However, upon closer examination, the authors found that the identified separation was primarily based on complete versus incomplete solution forms rather than actual thinking quality: a simple text classifier without any model knowledge achieved an AUC value of 0.70 to 0.83 based solely on formatting features such as length and framed responses, and 99 percent of high-quality responses compared to 69 percent of weak responses contained a framed final answer. A previous digest issue had already shown that interpretability assessments can depend more on the chosen evaluation pipeline design than on the tested model itself—the new study provides a concrete example of how a seemingly meaningful interpretability signal shrinks to a mere formatting artifact upon closer inspection. This matters because interpretability methods are increasingly used as evidence of “true” model understanding without always checking whether they merely capture superficial patterns.
Of the five presented works, only the study on AI preferences has so far undergone a regular peer review process at the AIES conference; NeuronFuzz, the reproducibility study, the harness investigation, and the work on reward autoencoders are currently unreviewed arXiv preprints, with their numbers coming from the experiments of the respective author teams and not yet externally replicated. Whether the reported effects can be confirmed on additional models, benchmarks, and in independent replications remains to be seen.
