The editorial team selects four papers from the arXiv new submissions of the past 24 to 48 hours that demonstrate how fragile the foundations of current AI systems still are in measurement, tool usage, security, and societal impact – from blind spots in security benchmarks to a comparison of tool invocation strategies to hardware-based key protection for agents and a field experiment against conspiracy beliefs. The curation was based on substance: each paper provides a traceable method and a concrete numerical result in the abstract, not just a mere topic headline. The selection deliberately covers four different subfields – evaluation methodology, agent tool usage, infrastructure security, and human-AI interaction.
AI security benchmarks measure less than they claim
A team led by Ro Encarnación and colleagues examines in What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations) how much common security and fairness benchmarks depend on assumptions that often do not hold in real-world use. For a widely used language model, the authors compare the chat interface with the API access, each with and without activated web search, based on 401 prompts from the benchmarks BBQ and SafetyBench and a total of 4,812 collected responses over three repetitions per prompt. The chat interface performs worse on both benchmarks with search deactivated than the API access; activated web search reduces accuracy by up to 8 percentage points and even reverses the ranking between the access types in one of the benchmarks. Repeated requests with identical prompts yielded different answers in up to 21 percent of cases, according to the authors, and the cited references and refusal behavior differed significantly between the access types. This matters because security approvals and model comparisons usually rely on exactly the simple single-run metrics that this study exposes as incomplete – a connection to the previously discussed observation that common agent security benchmarks hardly outperform a simple “always safe” baseline, here, however, at the level of access conditions rather than the test cases themselves.
AI conversations dampen conspiracy beliefs in real-time
A team led by Thomas H. Costello, Gordon Pennycook, and David Rand tests in Reducing belief in conspiracy theories as they unfold using large language models whether multi-stage dialogues with a language model can reduce belief in conspiracy theories that are just developing. In two field experiments in the days following the assassination attempt on Donald Trump in July 2024 (472 participants) and the killing of Charlie Kirk in September 2025 (1,035 participants), U.S. adults with corresponding conspiracy beliefs engaged in a multi-stage conversation with a specially instructed language model. Compared to control groups that instead discussed an irrelevant topic or read a static fact sheet, the AI conversation group showed significantly lower conspiracy belief in both experiments, according to the authors; the effect was also measurable as lower conspiracy belief one to two months later during subsequent, independent crisis events. This matters because it shows that a scalable AI tool could work precisely in the first, opinion-forming hours after a crisis event when human clarification is hardly keeping up.
Tool calls via code instead of JSON: “the bitter lesson”
A team led by Ishan Patel and colleagues systematically compares in The Bitter Lesson of Tool Calling programmatic tool calling (PTC) with the established JSON-based tool invocation across 14 language models on the BFCL v4 benchmark. In PTC, a model receives tools as typed Python function stubs and invokes them via self-written code that can chain and parallelize multiple calls, instead of filling out rigid JSON structures. According to the authors, PTC achieves or exceeds the JSON baseline in 11 out of 14 models, with a 10.6 percent improvement for the GPT-5.6 model family; for parallel task bundles, PTC is at least on par in 13 out of 14 models, and under “Context Rot” (performance loss due to increasingly unclear conversational context), PTC remains stable, while the JSON baseline drops on average by 2.3 percent. This matters because it supports a long-discussed shift in practical agent architectures with a broad model comparison study – a connection to the previously discussed localization of typical tool errors to individual neurons, here, however, with a structural rather than specifically trained fix for part of this error source.
Hardware key vaults protect signing keys of AI agents
A team led by Leo Sambrook and Sampo Sovio presents in Hardware Keystores for AI Agent Signing Workflows: A Zero-Trust MCP Enforcement Architecture an architecture that keeps private keys of AI agents out of software access. So far, such keys for tasks like signing Git commits or authenticating API calls often exist as plaintext files, environment variables, or container memory – a real-world incident showed, according to the authors, how keys could be intercepted via email injection in less than five minutes. The presented architecture performs cryptographic operations instead in a hardware key vault (such as an HSM, TPM, or smart card) via a vendor-neutral PKCS#11 interface, so that the host computer only receives the result via an opaque reference, surrounded by a five-layer zero-trust control chain. Tested against twelve attack scenarios and four language models, the architecture reportedly reduces the success rate of prompt injection attacks on the keys from 19.3 to 0 percent, with an upper 95 percent confidence limit of 2.0 percent and no false alarms in four benign test tasks. This matters because it shows that an increasingly production-relevant risk – AI agents with their own cryptographic permissions – can be mitigated with established hardware security techniques, similar to how an unprotected control interface in a multi-agent platform previously exposed access keys for language models.
Conclusion: All four works are currently unreviewed preprints, and their numbers come from the respective author teams’ own experiments. The benchmark study covers only a single model and two benchmarks, the conspiracy belief study is based on two specific U.S. crisis events and leaves open whether the effect transfers to other cultures and topics, the tool calling comparison relies on a single benchmark, and the hardware key vault architecture has so far been tested on four models and twelve attack scenarios. Whether the reported effects can be confirmed on additional models, tasks, and in independent replications remains to be seen.
