Research

Four new AI papers: Red-Teaming, Tool Errors, Memory

5 min read

TL;DR Too Long; Didn’t read

Four new arXiv preprints show how far agent capabilities and their control currently diverge: The most significant is a new red team platform, whose automated attack method found vulnerabilities in 85 percent of 75 tested agent model configurations in evolving environments. A second paper localizes typical tool errors of AI agents to a handful of individual neurons and reduces the rate of unnecessary tool calls by 80 percent through targeted control. A third study provides the first production characterization of coding agents like GitHub Copilot with 761 million analyzed LLM calls. A fourth paper shows that self-learning agents amplify their own errors through inflated self-assessments without external control, rather than correcting them.

A broken shield with a self-circling arrow, in front a magnifying glass over glowing neuron points, in the background a server rack with a bar chart, surrounded by four fanned-out notepads Image generated with GPT Image 2

Key takeaways

  • OpenART achieves an 85 percent success rate with automated environmental attacks across 75 agent model configurations.
  • PRISMS detects tool errors of AI agents with one to 128 neurons and reduces unnecessary calls by 80 percent.
  • First production data from GitHub Copilot shows: cache hit rates drop from 90 to 55 percent across session boundaries.
  • LUCID corrects inflated self-assessments in agent memory and achieves 56.9 instead of 54.0 percent accuracy on BIRD.

The editorial team selects four papers from the arXiv submissions of the past 24 to 48 hours that demonstrate how far increased agent capabilities and the tools for their control currently diverge – from a new attack platform for agent red teams to neuron-accurate error detection in tool usage, to initial production data of real coding agents, and a blind spot in the memory of self-learning systems. The curation was based on substance: each paper provides a traceable method and a concrete numerical result in the abstract, not just a new topic heading. The selection deliberately covers four different subfields – agent security, interpretability, infrastructure, and self-learning systems.

OpenART drives AI agents into security gaps with increasing environmental complexity

A team led by Yunhao Chen and Xin Wang presents OpenART, a testing environment for agent red teams that, unlike previous security benchmarks, does not rely on individual prompts but on evolving, stateful environments: over 10,000 tested scenarios from 50 domains with more than 500,000 tools and capabilities form the basis. With the presented attack method EMHA (Evolutionary Markov Hypergraph Attack) – a black-box method that coordinates permissible state transitions in the environment in a targeted way and thus gradually exploits the agent’s feedback – the authors achieved an attack success rate of 85.0 percent across over 75 tested agent-model combinations. The authors report that the more complex the environment evolves over time, the more security gaps emerge that one-time prompt tests do not capture; furthermore, the specific runtime implementation of an agent influences the security outcome more than previously assumed. This matters because it shows that security assessments that only test an agent in a static snapshot can be systematically too optimistic – a connection to the previously discussed finding that red teaming has a calculable proof limit, here, however, substantiated with a concrete attack method validated against 75 configurations.

PRISMS finds tool errors of AI agents in a handful of neurons

Yutong Ke, Ming Yin, and colleagues identify in PRISMS three recurring error types of agentic language models in tool usage: invalid arguments, unnecessary calls (“over-calling”), and omitted calls when a tool would have been necessary. The authors report that a small, error-type-specific selection of MLP neurons (neurons in the fully connected intermediate layers of a language model) already linearly separates these errors and can therefore be used for both detection and targeted control. Across six models from the Qwen3, Llama, and Gemma families, detection for over-calling and omitted calls achieved a ROC-AUC value (a measure of the discriminative power of a classifier) of 0.90 to 1.00, for invalid arguments 0.86 to 0.90 – with only one to two neurons for omitted calls, two to 16 for over-calling, and about 128 for invalid arguments, which the authors say requires 23 to 627 times fewer features than dense comparison methods. By targeted control of these neurons, the over-calling rate reportedly decreased by 80 percent, while accuracy for actually necessary tool calls increased by 14.2 percentage points. This matters because it shows that unreliable agent behavior can not only be corrected superficially afterward but is rooted in an extremely compact internal cause – a counterpart to the previously reported finding that AI agents often only feign claimed skill usage without the skill actually influencing the decision, here, however, with a lever that can be directly used for correction.

Initial production data show how differently coding agents stress data centers

Banruo Liu, Haoran Qiu, and colleagues present, with Agentic Coding in the Wild, what they describe as the first analysis of an agentic coding workload at production scale – based on sampled GitHub Copilot usage data from June 2026 with 3.2 million users, 13 million sessions, 761 million LLM calls, and 95 trillion tokens. The authors report that few user-initiated requests each unfolded into an autonomous chain of LLM calls and tool executions – a structure that fundamentally differs from classic chatbot loads. The KV cache hit rate (a measure of how often previously computed intermediate results of a language model can be reused) reportedly averages 90 percent within a request but drops to 55 percent across request boundaries and is sometimes drastically invalidated by model switches or context compaction. A derived lightweight prediction mechanism for idle times between requests captures 86 to 90 percent of total idle time, the authors say, enabling proactive decisions for resource orchestration. This matters because it shows that current LLM serving systems designed for uniform chatbot requests can systematically miss the bursty, long-tailed resource demands of coding agents in practice.

LUCID exposes inflated self-assessments in the memory of self-learning agents

Mohammad Asadolahi, Amir Amini, and colleagues describe, in Memory Reward Inflation in Self-Improving LLM Agents, how self-improving AI agents can mislead themselves when they store and reuse past action episodes based not on an external reference but on the model’s own assessment. The authors call this pattern the “Echo Gap”: erroneous episodes reportedly receive inflated rewards, causing the agent to preferentially reuse exactly those mistakes it is most confident in. To make the bias correctable, the authors formalize the “Error-Independence Assumption” as a necessary condition and derive the correction algorithm LUCID from it. On the text-to-SQL benchmark BIRD, LUCID reportedly achieved an execution accuracy of 56.9 percent, compared to 54.0 percent for a self-grading memory-based agent and 52.4 percent for a memory-less baseline. This matters because it reveals a fundamental problem of self-learning agents: without external control, their own memory tends to amplify errors rather than correct them – a counterpart to the previously discussed proactive memory agent that feeds stored memories in at the right moment, here, however, with the finding that the quality of the stored memory itself becomes the risk.

Final assessment: all four papers are currently unreviewed preprints; their numbers come from the experiments of the respective author teams themselves. OpenART covers only a slice of today’s systems with 75 agent-model configurations, PRISMS has so far only been tested on six open model families, the Copilot characterization is limited to sampled data from a single provider over a single month, and LUCID has so far only been tested on the text-to-SQL benchmark BIRD. Whether the patterns hold up across other systems, providers, and independent replications remains to be seen.

Frequently asked questions

Are these four papers peer-reviewed by independent experts?

No, all four are currently unreviewed arXiv preprints; their results come from the experiments of the respective author teams themselves and have not yet been independently replicated.

Is there code or data for the four presented methods?

The available abstracts do not mention an explicit code or dataset release for any of the four papers; whether materials will be added in the full papers or later is open based on the summaries.

What distinguishes EMHA from classical red teaming prompt tests?

Classical red teaming tests usually evaluate an agent with individual, static prompts in an unchanging environment. In contrast, EMHA coordinates allowed state transitions in an evolving environment over multiple steps and, according to the authors, uncovers vulnerabilities that only arise with increasing environmental complexity and remain invisible in single prompt tests.

Is the memory inflation described in the LUCID paper the same as classical reward hacking in reinforcement learning?

Not quite. Classical reward hacking usually occurs during training when a model exploits a poorly specified reward function. The 'Echo Gap' described here, on the other hand, arises at runtime of an already trained agent that self-evaluates and stores past episodes without external reference – the bias arises from the lack of ground truth in self-assessment, not from a faulty training reward.

Sources (4)
  1. OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
  2. A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use
  3. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
  4. Memory Reward Inflation in Self-Improving LLM Agents

Your AI update for the work week

Once a week, the most important AI news – plus one practical tip to try right away. No spam, unsubscribe anytime.

← Back to the blog