Skip to content
NewA calmer, faster ZeroHour

Probes partly explain Qwen2.5 web-agent action hallucinations

Original title: Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

arXiv cs.AI / cs.LG / cs.CLby Mohamed Aghzal ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

A last-token linear probe scores 0.80 AUC, and steering fixes 85% of hallucinations while disrupting 7% of valid actions.

The paper applies mechanistic interpretability to Qwen2.5-7B-Instruct while it acts on live websites and invents actions that are not available. A linear probe on the last prompt token predicts those hallucinations before generation, and steering along the probe direction removes most of them at a cost to some valid actions. The authors say the account is only partial: the direction depends on prompt formatting, and the computation is spread across components, positions, and examples, which complicates circuit discovery.

Unconfirmed: The authors say the probe and steering results only partially explain why the agent hallucinates.

Key facts
  • The studied failure is web agents proposing actions absent from the site.
  • Steering loses much of its effect after a minor change in how actions are represented.
  • The probe direction is sensitive to prompt formatting.
  • Circuit identification is described as difficult because the computation is distributed.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Mechanistic Interpretability (MI) has made significant progress in explaining the internal computations of Large Language Models (LLMs); however, these advances have yet to demonstrate comparable success in explaining the decision-making processes of LLM-based agents. This paper examines the challenges MI tools face when applied to realistic agentic failures.

arXiv cs.AI / cs.LG / cs.CL

By Mohamed Aghzal · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.