Benign-trained probes miss tool-action errors under adversarial conditions
Original title: Do Benign-Trained Hidden-State Probes Detect Tool Errors Under Adversarial Conditions?
Scalar uncertainty beat the probe on AUROC and average precision for Qwen2.5-14B-Instruct under system override.
Researchers tested whether logistic-regression probes trained only on benign tool-use trajectories detect incorrect tool actions under attack. They used four AgentDojo suites and three models: Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and GLM-4.6V-Flash. On held-out prompt-injection, system-override, and tool-poisoning runs, paired tests found six significant differences after Holm correction. All favored scalar uncertainty metrics over the probe, and only for Qwen2.5-14B-Instruct under system override.
- Probes were trained only on benign, non-defended trajectories.
- Evaluation used five-fold, task-grouped out-of-fold tests.
- Comparators were entropy, negative log-probability, and self-consistency uncertainty.
- Authors say benign-trained probes should not be assumed reliable under attack.
- Products
- AgentDojo
From the original article
1 min read at arxiv.orgAlthough hidden-state probes efficiently monitor model operation, it remains unclear whether probes trained on benign tool use remain useful for detecting incorrect actions under adversarial conditions. In this research, we assess this problem from the perspective of operational tool-action correct- ness. we use four AgentDojo suites, four conditions, and three models: Qwen2.
By Bibek Bhandari · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.