Benign-trained probes miss tool-action errors under adversarial conditions
Scalar uncertainty beat the probe on AUROC and average precision for Qwen2.5-14B-Instruct under system override.
Stories per day · 30 days
3 RSS
Relatedprompt-injection4agent-security3false-positives2llm-agents2llm-guards2agent-authorization1agent-safety1ai-agents1benchmark1benchmarks1evaluation1evidence-reuse1hidden-state-probes1indirect-prompt-injection1injecagent1matoi1monitoring1multilingual1
19 stories, newest first · one card per story
Stories tagged #agentdojo
SEAL cuts AgentDojo prompt-injection success from 34.6% to 0.1%
Joint task completion without attack success rises from 51.4% to 77.5%, using about 127 times fewer defense tokens than PISanitizer.
Tool-call guards fail when actions are rendered as shell commands
Four of five specialized guards raise AgentDojo false-positive rates by 21 to 47 points under shell form.
Prompt-injection outcomes often form after the agent reads
Thinking cuts Qwen3 variance fixed at read time from 0.91 to 0.62, and single-run labels disagree up to 35%.
Low prompt-injection rates mostly mean small agents ignore attacks
Across 33,000 trajectories, 76-80% of failed injections were ignored and only 17-18% were refused.
Language Is a Security Surface for Agent Guards
Most smaller agent guards change safety verdicts when requests are translated, especially into rarer languages.
Shortcut Sensitivity in Agent-Safety Evaluations
Shortcut cues in agent-safety benchmarks can change guard decisions and measured accuracy.
Matoi: Selective Revalidation of Consequence Evidence for Agent Authorization
Matoi reuses checked consequence evidence for agent authorization, cutting lifecycle cost a few percent in tests.
The Agent Knew: Internal States Forecast Unauthorized Actions and Stop Them, and the Lies That Follow
Internal-state readouts forecast unauthorized agent actions and lies, cutting violations and prompt-injection success.
Formal Runtime Verification for Tool-Using LLM Agents: An Offline Same-Benchmark Study on AgentDojo and STAC
Offline MFOTL monitoring flags about 70% of AgentDojo and STAC agent attacks but also 29% of benign runs.
Correct Answers, Unsupported Findings: Evidence Binding in Forensic Reconstruction of LLM Agent Logs
Study finds LLM readers can match agent-log facts while asserting unsupported citation sources.
Compromise Is Not Consequence: Evaluating Task-Scoped Authorization in LLM Agents with Paired Replay
Scoped authorization stopped harmful LLM-agent tool execution that broad bearer tokens still allowed.
Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks
Agent-security benchmark ASR scores shift up to 13 percentage points when tool names change, showing single-representation robustness scores may not generalize.
Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
Re-evaluation shows prompt-injection detector benchmark scores transfer poorly to real LLM agents; rankings on BIPIA, AgentDojo and tau-bench diverge sharply.
False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Safety-router benchmarks overstate routing gains by picking the best model on test data.
ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents
ToolFence authorizes tool-using LLM agents before execution, driving AgentDojo attack success near zero on Qwen3-max.
MetaPermit: Scalable and Auditable Access Control for AI Agents via LLM-Inferred Meta-Attributes
MetaPermit uses inferred meta-attributes and a fixed policy to authorize AI-agent tool calls more consistently and resist prompt injection.
ActGov: Governing LLM Agent Actions via Policy-Constrained Validation
ActGov validates each LLM agent tool call at runtime, cutting indirect prompt-injection success while preserving task utility.
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
EvoSafeHarness auto-synthesizes per-model, per-domain safety harnesses, cutting prompt-injection attack success on AgentDojo to 0.0% at 82.8% utility.