LLM agent detector evaluations hide cross-benchmark issues
Detectors trained on one benchmark fail on another with AUROC as low as 0.47.
Stories per day · 30 days
2 RSS
Relatedaadd-20251adversarial-examples1adversarial-ml1benchmarks1deepfake1graph-learning1hijacked-agents1mig-cow1relational-graphs1r-gcn1
5 stories, newest first · one card per story
Stories tagged #detectors
Study finds when trusting graph relations hurts detectors
Clean-trained graph detectors fell below graph-free counterparts in 58 of 90 settings on six graphs.
MIG-COW fools deepfake detectors in white-box tests
On AADD-2025, the method reports 99.96% white-box success but only 7.16% against official black-box targets.
Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
Re-evaluation shows prompt-injection detector benchmark scores transfer poorly to real LLM agents; rankings on BIPIA, AgentDojo and tau-bench diverge sharply.
PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift
PIDS-Bench shows prompt-injection detectors scoring F1 above 0.98 still misclassify about one-third of external benign security-adjacent prompts, revealing provenance-sensitive over-defense.