PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift
PIDS-Bench shows prompt-injection detectors scoring F1 above 0.98 still misclassify about one-third of external benign security-adjacent prompts, revealing provenance-sensitive over-defense.
PIDS-Bench is a frozen multi-axis benchmark that jointly evaluates prompt-injection detectors on attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution inputs, hard-benign prompts, obfuscated attacks, and domain/structural distribution shifts. It evaluates seven detectors plus a rule-based lower-bound reference. A detector exceeding F1 = 0.98 on held-out data still misclassifies roughly one-third of an externally-sourced benign security-adjacent subset, and no internal detector reaches F1 >= 0.95 with hard-benign FPR <= 0.10 on the stress distribution. Hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it intact on externally-sourced prompts, a pattern termed provenance-sensitive over-defense.
- Multi-axis benchmark measures over-defense, obfuscation resistance, distribution shift at fixed thresholds
- F1>0.98 detector misclassifies ~one-third of external benign security-adjacent prompts
- No internal detector achieves F1>=0.95 with hard-benign FPR<=0.10 on stress set
- Hard-negative augmentation fails on externally-sourced prompts (provenance-sensitive over-defense)
From the original article
1 min read at arxiv.orgPrompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under distribution shift, particularly on the benign side of the decision boundary, where false positives impose direct operational cost yet are seldom measured.
By Yusuf Khalid Shire · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.