Research Detects Prompt-Injection Hijacks via Model Probabilities
Original title: Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities
Monitor using output probabilities detects some hijacks on AG News classifier but misses most when attackers add certainty claims.
Programs using language models as classifiers can be hijacked by injected text that changes the predicted label. A monitor reading the model's own output probabilities ranks hijacked answers as less confident. Tests on five systems including API-served ones show the monitor separates clean and hijacked answers with AUROC 0.86-0.90 but misses 51-70% of plain hijacks and nearly all when attackers add sentences claiming certainty.
- Pooled AUROC 0.86-0.90 for ranking hijacks
- Misses 51-70% of plain injected note hijacks
- Misses 99-100% of hijacks with added certainty claims
- Threshold set for 5% false positives still misses more confident hijacks
From the original article
1 min read at arxiv.orgPrograms that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low.
By Junwei Yao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.