Selecting The Most Informative Tokens in Natural Language Autoencoders
Explaining five percent of tokens retains nearly all prompt-injection audit success on three of four datasets.
The authors ask which token positions natural-language autoencoders should explain when auditors look for prompt injection and concealment, using 4.7 million explanations. A ranker trained only on chat structure usually selects more relevant positions than signals from model computation and needs no extra forward pass for selection. On three of four datasets, explaining 5% of positions retains nearly the success rate of explaining every position. Pretrained verbalizers also recover words that models learned to conceal through fine-tuning, without further verbalizer training.
- Study uses 4.7 million explanations of injection and concealment.
- A chat-structure ranker beats signals from model computation.
- Explaining 5% of positions keeps nearly full success on three datasets.
- Pretrained verbalizers recover words models were tuned to conceal.
From the original article
1 min read at huggingface.co
Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an auditor inspect to understand a potential threat?
By Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.