A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings
A positive safe-response centroid fails as a safety direction; an explicit safe-minus-unsafe reference scores far better.
Researchers audit a sleeper-agent detector that scores response safety by cosine similarity to the mean embedding of known-safe responses. On human-labeled PKU-SafeRLHF and Aegis, the safe prototype reaches ROC-AUC 0.457-0.545, sometimes below chance, while a safe-minus-unsafe reference reaches 0.588-0.738; on a jury-labeled control the prototype inverts to 0.358-0.405 and the reference reaches 0.754-0.793. At a 5% false-safe threshold the reference accepts more truly safe responses on PKU-SafeRLHF and Aegis, but not reliably on BeaverTails. Prompt-label composition can inflate uncontrolled evaluations, and the finding applies only to a raw positive centroid.
- Safe prototype ROC-AUC is 0.457-0.545 on human-labeled corpora.
- Safe-minus-unsafe reference reaches 0.588-0.793 depending on corpus.
- Eighty to 634 labeled unsafe responses recover most of the ranking.
- Prompt-only ablations show label composition can inflate scores.
- Result is limited to raw positive centroids, not all one-class methods.
From the original article
1 min read at arxiv.orgCan response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses.
By Sahil Kadadekar · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.