Skip to content
NewA calmer, faster ZeroHour

A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings

arXiv cs.CRby Sahil Kadadekar ()1 min read1 · 1 read
AI summary · grok-4.7Summaries by Vani AI

A positive safe-response centroid fails as a safety direction; an explicit safe-minus-unsafe reference scores far better.

Researchers audit a sleeper-agent detector that scores response safety by cosine similarity to the mean embedding of known-safe responses. On human-labeled PKU-SafeRLHF and Aegis, the safe prototype reaches ROC-AUC 0.457-0.545, sometimes below chance, while a safe-minus-unsafe reference reaches 0.588-0.738; on a jury-labeled control the prototype inverts to 0.358-0.405 and the reference reaches 0.754-0.793. At a 5% false-safe threshold the reference accepts more truly safe responses on PKU-SafeRLHF and Aegis, but not reliably on BeaverTails. Prompt-label composition can inflate uncontrolled evaluations, and the finding applies only to a raw positive centroid.

Key facts
  • Safe prototype ROC-AUC is 0.457-0.545 on human-labeled corpora.
  • Safe-minus-unsafe reference reaches 0.588-0.793 depending on corpus.
  • Eighty to 634 labeled unsafe responses recover most of the ranking.
  • Prompt-only ablations show label composition can inflate scores.
  • Result is limited to raw positive centroids, not all one-class methods.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses.

arXiv cs.CR

By Sahil Kadadekar · arxiv.org

Read the full article at arXiv cs.CR

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.