TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
TRACE cut multi-turn attack success to the lowest rate in all 35 model-attack pairs with at most a 1.23-point utility drop.
TRACE is a token-level training objective intended to reduce multi-turn safety failures in aligned language models. It attributes refusal-related advantage across a safe response and applies contrastive erasure at high-gap positions on rejected responses, replacing a retain set with a gradient-norm penalty. Across five open-weight models and seven multi-turn attacks, it produced the lowest attack success rate in all 35 pairs. Utility on MMLU and HellaSwag fell by at most 1.23 points.
- Single-turn preference training does not by itself control multi-turn risk.
- TRACE credits safe tokens using a refusal-attributable discounted return.
- It recorded the lowest attack success rate in all 35 model-attack pairs.
- MMLU and HellaSwag scores dropped by at most 1.23 points.
From the original article
1 min read at arxiv.orgSafety-aligned large language models (LLMs) often refuse a harmful request but comply once the same goal is spread over several turns. Preference objectives score whole responses to single prompts, so their training loss alone cannot control risk on unseen histories. Our analysis gives sufficient conditions under which suppression at supervised single-turn contexts yields a bound on multi-turn trajectory risk.
By Fengpeng Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.