Alignment tax is graded and task-selective in DPO
Original title: The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention
Length-normalized DPO with SFT retention causes selective capability loss, preserving extraction but harming router tasks.
The study fine-tunes Qwen3.5-4B with sparse LoRA and length-normalized DPO plus SFT retention on structured financial tasks. It tracks downstream capability as implicit reward margin grows. Findings show the alignment tax is graded: extraction is preserved while router degrades severely. The margin acts almost entirely via suppression of rejected completions.
- Alignment tax is task-selective
- Extraction preserved with minimal loss
- Router degrades severely
- Margin accounted for by rejected completions
- Gold-logp deflation varies by task
- AI models
- Qwen3.5-4B
From the original article
1 min read at arxiv.orgSetting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.
By Ye Yuan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.