You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs
Study quantifies DPO-tuned LLMs undershooting requested emotional intensity, tracing the gap to candidate-pool extremity rather than conditioning format.
Conditioning an instruction-tuned LLM on continuous valence-arousal targets yields gain of only 0.26 for valence and 0.13 for arousal on Llama-3.1-8B, far below faithful control of 1.0. The authors attribute undershoot to neutral-heavy preference corpora like EmoBank and candidate pools lacking extreme affect, leaving DPO without extreme exemplars. Uniform target coverage with a hotter candidate pool raises valence gain to 0.40 on Llama-3.1-8B and 0.44 on Qwen3-8B, with modest in-distribution cost; arousal gains remain unstable across seeds.
- Measured valence gain 0.26 and arousal gain 0.13 on Llama-3.1-8B
- Candidate-pool extremity, not conditioning format, bottlenecks intensity control
- Resampling raises valence gain to 0.40 (Llama) and 0.44 (Qwen3-8B)
- Arousal control remains unreliable across seeds (0.14 +/- 0.07)
- DPO lacks extreme exemplars when preference corpora are neutral-heavy
- Organizations
- EmoBank (dataset)
- AI models
- Llama-3.1-8BQwen3-8B
From the original article
1 min read at arxiv.orgAsk a language model to respond "very excitedly," and its output is typically only mildly more energetic. We quantify this effect. We condition an instruction-tuned LLM on a continuous Valence-Arousal (VA) target, where valence measures how pleasant a state is and arousal how activated it is, measure the achieved affect with a frozen regressor, and sweep the requested target from -1 to +1.
By Hyunwoo Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.