Predicting Alignment Generalization with Value Representations
Activation-based value representations predict LLM alignment generalization far better than textual descriptions of values.
The paper defines alignment generalization prediction: estimating how fine-tuning an LLM to follow one value changes behavior across held-out values. Across 66 values from modern alignment targets, representations from model activations correlate 0.45 with the generalization matrix, compared with 0.05 for textual-description baselines. Similarity among values in a multi-value alignment target, measured with these representations, correlates with model robustness. The authors also report initial evidence of a shared, model-independent value space and an empirical taxonomy of LLM values.
- Predicts how fine-tuning on one value changes held-out value behavior.
- Analysis covers 66 values drawn from modern alignment targets.
- Activation methods correlate 0.45 versus 0.05 for text baselines.
- Value similarity inside multi-value targets correlates with robustness.
- Authors report early evidence of a shared value space and taxonomy.
From the original article
1 min read at arxiv.orgLLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on alignment evaluations, training models on sets of narrow behaviors still influences their behavior across unseen contexts and environments in unexpected ways.
By Andy Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.