Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Researchers build a 12,000-dilemma multilingual dataset and use orthogonalized task vectors to isolate and steer ethical value preferences in LLMs.
The paper introduces a 12,000-instance dataset of two-option moral dilemmas spanning Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty conflicts, translated into Hindi, Arabic, Spanish, and Chinese. GPT-5-mini consistently favored Honesty over Autonomy across all five languages, while Llama-3.2-1/3B showed strong first-option bias that plain fine-tuning and DPO removed, lifting accuracy above 98%. Orthogonalizing value preference task vectors against the general instruction-following vector enabled task arithmetic to produce a model with the opposite value stance.
- 12,000-instance dilemma dataset covers three value conflicts in five languages
- GPT-5-mini consistently prefers Honesty over Autonomy when no policy is given
- DPO and fine-tuning remove Llama-3.2 first-option bias, exceeding 98% accuracy
- Orthogonalized task vectors enable steering a model to opposite value stances
Full article190 words · extracted from huggingface.co · click to collapse
Large Language Models (LLMs) are increasingly deployed in applications that must weigh clashing moral values, yet even strong models exhibit hidden biases and brittle instruction-following across languages. We introduce a 12,000-instance dataset of two-option dilemmas covering pairwise three value conflicts: Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty, along with their translations into Hindi, Arabic, Spanish, and Chinese, to probe cross-lingual behavior. Benchmarking on GPT-5-mini reveals that it consistently favors Honesty over Autonomy across all five languages when no policy is given. The Llama-3.2-1/3B models exhibit strong first-option bias; however, both plain fine-tuning and Direct Preference Optimization fine-tuning effectively remove this bias, increasing accuracy to greater than 98%. In order to decouple the effect of learning correlations in the dataset from abstract values, we propose a task vector transfer based experiment where after computing the task vectors for a direction of value preference we orthogonalize it with respect to the general instruction following vector. Our experiment shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.21094