Direct Nash Optimization aligns language models with general preferences
Original title: Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization
DNO-Prct iterates on-policy sampling, rating, and contrastive updates to raise length-controlled win rates on AlpacaEval 2.0 for Llama 3.1 8B and Mistral 7B Instruct models.
Direct Nash Optimization approximates the Nash equilibrium of a two-player preference game. The practical DNO-Prct variant performs on-policy sampling, uses LLM-based ratings against a teacher, selects large-margin pairs, and applies DPO-style updates. On AlpacaEval 2.0 it improves GPT-4o win rates from 31.7% to 45.3% for Llama 3.1 8B and 6.8% to 18.4% for Mistral 7B. Open-weight annotators retain most gains, and ablations show the importance of gold responses and data look-back. Results are single runs using GPT-4o as both teacher and judge.
- LC win rate gains: 31.7% to 45.3% (Llama 3.1 8B), 6.8% to 18.4% (Mistral 7B)
- Retains 62% (Llama) and 82% (Mistral) gains with open-weight annotators
- Ablations show gold responses and data look-back improve results by 6-10 points
- On-policy sample count and margin vary by at most 2.4 points
From the original article
1 min read at arxiv.orgDirect Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update.
By Austin Tsai · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.