LambdaMix learns steerable policies with jointly trained low-rank adapters
Original title: Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters
One training run mixes per-objective adapters so preferences can change at inference without more optimization.
Researchers introduce LambdaMix, a parameter-efficient method that jointly trains one low-rank adapter per objective over a frozen base model using group-relative online reinforcement learning. A preference vector selects the adapter mixture and weights separately normalized rewards. Experiments cover four models on helpfulness and harmlessness, math with instruction following, and comment rewriting that preserves wording while reducing personal-attribute clues. On the first two tasks, LambdaMix reports higher hypervolume than conditioned and policy-merging baselines, using fewer completions and adapter parameters than specialist GRPO policies.
- The two largest models were evaluated on attribute-reducing comment rewrites.
- LambdaMix stores fewer adapter parameters than a set of GRPO specialist policies.
- Varying mixtures and objective weights together widened score ranges and coverage.
- Architecture and initialization were held fixed in that comparison.
From the original article
1 min read at huggingface.co
Language model deployment often requires different trade-offs between competing objectives as user preferences change, yet standard post-training methods typically produce a single policy for a fixed objective weighting. We introduce LambdaMix, a new parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning.
By Constantinos Karouzos, Nikolaos Aletras · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.