Collaborative Personalized Preference Alignment for LLMs under Data Deficiency
Approximate Pareto Optimality learns shared aligner initializations for few-shot LLM personalization from 20 examples.
The paper studies personalized LLM preference alignment when user feedback is scarce and objectives conflict. Approximate Pareto Optimality groups users whose updates are compatible, then combines gradient descent with controlled ascent so a shared initialization lies near each group's preference-specific Pareto points. Few-shot local adaptation iteratively refines that initialization, with bounds on one-step collaborative suboptimality. On Fed-ChatbotPA and UltraFeedback, APO improves over existing methods using only 20 local examples.
- Groups users with compatible gradients to limit cross-user interference.
- Mixes descent and controlled ascent toward preference-specific Pareto points.
- Improves Fed-ChatbotPA and UltraFeedback personalization with 20 local examples.
From the original article
1 min read at huggingface.co
Real-world users often exhibit highly heterogeneous preferences over multiple objectives for LLM responses. A lightweight aligner can tailor these responses to individual preferences, but scarce user-specific feedback makes personalized training difficult. Learning shared initializations across users can support few-shot adaptation.
By Liyan Yang, Yige Yuan, Zhiqin Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.