Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models
Million-parameter ranking models personalize LLM generation more accurately and cheaply than billion-parameter reward models.
The paper argues personalized LLM generation has large unused test-time headroom and is better treated as candidate matching than as a generator-capacity problem. Billion-parameter reward models are poorly calibrated for personalization and too expensive to score large candidate pools. The authors introduce roughly million-parameter MLP ranking models that reuse the base generator's internal embeddings. Across nine datasets in three settings, these rankers outperform generalist reward models on every dataset using under 0.4% of the parameters and four orders of magnitude lower scoring latency.
- Personalized generation is framed as candidate matching suited to Best-of-N.
- Million-parameter MLPs reuse generator embeddings to score large candidate pools.
- Rankers beat billion-parameter reward models on all nine datasets.
- They use under 0.4% of parameters and far lower scoring latency.
From the original article
1 min read at arxiv.orgAligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment.
By Qiyao Ma · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.