Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
P-TTT personalizes reward models with test-time fast weights, beating in-context preference conditioning without backprop.
Most RLHF pipelines train one reward model and miss individual preference differences, while in-context personalized reward models often fail to use the relations in a user's historical comparisons. Preference-Aligned Test-Time Training (P-TTT) encodes those pairwise relations into user-specific fast weights. Sequence-level update and apply operations match the granularity of preference feedback, and adaptation runs in a single forward pass without inference-time backpropagation. Experiments show P-TTT captures historical preferences more effectively and outperforms prior personalized reward methods by a large margin.
- In-context personalized reward models miss relations in historical preference pairs.
- P-TTT writes those relations into user-specific fast weights at test time.
- Updates happen in one forward pass without inference-time backpropagation.
- Sequence-level update and apply match response-level preference feedback.
- Experiments report a large margin over state-of-the-art personalized reward methods.
From the original article
1 min read at arxiv.orgReinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences.
By Bohao Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.