PoEM: Predicting RL Outcomes from Existing Policies
PoEM approximates a reinforcement-learning policy for a new reward from existing post-trained models, without extra RL.
PoEM predicts the policy that reinforcement learning would produce for a new reward using models already post-trained on other rewards, avoiding another full RL run when the reward changes. If the new reward is a linear combination of existing rewards, the new log-policy is a linear combination of the existing log-policies; the authors also observe that log-policies often span an approximately low-rank subspace even when rewards are not linearly related. Combination weights can be estimated from reward or basis-policy outputs on samples. The method is validated on synthetic and real rewards across text and image modalities.
- PoEM predicts a new reward's RL policy without running additional RL training.
- A linear reward mix corresponds to a linear mix of existing log-policies.
- Log-policies often lie in an approximately low-rank subspace across rewards.
- Weights are estimated from reward or basis-policy outputs on samples only.
- Validation covers synthetic and real rewards in text and image modalities.
- Products
- PoEM
From the original article
1 min read at arxiv.orgFoundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as human alignment, correctness, or instruction following. This post-training process is computationally intensive, sometimes unstable, and has to be run from scratch every time the reward model changes or when we want to combine multiple rewards.
By Kimia Hamidieh · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.