Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
Explanatory World Models reassign delayed RL credit and beat PPO and LaRe on long-horizon tasks.
Explanatory World Models assign credit for delayed reinforcement-learning outcomes by using a frontier language model to explain which earlier events contributed, then redistributing reward with a conservative hierarchical operator that preserves the undiscounted return. Variants draw on actual causation, explanatory simplicity, and hard-to-vary explanations. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with the largest gains when rewards are most delayed, and it also beats SFT and LaRe offline and A2C online on KuaiRand. An offline industry generative-recommender evaluation shows improved Recall and higher behavioral proxies.
- EWM attributes delayed rewards using causal, simple, and hard-to-vary explanations.
- A hierarchical operator redistributes reward while preserving undiscounted episode return.
- Largest gains versus PPO and LaRe occur when rewards are most delayed.
- An offline industry recommender evaluation improved Recall and behavioral proxies.
From the original article
1 min read at arxiv.orgGenerative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency.
By Oriol Corcoll · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.