WM-PPO values world-model predictions to improve sparse-reward agents
Original title: Learning to Act with World Models by Valuing What Comes Next
With a similar number of real steps, success rises from 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld.
WM-PPO combines a world model with PPO so that, for each sampled action, the model predicts the next observation and the critic scores that successor. Those scores supply auxiliary advantages to the policy and an auxiliary target to the critic, weighted by a shared gate based on earlier prediction reliability. Policy, critic, and world model train only on the agent's interactions, and only the policy is used at inference. The paper reports higher success than PPO on ALFWorld and ScienceWorld and a 6.5-point gain over the best tested PPO refinement on ScienceWorld.
- ALFWorld success rises from PPO's 63.5% to 87.5%.
- ScienceWorld success rises from 39.3% to 52.9%.
- No teacher or privileged information is used in training.
- The policy more often finishes prerequisite steps before committing.
From the original article
1 min read at huggingface.co
A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely.
By Chenghao Gao, Zhi Zhang, Dian Jin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.