When Do Intrinsic Rewards Lead to Exploration?
A formal criterion shows common intrinsic rewards can fail to produce optimally informative exploration.
The paper defines exploration quality by how well a policy's history can substitute for experience under alternative policies. In a simple environment, maximizing count-based, prediction-error, empowerment, and information-gain rewards yields policies that are Pareto-suboptimal under this criterion. The authors explain those failures and state conditions under which existing intrinsic rewards encourage optimal exploration. They also construct an objective that assigns higher value whenever exploration strictly improves.
- Exploration is scored by counterfactual information acquired.
- Count-based, prediction-error, empowerment, and information-gain can fail.
- Conditions are given for when intrinsic rewards succeed.
- A new objective rewards strictly better exploration.
From the original article
1 min read at arxiv.orgIntrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available.
By Scott W. Viteri · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.