Paper ties RL memory retrieval and credit to value of information
Original title: Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
A learned world model estimates that quantity from archived rollouts, defining an optimal retrieval set and a TD-target floor.
The paper argues that costly memory attention and delayed credit assignment in reinforcement learning are both governed by the value of information. It proposes estimating that quantity from archived rollouts with a learned world model. Read at the observation, the estimate is a provable lower bound on the optimal value of the acquiring action and floors its TD target. Read at the decision, it defines the optimal retrieval set. A pilot system trained with this signal solves a memory task with reduced context.
Unconfirmed: The memory-task result is a pilot reported by the authors, not an independent evaluation.
- Attention over the whole memory store is costly under robot latency and energy limits.
- Credit for a useful retrieved memory must cross the entire sequence.
- The observation-time estimate floors the TD target of the acquiring action.
- The decision-time reading defines which memories are worth retrieving.
- A pilot retrieval system solves a memory task with reduced context.
From the original article
1 min read at arxiv.orgMemory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence.
By Daniel Strauss · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.