Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision
Researchers propose workspace tokens, a lightweight latent robot memory distilled from train-time VLM saliency queries, improving policy performance without in-loop VLM calls.
The paper introduces workspace tokens, a latent memory representation for robotic manipulation policies trained by using a VLM to identify task-relevant current and historical information and distilling it via a set-reconstruction decoder loss. At deployment the workspace token serves as a drop-in replacement for observations, letting policies solve memory-intensive tasks without expensive in-the-loop VLM queries. Experiments in simulation and on hardware show it is both more lightweight and yields better policy performance than conditioning on full histories or in-loop VLM compression.
- Train-time VLM queries distill task-salient history into a compact workspace token
- Deployment requires no in-loop VLM reasoning, cutting compute cost
- Works as a drop-in replacement for observations in memory-intensive tasks
- Improves policy performance versus full-history conditioning in simulation and hardware
- AI models
- workspace token
From the original article
1 min read at arxiv.orgComplex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information.
By Nitish Dashora · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.