CuraEval finds demo curation rarely beats full robot training data
Original title: CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data
On robomimic, no curated subset significantly beat full-data training after Holm correction, though intervals still allow gains.
CuraEval compares demonstration scores and fixed-size subsets against random selection and full-data training on mixed-skill robomimic data with Diffusion Policy and BC-RNN. The evaluated PSD score depends only on episode length and squared position magnitudes, and nearly matches episode-length ranking on square (ρ=0.999). Selecting the shortest 50% of demonstrations raises square success from 70.7% under random selection to 76.5% with Diffusion Policy. No curated condition significantly outperforms full-data training after Holm correction.
Unconfirmed: Confidence intervals still permit useful gains even though no curated condition is significant after Holm correction.
- An adapted geometric-entropy LOO selector drops can success at 25% retention from 94.7% to 74.0%.
- On square, that selector reaches 41.7%, below three random subsets at 59.8–62.5%.
- Length selection at 75% retention moves square success from 76.7% to 86.7% under the original protocol.
- After checkpoint, rollout, and seed changes, the same comparison falls from 80.5% to 79.5%.
- Code is at github.com/jiuyixu25/CuraEval.
- AI models
- Diffusion PolicyBC-RNN
From the original article
1 min read at arxiv.orgRobot imitation learning uses demonstrations from operators of varying skill, but selecting training data remains challenging. Existing methods score demonstrations using smoothness, diversity, or estimated influence on policy performance, but differences in selection and evaluation protocols make it difficult to determine whether they outperform simple baselines.
By Jiuyi Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.