RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors
RESCUE finds robot-policy monitor accuracy does not imply lower intervention cost on LIBERO tasks.
RESCUE is a counterfactual framework that judges robot-policy runtime monitors by whether acting on alarms reduces expected task-completion cost, not only by detection accuracy. Using OpenVLA-OFT and pi0.5 on LIBERO-Spatial and LIBERO-Plus, the authors ran 128,340 counterfactual branches. At tau 0.5, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures had a recoverable checkpoint under continuation, versus 34 of 66 when scripted recovery was also allowed. Under a stop-and-reset model, even an ideal detector can afford only a 2.0-3.3% false-alarm rate before unnecessary resets outweigh early stops.
- RESCUE restores simulator and policy states to compare alternative interventions.
- OpenVLA-OFT and pi0.5 were tested on LIBERO with 128,340 counterfactual branches.
- Only 14 of 66 OpenVLA-OFT failures were recoverable by continuation at tau 0.5.
- Ideal detectors tolerate only a 2.0-3.3% false-alarm rate under stop-and-reset.
- Products
- LIBERO-SpatialLIBERO-Plus
- AI models
- OpenVLA-OFTpi0.5
From the original article
1 min read at arxiv.orgRuntime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions. Yet an accurate alarm may arrive too late for recovery, while an unnecessary intervention may disrupt an execution that would otherwise succeed. We introduce RESCUE, a counterfactual framework for evaluating whether a monitor's alarms enable useful interventions.
By David Ahmedt-Aristizabal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.