When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence
Open vision-language models miss robot-failure causes and slash refusal when the refusal option is listed first.
The authors build a simulated robot-failure benchmark with known injected causes and compare what camera images and force sensors reveal, with checks against data leakage. Force data reaches 0.99 diagnostic accuracy, while no image method exceeds 0.55. Six open vision-language models stay at or below a majority-class baseline on frames, and moving the refusal option from last to first collapses refusal from 78-100% to 0-6% in three of six model-family pairs. Giving force readings as text produces the first above-baseline diagnoses in four models, and one human answer lifts accuracy to about 0.70-0.81, but ask rates ignore a fourfold change in question cost.
- Force data scores 0.99; no image method exceeds 0.55
- Moving refusal first drops rates from 78-100% to 0-6%
- Image accuracy stays at or below a majority baseline
- Force text lifts four of six models above baseline
- Ask rates ignore a fourfold rise in question cost
From the original article
1 min read at arxiv.orgA robot that fails at a task faces the first decision in corrective dialogue: act on its own diagnosis, consult another onboard sensor, or interrupt a person. Choosing well requires knowing how much the robot's sensors reveal about the cause and how reliable the robot's own diagnosis is.
By Eshika Pathak · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.