RoboJev-27B open model scores robot manipulation success from video
Original title: RoboJev: An Open Vision-Language Judge for Robot Success Detection
On 268 held-out SO-101 episodes the authors report 95.2% balanced accuracy and 0.993 AUROC.
RoboJev-27B, an open-weight variant of JEV-27B-VL, judges whether a robot finished a task from an instruction, ten frames, and optional auxiliary finals. A rank-32 LoRA trains only the language model on 52,598 episodes, with the vision encoder frozen and a KL penalty on research data. Authors report 0.890 AUROC on 459 unseen-lab or robot episodes and 93.3% accuracy on 135 community episodes. Weights, code, and 20,628 logged predictions are released.
Unconfirmed: Accuracy figures are the authors' results, and the frontier-model comparison is not like-for-like.
- Median serial inference is 0.26 seconds per episode on one H100.
- The base model reaches 0.900 AUROC on the same unseen-lab inputs.
- A calibrated two-token readout emits success and failure probabilities.
- Frontier VLM comparisons use four primary-camera images, so they are not matched.
- Evaluation logs contain 29 model and protocol entries.
- Products
- SO-101
- AI models
- RoboJev-27BJEV-27B-VL
From the original article
1 min read at huggingface.co
Many robot manipulation videos do not indicate whether the task was successfully completed. RoboJev-27B is an open-weight vision-language model that takes a task instruction, 10 frames of a manipulation attempt, and (if available) final frames from auxiliary cameras and predicts whether the robot accomplished the task.
By Josef Liyanjun Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.