Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
Embodied-BenchForge automates embodied benchmark construction via closed-loop synthesis with verification and repair, yielding seven benchmarks for MLLM evaluation.
Embodied-BenchForge is an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts via Closed-Loop Benchmark Synthesis. Skill-Orchestrated Artifact Synthesis composes typed reusable skills while an artifact dependency graph records intermediate outputs; Requirement-Guided Verification and Repair triggers local re-execution or upstream rollback on failures. It constructs six Offline EQA benchmarks plus one interactive benchmark with 220 executable tasks, distinguishing MLLM and embodied agent capabilities in observation-based understanding and closed-loop execution.
- Closed-loop synthesis integrates forward artifact generation with backward verification and repair
- Artifact dependency graph enables localized re-execution or upstream rollback
- Built six offline EQA benchmarks and one interactive benchmark with 220 tasks
From the original article
1 min read at arxiv.orgAgentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families.
By Baoyang Jiang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.