SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation
SimuVerity benchmarks agents on 101 engineering-grade Simulink tasks; the best system scores only 42.86.
SimuVerity is a benchmark of 101 text-to-executable Simulink model-generation tasks spanning ten engineering domains, with executable-system profiles and four families of native simulation scenarios. A hierarchical evaluator checks artifact delivery, native executability, and engineering qualification, then scores qualified models on six dimensions including accuracy, mechanistic fidelity, and robustness. Six agent systems were evaluated, and the best overall score was only 42.86. The authors argue that structural similarity is a poor proxy for engineering performance.
- SimuVerity contains 101 text-to-executable Simulink tasks across ten domains.
- Evaluation checks delivery, native executability, and engineering qualification.
- Qualified models are scored on six engineering performance dimensions.
- Six agent systems were tested; the best overall score was 42.86.
- Structural similarity poorly predicts engineering performance.
From the original article
1 min read at huggingface.co
Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains.
By Ruiqi Zhang, Jiahao Wang, Mingxuan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.