RealEmbedBench scores LLMs on 5,142 embedded software tasks
Original title: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks
Harmony-to-RT-Thread translation averages a 0.1667 pass rate, and executable state-machine code remains weak.
RealEmbedBench covers code generation and completion, state machine modeling and implementation, and cross-OS translation for application-layer embedded software. Six unnamed LLMs show large differences by task and model. Local project context, especially the code window, strongly affects generation and completion. Models handle state-machine modeling better than producing executable implementations, and translation quality varies sharply by direction.
Unconfirmed: The six evaluated LLMs are not named in the abstract.
- The benchmark contains 5,142 samples across three task types.
- Average pass rate for Harmony-to-RT-Thread translation is 0.1667.
- Authors say prior embedded benchmarks emphasize simulated hardware-in-the-loop checks.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) perform well on general coding tasks, but their capabilities in application-layer embedded software development remain underexplored. Existing embedded benchmarks primarily focus on hardware-in-the-loop validation in simulated environments.
By Yujun Huang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.