CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices
CESBench scores 11 LLMs on 380 IoT cryptographic-engineering tasks, with judgment far weaker than code or recall.
CESBench is a public benchmark of 380 expert-written items on cryptographic engineering security for IoT devices, covering side-channel analysis, fault injection, implementation, countermeasures, evaluation, and integration. Tasks include 209 multiple-choice items, 67 judgment items, 63 scenario diagnoses, and 41 code tasks graded by 572 test cases. Eleven open-weight and proprietary LLMs scored from 54.4% to 83.6%. Strongest results were 98.6% on multiple choice, 95.1% on code, and 88.4% on scenarios, but only 58.8% on judgment; correct verdicts often lacked sound justifications.
- 380 expert items span side-channel, fault injection, implementation, countermeasures, evaluation, and integration.
- Eleven LLMs scored 54.4% to 83.6% overall.
- Judgment peaked at 58.8%, versus 98.6% multiple choice and 95.1% code.
- 88.5% of verdicts were correct, but justifications earned 53.4% of rubric marks.
From the original article
1 min read at arxiv.orgFor Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering.
By Wenquan Zhou · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.