CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices
CESBench scores 11 LLMs on 380 IoT cryptographic-engineering tasks, with judgment far weaker than code or recall.
CESBench is a public benchmark of 380 expert-written items on cryptographic engineering security for IoT devices, covering side-channel analysis, fault injection, implementation, countermeasures, evaluation, and integration. Tasks include 209 multiple-choice items, 67 judgment items, 63 scenario diagnoses, and 41 code tasks graded by 572 test cases. Eleven open-weight and proprietary LLMs scored from 54.4% to 83.6%. Strongest results were 98.6% on multiple choice, 95.1% on code, and 88.4% on scenarios, but only 58.8% on judgment; correct verdicts often lacked sound justifications.
- 380 expert items span side-channel, fault injection, implementation, countermeasures, evaluation, and integration.
- Eleven LLMs scored 54.4% to 83.6% overall.
- Judgment peaked at 58.8%, versus 98.6% multiple choice and 95.1% code.
- 88.5% of verdicts were correct, but justifications earned 53.4% of rubric marks.
Full article248 words · extracted from arxiv.org · click to collapse
For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.21344