Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks
A 267-word specification frame cut security defects in code from five frontier models, and never made any model worse.
A pre-registered paired study tested whether a fixed 267-word specification frame reduces defects in LLM-generated backend code covering money arithmetic, time handling, retry safety, and access control. Fifty realistic finance, healthcare, and insurance tasks were run through five frontier models from five vendor lineages, once bare and once with the frame, and scored by nine deterministic AST checkers plus the Bandit scanner. Mean findings fell by 0.16 to 0.70 per task in every model; where the arms differed, the frame won 95 of 100 times and never made any model worse in any domain. Bandit found 53 medium-or-high issues in the bare arm and 11 in the frame arm, and all 500 outputs plus analysis code are published with a DOI.
- Fifty finance, healthcare, and insurance backend tasks were pre-registered.
- Five frontier models each produced bare and specification-framed code.
- Mean defects fell 0.16 to 0.70 per task in every model.
- Bandit medium-or-high findings dropped from 53 to 11.
- All 500 outputs, checkers, and the pre-registration are published.
From the original article
1 min read at huggingface.co
Code generated by large language models passes security checks at a rate that has barely moved in four years. In regulated backends, the defect classes that matter most are money arithmetic, time handling, retry safety, and access control. Teams answer with instruction files, yet the largest controlled study of instruction files to date found no benefit.
By Sandeep Dhuri · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.