Rules to Tools: Executable Checks for LLM Agents in Scientific Computing
Executable checks slightly improve LLM scientific-coding agents' program repair versus written requirements on SciCode.
Rules to Tools (R2T) gives scientific coding agents prepared executable checks of public requirements rather than written specifications alone. On matched SciCode repair groups, complete repair was 26/30 with text versus 29/30 with checks, and 13/16 versus 15/16 on an eight-ID cohort. A larger shared-definition cohort tied at 13/24, while a PDE comparison scored 23/24 for detailed text and 24/24 for checks with 31.2% lower model output. Gains are task-dependent, and public CPU use rises when checks are available.
- Complete repair reached 29/30 with checks versus 26/30 with text.
- Eight-ID cohort scored 15/16 versus 13/16; interval includes zero.
- Shared-definition SciCode cohort tied at 13/24 per group.
- PDE checks scored 24/24 and cut model output 31.2 percent.
From the original article
1 min read at huggingface.co
Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation.
By Jingjie Ning, Guojiang Zhao, Chen Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.