Local Open-Weight LLMs Solve Most Vulnerability Reproduction Tasks
Original title: Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks
All tested configurations solve most of six easier CyberGym tasks down to 4B size with 27B required for the rest.
Twelve configurations of local models spanning three generations, two architectures, and three quantization levels evaluated on ten CyberGym tasks. Every configuration solves most of the six easier tasks. For harder tasks the 27B model is required. Quantization affects speed but not which tasks are solved. Agent trajectories often name the vulnerable function and correct hypothesis.
- Every configuration down to 4B model solves most of six easier tasks
- 27B model required for harder tasks
- Quantization changes speed not solved status
- Mixture of experts reaches scores faster
- AI models
- LLMs
From the original article
1 min read at arxiv.orgOpen-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service.
By Jorge García-Carrasco · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.