Skip to content
NewA calmer, faster ZeroHour
StoryAI researchinfo

Local Open-Weight LLM Agents Solve CyberGym Tasks with Best-of-N Sampling

2 sources · 2 articlesFirst reported Updated ()2
What's new

First merged summary.

Merged summary · x-ai/grok-4.7 · rewritten as coverage arrivesSummaries by Vani AI

Local open-weight LLM agents solve most of six easier CyberGym tasks down to 4B size, with turn-level best-of-N sampling yielding 5-10% accuracy gains on long-horizon tasks.

Twelve configurations of local models spanning three generations, two architectures, and three quantization levels were evaluated on ten CyberGym tasks for vulnerability-reproduction. Every configuration solves most of the six easier tasks. For harder tasks the 27B model is required. Turn-level best-of-N sampling with generator-selector LLMs yields 5-10% accuracy gains and improved consistency on long-horizon tasks. Agent trajectories often name the vulnerable function and correct hypothesis.

Key facts
  • Configurations down to 4B solve most of six easier CyberGym tasks
  • 27B model required for harder tasks
  • Turn-level best-of-N sampling provides 5-10% accuracy improvement
  • Improved consistency across repeated attempts
  • Quantization affects speed but not solved tasks
  • Agent trajectories name vulnerable function
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

What each outlet adds

beyond the shared facts
  • arXiv cs.CRAnalysis of local open-weight LLM agents on vulnerability-reproduction tasks with CyberGym benchmark
  • arXiv cs.AITurn-level best-of-N sampling for improved accuracy and consistency in LLM agents
AI models
LLMs

Coverage

oldest first · one row per article · UTC
  1. · 5d ago arXiv cs.CRFirst report

    Local Open-Weight LLMs Solve Most Vulnerability Reproduction Tasks

    All tested configurations solve most of six easier CyberGym tasks down to 4B size with 27B required for the rest.

  2. · 4d ago arXiv cs.AI / cs.LG / cs.CL

    Turn-level best-of-N sampling improves LLM agent accuracy by 5-10%

    Turn-level best-of-N sampling with generator-selector LLMs yields 5-10% accuracy gains and improved consistency on long-horizon tasks by selecting optimal actions at each step.