Skip to content
NewA calmer, faster ZeroHour

CyberGym material audit flags inconsistency in 338 of 1,507 tasks

Original title: On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity

arXiv cs.CRby Yule Ding ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

Requiring a target hit on one GLM-5.3 Level-1 run drops the differential pass rate from 85.3% to 78.8%.

A material audit of all 1,507 CyberGym tasks labels 338 (22.4%) inconsistent after correcting two labels found in stratified blind review. Coverage inconsistency accounts for 205 tasks, and 110 have descriptions and patches that jointly point to another defect. On 1,286 differential passes from one complete Level-1 GLM-5.3 run, reviewers assign 89 non-target and 10 insufficient-evidence verdicts.

Unconfirmed: Identity verdicts come from the authors' procedure, and they say those verdicts require independent validation.

Key facts
  • Non-target verdicts also appear in 39 of 1,052 consistent-task passes (3.7%).
  • Identity checks used execution, crash-signature matching, and LLM-assisted review.
  • The target is the defect tied to each reference crash report.
  • Authors say identity verdicts still need independent validation.
Analyze this story with Vani AIImpact, who is affected and what defenders should do, in a new chat

From the original article

1 min read at arxiv.org

CyberGym evaluates vulnerability reproduction by checking whether a proof-of-concept input (PoC) crashes a vulnerable program version but not a fixed version. Interpreting this differential pass as successful reproduction requires consistent task materials and evidence that the PoC triggers the intended defect.

arXiv cs.CR

By Yule Ding · arxiv.org

Read the full article at arXiv cs.CR

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.