CyberGym material audit flags inconsistency in 338 of 1,507 tasks
Original title: On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity
Requiring a target hit on one GLM-5.3 Level-1 run drops the differential pass rate from 85.3% to 78.8%.
A material audit of all 1,507 CyberGym tasks labels 338 (22.4%) inconsistent after correcting two labels found in stratified blind review. Coverage inconsistency accounts for 205 tasks, and 110 have descriptions and patches that jointly point to another defect. On 1,286 differential passes from one complete Level-1 GLM-5.3 run, reviewers assign 89 non-target and 10 insufficient-evidence verdicts.
Unconfirmed: Identity verdicts come from the authors' procedure, and they say those verdicts require independent validation.
- Non-target verdicts also appear in 39 of 1,052 consistent-task passes (3.7%).
- Identity checks used execution, crash-signature matching, and LLM-assisted review.
- The target is the defect tied to each reference crash report.
- Authors say identity verdicts still need independent validation.
From the original article
1 min read at arxiv.orgCyberGym evaluates vulnerability reproduction by checking whether a proof-of-concept input (PoC) crashes a vulnerable program version but not a fixed version. Interpreting this differential pass as successful reproduction requires consistent task materials and evidence that the PoC triggers the intended defect.
By Yule Ding · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.