1Password's AI patching benchmark is misleading
Trail of Bits reanalysis says 1Password's 26% AI clean-fix rate is misleading; 86% of eligible patches blocked exploits.
Trail of Bits critiques 1Password's FLAWED AI patching benchmark, arguing its 26% clean-fix headline mixes trials where agents were instructed to apply wrong fixes (22% of data) with trials that prohibited compiling or testing (36%). Restricting to reasonable conditions, 2,634 of 3,067 patches (86%) blocked the supplied exploit. Trail of Bits also reports 12.5% of 2,265 developer first fixes failed in its own 2024-2026 assessments, and released post-patch-validation and review-walkthrough agent skills.
- 1Password's 26% headline mixes bad instructions and no-testing trials
- 86% of eligible patches blocked supplied exploits under fair conditions
- Human developers fail 1 in 8 first fixes per Trail of Bits consulting data
- Trail of Bits released post-patch-validation and review-walkthrough agent skills
From the original article
11 min read at blog.trailofbits.com
1Password’s FLAWED report , published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches.
By blog.trailofbits.com via sarciszewski · blog.trailofbits.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.