SCARVA checks evidence behind LLM supply-chain vulnerability calls
Original title: SCARVA: Interpretable Agentic Reasoning in Software Supply-Chain Analysis
On 54 cases, Qwen 3.8 reached 82% balanced accuracy; Claude Opus 5.5 matched annotators on 94% of claim checks.
Researchers present SCARVA, a framework for interpretable agent assessments of whether vulnerable dependencies actually endanger an application. An ANALYST agent performs the assessment while every tool call, file access, and web request is recorded; independent JUDGE agents then produce an evidence-backed report. On 54 curated cases spanning exploitable, non-exploitable, and vulnerability-free projects, the authors report Qwen 3.8 at 82% balanced accuracy and Claude Opus 5.5 agreeing with human annotators on 94% of claim checks.
Unconfirmed: Accuracy and annotator-agreement figures are author-reported and not independently replicated here.
- Evaluation uses 54 cases balanced across three exploitability classes.
- SCARVA logs tool calls, file access, and web requests.
- Authors say Qwen 3.8 beat the proprietary models they tested.
- Claude Opus 5.5 was the strongest JUDGE in their annotation check.
- The paper discusses common agent failure modes on this task.
- Products
- SCARVA
- AI models
- Qwen 3.8Claude Opus 5.5
From the original article
1 min read at arxiv.orgComplex applications may inherit security vulnerabilities from their dependencies, though a vulnerable dependency does not necessarily endanger the application. Existing tools flag dependencies associated with public vulnerabilities, and more advanced ones analyze the call graph to assess whether the vulnerable code is reachable. However, these approaches are limited to specific programming languages and ecosystems.
By Tim Barsch · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.