More specialist model contexts do not beat tool evidence in assurance tests
Original title: Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance
Across 13,638 requests, extra contexts raised token use without safer clearances, and raw verifier feedback also increased incorrect verdicts.
A study asks whether extra specialist language-model contexts improve software assurance when deterministic tool evidence already exists. Systems using Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, completed 13,638 requests across Java warning controls, OWASP Benchmark, Defects4J test generation, SV-COMP assertion verdicts, and misleading evidence packages. Raising context count from 1 to 50 increased token use 49-fold on eight warnings without improving unsafe clearance. On 32 SV-COMP tasks, raw verifier feedback raised mean correct verdicts from 12.7 to 22.0 but incorrect verdicts from one to seven. No target defect was found in three Defects4J bugs.
Unconfirmed: Findings are descriptive for the tested settings; one Defects4J bug was unsupported by the test adapter.
- On 160 OWASP cases, the ten-context arm assessed only 96 cases.
- A JBMC-plus-JVM baseline returned 26 correct verdicts and six unresolved outcomes.
- Models rejected 70-100% of applicable counterexamples in 120-package sets.
- Scope checks classified every constructed evidence package correctly.
- AI models
- Qwen2.5-Coder-7B-Instruct
From the original article
1 min read at arxiv.orgCan additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.
By S M Abdulla Al Mamun · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.