LLM cross-checker finds OpenTitan bugs that pass UVM scoreboards
Original title: Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
Of 564 inserted bugs in four OpenTitan IPs, 68 were candidate escapes and 41 were confirmed.
Verify the Verifier uses an LLM Anomaly Cross-Checker to compare traces from a passing UVM regression with the specification. Scoreboard escapes are bugs that tests exercise and the scoreboard still passes, yet the design violates the specification. On four OpenTitan IPs, the framework confirmed 41 of 68 candidates with assertions that fail on the buggy design and hold on the original.
Unconfirmed: The authors say this is, to their knowledge, the first such framework.
- 564 bugs were inserted into four OpenTitan IPs.
- 68 were non-equivalent, exercised by tests, and still passing.
- Assertions failed on the buggy design and were proven on the original for 41 cases.
From the original article
1 min read at arxiv.orgA passing UVM regression shows the design agrees with the reference model on what the scoreboard checks, not that it meets the specification. Bugs that pass every test yet violate the specification, termed scoreboard escapes, survive mature testbenches.
By Ana E. Sanchez Villalobos · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.