RTL repair scores can miss whether models used the broken design
Original title: What Do RTL Repair Benchmarks Actually Measure?
A new 459-case benchmark finds simulation and formal scoring can disagree, and specification-only baselines swing widely.
The authors release RTLRepairDataset with 5,784 broken-and-fixed training pairs, RTLRepairBench with 459 held-out cases, and a harness for simulation and formal equivalence. On one cohort, simulation and formal scores gave opposite signs for the effect of providing a specification; reset-aware rescoring attributed nearly all disagreement to reset and first-cycle semantics. Across six models and 274 mined failures, the specification-only minus specification-conditioned recovery estimate ranged from -8.5 to +18.3 percentage points. They recommend reporting comparison semantics, oracle coverage, and a specification-only baseline. Materials are at github.com/Evaaaz/RTLRepairBench.
- RTLRepairBench has 374 syntactic and 85 behavioral mutations.
- Held-out evaluation covers 459 cases; training has 5,784 pairs.
- The recovery-point spread across six models is -8.5 to +18.3 points.
- Recommended controls include a specification-only baseline and undecided-case disclosure.
From the original article
1 min read at arxiv.orgRTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds.
By Yeyin Eva Zhu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.