How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
A self-audit shows small LLM evaluation tables reliably flag the worst models but not the best.
The authors self-audit an LLM prompt-structure evaluation covering eight open model variants from five families, from 8B to 675B parameters, with caching disabled and 293 raw intermediate representations stored. Repeated identical calls are unstable: mean node-set Jaccard ranges from 0.39 to 0.96, and 72% of prompt-model cells are never perfectly consistent. A joint cluster bootstrap keeps the two worst models' ranks in 99% and 86% of replicates, but the top two only in 68% each. Two defensible ways of merging repeats change four of eight rows and shift the headline by 7 points, reproducibility does not equal accuracy, and four endpoints were withdrawn within ten weeks.
- Eight open models from 8B to 675B were audited with caching off.
- Mean node-set Jaccard ranged from 0.39 to 0.96 across cells.
- Only the worst models kept stable ranks under cluster bootstrap.
- Alternate merge rules changed four of eight rows and moved the headline 7 points.
- Four of eight endpoints were withdrawn within ten weeks.
From the original article
1 min read at huggingface.co
Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begin with.
By Dipankar Sarkar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.