Mycelium meta-study maps hallucination detection across benchmarks
Original title: Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities
The authors sort benchmarks by provenance, typology and granularity and propose Span Coverage F1.
Mycelium treats hallucination detection as a benchmark-conditioned measurement problem. Existing scores are hard to compare because benchmarks define hallucination differently. The study organizes benchmarks by provenance of outputs and evidence, factuality and faithfulness failure types, and the unit of annotation. It also examines span-level evaluation, where annotation-boundary variation changes reported results, and proposes Span Coverage F1 as a containment-based metric.
- Three axes are provenance, typology and granularity.
- Span-boundary differences can substantially change reported detector scores.
- Span Coverage F1 is proposed as a complementary containment metric.
- The paper argues against treating cross-benchmark scores as directly comparable.
From the original article
1 min read at huggingface.co
Hallucination detection is increasingly important for evaluating and deploying large language models. However, existing methods are often compared across benchmarks that define and measure hallucination in substantially different ways, making reported scores difficult to interpret across settings.
By Julia Belikova, Elisei Rykov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.