Token readout choice confounds POPE VLM hallucination scores
Original title: Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation
A two-token yes/no logit lookup never matches LLaVA-1.5-7B outputs, shifting adversarial F1 by 6.13 points.
Authors reproducing a POPE hallucination-mitigation result found their LLaVA-1.5-7B adversarial F1 already sat above published baselines because evaluation compared the wrong vocabulary IDs. The model emits space-prefixed " Yes" and " No" after the ASSISTANT suffix. An eight-token lookup scores F1 0.8221, 0.8498, and 0.8713. The same protocol confound appears across four further VLMs from three tokenizer families, and nine inference-time corrections did not beat the fixed baseline.
Unconfirmed: How much of surveyed papers' reported adversarial gains would survive a matched readout and precision is unresolved.
- LLaVA-1.5-7B adversarial F1 was 82.21, above baselines of 81.00, 79.30, and 73.54.
- IDs 3582 and 1217 never appear across 9,000 greedy-decode questions.
- Porting LLaVA-1.5 IDs drops F1 to 0.00 on two disjoint-vocabulary models.
- Wrong predictions draw more image attention than correct ones (Cohen's d=-0.46).
- The reference run used 4-bit NF4, so retained gains stay unresolved.
From the original article
1 min read at huggingface.co
We set out to reproduce a recent hallucination-mitigation result on the POPE benchmark. The LLaVA-1. 5-7B baseline we measured, 82.
By Kesav Kumar Jayakumar, Karthigeyan Thilak · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.