When MLLMs miss what humans read: diagnosing and learning to read squint text
Squint OPSD lifts multimodal LLM reading of hard-to-see image text from 26.50 to 88.61 ANLS.
The authors define Squint Text as image text that humans recover by squinting but multimodal LLMs often report as absent, motivated by a production moderation pipeline. SquintBench contains 1,076 production-derived moderation images, and SquintCtrl is a controlled synthetic suite for diagnosing the failure. Frozen-weight analysis finds a coarse signal for hidden text but weak glyph identity, implicating spatial selection; suppressing high-frequency content also helps at a fixed visual-token budget. Squint OPSD trains a student from a frozen teacher given a downscaled view and the ground-truth transcription, reaching 88.61 ANLS versus 26.50 for the unadapted backbone and 24.44 for the best general-purpose model on the original view.
- SquintBench contains 1,076 production-derived moderation images.
- Spatial selection, not a missing coarse signal, is a main bottleneck.
- The student reads the original image alone at inference.
- ANLS rises from 26.50 to 88.61 versus the unadapted backbone.
From the original article
1 min read at arxiv.orgMultimodal LLMs can fail to read text embedded in an image's coarse structure (text a human recovers by squinting) and report that no text is present. We name this class Squint Text and study it as a recognition deficiency, motivated by cases from a production moderation pipeline.
By Chen Peng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.