Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
semCER weights OCR character edits by judged severity, matching readers better than standard CER.
semCER keeps Character Error Rate's alignment and denominator but assigns each grapheme edit a severity from an LLM judge, without word segmentation. The severity scores track human reader ratings edit by edit in English, Vietnamese, and Chinese, and readers preferred semCER over normalized CER when the metrics disagreed in English and Chinese. Across 16 languages, a wrong diacritic is charged almost nothing in Spanish but nearly as much as a wrong letter in Telugu. The authors release 61 million judged edits, aggregated reader ratings, and a distilled scorer.
- semCER charges each grapheme edit a severity score from an LLM judge.
- It needs no word segmentation and keeps CER's alignment and denominator.
- Readers sided with semCER over normalized CER in English and Chinese.
- Diacritic penalties vary widely across 16 languages, from Spanish to Telugu.
- Authors release 61 million judged edits and a distilled scorer.
From the original article
1 min read at huggingface.co
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation.
By Nguyen Nguyen, Yongyi Zang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.