Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks
Datalab released OmniExtractBench, an open 620-document benchmark scoring PDF-to-JSON extraction with auditable verdicts.
Datalab released OmniExtractBench, an open benchmark that scores how accurately systems fill a JSON schema from a PDF. It combines 620 documents from four existing suites and grades every value with one deterministic scorer that emits six verdict types. Content-based row alignment and dropping null addresses are meant to stop positional table failures and schema-padding from distorting scores. On the full corpus, Datalab’s accurate mode led at 93.85 accuracy, with Datalab balanced at 93.48 and Reducto deep_extract v2 at 93.47.
- Pools 620 PDFs from ExtractBench, Datalab, LongExtractBench, and LongArray-Extract.
- Deterministic scorer assigns one of six auditable verdicts per value.
- Content-based Hungarian row pairing avoids zeroing a table after one missed row.
- Datalab accurate scored 93.85; balanced and Reducto deep_extract v2 tied near 93.5.
- Scorer is Apache 2.0 on PyPI; dataset is CC BY 4.0 on Hugging Face.
- Organizations
- DatalabLlamaIndexReductoExtendmicro1Hugging Face
From the original article
4 min read at marktechpost.comContent-based row matching, 6 per-value verdicts and a null rule make OmniExtractBench an extraction benchmark anyone can audit. The post Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks appeared first on MarkTechPost.
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.