Skip to content
NewA calmer, faster ZeroHour

Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks

MarkTechPostby Asif Razzaq ()4 min read4 · 2 reads
AI summary · grok-4.7Summaries by Vani AI

Datalab released OmniExtractBench, an open 620-document benchmark scoring PDF-to-JSON extraction with auditable verdicts.

Datalab released OmniExtractBench, an open benchmark that scores how accurately systems fill a JSON schema from a PDF. It combines 620 documents from four existing suites and grades every value with one deterministic scorer that emits six verdict types. Content-based row alignment and dropping null addresses are meant to stop positional table failures and schema-padding from distorting scores. On the full corpus, Datalab’s accurate mode led at 93.85 accuracy, with Datalab balanced at 93.48 and Reducto deep_extract v2 at 93.47.

Key facts
  • Pools 620 PDFs from ExtractBench, Datalab, LongExtractBench, and LongArray-Extract.
  • Deterministic scorer assigns one of six auditable verdicts per value.
  • Content-based Hungarian row pairing avoids zeroing a table after one missed row.
  • Datalab accurate scored 93.85; balanced and Reducto deep_extract v2 tied near 93.5.
  • Scorer is Apache 2.0 on PyPI; dataset is CC BY 4.0 on Hugging Face.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

4 min read at marktechpost.com

Content-based row matching, 6 per-value verdicts and a null rule make OmniExtractBench an extraction benchmark anyone can audit. The post Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks appeared first on MarkTechPost.

MarkTechPost

By Asif Razzaq · marktechpost.com

Read the full article at MarkTechPost

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.