Agent audit finds a machine-generated farm in Slovak FineWeb2
Original title: Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2
Three runs on 26.5 million documents found seven issues; the largest slice is 1,408,437 documents, 5.32 percent.
Researchers used a budgeted, read-only LLM agent to explore the Slovak portion of FineWeb2 and turn findings into executable tests that a native speaker then checked. Three runs costing $2.21–$2.84 each reported seven findings across four phenomena. Blinded labels supported six findings and not one. A static sampling baseline validated more selectors at under half the cost, while the compiled suite scans the full corpus in 0.7 seconds without a GPU.
Unconfirmed: Blinded labels did not support one of the seven findings.
- The Slovak slice contains 26.5 million documents.
- The content farm is 1,408,437 documents, and 86.6 percent passed stricter thresholds.
- Supported defect rates ranged from 0.53 to 1.00 versus 0.08 in a random control.
- An LLM classification pass is estimated at $12,000–$90,000.
- The framework, Slovak tests, validation data, and agent trajectories are released.
From the original article
1 min read at arxiv.orgMultilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.
By Marek Šuppa · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.