arXiv audit finds AI humanizers alter facts in most documents
Original title: Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers
Eleven commercial AI humanizers changed facts in 35-65% of test documents while an open model retained most numbers and names.
Researchers audited 11 commercial AI humanizers plus one open model on whether they preserve checkable facts such as numbers, dates, names and quoted spans. In the October 2026 leaderboard cycle all 11 tools altered at least one fact in 35-65% of documents containing them, mostly quoted spans. Counting numbers and names alone, 10 of 11 tools still changed at least one in 20% or more of the documents. A general-purpose detector called Jev separated human originals from humanized rewrites with AUROC 0.864. Hypotheses were frozen before runs.
- All 11 commercial tools altered facts in 35-65% of documents
- Open 12B model kept 97% numbers and names in federal text but reworded 67-78% quoted spans
- No clear link between fact retention and evasion (Spearman rho = -0.46)
- Jev detector AUROC 0.864 versus human texts
- Released rule-based checker validated at precision 0.84-1.00
From the original article
1 min read at arxiv.orgAI "humanizers" rewrite machine-generated text so that detectors classify it as human-written. Public leaderboards score evasion and embedding similarity, but not whether checkable details survive. We check whether numbers, dates, person and organization names, and quoted spans survive each rewrite, using a rule-based checker validated by two AI annotators (precision 0.
By Ziyuan Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.