DocXEval benchmarks explainability methods for multimodal document models
Original title: DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding
Occlusion ranked most faithful and interpretable across 13 methods, while some metrics were unreliable.
DocXEval adapts existing explainability methods and metrics to multimodal document understanding that uses text, layout, and image, including multi-target variants for semantic entity recognition. The authors evaluate 13 methods on 3 tasks, 8 datasets, 4 models, and 17 metrics, and report that few methods are consistent, with Occlusion strongest overall. A meta-evaluation finds moderate to strong agreement among some faithfulness and complexity metrics and flags others as unreliable. Code is released at the stated GitHub repository.
Unconfirmed: The abstract does not name the four models or the datasets used in the evaluation.
- Evaluation covers 13 XAI methods, 8 datasets, 4 models, and 17 metrics.
- Analyses cover modality contribution, spatial attribution, and possible dataset bias.
- Authors publicly release code at github.com/saifullah3396/docxeval.
- Only a few methods performed consistently across all evaluation criteria.
From the original article
1 min read at arxiv.orgMultimodal transformer models have driven unprecedented performance breakthroughs in modern document understanding. However, the inherent black-box nature of these models remains a major obstacle to their safe deployment in high-stakes sectors.
By Saifullah Saifullah · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.