CRANE retrieves real examples as multimodal counterfactual explanations
Original title: CRANE: Post-Hoc Counterfactual Retrieval for Multimodal Classifiers
Multimodal Consensus Retrieval was best or joint-best on three of four datasets across three fusion strategies.
The paper presents CRANE, a post-hoc, model-agnostic framework that retrieves real training examples from the target class as nearest-neighbour counterfactuals for multimodal classifiers. It argues that applying modality-specific attribution separately fails to produce coherent explanations and does not transfer to arbitrary pretrained multimodal models. Across four heterogeneous datasets, intersecting per-modality neighbour sets was the strongest strategy, while unimodal retrieval stayed competitive when one modality dominated.
- Explanations are drawn from the joint training distribution, so plausibility is by construction.
- MC-R intersects per-modality neighbour sets to find instances close in every modality.
- Unimodal strategies remain competitive when a single modality dominates the decision boundary.
From the original article
1 min read at arxiv.orgCounterfactual explanation methods have been introduced for unimodal settings, with dedicated frameworks for tabular, timeseries, text, and image data, yet no post-hoc method has been designed for classifiers that learn from multimodal inputs.
By Franco Rugolon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.