Disagreement among faithfulness evaluation criteria
Original title: Navigating Disagreement Among Criteria for Evaluating Explanation Faithfulness
Different operationalizations of explanation faithfulness disagree on evaluations, with choices justified by explanatory and normative factors.
Faithfulness requires explanations to reflect model behavior, yet existing criteria often disagree on whether the same explanation qualifies. The paper reconstructs these criteria within a faithfulness functional defined by claims, comparisons, and aggregation. Auditors and practitioners gain a framework to navigate and justify these choices in concrete settings such as adverse action notices.
- Disagreements arise from choices in claims about the model
- Reconstruction supports contextually justified evaluations
- Industry procedure choices justified by setting-specific factors
- AI models
- LLM
From the original article
1 min read at arxiv.orgFaithfulness requires that algorithmic explanations accurately reflect the model's behavior. Yet existing operationalizations of faithfulness can disagree about whether the same explanation is faithful, leaving developers and auditors to choose among conflicting evaluations. In this paper, we investigate disagreement among evaluation criteria and offer a practical path forward for evaluating explanation faithfulness.
By Grant Shanklin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.