Disagreement among faithfulness evaluation criteria
Different operationalizations of explanation faithfulness disagree on evaluations, with choices justified by explanatory and normative factors.
Stories per day · 30 days
Relatedbenchmarks2evaluation2autoformalization1chain-of-thought1context-compression1explanation1formal-verification1interpretability1jacobian1keep-bench1lean1llm1llm-judges1long-context1rag1retrieval1veritas1
11 stories, newest first · one card per story
Stories tagged #faithfulness
LASAGNA finds RAG component gains often fail to compose
Three techniques predicted a 47% combined gain but delivered 17% together across 450 configurations.
KEEP-Bench finds context compressors often drop required evidence
The best open-weight 31B model keeps all required evidence in 66.8% of 4,102 cases, and the best proprietary model in 61.6%.
Verbalising Model's Computation for Chain-of-Thought Faithfulness Detection
VERITAS verbalizes layer-wise computation to detect unfaithful chain-of-thought, beating LLM judges by 28.4% balanced accuracy.
Lean compilation overstates faithfulness of statement autoformalization
A tool-augmented agent compiled 87.2% of 227 statements, but only 57.7% passed a two-judge semantic check.
Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
Efficient reasoning fine-tuning usually reduces chain-of-thought faithfulness, while monitorability stays more robust.
MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations
MEA trains a multi-agent system to produce more faithful natural-language explanations across tabular, text, and vision models.
Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment
A new metric shows LLMs' chain-of-thought often mismatches internal reasoning, and post-training can improve that alignment.
Memory Is a Derivation: The Distributed-Evidence Paradox in Long-Term Agents
DerivAudit checks whether long-term agent memories are supported by the history available when written.
Right Answer, Wrong Reason: Accuracy, Consistency, and Consensus Are Misleading Indicators of LLM Faithfulness in Clinical Decision Support
Only 23.3% of concepts cited by clinical LLMs were causally necessary on MedQA.
ReCite: Agentic Reasoning for Faithful Citation
ReCite is an agentic citation framework using claim-level reasoning and verification, outperforming large generative models in strict citation accuracy.