Correct Answers, Unsupported Findings: Evidence Binding in Forensic Reconstruction of LLM Agent Logs
Study finds LLM readers can match agent-log facts while asserting unsupported citation sources.
Researchers audited whether LLM readers can tie forensic findings to preserved records in LLM-agent logs, using 64 checkable cases from saved AgentDojo Banking executions. With original identifiers and no binding table, Sonnet recovered every literal source location but made unsupported citation-source assertions in 26 of 28 cases that needed the missing relation; 22 still matched the complete reference. Explicit bindings improved grounded reconstruction for both readers, while renaming identifiers did not. A deterministic same-packet comparator resolved the bounded task or abstained throughout.
- Audited 64 checkable AgentDojo Banking executions for citation-to-record binding.
- Without a binding table, Sonnet made unsupported source assertions in 26 of 28 cases.
- Explicit bindings improved grounded findings; identifier renaming alone did not.
- A deterministic comparator solved the bounded task or abstained every time.
Full article183 words · extracted from arxiv.org · click to collapse
Forensic reconstruction of LLM-agent actions requires not only recovering the correct value, but establishing which preserved record supports that finding. Tool logs, generated explanations, and local citation identifiers capture different parts of this evidence, yet a citation identifier does not establish a source unless its binding to a record is preserved. We audit this distinction using 64 mechanically checkable cases from saved AgentDojo Banking executions. Two LLM readers reconstruct source relationships under controlled variations in visible evidence and identifier-to-record bindings. We separately evaluate complete-record agreement, evidence-grounded findings, justified abstention, and unsupported assertions. With original identifiers and no binding table, Sonnet recovered every literal source location but made unsupported citation-source assertions in 26 of 28 cases requiring the missing relation; 22 nevertheless matched the complete reference. Adding explicit bindings improved grounded reconstruction for both readers, whereas identifier renaming alone provided no consistent remedy. A deterministic same-packet comparator correctly resolved the bounded task or abstained throughout. These results show that factual agreement alone is insufficient for evaluating forensic reconstruction of agent logs and motivate preserving explicit record bindings to distinguish supported findings from correct guesses.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.09581