LLM safety monitors can drop recognized evidence before enforcement
Original title: Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries
Authors confirm 18 evidence-path failures across 11 project groups, 14 of which reach enforcement boundaries.
A paper studies cases where LLM and agent runtime monitors recognize policy evidence but never deliver it to the block decision. Using canary contracts on public framework repairs, the authors report 18 evidence-path failures across 11 project groups and four ecosystems. Nine are established-boundary repairs, eight are new-surface coverage gaps, and one is boundary-only. They argue monitors need regression tests as schemas, streams, tools, and adapters change.
Unconfirmed: The 18 confirmed failures are the authors' protocol results; named projects are not listed in the abstract.
- Fourteen cases reach E1 enforcement boundaries, including seven native action outcomes.
- Eight failures are new-surface policy-coverage gaps and nine are established-boundary repairs.
- Remaining cases affect verifier, safety-prompt, or observability channels.
- Failures fall into carrier, adapter, authority, and surface families.
From the original article
1 min read at arxiv.orgModern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision.
By Junchi Liao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.