SUMMARK checks hosted LLM identity from reasoning summaries
Original title: Auditor-Assisted Summary-Channel Verification for Hosted LLM Identity Substitution
A trusted auditor enrolls identities in advance; audit time uses only the claimed identity and the released summary, not provider logs.
The paper studies verifying which model served a hosted LLM request when outsiders cannot see configuration, logs, routing, or provider metadata. SUMMARK treats user-visible reasoning summaries as a keyed audit channel: setup builds identity-specific vocabulary carriers, trains lightweight summary adapters, and sets per-identity thresholds. A statistical test then checks the claimed identity and can attribute the most likely enrolled one. Across families, scales, and tasks, the authors report reliable detection of cross-family and cross-scale substitution at low false-positive rates, with stated limits under precision changes, rewriting, and text-only spoofing.
- Verification needs no runtime access to model internals, inference logs, or routing decisions.
- The study includes FP16 to INT8 and INT4 precision-consistency diagnostics.
- Robustness was tested under bounded post-processing, rewriting, and text-only spoofing.
- Code is published at github.com/trinitr0toluene/Summark.
From the original article
1 min read at arxiv.orgHosted LLM services increasingly expose model identity as a product claim, but external users and auditors often cannot inspect which model configuration served a request.
By Ziyi Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.