Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation
Routing metadata from LLM gateways can reveal sensitive topics, including medical and self-harm requests, without content logs.
A pre-registered measurement on 1.7 million WildChat-1M and LMSYS-Chat-1M requests finds that LLM router decisions can shift with sensitive-topic category even at matched prompt length, including when content logging is off. For RouteLLM at the 50% operating point, post-hoc checks showed harassment and self-harm prompts reaching the strong model 19 points less often and sexual prompts 10 points more often; an exploratory medical split was 31 points lower but its labels failed a quality gate. Separating frequent medical askers from 20 RouteLLM decisions scored AUC 0.71, below the pre-registered 0.75 endpoint, while a domain router scored 0.92 as an upper estimate. Per-category length-matched parity removed the real-traffic gap at a cost of at most 0.2 RouterBench accuracy points, whereas stickiness, budget bands, and pooled parity failed.
- Measurement uses 1.7 million WildChat-1M and LMSYS-Chat-1M requests and two cost/quality routers.
- RouteLLM harassment and self-harm prompts hit the strong model 19 points less often, post hoc.
- Medical-asker separation from 20 RouteLLM decisions scored AUC 0.71, below the 0.75 endpoint.
- Per-category length-matched parity closes the gap for at most 0.2 RouterBench accuracy points.
- A post-hoc log attack still reaches AUC 0.73 after 20 RouteLLM requests.
- Products
- RouteLLMRouterBench
- Organizations
- LMSYS
From the original article
1 min read at arxiv.orgLLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off. We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts. We run pre-registered studies on 1.
By Teng-Ruei Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.