Engineer argues teams should run their own LLM proxy
Original title: If you don't run your own LLM proxy, how do you know whats going on?
The post says vendor dashboards lack engineer-grade token time series, while a self-hosted proxy can log keys, models, clients, and status.
A blog post argues that paid LLM front ends and vendor analytics do not give operators enough data to make engineering decisions. The author favors a self-hosted proxy with one endpoint for multiple providers and models, logs of which key called which model, virtual API keys, and fast provider switching. The author says they modified their proxy for custom MCP endpoints and can trace a cache miss by hand in about 20 seconds.
- No specific proxy product is recommended.
- Suggested logs cover key, model, client, time, and status.
- The author traces coding-session cache misses by hand in about 20 seconds.
- The author says they added custom MCP server endpoints to their proxy.
From the original article
2 min read at jrdsgl.comI think the title captures the idea fully but I'll dig into it because this ain't Twitter. You're working an idea, a piece of code, or a multibillion a year SaaS app which is probably saving the world.
By amrrs · jrdsgl.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.