PostHog uses decision models to filter most LLM calls
Original title: Not Everything Needs an LLM
In Replay Vision, 98.7% of summaries skip Gemini after two yes/no checks at a 0.4 threshold.
PostHog describes production use of small decision models that return typed answers with probabilities instead of free-form LLM prose. In Replay Vision, two narrow checks decide whether a session summary names a person or shows contact details or an ID; scores of 0.4 or above send it to Gemini for rewriting. The same pattern routes the Slack bot and reranks search results. PostHog says the models are far cheaper and faster than frontier models, and that thresholds must be set per task and model version.
Unconfirmed: Cost, speed and accuracy figures are PostHog's own measurements, not independently verified.
- PostHog claims up to 444.6x lower cost and 193.6x higher speed than frontier models.
- A 0.85 score still implies about 150 wrong answers per 1,000 checks.
- Slack replies require both checks at 0.85 or above.
- Reranking improved quality by about 25% on 20 test queries.
- Thresholds changed between v1 and v2 of the same model.
- Products
- Replay Vision
- Organizations
- PostHog
From the original article
7 min read at newsletter.posthog.com
LLMs love to yap. Ask a simple question and you get a few hundred words of "it depends." It's annoying as hell.
By speckx · newsletter.posthog.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.