Skip to content
NewA calmer, faster ZeroHour

PostHog uses decision models to filter most LLM calls

Original title: Not Everything Needs an LLM

Hacker News · securityby speckx ()7 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

In Replay Vision, 98.7% of summaries skip Gemini after two yes/no checks at a 0.4 threshold.

PostHog describes production use of small decision models that return typed answers with probabilities instead of free-form LLM prose. In Replay Vision, two narrow checks decide whether a session summary names a person or shows contact details or an ID; scores of 0.4 or above send it to Gemini for rewriting. The same pattern routes the Slack bot and reranks search results. PostHog says the models are far cheaper and faster than frontier models, and that thresholds must be set per task and model version.

Unconfirmed: Cost, speed and accuracy figures are PostHog's own measurements, not independently verified.

Key facts
  • PostHog claims up to 444.6x lower cost and 193.6x higher speed than frontier models.
  • A 0.85 score still implies about 150 wrong answers per 1,000 checks.
  • Slack replies require both checks at 0.85 or above.
  • Reranking improved quality by about 25% on 20 test queries.
  • Thresholds changed between v1 and v2 of the same model.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

7 min read at newsletter.posthog.com

LLMs love to yap. Ask a simple question and you get a few hundred words of "it depends." It's annoying as hell.

Hacker News · security

By speckx · newsletter.posthog.com

Read the full article at Hacker News · security

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.