Meta runs JiTTests to fail risky diffs before code lands
Original title: Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands
Meta says the internal system caught 74 privacy SEVs in the first half of 2026, with a self-reported 60% true-positive rate.
Meta has moved Catching Just-in-Time Tests from a research prototype into production. JiTTests uses LLMs to write executable tests meant to fail when a diff changes behavior, then ranks failures with a judge ensemble and rule checks before contacting engineers. A paper analyzes 22,126 generated tests on Meta backend code. The system is internal only, and the production figures are Meta’s own.
Unconfirmed: Catch rates and the 60% true-positive figure are self-reported, and a weak catch is not a confirmed bug.
- Intent-aware tests produced weak catches in 6.4% of tests and on 7.9% of diffs.
- An early deployment confirmed 8 of 41 reported catches, a 19.5% rate.
- Judge models named in the paper are Llama3.3-70B, Gemini 3 Pro and Claude Sonnet 4.
- 589 engineers engaged, 64% of those contacted, according to an October 2026 post.
- Privacy SEV occurrence was reported down 20%.
From the original article
8 min read at marktechpost.comMeta has moved Catching JiTTests into production. The LLM system writes tests designed to fail on risky diffs, and caught 74 privacy SEVs before landing in H1 2026, with a 60% engineer-reported true-positive rate. The post Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands appeared first on MarkTechPost.
By Michal Sutter · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.