Foyer study: logistic regression beats LLMs on inbox routing
Original title: Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
On about 200 labels, logistic regression led every field, while the most accurate system ignored 18% of emails needing action.
The authors describe Foyer, a self-hosted personal inbox agent, and a measurement protocol that labels 739 real Italian and English emails before training. They compare keyword rules, TF-IDF plus logistic regression, typed decision models, and hosted generative models under thread-grouped cross-validation. Production rules scored 28% on message type against a 58% majority class. With about 200 labels, logistic regression was highest on every field and statistically tied with Gemma 4 on type. Composed into actions, the most accurate system ignored emails needing action 18% of the time versus 4% for the best LLM.
- Labels cover 231 unbiased and 500 adversarially sampled emails.
- Logistic regression scored type 0.79, urgency 0.74, needs-reply 0.83, and is-noise 0.90.
- Sender relationship was recovered at 98.6% when the contact had been seen.
- A cost-chosen noise threshold halved ignored actionable mail and was cheapest.
From the original article
1 min read at arxiv.orgPersonal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself.
By Federico Lucca · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.