LuthorSQL gates text-to-SQL locally before a slower LLM agent
Original title: LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent
The 7B model scores 68.03% execution accuracy on BIRD’s hidden test at 0.84 seconds per question on one GPU.
LuthorSQL is a non-autoregressive 7B text-to-SQL model meant to answer easy database questions locally and escalate uncertain ones to a hosted agent. It predicts slot scores over a bounded grammar in one pass, compiles up to 32 plans, and picks an answer with a trained validator. With Claude Opus 5.5 on 227 post-cutoff questions, the authors report it handled nearly half locally and cut API cost 33.3% at the same accuracy. They also report Opus reproduces many public gold queries verbatim without the schema.
Unconfirmed: Accuracy, cost, and contamination figures are the authors’ reported results, not an independent evaluation.
- Accuracy holds in 9 GB with 8-bit weights.
- Zero-shot transfer to Snowflake, BigQuery, MySQL, PostgreSQL, and DuckDB stays within 3.1 points of SQLite.
- Authors say a Claude Opus 5.5 agent at 10,000 queries a day exceeds $70,000 a year at list price.
- Opus tool-assisted accuracy is reported at 95.6–97.4% on post-cutoff databases.
- Code and data are at the project’s GitHub repository.
From the original article
1 min read at huggingface.coFrontier LLM agents that query databases with tools are among the most accurate text-to-SQL systems, but per-query calls to a hosted model are slow, costly, and expose sensitive organizational queries to external APIs. For example, a Claude Opus 5. 5 agent serving 10,000 queries a day costs over $70,000 a year at list price.
By Ibrahim Memon, Armin Karamzade, M. Hanif Jhaveri · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.