ARCS: Towards Precise Text-to-SQL via Structured Disambiguation
ARCS benchmark shows text-to-SQL remains weak when natural question ambiguity is present.
The paper argues that subtle, domain-specific ambiguity is a major error source for deployed text-to-SQL systems and proposes structured disambiguation instead of open-ended clarification chat. It releases ARCS, a benchmark of naturally occurring unconstrained ambiguities over real databases, with annotations of valid ambiguity points, interpretations, and SQL. Reported results show gpt-6-sol at 51% end-to-end execution accuracy, while no open-source model exceeds 27%.
- Structured disambiguation uses constrained interactions instead of free-form dialogue.
- ARCS annotates ambiguity points, interpretations, and SQL on real databases.
- gpt-6-sol reaches only 51% end-to-end execution accuracy.
- No open-source model exceeds 27% accuracy.
From the original article
1 min read at huggingface.co
As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent.
By Yihao Hu, Yanlin Feng, Naoki Otani · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.