FERN ranks frozen retrievers for Text-to-SQL table retrieval
Original title: FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking
A LightGBM ranker over three frozen retrievers approaches RASL's published recall without generative LLM calls.
FERN retrieves tables for Text-to-SQL by fusing three frozen retrievers into 100 candidates, describing each with up to 120 features, and reordering them with LightGBM rankers fitted once on BIRD training questions. It is compared with published RASL recall, which uses Claude 3.5 models, on pooled BIRD, Spider, and Fiben catalogs, plus 121 BEAVER questions. On Spider, the unfitted candidate pool matches RASL's published recall at fifteen tables, and ranking raises complete@5 from 83.9% to 95.6%. Median latency on BIRD dev is 208 ms per question on one A100.
Unconfirmed: Fiben results use different question sets from the RASL comparison.
- Spider recall at five tables is 96.7 versus RASL's 97.0.
- BIRD with evidence is 96.6 versus 97.5; without evidence, 89.3.
- BEAVER recall at fifteen is 82.1, between two trained retrievers at 80.8 and 84.5.
- Fiben recall at fifteen is 54.2 versus 69.2 on different question sets.
- Non-retrieval features raise BIRD complete@5 by 7.7 points.
- AI models
- Claude 3.5
From the original article
1 min read at arxiv.orgText-to-SQL systems over large pooled catalogs must first retrieve the tables a question needs. We compare against the published table recall of RASL, which uses Claude 3. 5 models to extract keywords and select tables, on pooled BIRD, Spider, and Fiben catalogs.
By Arjun Rao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.