BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence
Researchers introduce BI-Bench, the first end-to-end business intelligence benchmark for LLMs, and BI-Agent, a tool-augmented agent gaining up to 40 accuracy points.
The paper presents BI-Bench, the first benchmark evaluating LLMs on end-to-end business intelligence tasks such as table identification, transformation, and join construction, built from real-world BI projects and user dashboards. Frontier LLMs score below 50% accuracy on it. The proposed BI-Agent decomposes BI workflows into subtasks (search, join, transform) and orchestrates specialized data management methods. A post-training framework using synthesized trajectories with supervised fine-tuning and reinforcement learning yields gains of up to 30 additional percentage points.
- BI-Bench is the first benchmark for end-to-end LLM business intelligence tasks.
- Frontier LLMs achieve under 50% accuracy on BI-Bench.
- Tool-augmented BI-Agent gains up to 40 points over vanilla LLMs.
- SFT and RL post-training adds up to 30 further points.
Full article265 words · extracted from huggingface.co · click to collapse
Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20886