TauIndianBankBench scores banking agents on required procedure steps
Original title: TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents
On 1,000 retail-banking tasks, 6.5–12.0% of sessions pass a final-record check while skipping a required step.
TauIndianBankBench is an action-oriented benchmark of 1,000 Indian retail-banking tasks in which a language-model assistant serves a simulated customer. Besides end-of-conversation records, it checks that every required procedure step occurred and that no unrequested change was made. The authors say outcome-only scoring marks 6.5–12.0% of sessions successful even when a required step was skipped or an unrequested change was made.
- The set contains 1,000 simulated Indian retail-banking tasks.
- Example rule: obtain a closure quote before closing a deposit.
- Outcome-only false successes are 6.5–12.0% of sessions for every model.
- Scoring covers both procedure compliance and final records.
- Countries
- India
- AI models
- TauIndianBankBench
From the original article
1 min read at huggingface.co
Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit.
By Krishanu Adhikary, Prashant Devadiga · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.