IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
IndicBankBench finds banking language-model assistants pass all three trials on only 43.7–58.2% of cases.
IndicBankBench is a 799-case benchmark for Indian retail banking assistants across five operational domains, a capability and refusal domain, and twenty primary axes. Cases are scored for safety, tool use, response adequacy, and advisory quality, with strict pass^3 requiring success on all three trials. Across eleven models, strict reliability is 43.7% to 58.2%, while at-least-once success is 60% to 74%, so single-run success overstates dependable behavior. The authors release the cases, a mock environment, and an evaluation harness.
- 799 cases span five banking domains plus a refusal domain
- Scoring covers safety, tool use, adequacy, and advice
- Strict pass-all-three rates range from 43.7% to 58.2%
- At-least-once success of 60–74% overstates reliability
- Diagnostics separate unnecessary questions from failed context use
From the original article
1 min read at huggingface.co
Banking assistants must use account-specific information to answer requests and, in many cases, take actions through tools. Evaluating only the final response misses important errors. An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one.
By Suvradip Paul, Chandra Bhushan, Harsh Sharma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.