| title | TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents |
|---|---|
| authors | Krishanu Adhikary, Prashant Devadiga |
| date | 2026-09-30 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.30161 |
| affiliation | National Payments Corporation of India |
| upvotes | 10 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit. Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful. We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change. For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking. |
cite
@misc{adhikary2026tauindianbankbench,
title={TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents},
author={Krishanu Adhikary and Prashant Devadiga},
year={2026},
eprint={2610.30161},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.30161}
}