Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe
Researchers introduce KOPA-Bench, a 145-task Korean public API tool-calling benchmark, and EDGE, an execution-grounded data synthesis method.
An arXiv paper presents KOPA-Bench, a benchmark of 145 real-world tasks chaining multiple tool-calls across live Korean government APIs, motivated by data-sovereignty requirements for on-premise open-source LLM agents. It also introduces EDGE, an execution-grounded dynamic graph that keeps only tool-output-to-input links verified by live API calls before synthesizing executable multi-step trajectories. A 9B model fine-tuned with GRPO on the resulting dataset nearly matches its untuned 27B family sibling on KOPA-Bench and improves on the BFCL benchmark.
- KOPA-Bench defines 145 multi-step tool-calling tasks over live Korean public APIs
- EDGE keeps only tool links verified by live execution before synthesizing trajectories
- GRPO-tuned 9B model nearly matches its untuned 27B family sibling
- Gains also transfer to the BFCL function-calling benchmark
- Products
- KOPA-BenchEDGE
- Countries
- South Korea
From the original article
1 min read at arxiv.orgData-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks.
By Dain Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.