Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe
Researchers introduce KOPA-Bench, a 145-task Korean public API tool-calling benchmark, and EDGE, an execution-grounded data synthesis method.
An arXiv paper presents KOPA-Bench, a benchmark of 145 real-world tasks chaining multiple tool-calls across live Korean government APIs, motivated by data-sovereignty requirements for on-premise open-source LLM agents. It also introduces EDGE, an execution-grounded dynamic graph that keeps only tool-output-to-input links verified by live API calls before synthesizing executable multi-step trajectories. A 9B model fine-tuned with GRPO on the resulting dataset nearly matches its untuned 27B family sibling on KOPA-Bench and improves on the BFCL benchmark.
- KOPA-Bench defines 145 multi-step tool-calling tasks over live Korean public APIs
- EDGE keeps only tool links verified by live execution before synthesizing trajectories
- GRPO-tuned 9B model nearly matches its untuned 27B family sibling
- Gains also transfer to the BFCL function-calling benchmark
Full article136 words · extracted from arxiv.org · click to collapse
Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.05395