EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
EdgeGen synthesizes database-grounded edge-case tasks that improve tool-calling agents without human annotation.
EdgeGen extracts compliance rules from a tool-calling agent's specification and generates database-grounded tasks designed to violate those rules, addressing generic synthetic data that ignores agent state. Combined with existing generators, it supports finetuning and harness optimization in a fully automated loop with no human annotation. Finetuning produces mean progress gains of 2 to 42 percent on the tau2-bench airline domain, while some baselines degrade certain models. For Gemma-4-e4b, harness optimization improves mean progress about 10 percent over a human-curated harness and 30 percent over the base harness.
- Compliance rules from the agent spec drive database-grounded violating tasks.
- The closed loop combines generation, finetuning, and harness optimization without labels.
- Finetuning improves mean progress 2 to 42 percent on tau2-bench airline.
- Gemma-4-e4b harness gains are 10 percent over human-curated and 30 percent over base.
Full article177 words · extracted from huggingface.co · click to collapse
Tool-calling LLM agents are increasingly deployed in enterprise applications. However, effective evaluation and optimization require high-quality, diverse task datasets that are often difficult to obtain due to privacy and other constraints. Existing synthetic task generation methods often produce generic tasks that ignore an agent's underlying state or database and fail to reflect real-world usage diversity. We propose EdgeGen, a synthetic task generation framework that extracts compliance rules from an agent's specification and uses them to generate database-grounded edge-case tasks designed to violate these rules. When combined with existing synthetic data generation techniques, EdgeGen enables agent improvement through finetuning and harness optimization. The resulting pipeline forms a fully automated closed-loop system that requires no human annotation. Finetuning on data generated by EdgeGen yields a consistent mean progress improvement of 2 percent to 42 percent on tau2bench airline domain, while other baseline methods show degradation for some models. On the other hand, for harness optimization, our method shows a mean progress improvement of 10 percent and 30 percent over the human-curated and base harnesses, respectively, for the Gemma-4-e4b model.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24115