Annotrace: A Workbench for Step-Level Annotation of Agent Traces
Annotrace labels agent traces at step level and reports 91% model agreement on 235 units.
Annotrace is a workbench for step-level labeling of LLM agent traces. It reads six formats, including OpenTelemetry GenAI, LangSmith, and Langfuse, and converts each run into an ordered nested list of steps covering delegations and tool calls. Users supply a codebook; a frontier model can propose an initial codebook from sample traces and is asked the same questions as human annotators. On a multi-agent question-answering corpus it exactly reproduces 235 prior annotation units, and two models labeling the same traces agree 91% of the time with kappa 0.82.
- Annotrace ingests six formats, including OpenTelemetry GenAI, LangSmith, and Langfuse.
- Steps, delegations, and tool calls become nested annotation units.
- A model drafts the codebook and answers the same questions as humans.
- Two models agreed 91% of the time, with kappa 0.82, on 235 units.
From the original article
1 min read at arxiv.orgEvery study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be.
By Arun Josephraj Arokiaraj · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.