onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
onPanda cuts median alignment annotation time 52% via token-level correction that keeps data on-policy for SFT and preference training.
onPanda is an interactive annotation tool where the annotator locates the first inappropriate token, corrects it, and the model regenerates from the corrected prefix, looping until the response is satisfactory. A small controlled study reports a 52% reduction in median annotation time versus manual post-editing. Because most tokens come from the model itself, outputs preserve the sampling distribution for on-policy SFT and preference data, and corrections yield paired positive-negative samples. The authors release Panda-CVL, an annotated dataset, plus a benchmark for token-level correction.
- Locate-correct-continue loop yields 52% median annotation time reduction in a small study.
- Data stays largely on-policy, suited for SFT and preference training.
- Corrections provide fine-grained supervision with naturally paired positive-negative samples.
- Releases Panda-CVL dataset and a token-level correction benchmark.
From the original article
1 min read at arxiv.orgWe present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing.
By Lei Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.