Optimal Sequential Annotations for Off-Policy Evaluation
Paper optimizes sequential annotation budgets for doubly robust off-policy evaluation of complex outcomes.
The paper shows how a limited ground-truth annotation budget can support doubly robust off-policy evaluation when rewards are missing and cheap labels, including LLM-as-a-judge, may be biased. It characterizes variance-optimal annotation probabilities for sequential forward-monotone protocols and gives a batch-adaptive implementation. On homelessness-services casenotes, RMSE fell 34-65% for housing placement and 17-68% for progress toward a housing application at budgets of 40% annotation and above; on LMArena preference votes it fell 55-62% at every budget.
- Uses a limited annotation budget for doubly robust off-policy evaluation.
- Derives optimal probabilities for sequential forward-monotone annotation.
- Motivated by casenotes from a homelessness-services nonprofit.
- Housing-placement RMSE fell 34-65% at 40% annotation or more.
- LMArena RMSE fell 55-62% across every annotation budget.
- Organizations
- LMArena
From the original article
1 min read at arxiv.orgOffline reinforcement learning and off-policy evaluation evaluates dynamic treatment rules based on retrospectively collected data prior to deployment. In recent AI applications, state and reward information is recorded as complex text or image, which recent AI advancements such as LLM-as-a-judge can label with unknown bias. Expert annotation may be available but at a higher cost.
By Woojin Chae · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.