Optimal Sequential Annotations for Off-Policy Evaluation
Paper optimizes sequential annotation budgets for doubly robust off-policy evaluation of complex outcomes.
The paper shows how a limited ground-truth annotation budget can support doubly robust off-policy evaluation when rewards are missing and cheap labels, including LLM-as-a-judge, may be biased. It characterizes variance-optimal annotation probabilities for sequential forward-monotone protocols and gives a batch-adaptive implementation. On homelessness-services casenotes, RMSE fell 34-65% for housing placement and 17-68% for progress toward a housing application at budgets of 40% annotation and above; on LMArena preference votes it fell 55-62% at every budget.
- Uses a limited annotation budget for doubly robust off-policy evaluation.
- Derives optimal probabilities for sequential forward-monotone annotation.
- Motivated by casenotes from a homelessness-services nonprofit.
- Housing-placement RMSE fell 34-65% at 40% annotation or more.
- LMArena RMSE fell 55-62% across every annotation budget.
Full article232 words · extracted from arxiv.org · click to collapse
Offline reinforcement learning and off-policy evaluation evaluates dynamic treatment rules based on retrospectively collected data prior to deployment. In recent AI applications, state and reward information is recorded as complex text or image, which recent AI advancements such as LLM-as-a-judge can label with unknown bias. Expert annotation may be available but at a higher cost. For example, safety classification via cheap but imperfect classifiers vs. expensive expert review. We show how a limited budget for ground-truth data-annotation can be used via doubly-robust OPE with missing rewards, and we optimize variance-optimal annotation probabilities for sequential off-policy evaluation, where the target policy value is estimated from annotated data. We characterize the optimal annotation probabilities for sequential forward-monotone annotation protocols, and provide a feasible batch-adaptive implementation. Our work is motivated by a collaboration with a homelessness services nonprofit that writes casenotes for individuals over time. Our method can be used to unlock trustworthy inference from casenote data and answer new inferential questions such as: how does expanding outreach effort over time affect progress towards a housing application and improvement in housing placement? In simulations and on two real datasets - casenotes from the nonprofit and human-preference votes from LMArena - we see reductions in RMSE of 34-65% for housing placement and 17-68% for progress towards a housing application at budgets of 40% of full annotation and above, and by 55-62% at every budget on LMArena.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26707