Pairwise Bellman validation selects off-policy estimators without extra critics
Original title: Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Unweighted PBV is competitive on Hopper and Deep OPE; target-occupancy weighting can help under controlled overlap.
The paper introduces pairwise Bellman validation, a tuning-free minimax procedure for choosing among finite classes of fitted Q-functions and occupancy-ratio estimators from held-out transitions. Critics are built from pairwise candidate differences, using finitely many empirical moments and no auxiliary validation models. Finite-sample oracle inequalities are claimed even under candidate-class misspecification. Variants cover unweighted PBV, target-aligned PBV with marginalized importance weighting, and adjoint PBV for occupancy ratios.
- Unweighted PBV needs a pairwise coercivity condition for its oracle guarantee.
- Target-aligned PBV drops that coercivity requirement via importance weighting.
- Adjoint PBV evaluates adjoint Bellman moments against log-ratio critics.
From the original article
1 min read at arxiv.orgModel selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs.
By Lars van der Laan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.