Paper proves near-minimax rates for quantile distributional RL
Original title: Near Minimax Rate for Quantile Distributional Reinforcement Learning
A quantile fixed-point estimator matches local asymptotic minimax risk, with resolution-independent rates when reward densities are bounded away from zero.
The paper studies quantile distributional policy evaluation, approximating the return distribution by a quantile Bellman fixed point. With a generative model, the authors build an estimator from an empirical Markov decision process and prove non-asymptotic Wasserstein error bounds. When reward densities are bounded away from zero, they obtain a root-n high-probability bound for every resolution and Wasserstein order. They also show the estimator attains the exact local asymptotic minimax risk at each fixed resolution.
- Root-n bounds cover all resolutions and Wasserstein orders from 1 to infinity.
- For orders above 2, one bound scales with resolution to the power 1/2 minus 1/p.
- A regular one-state instance has resolution-independent local risk constants.
- An endpoint-vanishing two-state case grows with resolution for every order above 2.
From the original article
1 min read at arxiv.orgIn this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point $\bmη_m$. We investigate the statistical accuracy of estimating $\bmη_m$ across quantile resolutions under Wasserstein distances.
By Zijie Cheng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.