HQL balances one-step and multi-step returns for long horizons
Original title: Horizon-Scalable Q-Learning
A multi-step target is kept only when a dataset action beats best-of-N behavior-cloned actions.
Horizon-Scalable Q-Learning targets error accumulation in offline Q-learning as task horizons grow. One-step temporal-difference learning compounds per-step errors, while fixed multi-step returns add bias from suboptimal dataset actions. HQL extends a target through a dataset action only when the critic values it at least as highly as the best of N actions from the behavior-cloning policy; otherwise it uses a one-step target. The authors say the rule needs no action densities, works with flow and diffusion policies, and outperforms prior methods on long-horizon OGBench and OCBench tasks.
Unconfirmed: Task gains are reported by the authors and are not independently verified.
- The method does not require action density estimates.
- It is compatible with flow and diffusion policies.
- Authors report results on combination-lock, OGBench, and OCBench.
- No fixed multi-step length is applied to a whole trajectory.
From the original article
1 min read at huggingface.co
Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small.
By Haoran Xu, Pragya Srivastava, Xinjie Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.