Offline RL method handles domain shift and unmeasured confounding
Original title: Transportable Offline Reinforcement Learning in Presence of Unmeasured Confounding
CAT-CCB and CAT-CMDP use causal transportability and pessimistic optimization, with finite-sample suboptimality bounds.
The paper studies offline policy learning when source data differ from the target domain and logged variables omit confounders. Multiple heterogeneous sources supply complementary information, identified with instrumental variables, negative controls, and proxies. For confounded contextual bandits the method is CAT-CCB; for episodic confounded MDPs it is CAT-CMDP. Finite-sample guarantees cover approximation, estimation, shift, coverage, and sample size, and experiments support learning target policies when no single source suffices.
- Confounded bandit identification uses bridge-function integral equations and minimax estimation.
- For bounded linear classes the bandit bound recovers a Õ(N^{-1/2}) rate.
- CAT-CMDP assumes Gaussian dynamics and linear function approximation.
- Guarantees quantify source-to-target shift, offline coverage, and sample size.
From the original article
1 min read at arxiv.orgOffline reinforcement learning seeks to learn decision policies from previously collected data, but deployment environments may differ from the domains in which the data were collected, and logged variables may omit confounders that affected both historical actions and outcomes.
By Junyi Liao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.