CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
CARE-VI introduces CARS, SEVA, and DARE components for reliability-aware value targets, achieving highest mean returns across twelve SAC, TD3, and TD7 settings.
CARE-VI is a framework for reliable temporal-difference target construction in off-policy actor-critic learning, combining Conservative Adaptive Ranking and Screening (CARS), Selector-Evaluator Value Assessment (SEVA), and Dynamic Adaptive Risk-aware Enhancement (DARE). The paper bounds boundary error and selected-value overestimation and proves fixed-policy recovery after finite-stage perturbations. Across SAC, TD3, and TD7 on four MuJoCo tasks, it achieves the highest mean return in all twelve experimental settings.
- CARS retains an ordered candidate action prefix within a preset budget, narrowing only on large boundary gaps
- SEVA orders candidates with selector critics and caps reviewed values at the selector reference
- DARE scales residual corrections using candidate reliability, selector-evaluator gaps, and a finite stage factor
- Highest mean return in all twelve SAC, TD3, and TD7 MuJoCo experiment settings
Full article242 words · extracted from arxiv.org · click to collapse
Reliable temporal-difference targets are central to off-policy actor-critic learning. Direct value improvement refines the next-state target with alternative actions, but the reliability of this refinement depends on how candidate actions are ranked, reviewed, and weighted. Noisy rankings may force premature candidate commitment, reusing selection scores may bias target valuation, and fixed enhancement weights may amplify weak evidence. To address these risks, we develop Conservative Adaptive Ranking and Screening (CARS), which retains an ordered candidate prefix within a preset budget and narrows it only when the observed boundary gap exceeds a disagreement-scaled uncertainty radius. Selector-Evaluator Value Assessment (SEVA) uses selector critics to order candidates and a separately parameterized evaluator critic to review the selected value, then caps the reviewed value at the selector reference. Dynamic Adaptive Risk-aware Enhancement (DARE) then regulates each residual correction using candidate reliability, the gap between selector and evaluator signals, and a finite stage factor. Together, CARS, SEVA, and DARE form CARE-VI, an evidence-regulated target construction framework that preserves the backbone interfaces for critic regression and actor updates. The analysis bounds the CARS boundary error, the SEVA selected-value overestimation, and the one-sided deviation of the DARE residual displacement from its population counterpart, and establishes fixed-policy recovery after the finite-stage perturbation ends. Experiments with SAC, TD3, and TD7 on four MuJoCo tasks show that CARE-VI achieves the highest mean return in all twelve settings. Grouped ablations and scalar diagnostics support the roles of the three components in improving target reliability.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20098