FERPO: Forward Entropy-Regularized Policy Optimization
FERPO updates continuous-control policies from critic values without differentiating the critic by actions.
FERPO is an on-policy maximum-entropy reinforcement learning algorithm that derives an entropy- and KL-regularized target action distribution and fits the actor by minimizing a forward-KL objective. The objective is estimated with self-normalized importance sampling from the rollout policy, avoiding differentiation of the critic with respect to actions. Experiments on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains, and actor updates are faster than Relative Entropy Pathwise Policy Optimization.
40