OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher
OPTED decouples RL teacher training from closed-loop student post-training for end-to-end driving, raising TransFuser and VaVAM driving scores 1.6x and 9.5x.
OPTED trains a privileged teacher with reinforcement learning on vectorized HD-map and bounding-box inputs, then uses it to supervise pre-trained camera-based students during closed-loop post-training without costly sensor simulation. Applied to TransFuser and VaVAM in AlpaSim using 3DGS neural reconstructions of real driving logs, driving scores increased 1.6x and 9.5x respectively. In controlled experiments it matched direct RL post-training performance with roughly three orders of magnitude fewer simulator interactions.
- Privileged RL teacher on vectorized inputs avoids sensor simulation
- Driving scores up 1.6x for TransFuser and 9.5x for VaVAM
- Uses 3DGS reconstructions of real driving logs in AlpaSim
- About 1000x fewer simulator interactions than direct RL post-training
Full article177 words · extracted from arxiv.org · click to collapse
As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6$\times$ and 9.5$\times$, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: https://01dami23.github.io/opted/
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20756