MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving
MILER achieves zero-shot sim-to-real reinforcement learning for autonomous driving, completing 17.3 km on a test track without human intervention.
MILER is an end-to-end reinforcement learning policy framework with zero-shot sim-to-real transfer for unstructured autonomous driving. It trains on a custom semantic mid-level representation simulator and uses BEVFusion to convert camera and LiDAR data into a matching semantic bird's-eye-view representation at deployment, with trajectory alignment replacing direct action application. The system drove 17.3 km with two vehicles on a 3.0 km test track at up to 33.6 km/h without intervention, running entirely on a Jetson AGX Orin.
- Zero-shot sim-to-real transfer of both perception and control
- Uses trajectory alignment instead of applying policy actions directly
- Evaluated on obstacles, hairpin curves, off-road sections, and speeds up to 33.6 km/h
- Full software stack runs on a Jetson AGX Orin
Full article207 words · extracted from arxiv.org · click to collapse
Reinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer for unstructured environments. In this work, we present MILER, an end-to-end policy framework with zero-shot sim-to-real transfer. During offline training, we employ a custom semantic mid-level representation (MLR) simulator and train the policy network using reinforcement learning, with its control outputs applied directly to a bicycle model. During deployment on the real vehicle, camera and LiDAR data are processed by BEVFusion to generate a semantic bird's-eye-view representation consistent with that of the MLR simulator. The actions generated by the policy network are not applied directly to the real vehicle. Instead, we employ a trajectory-alignment strategy that enables zero-shot sim-to-real transfer of both perception and control. We extensively evaluate the proposed framework on a diverse test track comprising numerous challenges, including various obstacles, hairpin curves, velocities of up to 33.6 km/h, and off-road sections. In total, we drove 17.3 km with two different vehicles on a 3.0 km test track without human intervention, thereby demonstrating the effectiveness of our approach. Furthermore, the entire software stack runs on a Jetson AGX Orin.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20747