MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving
MILER achieves zero-shot sim-to-real reinforcement learning for autonomous driving, completing 17.3 km on a test track without human intervention.
MILER is an end-to-end reinforcement learning policy framework with zero-shot sim-to-real transfer for unstructured autonomous driving. It trains on a custom semantic mid-level representation simulator and uses BEVFusion to convert camera and LiDAR data into a matching semantic bird's-eye-view representation at deployment, with trajectory alignment replacing direct action application. The system drove 17.3 km with two vehicles on a 3.0 km test track at up to 33.6 km/h without intervention, running entirely on a Jetson AGX Orin.
- Zero-shot sim-to-real transfer of both perception and control
- Uses trajectory alignment instead of applying policy actions directly
- Evaluated on obstacles, hairpin curves, off-road sections, and speeds up to 33.6 km/h
- Full software stack runs on a Jetson AGX Orin
- Vendors
- NVIDIA
From the original article
1 min read at arxiv.orgReinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer for unstructured environments.
By Thomas Steinecker · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.