Foundation4D unifies driving perception and 3D reconstruction
Original title: Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles
The model and PAI-Foundation4D benchmark cover 1,253 clips with dense depth and occupancy labels.
Foundation4D is a feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention, and inference refines trajectories with pose graph optimization while assembling object tracks. The authors also release PAI-Foundation4D, with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, they report leading depth, detection, and occupancy scores among compared methods, plus gains from co-training, and use predicted geometry as priors for lidarless NuRec.
Unconfirmed: Benchmark rankings are versus the methods the authors compared; the comparison set is not fully listed here.
- PAI-Foundation4D labels 1,253 driving clips for depth and occupancy.
- Co-training detection raised top-class mAP by 1.2 points.
- Geometry co-trained with detection cut translation error 13.4% relatively.
- Predicted geometry and trajectories served as priors for lidarless NuRec.
- Products
- PAI-Foundation4D
- AI models
- Foundation4D
From the original article
1 min read at huggingface.co
We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time.
By NVIDIA, Sérgio Agostinho, Maximilian Apodaca · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.