Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Marigold V2 adapts diffusion transformers for monocular depth estimation, improving AbsRel 16-26% over the previous best on KITTI and ETH3D.
Huawei's Bayer lab revisits the Marigold approach to repurpose image generation and editing models built on the diffusion transformer (DiT) architecture into monocular depth estimators. The recipes target single-step inference from pretrained multi-step flow-matching models, with remedies including alignment to ground-truth semantic features and a two-stage fine-tuning protocol using a Sinkhorn-based loss. The resulting model produces crisper depth maps that generalize out-of-distribution and also achieves state-of-the-art results on surface normals estimation and intrinsic image decomposition.
- Targets single-step inference from multi-step flow-matching models
- Uses a Sinkhorn-based loss in a two-stage fine-tuning protocol
- 16-26% AbsRel improvement over prior best on KITTI and ETH3D
- Also state-of-the-art on surface normals and intrinsic decomposition
- Organizations
- Huawei
- AI models
- Marigold V2
From the original article
1 min read at huggingface.co
Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps.
By Igor Pavlovic, Thiemo Wandel, Anton Obukhov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.