PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
PixelUMM unifies encoder-free image and video understanding and generation in raw pixel space.
PixelUMM is an encoder-free unified multimodal model for image and video understanding and generation in pixel space. Images are spatial patches and videos are spatiotemporal tubelets, connected to a shared backbone by single-layer linear projections. A Mixture-of-Transformers design uses shared attention with task-specific parameters and extends clean-pixel prediction to video, jointly supporting autoregressive text prediction and pixel-space flow matching. The authors report competitive results on image and video understanding and generation and study decoder design and spatial-temporal patch size.
- PixelUMM is encoder-free and works directly in pixel space.
- Images use spatial patches; videos use spatiotemporal tubelets.
- Single-layer linear projections feed a shared multimodal backbone.
- Mixture-of-Transformers mixes shared attention with task-specific parameters.
- Supports autoregressive text prediction and pixel-space flow matching.
- AI models
- PixelUMM
From the original article
1 min read at huggingface.co
Unified Multimodal Models (UMMs) often rely on separate visual representations for understanding and generation, increasing visual context length and complicating integration with established vision-language pretraining pipelines.
By Cong Wei, Xuanchi Ren, Bryan Chu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.