NLD-Uni unifies visual understanding and generation in a diffusion LLM
Original title: Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation
Cross-paradigm distillation and progressive token corruption are used instead of conventional masked diffusion.
The authors propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a discrete diffusion large language model for visual understanding and generation. A cross-paradigm on-policy distillation framework transfers understanding from autoregressive vision-language models and generation from continuous flow models into one diffusion LLM. Training gradually corrupts each token over multiple steps rather than masking a clean image token in one step, and inference can commit tokens early for progressive refinement. The paper reports state-of-the-art results among unified discrete diffusion LLMs on visual question answering, multimodal reasoning, grounding, text-to-image generation, and image editing.
Unconfirmed: State-of-the-art claims are the authors' comparison among unified discrete diffusion LLMs.
- Distillation sources are autoregressive VLMs and continuous flow models.
- Progressive corruption replaces a single mask-token corruption step.
- Inference can make intermediate token commitments for self-refinement.
- Evaluated tasks include grounding, image editing, and text-to-image generation.
- Products
- Nemotron-Labs-Diffusion-Uni
- AI models
- NLD-Uni
From the original article
1 min read at huggingface.co
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations.
By Shufan Li, Hanrong Ye, Greg Heinrich · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.