Paper splits visual generation into autoregressive knowledge and diffusion rendering
Original title: Know, then Draw: A New Learning Paradigm for Visual Generation
A 3B text-to-image model trained in 6,600 H100 hours scored 86 on GenEval and 87 on DPG-Bench.
Authors assign semantic knowledge to autoregressive pretraining and pixel rendering to diffusion post-training. Controlled comparisons found autoregression a stronger pretraining objective for downstream rendering. Semantic quality mattered more than AR-only image quality, motivating a tokenizer trained with language supervision only. Compute-optimal scaling laws showed autoregressive pretraining lowered asymptotic rendering loss versus pixel diffusion trained from scratch.
- The 3B model used only 6,600 H100 GPU-hours.
- Reported scores were 86 on GenEval and 87 on DPG-Bench.
- The semantic tokenizer has no image decoder or reconstruction loss.
- Autoregressive pretraining lowered fitted asymptotic rendering loss versus scratch pixel diffusion.
From the original article
1 min read at huggingface.co
Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering.
By Yucheng Mao, Chongjian Ge, Zhuowen Tu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.