Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
Hierarchical continuous diffusion models jointly denoise tokens and clusters, improving generative perplexity by over 20 points.
The paper introduces Hierarchical Continuous Diffusion Language Models (H-CDLMs), which diffuse token embeddings and coarser clusters of pretrained embeddings in parallel, with per-modality samplers and schedules. Applied to CoBit, H-CoBit reaches generative perplexity of 49.4 on LM1B and 50.4 on OpenWebText, improving the baseline by 24.2 and 20.7 points and beating comparable discrete diffusion LMs, and scores 27.4% on GSM8K. The same hierarchy applied to the flow-matching model FLM (H-FLM) shows consistent gains. Code is promised at the authors' public GitHub repository.
- H-CDLMs jointly diffuse tokens and coarser embedding clusters.
- H-CoBit reaches GenPPL 49.4 on LM1B and 50.4 on OpenWebText.
- Those scores improve on the baseline by 24.2 and 20.7 points.
- On GSM8K, H-CoBit reaches 27.4% accuracy; H-FLM also gains.
Full article214 words · extracted from arxiv.org · click to collapse
Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at https://github.com/matol-16/HCDLM.git .
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.08738