dQwen3.5: Hybrid-Attention Diffusion Language Models
Researchers adapt Qwen3.5 hybrid-attention backbones (0.8B-9B) into diffusion language models, reaching training loss in half the tokens versus full attention.
Researchers converted pretrained Qwen3.5 models at 0.8B, 2B, 4B, and 9B scales into the dQwen3.5 family of diffusion language models. The work addresses hybrid attention-RNN backbones, whose RNN layers are structurally causal and hard to bidirectionalize. Against a full-attention control, the hybrid backbone reaches a given training loss in roughly half the tokens. The resulting models match full-attention DLMs in any-order decoding and perform strongly under parallel decoding.
- Adapts Qwen3.5 at 0.8B, 2B, 4B, and 9B scales into dQwen3.5 diffusion models
- Hybrid attention-RNN backbones are structurally causal, complicating bidirectionalization
- Hybrid backbone reaches target training loss in about half the tokens
- Shows any-order decoding behavior comparable to full-attention diffusion models
From the original article
1 min read at arxiv.orgAdapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize.
By Anton Xue · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.