dQwen3.5: Hybrid-Attention Diffusion Language Models
Researchers adapt Qwen3.5 hybrid-attention backbones (0.8B-9B) into diffusion language models, reaching training loss in half the tokens versus full attention.
Researchers converted pretrained Qwen3.5 models at 0.8B, 2B, 4B, and 9B scales into the dQwen3.5 family of diffusion language models. The work addresses hybrid attention-RNN backbones, whose RNN layers are structurally causal and hard to bidirectionalize. Against a full-attention control, the hybrid backbone reaches a given training loss in roughly half the tokens. The resulting models match full-attention DLMs in any-order decoding and perform strongly under parallel decoding.
- Adapts Qwen3.5 at 0.8B, 2B, 4B, and 9B scales into dQwen3.5 diffusion models
- Hybrid attention-RNN backbones are structurally causal, complicating bidirectionalization
- Hybrid backbone reaches target training loss in about half the tokens
- Shows any-order decoding behavior comparable to full-attention diffusion models
Full article125 words · extracted from arxiv.org · click to collapse
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20751