dQwen3.5: Hybrid-Attention Diffusion Language Models
Researchers adapt Qwen3.5 hybrid-attention backbones (0.8B-9B) into diffusion language models, reaching training loss in half the tokens versus full attention.
Researchers converted pretrained Qwen3.5 models at 0.8B, 2B, 4B, and 9B scales into the dQwen3.5 family of diffusion language models. The work addresses hybrid attention-RNN backbones, whose RNN layers are structurally causal and hard to bidirectionalize. Against a full-attention control, the hybrid backbone reaches a given training loss in roughly half the tokens. The resulting models match full-attention DLMs in any-order decoding and perform strongly under parallel decoding.
45