Cola2 studies continuous semantic compression for hierarchical text generation
Original title: Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation
Authors report an unconverged checkpoint averaging 60.10 on eight tasks and beating Llama-3-8B on two.
Cola2 jointly trains continuous semantic compression and discrete text generation. It segments text into semantic spans, controls compression with auxiliary-loss-free negative feedback, and uses a block-causal masked diffusion decoder with classifier-free guidance. Generated text is re-encoded under a fixed span layout at inference. The authors say experiments use about 210 billion training tokens and that a checkpoint trained on 346.03 billion source tokens scores 60.10 across eight tasks.
Unconfirmed: The 60.10 average and Llama-3-8B comparison are author-reported for an unconverged checkpoint.
- The decoder is trained with clean, noisy, and dropped latent conditions.
- Token-prediction gradients update the encoder through the latents.
- Flow Matching prior learning is stop-gradient decoupled from representation learning.
- Authors say it beats Llama-3-8B on HellaSwag and StoryCloze.
- AI models
- Llama-3-8B
From the original article
1 min read at huggingface.co
Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation.
By Hongcan Guo, Yian Zhao, Feng Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.