Cola2 studies continuous semantic compression for hierarchical text generation
Authors report an unconverged checkpoint averaging 60.10 on eight tasks and beating Llama-3-8B on two.
Stories per day · 30 days
2 RSS
Relatedattention1autonomous-research1cola21continuum-attention1diffusion-decoder1diffusion-language-models1gsm8k1hierarchical-language-models1linear-attention1multi-agent1openwebtext1retrieval1self-conditioning1semantic-compression1softmax1transformers1
12 stories, newest first · one card per story
Stories tagged #language-modeling
LAW bases language modeling entirely on self-conditioning
After TinyGSM training, LAW solves 42.2% of GSM8K at 1024 NFE versus 34.5% for uniform loopholing discrete diffusion.
Multi-agent AI team finds linear attention designs that beat GDN2
Agents with distinct disciplines, private histories, and adaptive allocation report gains over GDN2 at 1.3B scale.
Continuum Attention
Continuum Attention beats softmax and focal attention by integrating a continuous memory curve.
Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
Hierarchical continuous diffusion models jointly denoise tokens and clusters, improving generative perplexity by over 20 points.
Differentially Private Mixing of Public Datasets Improves Private Learning
Private mixing of public datasets improves DP pretraining utility on NIH X-rays and ENRON text.
Learning to Learn a Language
A 300M-parameter model trained only on synthetic priors learns to predict real languages in context.
Hierarchical Continuous Diffusion Language Models
HC-DLM ties discrete tokens to a continuous latent and beats diffusion baselines on reasoning and LM1B.
Learning to Re-Draft: A Variational Stackelberg Game for Discrete Diffusion
VSDD learns a corruption process so discrete diffusion models can re-draft tokens more effectively.
Fractional State Space Transition for Long Sequence Modeling
FRAC, a selective state-space model using fractional dynamics, improves long-context memory over exponential-forgetting SSM baselines.
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Complex KDA extends Kimi Delta Attention so one transition can represent 2D rotations and stronger state tracking.
Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning
DBTM achieves one-step text generation via a time-independent transport map trained directly from data, removing pretrained teacher distillation.