ECHO folds multi-token prediction into the backbone itself
Original title: ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Dynamic Depth Sampling draws training depth with decaying probabilities, at half the overhead of training every depth.
Existing multi-token prediction methods attach extra modules and extra passes for each horizon. ECHO has the backbone's final layer predict every horizon, so supervision lands on the parameters that generate the next token. In pretraining, added depth converted into downstream accuracy where auxiliary modules saturated, concentrating on reasoning and code. At inference the pretrained model drafts beyond its trained depth and keeps draft-target alignment on open-ended, high-entropy generation. The advantage survived a doubled token budget, a repeated pretraining run, and supervised finetuning.
- Future prediction becomes part of the autoregressive objective, not an attached task.
- The model is its own drafter and drafts beyond its trained depth.
- Post-hoc drafters lose alignment on open-ended, high-entropy generation.
- Gains held after a doubled token budget and a repeated pretraining run.
From the original article
1 min read at huggingface.co
Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs.
By Yuxuan Zhou, Junyuan Shang, Tianyu Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.