TokenGate learns token pruning for faster LLM prefill
Original title: TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
It reports 1.4–11.3× faster time-to-first-token and 1.5–3.4× lower peak memory with no accuracy loss.
TokenGate is a small drop-in component that adds end-to-end learnable token pruning for long-context prefill. Token-gated attention re-weights attention logits with token importance scores so training can bypass the non-differentiable top-K operator. The authors say it extends standard transformers such as Qwen3 and hybrid-attention models such as Qwen3.5 with lightweight adaptation similar to parameter-efficient fine-tuning. Code is released at the TokenGate GitHub repository.
- Reported speedups are 1.4–11.3× faster time-to-first-token.
- Peak memory is 1.5–3.4× lower with no stated accuracy degradation.
- Versus prior token-pruning methods, prefill is up to 6.4× faster and memory up to 2× lower.
- It adapts Qwen3 and Qwen3.5 with lightweight fine-tuning-style changes.
- Products
- TokenGate
From the original article
1 min read at huggingface.co
Expert human readers process large inputs (e. g. , code repositories and long documents) by skimming to closely read only what matters.
By Liana Patel, Rishabh Ranjan, Ion Stoica · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.