NgramRL speeds RL rollouts using drafts from recent history
Original title: NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
Warm-cache runs commit 3.60–7.77 tokens per forward, and a prefetcher raises Spider throughput 1.38x.
NgramRL targets RL post-training, where autoregressive rollout generation can exceed 70% of training time. It builds a per-prompt n-gram drafter from bounded recent rollouts, with no auxiliary model, drafter training, or GPU-resident drafter state, while target-model verification preserves the rollout distribution. An adaptive fan-out prefetcher hides CPU cache lookup. Across two model families and three workloads it accepts more tokens per forward than SuffixDecoding and prompt-lookup decoding.
- On Spider, the prefetcher covers 96.6% of Qwen3-8B outcomes at budget 16.
- Throughput is 1.38x synchronous NgramRL, near a 1.40x overlap ceiling.
- Token acceptance is up to 1.99x SuffixDecoding and 7.0x prompt-lookup decoding.
From the original article
1 min read at arxiv.orgReinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but its cost is dominated by autoregressive rollout generation, which can exceed 70% of training time.
By Nilesh Prasad Pandey · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.