Context-Answer Metamodels Forecast LLM Activations Pre-Answer
Original title: Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
Context activations can forecast mean-pooled answer activations with high fidelity and generalize zero-shot to out-of-domain datasets.
Researchers trained context-answer metamodels that map context activation sequences to point or distribution forecasts of mean-pooled answer activations under stochastic decoding. The four metamodel families forecast mean-pooled answer activations with high fidelity; distribution metamodels also capture rollout variability. Performance improves as the training set grows to 500,000 prompts and metamodels generalize zero-shot to out-of-domain datasets. Forecasts retain behavior-predictive signal and context activations from Qwen2.5 predict mean-pooled answer activations from Qwen3.5 and Gemma 4.
- Four metamodel families studied
- Forecasts achieve high fidelity on mean-pooled activations
- Distribution metamodels capture rollout variability
- Performance scales with training set to 500,000 prompts
- Zero-shot generalization to out-of-domain datasets
From the original article
1 min read at arxiv.orgUnder stochastic decoding, a single context can elicit many different answers from a language model. Characterizing this variation typically requires many expensive autoregressive rollouts, each revealing only one realization. We ask whether the activations produced while reading the context can directly forecast the model's rollout activations.
By Tommy He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.