Closing the Context Gap: Activation Alignment for Tabular In-Context Learning
Activation alignment lets tabular models use less context while recovering much of full-context accuracy.
The paper proposes activation alignment for tabular in-context learning so models can use fewer training examples without discarding the benefit of full context. A lightweight linear transformation, trained on synthetic unlabeled data without a GPU and converging in seconds to minutes, maps a partial-context student's activations toward those of a full-context teacher. On 38 TabArena classification datasets, aligned TabPFN-3 and TabFM students beat the unaligned baseline at every context budget and recover nearly half of the teacher's advantage in low-data regimes.
- A CPU-trained linear map aligns partial-context activations to full context
- Evaluated on TabPFN-3 and TabFM across 38 TabArena datasets
- Aligned students improve significantly at every context budget
- Low-data alignment recovers nearly half of the teacher's advantage
From the original article
1 min read at huggingface.co
Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance.
By Yoel Zeldes · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.