Convergent Emergence of In-Context Learning Across Modalities
Controlled experiments show few-shot in-context learning emerges across six modalities including language, genomes, images, and proteins, partially supporting a convergence hypothesis.
The paper tests the Convergent Emergence Hypothesis: that few-shot in-context learning, when it emerges, shares a common cross-modality difficulty profile. A controlled framework instantiated the same task suite across six modalities: language, genome, integer sequences, time series, images, and proteins. Paired-mapping ICL emerged in all six modalities, surpassed controlled baselines, and showed correlated per-task effects in five of them, providing partial support for the hypothesis.
- Same few-shot ICL task suite instantiated in six modalities for controlled comparison
- Paired-mapping ICL emerged in all six modalities and beat controlled baselines
- Per-task difficulty effects correlated across five of six modalities
- Extends prior ICL findings from language models to autoregressive genomic models
From the original article
1 min read at huggingface.co
Few-shot in-context learning (ICL), the capacity of a model to infer abstract patterns from input-output examples provided in its prompt and apply them to new inputs, has been extensively studied in large language models trained for next-token prediction on human text. Recently, few-shot ICL has been demonstrated in autoregressive genomic models as well.
By Nathan Breslow, Seungwook Han, Daniel Hyunsoo Lee · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.