Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism
Small transformers accumulate Bayesian evidence for latent common causes through a context-invariant, generalizing mechanism.
Researchers investigate how small transformers can perform Bayesian reasoning about latent common causes as a cross-context generalization. Their setup separates the model's causal mechanisms from the true data-generating process and emphasizes natural-language-style prediction. They examine whether evidence accumulation can be encoded in representations that transfer to novel test cases.
- Studies Bayesian common-cause inference in small transformers
- Separates learned mechanisms from the true data-generating process
- Tests whether evidence accumulation generalizes to novel contexts
From the original article
1 min read at arxiv.orgWe present an in-depth investigation of how a form of Bayesian reasoning about common causes can emerge as a cross-contextual generalization in small, tractable transformers.
By Amir Mohammadpour · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.