Breakdown of Local Denoising as Semantic Speciation
Theory shows semantic correlations place generative models' nonlocality window inside the speciation window.
The paper relates two temporal regimes in generative models: a speciation window, when a sample commits to a semantic class, and a nonlocality window, when local context becomes insufficient. Under a common-cause hypothesis, it proves nonlocality must occur within speciation when semantic labels explain part of distant-token correlation. It also gives conditions for both windows to shrink to one limiting time as system size grows, and verifies that phase-transition behavior analytically in Gaussian mixtures.
- Proves the nonlocality window lies inside the speciation window
- Assumes semantic labels explain correlations between distant tokens
- Both windows can collapse to one limiting time as size grows
- Behavior is verified analytically on Gaussian mixtures
From the original article
1 min read at arxiv.orgThe dynamics of generative models exhibit two apparently distinct temporal windows: a speciation window, in which a sample commits to a semantic class, and a nonlocality window, in which local context windows become insufficient for generation. Motivated by evidence of their near-concurrence in a variety of frontier models, we investigate their relationship through the spatial distribution of semantic information.
By Guangkuo Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.