ConSDL improves sparse dictionary consistency over SAE baselines
Original title: Beyond SAEs: Consistent Concept Discovery with Sparse Coding
On 16 synthetic settings recovery was near perfect; consistency rose 17.6% for dictionaries and 35.9% for coefficients.
The paper proposes Consistent Sparse Dictionary Learning, which replaces a sparse autoencoder's linear encoder with per-sample sparse coding. The authors say a single linear encoder pass yields suboptimal coefficients compared with per-sample optimization. Across four vision and language settings, they report higher cross-seed dictionary and coefficient consistency than the average of six SAE baselines.
Unconfirmed: State-of-the-art consistency is the authors' result versus six SAE baselines.
- ConSDL nearly recovered dictionaries and coefficients in 16 synthetic settings.
- Dictionary consistency improved 17.6% and coefficient consistency 35.9%.
- The comparison is against the average of six SAE baselines.
- The method is motivated by instability of SAE dictionaries across training runs.
From the original article
1 min read at arxiv.orgSparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding.
By Jinho Choi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.