Redistribution score tracks recurring SAE feature shifts
Original title: Representation Redistribution in Sparse Autoencoders
On two frozen safety readouts, median supports of about 20 features reached failure AUROCs of 0.935 and 0.958.
The paper studies how prompt transformations that preserve a request can redistribute activity across sparse autoencoder features. It introduces the redistribution sensitivity score to find large, consistently directed feature responses and checks them on held-out prompt variants. Across 27 pretrained SAEs and eight language models, the tested transformations produced a reproducible redistribution profile. An exact decomposition separates recurring representation change from signed readout contribution and decision failure.
Unconfirmed: AUROC figures are the authors' measurements on the two frozen safety readouts studied.
- Evaluation covered 27 pretrained SAEs spanning eight language models.
- Median supports used 20 and 19 features on two frozen safety readouts.
- Those supports gave recurrent failure AUROCs of 0.935 and 0.958.
- Full erosion scored 0.965 and 0.964 on the same readouts.
- Code is at the OpenSAE Redistribution-Sensitivity-Score repository.
From the original article
1 min read at huggingface.co
Sparse autoencoders (SAEs) decompose language model activations into sparse features, but prompt transformations designed to preserve the underlying request can redistribute activity across them. A feature change observed on one transformed prompt need not recur, and even a recurring change need not affect a downstream computation.
By Ziye Mei, Haotian Ma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.