Local Sparsity Enables Unsupervised LLM Safety Detection
Proposes locally masked sparse-autoencoder anomaly detection that flags unsafe LLM outputs unsupervised, using only 1-2% of SAE neurons.
The paper proposes an unsupervised deployment-time LLM safety method based on locally masked sparse-autoencoder (SAE) anomaly detection, avoiding the need for unsafe training data. Under the linear representation hypothesis, nearby activation points share a small common active support, making high-dimensional anomaly detection statistically feasible with theoretical justification. Validated across architectures and both capability-testing and safety-specific datasets, the method uses only 1-2% of SAE neurons and reaches near-optimal performance when calibrated with 1% out-of-distribution data.
- Unsupervised anomaly detection avoids assuming access to unsafe training data
- Local sparsity in SAE concept space justifies high-dimensional detection
- Uses only 1-2% of SAE neurons for computation
- Near-optimal performance with 1% out-of-distribution calibration data
Full article184 words · extracted from arxiv.org · click to collapse
Deployment-time safety methods for large language models (LLMs) are predominantly supervised and assume access to unsafe training data. Nevertheless, new attacks and harm categories regularly arise, not captured by models trained in such a supervised fashion. An alternative approach is to view this problem through the lens of anomaly detection, namely, to rely solely on modeling safe data and flagging out-of-distribution inputs. However, LLM activations lie in a high-dimensional space, raising concerns about whether anomaly detection is statistically feasible. We show that, under the linear representation hypothesis (LRH), there may indeed be hope. In the LRH concept space, which is typically recovered via a sparse autoencoder (SAE), nearby points share a small common active support. Using this local sparsity insight, we propose a framework for locally masked SAE-based anomaly detection, supported by theoretical justifications. We validate it on various architectures and datasets, including both capability-testing datasets and safety-specific datasets. Finally, when we allow algorithms to use 1% out-of-distribution data for calibration, locally sparse methods achieve near-optimal performance, demonstrating their ability to capture meaningful safety information while using only 1-2% of SAE neurons for computation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20129