Safety probe error follows scaling laws in size and labels
Original title: Scaling Laws for Safety Probes
Laws fitted on a family's smallest models predict the next model's error within 0.02 in 72% of cases.
A paper tests whether safety-probe performance can be predicted from model size and label count before training. Researchers built six new safety datasets and trained two probe architectures on 20 base models from five families, using 32 to 1,000,000 labels. Probe error, defined as 1 minus AUROC, falls as a power law in model size and approaches a floor as examples increase. The laws also pick the cheaper path to a target error in 73% of cases.
- Existing safety datasets were too small or already saturated for the study.
- The slope of error reduction steepens as the target model grows.
- Only fitted constants differ across tasks, families, and probe architectures.
- Picking the worse architecture added 0.006 error on average.
From the original article
1 min read at arxiv.orgSafety probes trained on a model's activations are a standard tool for monitoring language models in real time. Each time a new model is released or a new harm area needs to be monitored, the probe must be retrained, which requires new labels and a choice of probe architecture. Yet neither the model nor the labels tell us in advance how well the resulting probe will perform.
By Yuchen Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.