Watermark Basin Alignment detects distilled LLMs without model weights
Original title: Watermark Basins: Black-Box Detection of Distilled LLMs
At a 0.1% false-positive rate it detected 79% of distilled models at n-gram=1, versus 29% for a token baseline.
Researchers describe watermark basins, key-specific geometric signals left in parameter space when student models are fine-tuned on watermarked teacher traces. Their black-box detector, Watermark Basin Alignment, queries a suspect model through its API and uses a clean probe model without needing weights or logits. On mathematical-reasoning distillation from 135M to 8.2B parameters, it beat token-level detection. The signal survived quantization, pruning, and reinforcement-learning post-training.
Unconfirmed: Reported detection rates are the authors' experimental results on mathematical-reasoning distillation.
- Detection was 45% at n-gram=3 versus 1.3% for the token baseline.
- The false-positive rate used for those figures was 0.1%.
- Students ranged from 135M to 8.2B parameters.
- The basin was described as orthogonal to ordinary fine-tuning directions.
- Token-level signals can fail when traces are sampled at low temperature.
From the original article
1 min read at arxiv.orgProprietary language models and their reasoning traces are vulnerable to API-based distillation, where an adversary trains a student model on the teacher model's outputs. To obtain evidence of distillation, model providers can watermark the teacher's outputs, aiming to leave a radioactive signal in the student's outputs.
By Hongyan Chang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.