Poster: A Preliminary Study of LLM Distillation Inference
Distillation-inference test flags Llama-3.2-3B models distilled from Qwen2.5-7B at TPR 1.0 and p=0.02.
Unauthorized distillation trains a model on a proprietary LLM's outputs, and the authors study whether an auditor can tell distillation from independent training. They train shadow models—distilled ones on the teacher's reasoning traces, independent ones on reference answers only—and convert the suspect's score into a calibrated p-value. In a preliminary study with Qwen2.5-7B as teacher and Llama-3.2-3B suspects, the test achieved a true positive rate of 1.0 at significance 0.02.
- Casts unauthorized distillation detection as a calibrated hypothesis test
- Distilled shadow models learn teacher reasoning traces; independent ones use answers only
- Teacher is Qwen2.5-7B; suspect models are Llama-3.2-3B
- True positive rate of 1.0 at significance level 0.02
- AI models
- Qwen2.5-7BLlama-3.2-3B
From the original article
1 min read at arxiv.orgUnauthorized model distillation, in which a model is trained on the outputs of a proprietary large language model (LLM), is a growing threat to model providers. We study distillation inference: determining whether a suspect model was distilled from another model or trained independently.
By Edward Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.