Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Study shows specialists trained on question-answer pairs implicitly select latent reasoning trajectories, and tuning choices control the precision-generalization trade-off in distillation.
The work demonstrates that specialist optimization implicitly selects from a latent trajectory space when specialists are trained only on question-answer pairs without explicit reasoning supervision. Using student distillation as an agnostic probe across 27 specialist-student pairings, specialization-generalization profiles correlate exceptionally strongly. Explicitly controlling the specialist's distributional drift systematically shifts both teacher and distilled student along a controllable trade-off between domain precision and general-capability retention across chemistry, physics, and multilingual settings, even across divergent model families.
- Specialist optimization implicitly selects from a latent trajectory space
- 27 specialist-student pairings show strongly correlated specialization profiles
- Controlling distributional drift tunes domain precision versus general capability
- Findings hold across chemistry, physics, multilingual, and divergent model families
Full article175 words · extracted from huggingface.co · click to collapse
Specialist distillation effectively transfers domain expertise to student models via teacher-generated reasoning trajectories. However, when these specialists are trained solely on question--answer pairs without explicit reasoning supervision, what governs the trajectories they generate? In this work, we show that specialist optimization implicitly selects from this latent trajectory space. To isolate and observe this latent distribution, we leverage student distillation not as a downstream goal, but as an agnostic probe---since students inherit no parameterization or optimization constraints from the specialist, inheriting only the sampled trajectories themselves. Through this probe, our empirical analysis unveils a tight governing relationship: across 27 specialist--student pairings, their specialization--generalization profiles correlate exceptionally strongly. Crucially, explicitly controlling the specialist's distributional drift systematically shifts both the teacher and its distilled student along a controllable trade-off between domain precision and general-capability retention. Across chemistry, physics, and multilingual settings, distilled students systematically reflect these specialist-induced profiles, even across divergent model families. Our findings establish a new view of specialist training: when gold reasoning is absent, tuning choices directly control the latent supervision passed to downstream models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.13770