The Sequence Knowledge #898: The Trace Is the Teacher: Distilling Reasoning Into Small Models
From the release of DeepSeek R1, distillation in reasoning models have become one of the most common techniques in frontier AI.
TL;DR
- DeepSeek's R1 model generated approximately 800,000 worked solutions, including detailed thought processes, false starts, and self-corrections.
- These traces were filtered for correctness and readability and used for supervised fine-tuning on smaller open models (Qwen and Llama).
- The fine-tuning process did not involve advanced techniques like reinforcement learning, but simple next-token prediction on the teacher's transcripts.
- The distilled models exhibited emergent reasoning behaviors, such as solving advanced math problems and verifying their own work, outperforming their size.
- This result contradicts previous arguments that naive sequence-level imitation is ineffective for distilling reasoning, highlighting the importance of the reasoning trace itself.