The Sequence Knowledge #902: Learning About Distillation: When the Dataset Becomes the Teacher
Once models can generate their own curricula, data stops being a static resource and becomes a transmission medium for intelligence.
TL;DR
- Traditionally, machine learning data was a pre-existing resource to be excavated and cleaned.
- Large language models can now produce various forms of data, including questions, answers, and critiques.
- This allows for a new training paradigm where a capable model (teacher) generates a dataset to train a smaller model (student).
- This process is described as synthetic data used for distillation, where the teacher model's behavior is embedded in the student.