The Sequence Knowledge #920: The Physics of Teaching: Distillation Scaling Laws
The Sequence — Distillation Series
TL;DR
- Distillation was previously based on anecdotes and lacked predictive power.
- Pretraining has benefited from scaling laws, making model training size predictable.
- A new study, "Distillation Scaling Laws," by an Apple team provides a scientific framework for distillation.
- This study is the most compute-intensive controlled study of distillation to date.
- The new laws offer predictability regarding the influence of teacher models on student models.