The Sequence Knowledge #890: A Brief History of Model Distillation
The papers and techniques that laid out the ground work to evolve distillation to this level.
TL;DR
- The widely accepted story of knowledge distillation starts in 2015 with Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
- The actual conceptual development of distillation occurred between 2006 and 2015.
- Early papers from 2006-2015 laid the foundation for how distillation is understood and applied today.
- The core question of what knowledge is transferred from teacher to student remains consistent despite changes in vocabulary and diagrams.