The Sequence Knowledge #907: The Brain Transplant: Distilling Transformers Into Other Architectures
Weird but more common than you think. The type of distillation you were not thinking about.
TL;DR
- All previous distillation methods used models of the same architecture for teacher and student.
- Cross-architecture distillation involves a transformer teacher and a non-transformer student.
- The student model can be a state-space model, linear RNN, or other recurrent architecture.
- This process transfers the transformer's capability to a different computational substrate.
- The success of this method defies initial expectations.
- This area of distillation is economically significant.