Distilling the Knowledge in a Neural Network
Дистилляция знаний в нейронной сети
2015-03-09
SCID: 54.1/sx9q4xyw
Discuss with AI
MNISTensemble distillationknowledge distillationmodel compressionspecialist models
Figures from the paper
Abstract (AI)
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Key Findings
1
A new ensemble type combining one or more full models with many specialist models is introduced; specialists learn to distinguish fine-grained classes confused by full models.
2
Distillation produced surprising performance improvements on MNIST compared to typical single-model training.
3
Distilling an ensemble into a single model significantly improved the acoustic model of a heavily used commercial system.
4
Ensembles of models improve performance but are expensive to deploy as predictions require multiple large neural nets.
5
Knowledge from an ensemble can be compressed into a single model using a distillation/compression technique, enabling easier deployment.
6
Specialist models can be trained rapidly and in parallel, unlike traditional mixtures of experts.
Research Object
Knowledge distilled from an ensemble of neural network models into a single neural network (for tasks such as MNIST classification and acoustic modeling)
Research Subject
The efficacy of model compression/distillation: transferring ensemble predictive behavior into a single model to retain performance while reducing computational cost, including use of specialist models and evaluation on MNIST and commercial acoustic systems
Publication Details
Publication Date
2015-03-09
Journal
Publisher
ISSN
Cited by
26306
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai4
Cited by20
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices2018
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions2021
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)2018
Emerging Properties in Self-Supervised Vision Transformers2021
Object Detection With Deep Learning: A Review2019
Deep Reinforcement Learning: A Brief Survey2017
Continual lifelong learning with neural networks: A review2019
Deep Learning for Generic Object Detection: A Survey2019
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence2023
On the Opportunities and Risks of Foundation Models2021
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks2023
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence2023
Machine Learning Interpretability: A Survey on Methods and Metrics2019
TinyBERT: Distilling BERT for Natural Language Understanding2020
Relational Knowledge Distillation2019
Text Data Augmentation for Deep Learning2021
A survey of uncertainty in deep neural networks2023
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey2020