Correlation Congruence for Knowledge Distillation
Согласование корреляций для дистилляции знаний
2019-10-01
SCID: 54.1/eraxqx6q
Discuss with AI
Taylor series kernel expansioncorrelation congruence for knowledge distillation (CCKD)image classification and metric learning (CIFAR-100, ImageNet-1K, ReID, Face Recognition)instance-level knowledge distillationinter-instance correlation
Figures from the paper
Abstract (AI)
Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.
Key Findings
1
CCKD is compatible with and easily deployable in common teacher-student frameworks, including KD and hint-based learning methods.
2
CCKD substantially outperforms original KD and other state-of-the-art KD methods on image classification (CIFAR-100, ImageNet-1K) and metric learning (ReID, Face Recognition).
3
Introduced Correlation Congruence for Knowledge Distillation (CCKD) that transfers both instance-level information and inter-instance correlations.
4
Proposed a generalized kernel method based on Taylor series expansion to better capture correlations between instances.
Research Object
Teacher-student knowledge distillation frameworks for deep neural networks (models used in image classification and metric learning)
Research Subject
Transferring both instance-level information and inter-instance correlation via the proposed Correlation Congruence for Knowledge Distillation (CCKD), including a generalized kernel method (Taylor expansion) to better capture correlations and improve student performance
Publication Details
Publication Date
2019-10-01
Journal
Publisher
ISSN
Cited by
604
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
Deep Residual Learning for Image Recognition2016
Very Deep Convolutional Networks for Large-Scale Image Recognition2014
ImageNet: A large-scale hierarchical image database2009
Distilling the Knowledge in a Neural Network2015
Learning Multiple Layers of Features from Tiny Images2024
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices2018
Bilinear CNN Models for Fine-Grained Visual Recognition2015