Relational Knowledge Distillation
Реляционное сгущение знаний
2019-06-01
SCID: 54.1/mxzwcht7
Discuss with AI
Relational Knowledge Distillationangle-wise distillation lossdistance-wise distillation lossmetric learningteacher-student knowledge transfer
Figures from the paper
Abstract (AI)
Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.
Key Findings
1
Experiments across tasks show RKD significantly improves student model performance compared to prior methods.
2
In metric learning, RKD-enabled students can outperform their teachers and achieve state-of-the-art results on standard benchmarks.
3
Introduces relational knowledge distillation (RKD), which transfers mutual relations among data examples instead of individual example activations.
4
Proposes two concrete RKD loss formulations: distance-wise and angle-wise distillation losses that penalize structural relational differences.
Research Object
Relational knowledge (mutual relations among data examples) transferred from a teacher model to a student model
Research Subject
Distillation of inter-example relations via distance-wise and angle-wise losses to train student models (relational knowledge distillation) and its effect on student performance, particularly in metric learning
Publication Details
Publication Date
2019-06-01
Journal
Publisher
ISSN
Cited by
1778
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest