Relational Knowledge Distillation

Реляционное сгущение знаний
Minsu Cho, Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho
2019-06-01

Relational Knowledge Distillationangle-wise distillation lossdistance-wise distillation lossmetric learningteacher-student knowledge transfer
Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.
1
Experiments across tasks show RKD significantly improves student model performance compared to prior methods.
2
In metric learning, RKD-enabled students can outperform their teachers and achieve state-of-the-art results on standard benchmarks.
3
Introduces relational knowledge distillation (RKD), which transfers mutual relations among data examples instead of individual example activations.
4
Proposes two concrete RKD loss formulations: distance-wise and angle-wise distillation losses that penalize structural relational differences.

Relational knowledge (mutual relations among data examples) transferred from a teacher model to a student model

Distillation of inter-example relations via distance-wise and angle-wise losses to train student models (relational knowledge distillation) and its effect on student performance, particularly in metric learning

Publication Details
Publication Date
2019-06-01
Journal
Publisher
ISSN
Cited by
1778
Access Type
Author Information
Authors
Minsu Cho
Wonpyo Park
Dongju Kim
Yan Lu
Minsu Cho
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%