Fine-Grained Distillation for Long Document Retrieval

Мелкозернистая дистилляция для поиска по длинным документам
Jianbing Shen, Daxin Jiang, Guodong Long, Tao Shen, Chongyang Tao, Yucheng Zhou, Xiubo Geng, Can Xu
2024-03-24

cross-encoderdense retrievalfine-grained distillationknowledge distillationlong document retrieval
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
1
Experiments on two long-document retrieval benchmarks demonstrate state-of-the-art performance for the proposed framework.
2
FGD applies multi-granular aligned distillation only during training, improving long-document retriever learning without changing the conventional retrieval paradigm.
3
Long-document retrieval is hindered by the scope hypothesis, as documents may cover multiple topics and exhibit substantial structural heterogeneity.
4
The proposed Fine-Grained Distillation framework generates globally consistent representations across multiple granularities while preserving dense retrieval.
5
This heterogeneity creates a granular-mismatch problem that reduces the effectiveness of conventional knowledge distillation from cross-encoders.

long-document retrieval systems/retrievers operating on large-scale document collections

the effects of document scope and structural heterogeneity on knowledge-distillation efficacy, and fine-grained multi-granular representation alignment for improving retrieval performance

Publication Details
Publication Date
2024-03-24
Journal
Publisher
ISSN
Cited by
17
Access Type
Author Information
Authors
Jianbing Shen
Daxin Jiang
Guodong Long
Tao Shen
Chongyang Tao
Yucheng Zhou
Xiubo Geng
Can Xu
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%