Fine-Grained Distillation for Long Document Retrieval
Мелкозернистая дистилляция для поиска по длинным документам
2024-03-24
SCID: 54.1/dqdkheph
Discuss with AI
cross-encoderdense retrievalfine-grained distillationknowledge distillationlong document retrieval
Figures from the paper
Abstract (AI)
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
Key Findings
1
Experiments on two long-document retrieval benchmarks demonstrate state-of-the-art performance for the proposed framework.
2
FGD applies multi-granular aligned distillation only during training, improving long-document retriever learning without changing the conventional retrieval paradigm.
3
Long-document retrieval is hindered by the scope hypothesis, as documents may cover multiple topics and exhibit substantial structural heterogeneity.
4
The proposed Fine-Grained Distillation framework generates globally consistent representations across multiple granularities while preserving dense retrieval.
5
This heterogeneity creates a granular-mismatch problem that reduces the effectiveness of conventional knowledge distillation from cross-encoders.
Research Object
long-document retrieval systems/retrievers operating on large-scale document collections
Research Subject
the effects of document scope and structural heterogeneity on knowledge-distillation efficacy, and fine-grained multi-granular representation alignment for improving retrieval performance
Publication Details
Publication Date
2024-03-24
Journal
Publisher
ISSN
Cited by
17
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
REALM: Retrieval-Augmented Language Model Pre-Training2020
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021
Graph Neural Networks for Natural Language Processing: A Survey2023
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2020
SeDR: Segment Representation Learning for Long Documents Dense Retrieval2022
Qualitative Spatial Reasoning over Questions (Short Paper)2022