SeDR: Segment Representation Learning for Long Documents Dense Retrieval
SeDR: обучение представлений сегментов для плотного поиска по длинным документам
2022-11-20
SCID: 54.1/wukpmvgm
Discuss with AI
Late-Cache NegativeSegment-Interaction Transformerdense retrievallong document retrievalsegment representation learning
Figures from the paper
Abstract (AI)
Recently, Dense Retrieval (DR) has become a promising solution to document retrieval, where document representations are used to perform effective and efficient semantic search. However, DR remains challenging on long documents, due to the quadratic complexity of its Transformer-based encoder and the finite capacity of a low-dimension embedding. Current DR models use suboptimal strategies such as truncating or splitting-and-pooling to long documents leading to poor utilization of whole document information. In this work, to tackle this problem, we propose Segment representation learning for long documents Dense Retrieval (SeDR). In SeDR, Segment-Interaction Transformer is proposed to encode long documents into document-aware and segment-sensitive representations, while it holds the complexity of splitting-and-pooling and outperforms other segment-interaction patterns on DR. Since GPU memory requirements for long document encoding causes insufficient negatives for DR training, Late-Cache Negative is further proposed to provide additional cache negatives for optimizing representation learning. Experiments on MS MARCO and TREC-DL datasets show that SeDR achieves superior performance among DR models, and confirm the effectiveness of SeDR on long document retrieval.
Key Findings
1
Experiments on MS MARCO and TREC-DL show that SeDR achieves superior performance among dense retrieval models and is effective for long-document retrieval.
2
Late-Cache Negative supplies additional cached negatives to mitigate insufficient negative samples caused by GPU memory constraints during long-document training.
3
SeDR addresses long-document dense retrieval by learning document representations that preserve whole-document information rather than relying on truncation or simple splitting-and-pooling.
4
Segment-Interaction Transformer outperforms alternative segment-interaction patterns for dense retrieval.
5
The Segment-Interaction Transformer produces document-aware and segment-sensitive representations while maintaining the computational complexity of splitting-and-pooling.
Research Object
long documents in dense retrieval systems
Research Subject
document representation learning and retrieval performance under long-document encoding constraints
Publication Details
Publication Date
2022-11-20
Journal
Publisher
ISSN
Cited by
6
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Representation Learning with Contrastive Predictive Coding2018
Dense Passage Retrieval for Open-Domain Question Answering2020
Longformer: The Long-Document Transformer2020
ETC: Encoding Long and Structured Inputs in Transformers2020
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021