SeDR: Segment Representation Learning for Long Documents Dense Retrieval

SeDR: обучение представлений сегментов для плотного поиска по длинным документам
Junying Chen, Dongfang Li, Yutao Huang, Qingcai Chen
2022-11-20

Late-Cache NegativeSegment-Interaction Transformerdense retrievallong document retrievalsegment representation learning
Recently, Dense Retrieval (DR) has become a promising solution to document retrieval, where document representations are used to perform effective and efficient semantic search. However, DR remains challenging on long documents, due to the quadratic complexity of its Transformer-based encoder and the finite capacity of a low-dimension embedding. Current DR models use suboptimal strategies such as truncating or splitting-and-pooling to long documents leading to poor utilization of whole document information. In this work, to tackle this problem, we propose Segment representation learning for long documents Dense Retrieval (SeDR). In SeDR, Segment-Interaction Transformer is proposed to encode long documents into document-aware and segment-sensitive representations, while it holds the complexity of splitting-and-pooling and outperforms other segment-interaction patterns on DR. Since GPU memory requirements for long document encoding causes insufficient negatives for DR training, Late-Cache Negative is further proposed to provide additional cache negatives for optimizing representation learning. Experiments on MS MARCO and TREC-DL datasets show that SeDR achieves superior performance among DR models, and confirm the effectiveness of SeDR on long document retrieval.
1
Experiments on MS MARCO and TREC-DL show that SeDR achieves superior performance among dense retrieval models and is effective for long-document retrieval.
2
Late-Cache Negative supplies additional cached negatives to mitigate insufficient negative samples caused by GPU memory constraints during long-document training.
3
SeDR addresses long-document dense retrieval by learning document representations that preserve whole-document information rather than relying on truncation or simple splitting-and-pooling.
4
Segment-Interaction Transformer outperforms alternative segment-interaction patterns for dense retrieval.
5
The Segment-Interaction Transformer produces document-aware and segment-sensitive representations while maintaining the computational complexity of splitting-and-pooling.

long documents in dense retrieval systems

document representation learning and retrieval performance under long-document encoding constraints

Publication Details
Publication Date
2022-11-20
Journal
Publisher
ISSN
Cited by
6
Access Type
Author Information
Authors
Junying Chen
Dongfang Li
Yutao Huang
Qingcai Chen
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%