REALM: Retrieval-Augmented Language Model Pre-Training
REALM: предварительное обучение языковой модели с дополнением за счет извлечения информации
2020-02-10
SCID: 54.1/73jm728c
Discuss with AI
REALMlatent knowledge retrievermasked language modelingopen-domain question answeringretrieval-augmented language model pre-training
Figures from the paper
Abstract (AI)
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
Key Findings
1
Fine-tuning REALM for open-domain question answering outperforms prior explicit- and implicit-knowledge models by 4–16% absolute accuracy across three benchmarks.
2
REALM augments language-model pre-training with a latent knowledge retriever that retrieves and attends over documents from large corpora such as Wikipedia.
3
Retrieval-based knowledge storage provides qualitative benefits including greater interpretability and modularity compared with knowledge encoded solely in model parameters.
4
The retriever can be pre-trained unsupervised using masked language modeling, with backpropagation through retrieval over millions of documents.
Research Object
Retrieval-Augmented Language Model pre-training (REALM) with a latent knowledge retriever over a large document corpus
Research Subject
The effectiveness, interpretability, modularity, and knowledge-retrieval behavior of REALM for open-domain question answering
Publication Details
Publication Date
2020-02-10
Journal
arXiv (Cornell University)
Publisher
Cornell University
ISSN
2331-8422
Cited by
1335
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai4
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Distributed Representations of Words and Phrases and their Compositionality2013
Neural Machine Translation by Jointly Learning to Align and Translate2014
Transformer-Based Feature Learning for Algorithm Selection in Combinatorial Optimisation2025
Cited by19
Dense Passage Retrieval for Open-Domain Question Answering2020
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing2022
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks2020
On the Opportunities and Risks of Foundation Models2021
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
Text Data Augmentation for Deep Learning2021
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
Active Retrieval Augmented Generation2023
Atlas: Few-shot Learning with Retrieval Augmented Language Models2022
Retrieval-Augmented Generation for AI-Generated Content: A Survey2026
Benchmarking Large Language Models in Retrieval-Augmented Generation2024
Retrieval Augmentation Reduces Hallucination in Conversation2021
In-Context Retrieval-Augmented Language Models2023
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021
Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering2023
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval2020
Dense Hierarchical Retrieval for Open-domain Question Answering2021
Robust Retrieval Augmented Generation for Zero-shot Slot Filling2021
Fine-Grained Distillation for Long Document Retrieval2024