Atlas: Few-shot Learning with Retrieval Augmented Language Models
Atlas: обучение в режиме few-shot с использованием языковых моделей, дополненных поиском
2022-08-05
SCID: 54.1/h9k9r3n8
Discuss with AI
Natural Questionsdocument indexfew-shot learningknowledge-intensive tasksretrieval-augmented language models
Figures from the paper
Abstract (AI)
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.
Key Findings
1
Atlas is a carefully designed and pretrained retrieval-augmented language model that learns knowledge-intensive tasks from very few examples.
2
Atlas’s performance depends on document-index content, which can be easily updated to incorporate new or changed knowledge.
3
Evaluations cover diverse benchmarks, including MMLU, KILT, and NaturalQuestions, demonstrating Atlas’s broad few-shot capabilities.
4
On Natural Questions, Atlas exceeds 42% accuracy using only 64 examples and outperforms a 540B-parameter model by 3% with 50× fewer parameters.
5
The results show that retrieval augmentation can support strong few-shot knowledge-intensive learning without requiring massive parameter counts to store knowledge.
Research Object
Atlas, a retrieval-augmented language model for few-shot learning on knowledge-intensive tasks
Research Subject
Few-shot learning of knowledge-intensive tasks and the effect of document-index content and updates
Publication Details
Publication Date
2022-08-05
Journal
Publisher
ISSN
Cited by
993
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai6
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Dense Passage Retrieval for Open-Domain Question Answering2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks2020
Scaling Laws for Neural Language Models2020
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
REALM: Retrieval-Augmented Language Model Pre-Training2020
Cited by11
A Survey of Large Language Models2026
Retrieval-Augmented Generation for Large Language Models: A Survey2023
A Comprehensive Overview of Large Language Models2025
Active Retrieval Augmented Generation2023
Retrieval-Augmented Generation for AI-Generated Content: A Survey2026
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models2024
Benchmarking Large Language Models in Retrieval-Augmented Generation2024
In-Context Retrieval-Augmented Language Models2023
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy2023
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models2024
When Large Language Models Meet Vector Databases: A Survey2025