Lost in the Middle: How Language Models Use Long Contexts
Затерянные в середине: как языковые модели используют длинные контексты
2024-01-01
SCID: 54.1/mh3r4y94
Discuss with AI
key-value retrievallong-context evaluationlong-context language modelsmulti-document question answeringposition-dependent performance
Figures from the paper
Abstract (AI)
Abstract While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Key Findings
1
Accessing relevant information in the middle of long contexts causes significant performance degradation, including for explicitly long-context models.
2
Model performance can degrade substantially when the position of relevant information changes within the input context.
3
Models generally perform best when relevant information appears at the beginning or end of long contexts.
4
The analysis introduces evaluation protocols for assessing how robustly future language models use long input contexts.
5
The study evaluates long-context language models on multi-document question answering and key-value retrieval tasks requiring relevant-information identification.
Research Object
language models processing long input contexts
Research Subject
the dependence of relevant-information retrieval and task performance on information position within long contexts, including the degradation of performance for information located in the middle
Publication Details
Publication Date
2024-01-01
Journal
Publisher
ISSN
Cited by
3476
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai6
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
LLaMA: Open and Efficient Foundation Language Models2023
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
Longformer: The Long-Document Transformer2020
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
In-Context Retrieval-Augmented Language Models2023
Cited by19
A Survey of Large Language Models2026
Retrieval-Augmented Generation for Large Language Models: A Survey2023
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education2024
Retrieval-Augmented Generation for AI-Generated Content: A Survey2026
Benchmarking Retrieval-Augmented Generation for Medicine2024
Explainability for Large Language Models: A Survey2024
Graph Retrieval-Augmented Generation: A Survey2025
Knowledge Graph-Guided Retrieval Augmented Generation2025
Exploring Large Language Model‐Based Intelligent Agents: Definitions, Methods, and Prospects2026
Seven Failure Points When Engineering a Retrieval Augmented Generation System2024
Mathemyths: Leveraging Large Language Models to Teach Mathematical Language through Child-AI Co-Creative Storytelling2024
Clinical entity augmented retrieval for clinical information extraction2025
Maintaining research integrity in the age of GenAI: an analysis of ethical challenges and recommendations to researchers2025
BioMaster: Multi-agent system for automated bioinformatics analysis workflow2026
Overloaded minds and machines: a cognitive load framework for human-AI symbiosis2026
Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation2025
Benchmarking LLM-based agents for single-cell omics analysis2026
Toward Sustainable Agentic AI Systems: A Survey of Architectures and Methodologies2026