Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Интеллект, компилируемый через аффордансы: наблюдаемое сопоставление когнитивных импедансов для LLM-интегрированных систем без мета-доступа
2020-05-22
SCID: 54.1/95m8uymm
Discuss with AI
ARCATLASAURORACPIContextual Personal IntelligencePolymeta
Figures from the paper
Abstract (AI)
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
Key Findings
1
CPI integrates four co-evolved systems—Polymeta, ARC, ATLAS, and AURORA—to create adaptive, relational, and life-aware AI.
2
CPI is shaped by four core dimensions: memory, meaning, rhythm, and developmental phase.
3
Contextual Personal Intelligence (CPI) is proposed as a new AI paradigm that evolves with an individual over time.
4
The paper presents an architectural foundation, philosophical implications, and a roadmap for developing CPI.
Research Object
Contextual Personal Intelligence (CPI) system integrating Polymeta, ARC, ATLAS, and AURORA
Research Subject
The paradigm-level design and evolution of an AI that adapts to an individual over time via memory, meaning, rhythm, and developmental phase — i.e., architectural foundation, adaptive/relational behavior, and roadmap for life-aware AI
Publication Details
Publication Date
2020-05-22
Journal
Publisher
ISSN
Cited by
4480
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Dense Passage Retrieval for Open-Domain Question Answering2020
HuggingFace's Transformers: State-of-the-art Natural Language Processing2019
REALM: Retrieval-Augmented Language Model Pre-Training2020
Data Driven VRP: A Neural Network Model to Learn Hidden Preferences for VRP2021
Cited by20
Dense Passage Retrieval for Open-Domain Question Answering2020
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
Text Data Augmentation for Deep Learning2021
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
Active Retrieval Augmented Generation2023
Atlas: Few-shot Learning with Retrieval Augmented Language Models2022
Benchmarking Large Language Models in Retrieval-Augmented Generation2024
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation2025
Retrieval Augmentation Reduces Hallucination in Conversation2021
A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges2024
The Power of Noise: Redefining Retrieval for RAG Systems2024
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021
Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering2023
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges2025
A review of large language models and autonomous agents in chemistry2024
Retrieval augmented generation for large language models in healthcare: A systematic review2025
Vision-language models for medical report generation and visual question answering: a review2024
Rag-Fusion: A New Take on Retrieval Augmented Generation2024
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models2023