Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Интеллект, компилируемый через аффордансы: наблюдаемое сопоставление когнитивных импедансов для LLM-интегрированных систем без мета-доступа
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela, Fabio Petroni, Aleksandra Piktus
2020-05-22

ARCATLASAURORACPIContextual Personal IntelligencePolymeta
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
1
CPI integrates four co-evolved systems—Polymeta, ARC, ATLAS, and AURORA—to create adaptive, relational, and life-aware AI.
2
CPI is shaped by four core dimensions: memory, meaning, rhythm, and developmental phase.
3
Contextual Personal Intelligence (CPI) is proposed as a new AI paradigm that evolves with an individual over time.
4
The paper presents an architectural foundation, philosophical implications, and a roadmap for developing CPI.

Contextual Personal Intelligence (CPI) system integrating Polymeta, ARC, ATLAS, and AURORA

The paradigm-level design and evolution of an AI that adapts to an individual over time via memory, meaning, rhythm, and developmental phase — i.e., architectural foundation, adaptive/relational behavior, and roadmap for life-aware AI

Publication Details
Publication Date
2020-05-22
Journal
Publisher
ISSN
Cited by
4480
Access Type
Author Information
Authors
Patrick Lewis
Ethan Perez
Aleksandara Piktus
Fabio Petroni
Vladimir Karpukhin
Naman Goyal
Heinrich Küttler
Mike Lewis
Wen-tau Yih
Tim Rocktäschel
Sebastian Riedel
Douwe Kiela
Fabio Petroni
Aleksandra Piktus
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%