Shared computational principles for language processing in humans and deep language models
Общие вычислительные принципы обработки языка у человека и в глубоких языковых моделях
2022-03-01
SCID: 54.1/mza9xwuh
Discuss with AI
autoregressive deep language modelscontextual embeddingselectrocorticography (ECoG)next-word predictionpost-onset surprise
Figures from the paper
Abstract (AI)
Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.
Key Findings
1
Autoregressive deep language models may offer a biologically feasible computational framework for investigating the neural basis of language.
2
Both neural responses and deep language models compare pre-onset predictions with incoming words to compute post-onset surprise.
3
ECoG recordings from nine participants listening to a 30-minute podcast provide empirical evidence for shared computational principles between brains and deep language models.
4
Human brains and autoregressive deep language models continuously predict upcoming words before they are spoken during natural narrative processing.
5
Humans and autoregressive deep language models represent words in natural contexts using contextual embeddings.
Research Object
Human brain language processing and autoregressive deep language models processing the same natural narrative
Research Subject
Shared computational principles of continuous next-word prediction, post-onset surprise computation, and contextual word representation
Publication Details
Publication Date
2022-03-01
Journal
Publisher
ISSN
Cited by
480
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest