Memory for prediction
A Transformer-based theory of sentence processing
Bibliographic Data
| ID | 14810679 |
|---|---|
| Authors | Soo Hyun Ryu (0000-0001-6636-9765, University of Michigan, corresponding author), Richard L Lewis (0000-0001-6403-489X, Center for Applied Linguistics) |
| Year | 2025 |
| Volume | 145 |
| Pages | 104670 |
| Publication date | 2025-12-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Journal of Memory and Language (JOURNAL) |
| Journal identifiers | ISSN: 0749-596X • E-ISSN: 1096-0821 |
| Publisher | Elsevier BV (PUBLISHER) |
| DOI | 10.1016/j.jml.2025.104670 |
| OpenAlex | W4412717590 |
| Language | EN |
| Citations received | 1 |
| References cited | 68 |
We demonstrate that Transformer-based neural network language models provide a new foundation for mechanistic theories of sentence processing that seamlessly integrate expectation-based and memory-based accounts. First, we show that the attention mechanism in GPT2-small operates as a kind of cue-based retrieval architecture that is subject to similarity-based interference. Second, we show that it provides accounts of classic memory effects in parsing, including contrasts involving relative clauses and center-embedding. Third, we show that a simple word-by-word entropy metric computed over the internal attention patterns provides an index of memory interference that explains variance in eye-tracking and self-paced reading time measures (independent of surprisal and other predictors) in two natural story reading time corpora. Because the cues and representations are learned, there is no need for the theorist to postulate representational features and cues. Transformers provide practical modeling tools for exploring the effects of memory and experience, given the increasing availability of both pre-trained models and software for training new models, and the ease with which surprisal and attention entropy metrics may be computed
Cognition · Cognitive psychology · Natural language processing · Sentence · Sentence processing · Computer Science · Natural Language Processing Techniques · Neurobiology of Language and Bilingualism · Neuroscience · Psychology · Topic Modeling
The Stanford CoreNLP Natural Language Processing Toolkit
Consequences of the Serial Nature of Linguistic Input for Sentenial Complexity
Contextual effects on word perception and eye movements during reading
Making and correcting errors during sentence comprehension
The E-Z Reader model of eye-movement control in reading
Linguistic complexity
An Activation‐Based Model of Sentence Processing as Skilled Memory Retrieval
Computational principles of working memory in sentence comprehension
Swift
Toward a Universal Law of Generalization for Psychological Science
The sausage machine
Lossy‐Context Surprisal
Expectation-based syntactic comprehension
BRMS
A Mathematical Theory of Communication
Individual Differences in Cue Weighting in Sentence Comprehension
When missing NPs make double center-embedding sentences acceptable
Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty
Neurocomputational Models of Language Processing
A Performance Theory of Order and Constituency
Interference in short-term memory
A Probabilistic Model of Lexical and Syntactic Access and Disambiguation
Uncertainty About the Rest of the Sentence
Comparisons of Online Reading Paradigms
Information-theoretical Complexity Metrics
A Usage-Based Approach to Recursion in Sentence Processing
A Theory of Syntactic Recognition for Natural Language
| Citations per year | 1 |
|---|---|
| Citation span | 2026 - 2026 (1) |
| Highly cited | No |