A Computational Framework to Study Hierarchical Processing in Visual Narratives
Bibliographic Data
| ID | 7150325 |
|---|---|
| Authors | Aditya Upadhyayula (0000-0001-5146-8680, Department of Psychological & Brain Sciences Washington University in St. Louis, corresponding author), Neil Cohn (0000-0003-0074-1995, Department of Communication & Cognition Tilburg University) |
| Year | 2025 |
| Volume | 49 |
| Issue | 5 |
| Pages | e70050-e70050 |
| Publication date | 2025-05-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Cognitive Science (JOURNAL) |
| Journal identifiers | ISSN: 0364-0213 • E-ISSN: 1551-6709 |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1111/cogs.70050 |
| PMID | 40317561 |
| OpenAlex | W4410031365 |
| Language | EN |
| References cited | 74 |
Theories of visual narrative comprehension have advocated for a hierarchical grammar‐based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task—where participants ranked all possible segmental boundaries—demonstrated that participants’ preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory—an Earley parser, a hidden Markov model (HMM), and an n‐gram model—were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n‐grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations
Comprehension · Grammar · Hierarchy · Linguistics · Narrative · Natural language processing · Parsing · Visual reasoning · Computer Science · Language, Metaphor, and Cognition · Multimodal Machine Learning Applications · Subtitles and Audiovisual Media · Artificial Intelligence
The effect of word predictability on reading time is logarithmic
The Faculty of Language
Remembrance of things parsed
Contextual prerequisites for understanding
A probabilistic earley parser as a psycholinguistic model
An Overview of Hierarchical Structure in Music
Notes on a Schema for Stories
Three models for the description of language
The Construction of Situation Models in Narrative Comprehension
The Brain Basis of Language Processing
Expectation-based syntactic comprehension
Agents’ goals affect construal of event endpoints
Expectation-based Comprehension
Factors That Affect Crossover Between Multiple Worlds Within a Narrative
Events shape long-term memory for story information
Logical necessity and transitivity of causal relations in stories
The Psychology of Language
The perceptual organization of ongoing behavior
Attribution and the unit of perception of ongoing behavior
Parallels and Nonparallels between Language and Music
Problems of projection
Information-theoretical Complexity Metrics
Event structure in perception and conception
Situation models in language comprehension and memory
| Highly cited | No |
|---|