Adults interpret iconicity in speech and gesture via the same modality-independent process

Bibliographic Data

ID21641125
AuthorsMingtong Li (0000-0002-0497-6030, University of Warwick, corresponding author), Suzanne Aussems (0000-0002-0781-5849, University of Warwick), Sachiko Kita (0000-0002-0088-3654, University of Warwick)
Year2025
Volume32
Issue5
Pages2331-2343
Publication date2025-10-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenuePsychonomic Bulletin & Review (JOURNAL)
Journal identifiersISSN: 1069-9384 • E-ISSN: 1531-5320
PublisherSpringer Science and Business Media LLC (PUBLISHER)
DOI10.3758/s13423-025-02698-2
PMID40355683
OpenAlexW4410288662
LanguageEN
References cited39

Iconicity is the resemblance or similarity between the form of a signal and its meaning. In two studies, we investigated whether adults interpret iconicity in speech and gesture via a modality-independent mechanism (Study 1, N = 40; Study 2, N = 348). Participants in both studies completed two verb–action matching tasks. In these tasks, either spoken verbs or co-speech gestures were manipulated at different rates to iconically represent actions (i.e., manners of locomotion) performed at corresponding rates. We hypothesized that adult participants would map verbs spoken at a fast rate to fast actions and verbs spoken at a slow rate to slow actions. Similarly, we hypothesized that adult participants would map gestures produced with faster hand movements to fast actions and gestures produced with slower hand movements to slow actions. Moreover, we predicted a positive correlation between the two task performances. In both a pilot study (Study 1) and a preregistered replication (Study 2), participants successfully used iconic cues to match verbs to actions in both verb–action matching tasks. Crucially, adult participants who performed well in the verb–action matching task with iconic speech cues also performed well in the verb–action matching task with iconic gesture cues. This weak positive correlation remained significant even after controlling verbal working memory performance. These findings suggest that adults may rely on a modality-independent cognitive mechanism for interpreting iconicity both within and across different sensory modalities, using both visual information from gestures and auditory information from speech to identify referent actions

Cognitive psychology · Gesture · Iconicity · Linguistics · Speech recognition · Utterance · Verb · Communication · Computer Science · Hearing Impairment and Communication · Language, Metaphor, and Cognition · Multisensory perception and integration · Psychology · Artificial Intelligence

Tools

Open DOIOpen Access