Audiovisual Perception of Lexical Stress

Beat Gestures and Articulatory Cues

Bibliographic Data

ID6440690
AuthorsRonny Bujok (0000-0001-6557-9808, Max Planck Institute for Psycholinguistics, The Netherlands, corresponding author), Andrea S Meyer (0000-0002-7735-9025), Antje S Meyer, Hans Rutger Bosker (0000-0002-2628-7738, Max Planck Institute for Psycholinguistics, The Netherlands)
Year2024
Volume68
Issue1
Pages181-203
Publication date2024-06-14
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueLanguage and Speech (JOURNAL)
Journal identifiersISSN: 0023-8309 • E-ISSN: 1756-6053
PublisherSAGE Publishing (PUBLISHER • US)
DOI10.1177/00238309241258162
PMID38877720
OpenAlexW4399706433
LanguageEN
Citations received3
References cited51

Human communication is inherently multimodal. Auditory speech, but also visual cues can be used to understand another talker. Most studies of audiovisual speech perception have focused on the perception of speech segments (i.e., speech sounds). However, less is known about the influence of visual information on the perception of suprasegmental aspects of speech like lexical stress. In two experiments, we investigated the influence of different visual cues (e.g., facial articulatory cues and beat gestures) on the audiovisual perception of lexical stress. We presented auditory lexical stress continua of disyllabic Dutch stress pairs together with videos of a speaker producing stress on the first or second syllable (e.g., articulating VOORnaam or voorNAAM ). Moreover, we combined and fully crossed the face of the speaker producing lexical stress on either syllable with a gesturing body producing a beat gesture on either the first or second syllable. Results showed that people successfully used visual articulatory cues to stress in muted videos. However, in audiovisual conditions, we were not able to find an effect of visual articulatory cues. In contrast, we found that the temporal alignment of beat gestures with speech robustly influenced participants’ perception of lexical stress. These results highlight the importance of considering suprasegmental aspects of language in multimodal contexts

Gesture · Linguistics · Motor theory of speech perception · Perception · Speech perception · Speech recognition · Stress (linguistics · Syllable · Artificial Intelligence · Computer Science · Hearing Impairment and Communication · Language, Metaphor, and Cognition · Multisensory perception and integration · Psychology

Tools

Open DOIOpen Access