Crossmodal correspondence between lexical tones and visual motions

A forced-choice mapping task on Mandarin Chinese

Bibliographic Data

ID21227003
AuthorsFeier Gao (0000-0002-3872-8852, Southeast University, corresponding author)
Year2024
Volume10
Issue1
Pages721-729
Publication date2024-12-31
Open AccessNo
TypeARTICLE
VenueLinguistics Vanguard (JOURNAL)
Journal identifiersISSN: 2199-174X • E-ISSN: 2199-174X
PublisherWalter de Gruyter GmbH (PUBLISHER • DE)
DOI10.1515/lingvan-2023-0151
OpenAlexW4400487324
LanguageEN
References cited39

Crossmodal correspondence refers to the phenomenon in which individuals match stimulus features (e.g., auditory pitch) with different sensory modalities (e.g., visual size). While studies on correspondences exhibited by suprasegmentals have mostly focused on pitch-size and pitch-shape associations, audiospatial binding observed in the production and perception of Mandarin tones, where pitch of the syllable distinguishes word meanings, sheds light on the symbolic potential of auditory pitch. In the present study, a forced-choice mapping task was conducted in the form of a word guessing game, where native Mandarin listeners select the meaning of an auditory “alien” word from two visual motions. The results showed that: (1) listeners reliably match auditory tones with visual motions in the way that pitch trajectories are congruent with spatial movements, (2) vowel category impacts tone-motion correspondence when syllables are articulated in non-contour tones, and (3) the capacities in driving the tone-motion correspondence are different across tonal categories. These findings further contribute to our understanding of the sound symbolic potential of lexical tones and expand the boundary of crossmodal correspondence that can be demonstrated by pitch

Crossmodal · Linguistics · Mandarin Chinese · Natural language processing · Perception · Speech recognition · Visual perception · Categorization, perception, and language · Computer Science · Engineering · Language, Metaphor, and Cognition · Multisensory perception and integration · Neuroscience · Philosophy · Psychology

Tools

Open DOI