Will I speak louder if I see you struggling to understand? Speech modifications in response to non-verbal visual cues of listening effort

During face-to-face interactions, overt non-verbal cues from conversational partners can reflect the quality of the conversation. Here, we investigated the impact of specific visual non-verbal cues (i.e., facial expressions of confusion, approaching movements towards the speaker), on the speakers’ acoustic–phonetic adaptations typically employed to enhance clarity (i.e., speech rate, voice intensity, and pitch). Participants read sentences to a listener seated in a separate room and visible through a head-mounted display. Crucially, no auditory feedback was provided. Unknown to participants, the listener was a confederate whose behavior was pre-recorded in silence using a 360° video camera. The confederate simulated three different levels of listening effort: easy, medium, and hard. After each trial, participants rated the estimated listening effort and comprehension of the confederate, as well as their own speaking effort. In Experiment 1 ( N = 22), participants were told their voice was transmitted with varying background noise levels, providing a rationale for potential listener difficulties. In Experiment 2 ( N = 12), no mention of background noise was made, to further isolate the effects of visual cues. In both experiments, participants adjusted their speech to the seen listener’s behavior. Experiment 1 showed adaptations across all measured parameters, and significant modulations persisted for most parameters in Experiment 2. These adaptations were already present in the earliest phase of exposure to the visual effort cues and remained stable across the mini-block, suggesting a rapid and highly sensitive monitoring mechanism. These findings indicate that visible non-verbal cues of listening effort can elicit acoustic–phonetic adaptations, even in the absence of acoustic feedback or explicit communicative signals, highlighting their role in regulating communication and maintaining speech clarity

Active listening · CLARITY · Comprehension · Cue-dependent forgetting · Listening comprehension · Sensory cue · Silence · Multisensory perception and integration · Neurobiology of Language and Bilingualism · Phonetics and Phonology Research

• Physiologic and acoustic differences between male and female voices

  Ingo R Titze•The Journal of the Acoustical…•1989

  Añadir item

• Dialog as interpersonal synergy

  Riccardo Fusaroli, Joanna Rączaszek-Leonardi et al.•New Ideas in Psychology•2014

  Añadir item

• Multimodal Language Processing in Human Communication

  Open Access•Judith Holler, C Levinson et al.•Trends in Cognitive Sciences•2019

  Añadir item

• Language as a multimodal phenomenon

  Open Access•Gabriella Vigliocco, Pamela Perni et al.•Philosophical Transactions of the…•2014

  Añadir item

• Controlling the False Discovery Rate

  Open Access•Yoav Benjamini, Yosef Hochberg•Journal of the Royal Statistical…•1995

  Añadir item

• The Development of Fluency in Advanced Learners of French

  Richard Towell, Roger Hawkins et al.•Applied Linguistics•1996

  Añadir item

• Foreigner Talk, Baby Talk, Native Talk

  Barbara F Freed•International Journal of the…•1981

  Añadir item

• The current status of the motherese hypothesis

  Open Access•Lila R Gleitman, Elissa L Newport et al.•Journal of Child Language•1984

  Añadir item

• Mothers' Speech to Hearing‐Impaired Infants and Children With Cochlear Implants

  Open Access•Tonya R Bergeson, Rachel J Miller et al.•Infancy•2006

  Añadir item

• How Children and Adults Produce and Perceive Uncertainty in Audiovisual Speech

  Open Access•Emiel Krahmer, Marc Swerts•Language and Speech•2005

  Añadir item

• Auditory feedback on the regulation of vocal intensity of preschool children

  Gerald M Siegel, Herbert L Pick et al.•Developmental Psychology•1976

  Añadir item

• Inclusion of Other in the Self Scale and the structure of interpersonal closeness

  Arthur Aron, Elaine N Aron et al.•Journal of Personality and Social…•1992

  Añadir item

• The relation of empathy to prosocial and related behaviors

  Nancy Eisenberg, Paul Miller et al.•Psychological Bulletin•1987

  Añadir item