The Co‐Structuring of Gesture‐Vocal Dynamics
An Exploration in Karnatak Music Performance
Bibliographic Data
| ID | 7150294 |
|---|---|
| Authors | Lara Pearson (0000-0002-5073-8738, Max Planck Institute for Empirical Aesthetics, corresponding author), Thomas Nuttall (0000-0001-6316-1424, Music Technology Group, Pompeu Fabra University), Wim Pouw (0000-0003-2729-6502, Department of Computational Cognitive Science, Research Center for Cognitive Science and Artificial Intelligence Tilburg University) |
| Year | 2025 |
| Volume | 49 |
| Issue | 11 |
| Pages | e70137-e70137 |
| Publication date | 2025-11-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Cognitive Science (JOURNAL) |
| Journal identifiers | ISSN: 0364-0213 • E-ISSN: 1551-6709 |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1111/cogs.70137 |
| PMID | 41319266 |
| OpenAlex | W4416829370 |
| Language | EN |
| References cited | 60 |
In music performance contexts, vocalists tend to gesture with hand and upper body movements as they sing. But how does this gesturing relate to the sung phrases, and how do singers’ gesturing styles differ from each other? In this study, we present a quantitative analysis and visualization pipeline that characterizes the multidimensional co‐structuring of body movements and vocalizations in vocal performers. We apply this to a dataset of performances within the Karnatak music tradition of South India, including audio and motion tracking data of 44 performances by three expert Karnatak vocalists, openly published with this report. Our results show that time‐varying features of head and hand gestures tend to be more similar when the concurrent vocal time‐varying features are also more similar. While for each performer we find clear co‐structuring of sound and movement, they each show their own characteristic salient dimensions (e.g., hand position, head acceleration) through which movement co‐structures with singing. Our time‐series analyses thereby provide a computational approach to characterizing individual vocalists’ unique multimodal vocal‐gesture co‐structuring profiles. We also show that co‐structuring clearly reduces degrees of freedom of the multimodal performance such that motifs that sound alike tend to co‐structure with gestures that move alike. The current method can be applied to any multimodally ensembled signals in both human and nonhuman communication, to determine co‐structuring profiles and explore any reduction in degrees of freedom. In the context of Karnatak singing performance, the current analysis is an important starting point for further experimental study of gesture‐vocal synergies
Coarticulation · Gesture · Performing arts · Salient · Singing · Visualization · Action Observation and Synchronization · Music Technology and Sound Studies · Neuroscience and Music Perception
Gesture Visible Action as Utterance
The Primacy of Movement
Coding gestural behavior with the Neuroges-Elan system
Dynamic programming algorithm optimization for spoken word recognition
How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables
Gesture and speech in interaction
The Territory Between Speech and Song
Primary versus secondary musical parameters and the classification of melodic motives
The Role of the Body in the Production and Perception of Solo Vocal Performance
Gesture in Musical Declamation
Head movements encode emotions during speech and song
Recurrent gestures throughout bodies, languages, and cultural practices
Foreground gesture, background gesture
How recurrent gestures mean
Multimodal coordination and pragmatic modes in conversation
Hindustani raga and singer classification using 2D and 3D pose estimation from video recordings
Looking Beyond the Score
Dynamical systems theory in cognitive science and neuroscience
Linking facial animation, head motion and speech acoustics
The Analysis of Raga Alapana in South Indian Music
| Citation velocity | historical |
|---|---|
| Highly cited | No |