What multimodal components, tools, dataset and focus of emotion are used in the current research of multimodal emotion

A systematic literature review

Datos Bibliográficos

ID22110409
AutoresReny Rahmalina (0000-0003-1016-7448, Indonesia University of Education, autor de correspondencia), Wawan Gunawan (0000-0003-1792-8350, Indonesia University of Education)
Año2024
Volumen10
Número1
Fecha de publicación2024-12-31
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaCogent Social Sciences (JOURNAL)
Identificadores de la revistaISSN: 2331-1886 • E-ISSN: 2331-1886
EditorialInforma UK Limited (PUBLISHER • GB)
DOI10.1080/23311886.2024.2376309
OpenAlexW4400687235
IdiomaEN
Citas recibidas1
Referencias citadas94

Emotional engagement is essential in human communication, and the meaning of emotions often entails multimodal relationships. Besides language, multimodality and emotions are semiotic systems that have increasingly attracted the attention of researchers, especially concerning their contribution to the meaning-making process in communication. However, research trends that address how emotions and multimodal components (hereafter multimodal emotions) complement each other in the meaning-making process have not been extensively researched. Hence, this research aims to identify research trends on multimodality and emotions studied from 2018 to 2022. The method used in this research was systematic literature review (SLR) to review, identify, evaluate and interpret all existing research on this topic. Data were obtained from IEEE Xplore, Science Direct and Emerald journals which were then sorted based on the PRISMA method. This analysis acquired comprehensive information on the multimodal components of emotions, data acquisition technologies, datasets and emotional focus and trends. This SLR demonstrated that the research of emotions involving multimodality was conducted with a focus on audio-visual mode with the use of machines namely EEG and LSTM sourced from the IEMOCAP dataset and focused on positive and negative emotions, with anger being the largest focus. Moreover, the findings also showed the interconnectedness of each multimodal component of emotions. Based on these findings, this study suggests that multimodal emotion research should focus on identifying and investigating the meanings generated from emotions to produce good communication. Lack of subject privacy and unmanageable bias in research are other things to consider for future research directions

Cognitive psychology · Data science · Multimodal therapy · Psychotherapist · Computer Science · Digital Communication and Language · Discourse Analysis in Language Studies · Language, Metaphor, and Cognition · Psychology