Challenges in Observing the Emotions of Children with Autism Interacting with a Social Robot

Bibliographic Data

ID21199708
AuthorsDuygun Erol Barkana (0000-0002-8929-0459, Yeditepe University), Katrin D Bartl-Pokorny (0000-0002-2658-1056, University of Augsburg), Hatice Köse (0000-0003-4796-4766, Istanbul Technical University), Agnieszka Landowska (0000-0002-4728-689X, Gdańsk University of Technology), Manuel Milling (0000-0002-8842-2958, University of Augsburg), Ben Robin (0000-0002-1646-901X, University of Hertfordshire), Ben Robins, Björn W Schuller (0000-0002-6478-8699, University of Augsburg), Pınar Uluer (0000-0003-2923-6220, Galatasaray University), Pinar Uluer, Michal R Wrobel (0000-0002-1117-903X, Gdańsk University of Technology, corresponding author), Tatjana Zorcec (0000-0002-4956-5163, PHI University Psychiatric Clinic - Skopje)
Year2024
Volume16
Issue11-12
Pages2261-2276
Publication date2024-12-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueInternational Journal of Social Robotics (JOURNAL)
Journal identifiersISSN: 1875-4791 • E-ISSN: 1875-4805
PublisherSpringer Science and Business Media LLC (PUBLISHER)
DOI10.1007/s12369-024-01185-3
OpenAlexW4404165575
LanguageEN
Citations received1
References cited43

This paper concerns the methodology of multi-modal data acquisition in observing emotions experienced by children with autism while they interact with a social robot. As robot-enhanced therapy gains more and more attention and proved to be effective in autism, such observations might influence the future development and use of such technologies. The paper is based on an observational study of child-robot interaction, during which multiple modalities were captured and then analyzed to retrieve information on a child’s emotional state. Over 30 children on the autism spectrum from Macedonia, Turkey, Poland, and the United Kingdom took part in our study and interacted with the social robot Kaspar. We captured facial expressions/body posture, voice/vocalizations, physiological signals, and eyegaze-related data. The main contribution of the paper is reporting challenges and lessons learned with regard to interaction, its environment, and observation channels typically used for emotion estimation. The main challenge is the limited availability of channels, especially eyegaze-related (29%) and voice-related (6%) data are not available throughout the entire session. The challenges are of a diverse nature—we distinguished task-based, child-based, and environment-based ones. Choosing the tasks (scenario) and adapting environment, such as room, equipment, accompanying person, is crucial but even with those works done, the child-related challenge is the most important one. Therapists have pointed out to a good potential of those technologies, however, the main challenge to keep a child engaged and focused, remains. The technology must follow a child’s interest, movement, and mood. The main observations are the necessity to train personalized models of emotions as children with autism differ in level of skills and expressions, and emotion recognition technology adaptation in real time (e. g., switching modalities) to capture variability in emotional outcomes

Autism · Cognitive psychology · Cognitive science · Developmental psychology · Human–computer interaction · Mechatronics · Mobile robot · Robot · Robot control · Robotics · Social robot · Autism Spectrum Disorder Research · Child Development and Digital Technology · Computer Science · Psychology · Social Robot Interaction and HRI · Artificial Intelligence

Tools

Open DOIOpen Access