Multimodal Depression Detection Based on Self-Attention Network With Facial Expression and Pupil

Depression is a major mental health issue in contemporary society, with an estimated 350 million people affected globally. The number of individuals diagnosed with depression continues to rise each year. Currently, clinical practice relies entirely on self-reporting and clinical assessment, which carries the risk of subjective biases. In this article, we propose a multimodal method based on facial expression and pupil to detect depression more objectively and precisely. Our method first extracts the features of facial expressions and pupil diameter using residual networks and 1-D convolutional neural networks. Second, a cross-modal fusion model based on self-attention networks (CMF-SNs) is proposed, which utilizes cross-modal attention networks within modalities and parallel self-attention networks between different modalities to extract CMF features of facial expressions and pupil diameter, effectively complementing information between different modalities. Finally, the obtained features are fully connected to identify depression. Multiple controlled experiments show that compared to single modality, the multimodal fusion method based on self-attention networks has a higher ability to recognize depression, with the highest accuracy of 75.0%. In addition, we conducted comparative experiments under three different stimulation paradigms, and the results showed that the classification accuracy under negative and neutral stimuli was higher than that under positive stimuli, indicating a bias of depressed patients toward negative images. The experimental results demonstrate the superiority of our multimodal fusion method

Cognitive psychology · Facial expression · Pupil · Computer Science · Emotion and Mood Recognition · Neuroscience · Psychology · Artificial Intelligence

• ERBMA-Net

  Open Access•Muhammad Turyalai Khan, Yin Cao et al.•IEEE Transactions on Computational…•2026

  Incluir item

• Emotion Separation and Recognition From a Facial Expression by Generating the Poker Face With Vision Transformers

  Open Access•Jia Li, Jiantao Nie et al.•IEEE Transactions on Computational…•2025

  Incluir item

• The pupil as a measure of emotional arousal and autonomic activation

  Open Access•Margaret M Bradley, Laura Miccoli et al.•Psychophysiology•2008

  Incluir item

• Attentional Biases for Negative Interpersonal Stimuli in Clinical Depression.

  Ian H Gotlib, Elena N Krasnoperova et al.•Journal of Abnormal Psychology•2004

  Incluir item

• Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the Covid-19 pandemic

  Open Access•Damian F Santomauro, Damian Santomauro et al.•The Lancet•2021

  Incluir item

• Orthogonal-Moment-Based Attraction Measurement With Ocular Hints in Video-Watching Task

  Open Access•Minqiang Yang, Xiang Feng et al.•IEEE Transactions on Computational…•2023

  Incluir item

• What Does Your Bio Say? Inferring Twitter Users’ Depression Status From Multimodal Profile Information Using Deep Learning

  Open Access•Soumitra Ghosh, Asif Ekbal et al.•IEEE Transactions on Computational…•2022

  Incluir item

• Factors influencing suicidal tendencies during Covid-19 pandemic in Korean multicultural adolescents

  Open Access•Ju Young Park, Insook Lee•BMC Psychology•2022

  Incluir item