Human-AI collaboration in classroom video analysis

Evaluating the feasibility and usability of multimodal large language models for nonverbal behavior recognition

Dados Bibliográficos

ID21452856
AutoresYiqiu Zhou (0009-0000-9004-9307, University of Illinois Urbana-Champaign), Jina Kang (0000-0002-7043-5427, University of Illinois Urbana-Champaign, autor correspondente), Ha Nguyen (0000-0001-8327-5488, University of North Carolina at Chapel Hill)
Ano2026
Páginas1-20
Data de publicação2026-05-04
Open AccessNão
TipoARTICLE
PeriódicoBehaviour and Information Technology (JOURNAL)
Identificadores do periódicoISSN: 0144-929X • E-ISSN: 1362-3001
EditoraInforma UK Limited (PUBLISHER • GB)
DOI10.1080/0144929x.2026.2662406
OpenAlexW7160094245
IdiomaEN
Referências citadas58

Video-based analysis of classroom interactions offers valuable insights into user behaviour and engagement, yet manual coding remains labour-intensive and difficult to scale. Traditional computer vision models automate some tasks but rely on rigid taxonomies and extensive labelled data, limiting their usability for educators and researchers. This study explores the use of pre-trained video-based Multimodal Large Language Models (MLLMs) to support automated detection of nonverbal behaviours in classroom settings. We compare two models – LLaVA-7B and LLaVA-72B – in identifying key indicators of embodied and collaborative learning (gestures, eye contact, and joint attention). Using video from a simulation-based science activity, we evaluate each model’s accuracy and efficiency. Our results revealed a critical trade-off: the larger model achieved higher accuracy but required significant computational resources, while the smaller, faster model struggled to detect nuanced social cues. Error analysis reveals common misclassifications, underscoring the importance of careful prompt design and behaviour framing. Given these challenges, we conclude that MLLMs are currently best used as assistants within a human-in-the-loop workflow, where they function as a ‘first-pass’ filter while the human users retain final interpretation. This work provides empirical evidence and practical guidance for integrating generative AI into behaviour analysis workflows

Language acquisition · Multimodality · Natural language · Nonverbal communication · Usability · Emotion and Mood Recognition · Intelligent Tutoring Systems and Adaptive Learning · Social Robot Interaction and HRI

Ferramentas

Abrir DOI