Human-AI collaboration in classroom video analysis

Evaluating the feasibility and usability of multimodal large language models for nonverbal behavior recognition

Datos Bibliográficos

ID21452856
AutoresYiqiu Zhou (0009-0000-9004-9307, University of Illinois Urbana-Champaign), Jina Kang (0000-0002-7043-5427, University of Illinois Urbana-Champaign, autor de correspondencia), Ha Nguyen (0000-0001-8327-5488, University of North Carolina at Chapel Hill)
Año2026
Páginas1-20
Fecha de publicación2026-05-04
Open AccessNo
TipoARTICLE
RevistaBehaviour and Information Technology (JOURNAL)
Identificadores de la revistaISSN: 0144-929X • E-ISSN: 1362-3001
EditorialInforma UK Limited (PUBLISHER • GB)
DOI10.1080/0144929x.2026.2662406
OpenAlexW7160094245
IdiomaEN
Referencias citadas58

Video-based analysis of classroom interactions offers valuable insights into user behaviour and engagement, yet manual coding remains labour-intensive and difficult to scale. Traditional computer vision models automate some tasks but rely on rigid taxonomies and extensive labelled data, limiting their usability for educators and researchers. This study explores the use of pre-trained video-based Multimodal Large Language Models (MLLMs) to support automated detection of nonverbal behaviours in classroom settings. We compare two models – LLaVA-7B and LLaVA-72B – in identifying key indicators of embodied and collaborative learning (gestures, eye contact, and joint attention). Using video from a simulation-based science activity, we evaluate each model’s accuracy and efficiency. Our results revealed a critical trade-off: the larger model achieved higher accuracy but required significant computational resources, while the smaller, faster model struggled to detect nuanced social cues. Error analysis reveals common misclassifications, underscoring the importance of careful prompt design and behaviour framing. Given these challenges, we conclude that MLLMs are currently best used as assistants within a human-in-the-loop workflow, where they function as a ‘first-pass’ filter while the human users retain final interpretation. This work provides empirical evidence and practical guidance for integrating generative AI into behaviour analysis workflows

Language acquisition · Multimodality · Natural language · Nonverbal communication · Usability · Emotion and Mood Recognition · Intelligent Tutoring Systems and Adaptive Learning · Social Robot Interaction and HRI

Herramientas

Abrir DOI