Human-AI collaboration in classroom video analysis

Evaluating the feasibility and usability of multimodal large language models for nonverbal behavior recognition

Bibliographic Data

ID21452856
AuthorsYiqiu Zhou (0009-0000-9004-9307, University of Illinois Urbana-Champaign), Jina Kang (0000-0002-7043-5427, University of Illinois Urbana-Champaign, corresponding author), Ha Nguyen (0000-0001-8327-5488, University of North Carolina at Chapel Hill)
Year2026
Pages1-20
Publication date2026-05-04
Peer ReviewedYes
Open AccessNo
TypeARTICLE
VenueBehaviour and Information Technology (JOURNAL)
Journal identifiersISSN: 0144-929X • E-ISSN: 1362-3001
PublisherInforma UK Limited (PUBLISHER • GB)
DOI10.1080/0144929x.2026.2662406
OpenAlexW7160094245
LanguageEN
References cited58

Video-based analysis of classroom interactions offers valuable insights into user behaviour and engagement, yet manual coding remains labour-intensive and difficult to scale. Traditional computer vision models automate some tasks but rely on rigid taxonomies and extensive labelled data, limiting their usability for educators and researchers. This study explores the use of pre-trained video-based Multimodal Large Language Models (MLLMs) to support automated detection of nonverbal behaviours in classroom settings. We compare two models – LLaVA-7B and LLaVA-72B – in identifying key indicators of embodied and collaborative learning (gestures, eye contact, and joint attention). Using video from a simulation-based science activity, we evaluate each model’s accuracy and efficiency. Our results revealed a critical trade-off: the larger model achieved higher accuracy but required significant computational resources, while the smaller, faster model struggled to detect nuanced social cues. Error analysis reveals common misclassifications, underscoring the importance of careful prompt design and behaviour framing. Given these challenges, we conclude that MLLMs are currently best used as assistants within a human-in-the-loop workflow, where they function as a ‘first-pass’ filter while the human users retain final interpretation. This work provides empirical evidence and practical guidance for integrating generative AI into behaviour analysis workflows

Language acquisition · Multimodality · Natural language · Nonverbal communication · Usability · Emotion and Mood Recognition · Intelligent Tutoring Systems and Adaptive Learning · Social Robot Interaction and HRI

Tools

Open DOI