Non-Semantic Multimodal Fusion for Predicting Segment Access Frequency in Lecture Archives
Bibliographic Data
| ID | 22046004 |
|---|---|
| Authors | Ruozhu Sheng (0000-0002-3284-8490, Japan Advanced Institute of Science and Technology, corresponding author), Jinghong Li (0000-0002-2716-4801, Japan Advanced Institute of Science and Technology), Shinobu Hasegawa (0000-0002-0892-9629, Japan Advanced Institute of Science and Technology, corresponding author) |
| Year | 2025 |
| Volume | 15 |
| Issue | 8 |
| Pages | 978 |
| Publication date | 2025-07-30 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Education Sciences (JOURNAL) |
| Journal identifiers | ISSN: 2227-7102 • E-ISSN: 2227-7102 |
| Publisher | MDPI AG (PUBLISHER • IT) |
| DOI | 10.3390/educsci15080978 |
| OpenAlex | W4412756299 |
| Language | EN |
| References cited | 21 |
This study proposes a non-semantic multimodal approach to predict segment access frequency (SAF) in lecture archives. Such archives, widely used as supplementary resources in modern education, often consist of long, unedited recordings that are difficult to navigate and review efficiently. The predicted SAF, an indicator of student viewing behavior, serves as a practical proxy for student engagement. The increasing volume of recorded material renders manual editing and annotation impractical, making the automatic identification of high-SAF segments crucial for improving accessibility and supporting targeted content review. The approach focuses on lecture archives from a real-world blended learning context, characterized by resource constraints such as no specialized hardware and limited student numbers. The model integrates multimodal features from instructor’s actions (via OpenPose and optical flow), audio spectrograms, and slide page progression—a selection of features that makes the approach applicable regardless of lecture language. The model was evaluated on 665 labeled one-minute segments from one such course. Experiments show that the best-performing model achieves a Pearson correlation of 0.5143 in 7-fold cross-validation and 61.05% average accuracy in a downstream three-class classification task. These results demonstrate the system’s capacity to enhance lecture archives by automatically identifying key segments, which aids students in efficient, targeted review and provides instructors with valuable data for pedagogical feedback
Annotation · Deep learning · Information retrieval · Machine learning · Multimedia · Natural language processing · World Wide Web · Computer Science · Online and Blended Learning · Online Learning and Analytics · Visual and Cognitive Learning Processes · Artificial Intelligence
| Highly cited | No |
|---|