All eyes on the signal? - Mapping cohesive discourse structures with eye-tracking data of explanation videos

Bibliographic Data

ID22093127
AuthorsLeandra Thiele (0000-0002-8074-8507, University of Bremen, corresponding author), Florian Schmidt-Borcherding (0000-0002-9618-8946, University of Bremen), John A Bateman (0000-0002-7209-9295, University of Bremen)
Year2024
Volume9
Publication date2024-04-11
Open AccessYes
TypeARTICLE
VenueFrontiers in Communication (JOURNAL)
Journal identifiersISSN: 2297-900X • E-ISSN: 2297-900X
PublisherFrontiers Media SA (PUBLISHER • CH)
DOI10.3389/fcomm.2024.1356495
OpenAlexW4394723133
LanguageEN
Citations received3
References cited55

In this paper, we consider the issue of how the fine-grained multimodal design of educational explanation videos, such as those widely available on YouTube and other platforms, may be made accessible to empirical studies of reception and effectiveness. This is necessary because previous research has often led to conflicting conclusions concerning the roles of particular design elements. We argue that this may largely be due to insufficient characterizations of multimodal design itself. To achieve tighter control of this potential source of variation, we present a multimodal descriptive annotation framework drawing on multimodal (cohesive) film discourse analysis. This framework is seen as a critical first step toward being able to highlight just those differences in design that have functional consequences. For such consequences to accrue, however, viewers need to attend differently to corresponding design differences. The goal of the current paper, therefore, is to use eye-tracking techniques to explore the extent to which discourse structures revealed by our analytic framework relate to recipients' attention allocation. We hypothesize that any potentially emerging anomalies in regards to discourse organization, such as instances of unsuccessful cohesion signaling, may have correlations in the behavioral data. We report our current state of development for performing this kind of multimodal cohesion analysis and some of the unresolved challenges raised when considering how such analyses may be related to performance data

Cognitive psychology · Data science · Eye tracking · Human–computer interaction · Multimodality · World Wide Web · Computer Science · Psychology · Subtitles and Audiovisual Media · Video Analysis and Summarization · Visual and Cognitive Learning Processes · Artificial Intelligence

Tools

Open DOIOpen Access