Multimodal Corpus‐Based Approaches

Bibliographic Data

ID23699722
AuthorsJohn A Bateman (0000-0002-7209-9295, corresponding author)
Year2026
Pages1-10
Publication date2026-03-26
Peer ReviewedYes
Open AccessYes
TypeCHAPTER
VenueEncyclopedia of Applied Linguistics (SOURCE_BOOK)
PublisherWiley (PUBLISHER • GB)
DOI10.1002/9781405198431.wbeal20833
OpenAlexW7108207568
ISBN9781405198431
LanguageEN
References cited24

Growing out of earlier corpus work on verbal language, multimodal corpora are specially collected and curated representations of authentic communications where multiple expressive resources are at work. Such expressive resources are usually characterized in terms of “modes,” which is why the corresponding corpora are termed “multimodal.” “Modes” may include sensory modes (i.e., expressive resources distinguished by sight, hearing, touch, etc.), almost any source of measurements concerning some communicative behavior, such as pulse rate or eye‐tracking data, loose labels for medial forms, such as images, “language,” diagrams, conversations, or sounds, as well as more formally characterized semiotic modes, which generally combine specified aspects of materiality with defined principles and categories of composition and interpretation. Accommodating such diverse kinds of data raises many challenges that need to be respected both when designing tools for using corpora and when considering combinations of types of data, which is always a central concern with multimodal communication. Current approaches generally employ forms of stand‐off annotation but still require stronger theoretical foundations for motivating particular annotation schemes and for understanding the forms of annotation that will be most effective.

Annotation · Materiality (auditing) · Multimodality · Semantics (computer science) · Semiotics

Tools

Open DOILibgen