Beyond speech balloons and thought bubbles

The integration of text and image

Bibliographic Data

ID4980657
AuthorsNeil Cohn (0000-0003-0074-1995, University of California San Diego, corresponding author)
Year2013
Volume2013
Issue197
Publication date2013-01-25
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueSemiotica (JOURNAL)
Journal identifiersISSN: 0037-1998 • E-ISSN: 1613-3692
PublisherWalter de Gruyter GmbH (PUBLISHER • DE)
DOI10.1515/sem-2013-0079
OpenAlexW2321353576
LanguageEN
Citations received20
References cited7

Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture

Gestalt psychology · Gesture · Human–computer interaction · Image (mathematics) · Interface (matter) · Linguistics · Modalities · Natural language processing · Perception · Process (computing) · Artificial Intelligence · Comics and Graphic Narratives · Computer Science · EFL/ESL Teaching and Learning · Language, Metaphor, and Cognition · Psychology

Tools

Open DOISci-Hub