Generative AI Exhibits the Lightness/Pitch Correspondence in Image Generation and Classification Tasks

Bibliographic Data

ID23039671
AuthorsJohn Mcewan (0000-0002-9225-4934, The University of Queensland, corresponding author), Jennifer C Day (0000-0003-0487-9385, The University of Queensland)
EditorsSaad G Yaseen (0000-0002-3079-9410)
Year2025
Volume2025
Issue1
Publication date2025-01-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueHuman Behavior and Emerging Technologies (JOURNAL)
Journal identifiersISSN: 2578-1863 • E-ISSN: 2578-1863
PublisherWiley (PUBLISHER • GB)
DOI10.1155/hbe2/7109027
OpenAlexW7108775539
LanguageEN
References cited66

A crossmodal correspondence (CMC) is an association between sensory features from different modalities. For example, the lightness/pitch correspondence is the tendency to pair more luminant stimuli with higher pitches and dimmer stimuli with lower pitches. There has been a recent surge in research examining how these associations are present in current generative AI models and to what degree they are language dependent. Previous research on the relationship between CMCs and AI has focused on explicit judgements of association, typically with rating scales. In contrast, other psychology subfields such as social psychology are using implicit measures of associations in AI output. In the present study, we argue for the merits of an implicit functionalist approach to generative AI research in CMCs and use two psychophysical paradigms to demonstrate this approach. Experiment 1 explores how the lightness/pitch correspondence might manifest in text‐to‐image models when they are prompted to visually depict auditory characteristics. The results indicate that DALL‐E 3 consistently employs the lightness/pitch, as well as a contrast/pitch correspondence when attempting to visually depict auditory pitch. Experiment 2 then looks at ChatGPT‐4o′s performance in classifying the images from Experiment 1 as high or low in pitch. We find that ChatGPT‐4o uses both lightness and contrast information to inform its classifications of pitch. The implications of these results regarding the study of sensory associations with AI, as well as the specific future research directions of CMCs, are discussed.

Association (psychology) · Contrast (vision) · Crossmodal · Experimental psychology · Generative grammar · Generative model · Perception · Psychophysics · Multisensory perception and integration · Neural and Behavioral Psychology Studies · Neurobiology of Language and Bilingualism

Tools

Open DOIOpen Access