How Is a “Kitchen Chair” like a “Farm Horse”? Exploring the Representation of Noun-Noun Compound Semantics in Transformer-based Language Models

Bibliographic Data

ID12156136
AuthorsMark Ormerod (0000-0003-3454-2863, Queen’s University Belfast. [email protected], corresponding author), Jesús Martínez del Rincón (0000-0002-9574-4138, Queen's University Belfast), Barry Devereux (0000-0003-2128-8632, Queen's University Belfast)
Year2023
Volume50
Issue1
Pages49-81
Publication date2023-11-15
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueComputational Linguistics (JOURNAL)
Journal identifiersISSN: 0891-2017 • E-ISSN: 1530-9312
PublisherAssociation for Computational Linguistics (PUBLISHER • US)
DOI10.1162/coli_a_00495
OpenAlexW4388691779
LanguageEN
Citations received1
References cited41

Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel “compositional probe” setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds

Natural language processing · Noun · Noun phrase · Question answering · Security token · Transformer · Computer Science · Natural Language Processing Techniques · Text Readability and Simplification · Topic Modeling · Artificial Intelligence

Tools

Open DOIOpen Access