Codebook LLMs

Evaluating LLMs as Measurement Tools for Political Science Concepts

Bibliographic Data

ID6332293
AuthorsAndrew Halterman (0000-0001-9716-9555, Michigan State University, corresponding author), Katherine A Keith (0000-0002-8101-4572, Williams College)
Year2025
Pages1-17
Publication date2025-09-19
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenuePolitical Analysis (JOURNAL)
Journal identifiersISSN: 1047-1987 • E-ISSN: 1476-4989
PublisherCambridge University Press (CUP) (PUBLISHER)
DOI10.1017/pan.2025.10017
OpenAlexW4414362097
LanguageEN
Citations received8
References cited23

Codebooks—documents that operationalize concepts and outline annotation procedures—are used almost universally by social scientists when coding political texts. To code these texts automatically, researchers are increasingly turning to generative large language models (LLMs). However, there is limited empirical evidence on whether “off-the-shelf” LLMs faithfully follow real-world codebook operationalizations and measure complex political constructs with sufficient accuracy. To address this, we gather and curate three real-world political science codebooks—covering protest events, political violence, and manifestos—along with their unstructured texts and human-coded labels. We also propose a five-stage framework for codebook-LLM measurement: Preparing a codebook for both humans and LLMs, testing LLMs’ basic capabilities on a codebook, evaluating zero-shot measurement accuracy (i.e., off-the-shelf performance), analyzing errors, and further (parameter-efficient) supervised training of LLMs. We provide an empirical demonstration of this framework using our three codebook datasets and several pre-trained 7–12 billion open-weight LLMs. We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction-tuning can substantially improve performance. Rather than suggesting the “best” LLM, our contribution lies in our codebook datasets, evaluation framework, and guidance for applied researchers who wish to implement their own codebook-LLM measurement projects

Artificial Intelligence in Law · Legal Education and Practice Innovations