Codebook LLMs

Evaluating LLMs as Measurement Tools for Political Science Concepts

Datos Bibliográficos

ID6332293
AutoresAndrew Halterman (0000-0001-9716-9555, Michigan State University, autor de correspondencia), Katherine A Keith (0000-0002-8101-4572, Williams College)
Año2025
Páginas1-17
Fecha de publicación2025-09-19
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaPolitical Analysis (JOURNAL)
Identificadores de la revistaISSN: 1047-1987 • E-ISSN: 1476-4989
EditorialCambridge University Press (CUP) (PUBLISHER)
DOI10.1017/pan.2025.10017
OpenAlexW4414362097
IdiomaEN
Citas recibidas8
Referencias citadas23

Codebooks—documents that operationalize concepts and outline annotation procedures—are used almost universally by social scientists when coding political texts. To code these texts automatically, researchers are increasingly turning to generative large language models (LLMs). However, there is limited empirical evidence on whether “off-the-shelf” LLMs faithfully follow real-world codebook operationalizations and measure complex political constructs with sufficient accuracy. To address this, we gather and curate three real-world political science codebooks—covering protest events, political violence, and manifestos—along with their unstructured texts and human-coded labels. We also propose a five-stage framework for codebook-LLM measurement: Preparing a codebook for both humans and LLMs, testing LLMs’ basic capabilities on a codebook, evaluating zero-shot measurement accuracy (i.e., off-the-shelf performance), analyzing errors, and further (parameter-efficient) supervised training of LLMs. We provide an empirical demonstration of this framework using our three codebook datasets and several pre-trained 7–12 billion open-weight LLMs. We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction-tuning can substantially improve performance. Rather than suggesting the “best” LLM, our contribution lies in our codebook datasets, evaluation framework, and guidance for applied researchers who wish to implement their own codebook-LLM measurement projects

Artificial Intelligence in Law · Legal Education and Practice Innovations