Open-source LLMs for text annotation

A Practical Guide for Model Setting and Fine-Tuning

Datos Bibliográficos

ID7158796
AutoresMeysam Alizadeh (0000-0001-6696-6471, University of Zurich, autor de correspondencia), Mael Kubli (0000-0002-5592-9648, University of Zurich), Zeynab Samei (0000-0002-6802-5090, Institute for Research in Fundamental Sciences), Shirin Dehghani (Allameh Tabataba'i University), Mohammadmasiha Zahedivafa (Iran University of Science and Technology), Juan Diego Bermeo (0009-0000-6874-4848, University of Zurich), Maria Korobeynikova (University of Zurich), Fabrizio Gilardi (0000-0002-0635-3048, University of Zurich)
Año2025
Volumen8
Número1
Páginas17-17
Fecha de publicación2025-02-01
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaJournal of Computational Social Science (JOURNAL)
Identificadores de la revistaISSN: 2432-2725 • E-ISSN: 2432-2717
EditorialSpringer Science and Business Media LLC (PUBLISHER)
DOI10.1007/s42001-024-00345-9
PMID39712076
OpenAlexW4405552797
IdiomaEN
Citas recibidas13
Referencias citadas22

This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ - 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ - 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers

Annotation · Cartography · Geography · Open source · Computational and Text Analysis Methods · Computer Science · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence