Evaluating large language models on a hermeneutically complex text annotation task

Bibliographic Data

ID19487403
AuthorsAlexandra Fodor (0000-0003-1290-4560, Eötvös Loránd University), Eszter Katona (0000-0002-0303-6223, Eötvös Loránd University, corresponding author), Renáta Németh (0000-0001-6796-3563, Eötvös Loránd University)
Year2026
Volume16
Pages100270
Publication date2026-06-01
Open AccessYes
TypeARTICLE
VenueAmpersand (JOURNAL)
Journal identifiersISSN: 2215-0390 • E-ISSN: 2215-0390
PublisherElsevier BV (PUBLISHER)
DOI10.1016/j.amper.2026.100270
OpenAlexW7161247791
LanguageEN
References cited14

The study examines whether large language models (LLMs) are capable of interpreting individual framings of depression in online forums - a challenging annotation task involving categories that are neither mutually exclusive nor trivial from a hermeneutic perspective. Using a corpus annotated by two annotators, the texts and categories that posed difficulties for human interpretation were identified, and the LLMs were prompted to report their confidence in their classifications. The study addresses the following research questions: whether LLMs can perform complex, meaning-making tasks comparable to human interpretation; how model and prompt choices affect accuracy and reliability; the extent to which model confidence in its own classifications correlates with classification accuracy; whether LLMs and humans struggle with the same texts; and how these findings can inform human–LLM collaboration in annotation process. Two models, GPT-4o mini (closed-source) and Llama 3.3 70B (open-source) were evaluated using zero-shot and few-shot prompting strategies. While both models showed strong reliability in terms of run-to-run, model-to-model and prompt-to-prompt comparisons, their overall accuracy was moderate. Performance was highest on texts that were also easier for human annotators. Notably, higher model confidence was often associated with greater accuracy, suggesting that confidence scores may be useful in supporting collaborative annotation systems. The findings highlight both the potential and the interpretive limits of LLMs in social science applications, underscoring the importance of careful annotation design and the thoughtful integration of LLMs into human-centered workflows. • Evaluated LLMs on a complex depression annotation task • Few-shot prompting yielded only marginal improvements over zero-shot • The LLM also struggled to label posts that were challenging for human annotators • Model confidence correlated positively with annotation accuracy • Model confidence could be used in supporting collaborative annotation systems

Affect (linguistics) · Annotation · Construct (python library) · Expression (computer science) · Interpretation (philosophy) · Task (project management) · Computational and Text Analysis Methods · Digital Mental Health Interventions · Mental Health via Writing

Tools

Open DOIOpen Access