Text as data for evaluation
Natural language processing and large language models to generate novel insights from unstructured text data
Bibliographic Data
| ID | 6423912 |
|---|---|
| Authors | Thomas Wencker (0000-0002-8825-8529, DEval – Deutsches Evaluierungsinstitut der Entwicklungszusammenarbeit, corresponding author), Janos Borst (0000-0002-9166-4069, Leipzig University), Andreas Niekler (0000-0002-3036-3318, Leipzig University) |
| Year | 2025 |
| Volume | 31 |
| Issue | 3 |
| Pages | 369-393 |
| Publication date | 2025-06-13 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Evaluation (JOURNAL) |
| Journal identifiers | ISSN: 1356-3890 • E-ISSN: 1461-7153 |
| Publisher | SAGE Publishing (PUBLISHER • US) |
| DOI | 10.1177/13563890251330911 |
| OpenAlex | W4411267265 |
| Language | EN |
| Citations received | 2 |
| References cited | 57 |
Policy formulation and implementation generate large volumes of text. However, since reading all relevant sources is often impossible, evaluators must navigate the complexities of selecting the appropriate technology to efficiently extract meaningful information from growing amounts of unstructured text. Text mining blends interpretative and statistical methods to generate novel insights, potentially contributing to evidence-based policy-making. At the same time, biases, a potential lack of accuracy, explainability, and transparency create ethical concerns and make it necessary to combine natural language processing and human judgment to avoid over-reliance on the capabilities of these methods and, in particular, large language models. This article provides practical guidance on how evaluators can use natural language processing to convert unstructured data from text to structured data. It presents a decision framework that accounts for the characteristics of the data, the nature of the task, and the expected results, facilitating the selection of the appropriate technique
Big data · Data mining · Language model · Natural language · Natural language processing · Unstructured data · Computational and Text Analysis Methods · Computer Science · Topic Modeling · Artificial Intelligence
Explanation in artificial intelligence
Transformers
Persistent Anti-Muslim Bias in Large Language Models
An open source machine learning framework for efficient and transparent systematic reviews
A Survey on Transfer Learning
Deep Contextualized Word Representations
STM
ChatGPT outperforms crowd workers for text-annotation tasks
Text Mining for Qualitative Data Analysis in the Social Sciences
Quantitative, Qualitative, Mixed or Holistic Research? Combining Methods in Linguistic Research
Transparency in artificial intelligence
Consistent and replicable estimation of bilateral climate finance
Estimating the common agricultural policy milestones and targets by neural networks
Applying LDA Topic Modeling in Communication Research
Applying Big Data visualization to detect trends in 30 years of performance reports
Artificial intelligence and big data-driven evaluation research and practices
Integrating Big Data Into Evaluation
There’s So Much to Do and Not Enough Time to Do It! A Case for Sentiment Analysis to Derive Meaning From Open Text Using Student Reflections of Engineering Activities
More human than human
Text as Data
Reliability in Content Analysis
Text mining for social science - The state and the future of computational text analysis in sociology
| Unique citing works | 2 |
|---|---|
| Citations per year | 2 |
| Citation span | 2025 - 2026 (2) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 2 |