Large language models for scientometric mapping of scientific controversy
A validated hybrid AI–Human framework
Bibliographic Data
| ID | 21442243 |
|---|---|
| Authors | Teo Sušnjak (0000-0001-9416-1435, Massey University, corresponding author), Cole Palffy (Massey University), Tatiana Zimina (Massey University), Nazgul Altynbekova (Massey University), Kunal Garg (0000-0003-4346-027X, University of Jyväskylä), Leona Gilbert (0000-0002-7470-5770, University of Jyväskylä) |
| Year | 2026 |
| Publication date | 2026-06-13 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Scientometrics (JOURNAL) |
| Journal identifiers | ISSN: 0138-9130 • E-ISSN: 1588-2861 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.1007/s11192-026-05681-3 |
| OpenAlex | W7164662816 |
| Language | EN |
| References cited | 56 |
This study advanced large language model (LLM)-enabled scientometric methods for analysing contested scholarly communication through a hybrid AI–human workflow for large-scale mapping of stance and themes in scientific corpora. We treated the literature as a large textual record and examined how competing knowledge claims were distributed across venues and over time. We used the Lyme disease controversy as a testbed, applied multiple LLMs to thousands of abstracts, and validated the outputs with domain experts to trace longitudinal shifts in authorial stance and thematic focus. We examined how alternative perspectives were distributed across venues and themes, how these distributions changed over time, and how venue stratification shaped the visibility of different epistemic positions. Results showed persistent asymmetries in publication placement and citation attention, with higher-impact venues disproportionately hosting one perspective relative to the other, while thematic mapping identified recurring fault lines around disputed claims. Validation showed substantial alignment between LLM outputs and expert ratings across classification tasks (Cohen’s kappa 0.58–0.71), broadly comparable to expert–expert agreement and sufficient for corpus-level scientometric analysis. The framework offers a reproducible model of human–LLM integration in literature-based scientometrics and enables venue- and time-resolved indicators for monitoring controversies and publication imbalances
Bibliometrics · Citation · Perspective (graphical) · Scientific communication · Scientometrics · Thematic map · Visibility · Workflow · Biomedical Text Mining and Ontologies · Computational and Text Analysis Methods · scientometrics and bibliometrics research
Qualitative Content Analysis in Practice
Computing Inter-Rater Reliability for Observational Data
BioBert
A survey on sentiment analysis methods, applications, and challenges
Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective
The Kappa Statistic in Reliability Studies
ChatGPT outperforms crowd workers for text-annotation tasks
Interrater reliability
Growth rates of modern science
A systematic literature review of open innovation in the public sector
Using language models to label clusters of scientific documents
Journal article classification using abstracts
A comparison of citation-based clustering and topic modeling for science mapping
Semantic filtering meets knowledge elements
Co-Reviewer
Large language models and responsible research evaluation
A Coefficient of Agreement for Nominal Scales
Divided Bodies
Laboratory Life
The value-ladenness of transparency in science
Using thematic analysis in psychology
Framing
Critical Questions for Big Data
Measuring nominal scale agreement among many raters
| Highly cited | No |
|---|