Evaluating human versus machine learning performance in classifying research abstracts

Bibliographic Data

ID21443515
AuthorsYeow Chong Goh (Nanyang Technological University), Xin Cai (0000-0002-7191-2571, Nanyang Technological University), Xin Qing Cai, Walter Theseira (0000-0002-8738-2341, Singapore University of Social Sciences), Giovanni Ko (Singapore Management University), Khiam Aik Khor (0000-0003-1954-8423, Nanyang Technological University, corresponding author)
Year2020
Volume125
Issue2
Pages1197-1212
Publication date2020-11-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueScientometrics (JOURNAL)
Journal identifiersISSN: 0138-9130 • E-ISSN: 1588-2861
PublisherSpringer Science and Business Media LLC (PUBLISHER)
DOI10.1007/s11192-020-03614-2
PMID32836529
OpenAlexW3043669475
LanguageEN
Citations received9
References cited31

We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance against the support vectors machine ML algorithm at classifying European Research Council Starting Grant project abstracts to their actual evaluation panels, which are organised by discipline groups. On average, ML is more accurate than human classifiers, across a variety of training and test datasets, and across evaluation panels. ML classifiers trained on different training sets are also more reliable than human classifiers, meaning that different ML classifiers are more consistent in assigning the same classifications to any given abstract, compared to different human classifiers. While the top five percentile of human classifiers can outperform ML in limited cases, selection and training of such classifiers is likely costly and difficult compared to training ML models. Our results suggest ML models are a cost effective and highly accurate method for addressing problems in comparative bibliometric analysis, such as harmonising the discipline classifications of research from different funding agencies or countries

Machine learning · Meaning (existential) · Percentile · Set (abstract data type) · Statistics · Task (project management) · Training set · Variety (cybernetics) · Advanced Text Analysis Techniques · Artificial Intelligence · Biomedical Text Mining and Ontologies · Computer Science · Engineering · Mathematics · Psychology · Topic Modeling

Tools

Open DOIOpen Access