Using Natural Language Processing to Automatically Assess Feedback Quality

Findings From 3 Surgical Residencies

Bibliographic Data

ID21614166
AuthorsErkin Ötleş (0000-0003-3169-6832, E. Ötleşis Medical Scientist Training Program fellow, Department of Industrial and Operations Engineering, University of Michigan Medical School, Ann Arbor, Michigan.), Daniel E Kendrick (0000-0001-8589-8603, D.E. Kendrickis assistant professor, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.), Quintin P Solano (0000-0002-5195-8601, Q.P. Solanois a third-year medical student, University of Michigan Medical School, Ann Arbor, Michigan., corresponding author), Mary Schuller (M. Schulleris senior project manager, Department of Surgery, University of Michigan Medical School, Ann Arbor, Michigan.), Mary C Schuller (University of Michigan), Samantha L Ahle (S.L. Ahleis a resident, Department of Surgery, Yale School of Medicine, New Haven, Connecticut.), Mickyas H Eskender (M.H. Eskenderis a resident, Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois.), Emily Carnes (E. Carnesis research assistant, Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois.), Barbara Crutchfield George (0000-0002-9404-5255, University of Michigan), Brian C George (B.C. Georgeis assistant professor and director, Center for Surgical Training and Research, Department of Surgery, University of Michigan Medical School, Ann Arbor, Michigan.)
Year2021
Volume96
Issue10
Pages1457-1460
Publication date2021-10-01
Peer ReviewedYes
Open AccessNo
TypeARTICLE
VenueAcademic Medicine (JOURNAL)
Journal identifiersISSN: 1040-2446 • E-ISSN: 1938-808X
PublisherOxford University Press (OUP) (PUBLISHER)
DOI10.1097/acm.0000000000004153
PMID33951682
OpenAlexW3157626300
LanguageEN
Citations received7
References cited19

PURPOSE: Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment. METHOD: During the 2016-2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy. RESULTS: The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM. CONCLUSIONS: To the authors' knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy

Formative assessment · Logistic regression · Machine learning · Naive Bayes classifier · Natural language processing · Random forest · Statistics · Support vector machine · Computer Science · Diversity and Career in Medicine · Innovations in Medical Education · Surgical Simulation and Training · Artificial Intelligence

Tools

Open DOI