Do outlier assessors provide useful narrative comments
Bibliographic Data
| ID | 21448386 |
|---|---|
| Authors | Sebastian Dewhirst (0000-0003-1996-6692, University of Ottawa, corresponding author), Nora D Szabo (0009-0002-9492-2163, University of Ottawa), Andrew K Hall (0000-0003-1227-5397, University of Ottawa), Warren J Cheung (0000-0002-2730-8190, University of Ottawa) |
| Year | 2026 |
| Volume | 48 |
| Issue | 5 |
| Pages | 836-843 |
| Publication date | 2026-05-04 |
| Open Access | No |
| Type | ARTICLE |
| Venue | Medical Teacher (JOURNAL) |
| Journal identifiers | ISSN: 0142-159X • E-ISSN: 1466-187X |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/0142159x.2025.2596907 |
| PMID | 41368758 |
| OpenAlex | W4417168293 |
| Language | EN |
| References cited | 50 |
Introduction Outlier stringent/lenient and range-restricted assessors provide low utility assessment scores, reflecting assessor tendency more than learner performance. The quality of narrative comments generated by these assessors remains unknown.Methods End-of-shift assessments from an academic emergency department were scored using the Quality of Assessment of Learning (QuAL) score. The mean Qual score was calculated for each assessor. Stringency/leniency and range restriction were quantified using the mean-delta method and standard deviation of awarded scores respectively. Outlier vs. non-outlier assessor mean QuAL scores were compared using T-tests. Linear regression was conducted with mean QuAL score as the dependent variable, and assessor range-restriction as the independent variable.Results 2034 assessments were completed by 81 assessors. We identified 20 outlier assessors (10 stringent, 10 lenient). Assessor mean QuAL scores ranged from 1.2 to 5 (mean = 3.9, SD = 0.9). Mean QuAL score of non-outliers (4.0) was significantly higher than outlier lenient assessors (3.1, p = 0.02) but not different from outlier stringent assessors (3.8, p = 0.5). Range restriction was negatively correlated with mean QuAL scores (p = 0.003, R2=0.11)Conclusions Outlier lenient and range-restricted assessors provided lower-than-average quality narrative assessments. These assessors, where both numeric and narrative assessments are of limited utility, are ideal candidates for targeted interventions to improve assessment quality
Ideal (ethics) · Narrative · Narrative review · Outlier · Psychological intervention · Quality (philosophy) · Language Development and Disorders · Psychometric Methodologies and Testing · Reading and Literacy Development
Assessment, feedback and the alchemy of learning
How self-determination theory can assist our understanding of the teaching and learning processes in medical education. Amee Guide No. 59
Competency-based medical education
Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial
Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education
Predictors of faculty narrative evaluation quality in medical school clerkships
| Highly cited | No |
|---|