Mitigating Bias in Clinical Assessment

Recent investigations of medical school clerkship grades have shown systematically lower scores for Black students1 and less representation of non-White students amongst those receiving clerkship honors grades2 or induction into Alpha Omega Alpha. 3 These and other studies raise concerns about grading bias for students with nondominant identities in the clerkship setting. While recommendations to address grading bias exist,3,4 the efficacy of these practices in mitigating disparate grading outcomes remains unknown. While the use of grading committees for final clerkship assessment is among these recommendations, few schools appear to use these committees at the undergraduate medical education (UME) level, although they are a standard requirement in graduate medical education. Our study aims to address this lack of evidence by examining the impact of grading committees on numerical and narrative clinical performance assessments and exploring characteristics of these committees that might contribute to any reduction in bias. Data consisted of clerkship assessments from 97 students in the classes of 2020 and 2021 drawn from both individual faculty ratings (n = 3,300) and the grading committee summary forms (n = 519) used for the students’ permanent records. We compared mean scores by underrepresented in medicine (URiM) status on each of 17 items on a shared clinical performance assessment form using Welch’s unequal variance independent-samples t tests with Bonferroni correction to adjust for multiple comparisons (alpha = 0.05/17 = 0.0029). We then examined the narratives on a sample of 10 URiM and 10 non-URiM students from both individual faculty and summary grading committees using qualitative analysis to look for differences in language and content themes. For individual forms, 4 of 17 items showed significantly lower scores for URiM students with moderate effect size (Cohen’s d of 0.47–0.56). The grading committee summary forms only had 1 of 17 items that differed significantly by URiM status (oral presentation skills; mean diff = 0.196, 95% CI [0.08, 0.313]; t [130.95] = 3.327, P = .001, Cohen’s d = 0.51). Content analysis of the individual narratives showed that for non-URiM students, the most common themes were around specific clinical behaviors and skills. In contrast, for URiM students, the most common themes were personal qualities. This difference was not present in the grading committee narratives, which showed similar themes across URiM status. The largest number of comments for both groups addressed work ethic, personal qualities, self-regulated learning habits, and patient-centered communication. Grading committees used a commonly agreed-upon practice for determining honors-level clinical performance and test scores. However, they varied in size, use of other data sources, and process of review across clerkships. All relied heavily on group discussion and close reading of all individual assessments. These findings demonstrate the mitigating effect of a grading committee on clerkship grading bias both numerically and narratively and support the recommendations in the literature. 5 Despite variability by clerkship, all grading committees discussed grade assignments as a group and considered multiple data sources before assigning a grade or deciding on the final narrative summary. We hold that the process of group discussion itself may be the most salient feature that led the committees to mitigate grading bias for students underrepresented in medicine. Our findings strengthen calls for the widespread adoption of UME grading committees and suggest that they are efficacious despite variability. The authors would like to acknowledge the students whose performance forms the basis for this study, and the members of the multiple grading committees

Bonferroni correction · Curriculum · Educational measurement · Family medicine · Medical education · MEDLINE · Pedagogy · Political science · Statistics · Diversity and Career in Medicine · Innovations in Medical Education · Mathematics · Medical Education and Admissions · Medicine · Psychology

• A Narrative Study of Equity in Clinical Assessment Through the Antideficit Lens

  Arianne Teherani, Sandra Pérez et al.•Academic Medicine•2020

  Add to list

• Medical Education’s Wicked Problem

  Catherine R Lucey, Karen E Hauer et al.•Academic Medicine•2020

  Add to list