Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace
Findings From One Institution
Datos Bibliográficos
PURPOSE: Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships. METHOD: During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability. RESULTS: A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70). CONCLUSIONS: The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment
Curriculum · Educational measurement · Formative assessment · Generalizability theory · Medical education · Pedagogy · Summative assessment · Diversity and Career in Medicine · Innovations in Medical Education · Medicine · Psychology · Surgical Simulation and Training
A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace
Clerkship Assessment Reliability Based on Assessor Type and Assessor-Student Contact Frequency
How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study
Competency-based medical education across the continuum
‘Next steps are…’
Entrustment Decision Making in Clinical Training
Workplace-based assessment as an educational tool
Validity
Entrustability of professional activities and competency‐based training
Curriculum development for the workplace using Entrustable Professional Activities (EPAs)
The Case for Use of Entrustable Professional Activities in Undergraduate Medical Education
A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments
Closing the Gap Between Preclinical and Clinical Training
The Association of American Medical Colleges
| Obras citantes distintas | 5 |
|---|---|
| Citas por año | 1,25 |
| Intervalo de citas | 2022 - 2025 (4) |
| Velocidad de citación | recent |
| Altamente citado | No |
| Tipos de cita | Neutras: 5 |