Agreement, calibration, and failure of three large language models as high-stakes multimodal ospe graders
A comparative psychometric analysis
Bibliographic Data
| ID | 21650342 |
|---|---|
| Authors | Shahid Akhtar Akhund (0000-0003-1820-1276, Alfaisal University), Sadia Qazi (0009-0009-9275-8022, Alfaisal University, corresponding author), Muhammad Atif Mazhar (0009-0007-1588-3909, Alfaisal University), Aftab Ahmed Shaikh (0009-0009-4236-5933, Alfaisal University), Eshal Atif (0009-0006-7010-6910, Alfaisal University), Adeen Shaikh (0009-0009-7497-8662, Alfaisal University), Hassan Shaibah (0000-0002-3575-6460, Alfaisal University), Mohammed Alged Elsheikh Musa (0009-0005-9890-5469, Alfaisal University), Mohd Khirulnizam Musa (Alfaisal University), Sabiya N Qazi (0009-0004-4054-7809, Alfaisal University), Mateen A Khan (0000-0002-2300-4595, Alfaisal University), Akef Obeidat (0000-0002-4666-5118, Alfaisal University) |
| Year | 2026 |
| Volume | 31 |
| Issue | 1 |
| Pages | 2684837-2684837 |
| Publication date | 2026-12-31 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Medical Education Online (JOURNAL) |
| Journal identifiers | ISSN: 1087-2981 • E-ISSN: 1087-2981 |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/10872981.2026.2684837 |
| PMID | 42421433 |
| OpenAlex | W7167836367 |
| Language | EN |
| References cited | 63 |
Background Medical education faces increasing demand for scalable grading solutions. Large language models have been proposed as automated graders for high-stakes assessments, but evidence for their reliability in multimodal practical examinations remains limited.Methods This retrospective inter-rater reliability study compared three LLMs: ChatGPT-4o, Gemini 2.5 Flash, and Claude 3.5 Haiku with original human reference scores on 19 integrated anatomy, histology, and physiology objective structured practical examination items completed by 309 pre-medical students. Human reference scores were assigned during live summative grading by two content experts who divided items, with each item marked by one expert across all students; no human inter-rater reliability estimate was available. Items required simultaneous image interpretation and short-answer responses. Models were evaluated using zero-shot prompting reflecting deployment-realistic conditions. Agreement was assessed using Spearman's ρ, Cohen's κ, intraclass correlation coefficients, and Bland-Altman analysis. Item-level gap analysis compared student success rates with LLM performance across items.Results Rank-order correlations were strong across all models (ρ = 0.784–0.921), but categorical agreement diverged substantially. Pass/fail agreement ranged from 52.1% (ChatGPT; κ = 0.067, slight) to 84.1% (Claude; κ = 0.680, substantial). Bland-Altman analysis showed inconsistent systematic bias: Claude over-scored by +2.5 points, Gemini under-scored by −5.0 points, and ChatGPT by −10.0 points. Item-level gap analysis highlighted three recurring divergence patterns: non-standard histological staining, three-dimensional spatial reasoning from two-dimensional images, and semantic inflexibility in short-answer evaluation. The most extreme case was a pelvic three-dimensional model item on which 93.3% of students succeeded by human grading but all three LLMs assigned a mean score of zero.Conclusions Strong rank-order correlation alone does not support LLM grading for categorical decisions in high-stakes assessment contexts. LLMs may serve as assessment assistants for first-pass ranking and discrepancy flagging, but item-level human review of flagged discordances, error-profile monitoring, and preserved human authority over pass/fail decisions are required before high-stakes deployment
Categorical variable · Correlation · Human reliability · Inter-rater reliability · Intraclass correlation · Summative assessment · Artificial Intelligence in Healthcare and Education · Clinical Reasoning and Diagnostic Skills · Innovations in Medical Education
Propriedades psicométricas na avaliação de instrumentos
Formative Assessment
Exploring the potential of using an AI language model for automated essay scoring
High agreement but low kappa
Performance of ChatGPT on USMLE
High agreement but low Kappa
On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha
Artificial intelligence in higher education
A Psychometric Toolbox for Testing Validity and Reliability
Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.
The Measurement of Observer Agreement for Categorical Data
A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research
Making sense of Cronbach's alpha
A systematic review of large language models and their implications in medical education
Gaps, guesswork, and ghosts lurking in technology integration
A Coefficient of Agreement for Nominal Scales
Assessment criteria in a large-scale writing test
Assessment rubrics
ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review
The Impact of Timely Formative Feedback on University Student Motivation
Analysis of college students' attitudes toward the use of ChatGPT in their academic activities
Evaluating large language models as graders of medical short answer questions
Systematic review of research on artificial intelligence applications in higher education – where are the educators
Bias and Fairness in Large Language Models
The development of student feedback literacy
| Highly cited | No |
|---|