Consistency over accuracy
Run-to-run stability of contemporary large language models on Turkish curriculum-aligned theoretical anatomy multiple-choice questions
Bibliographic Data
| ID | 22423966 |
|---|---|
| Authors | Ömer Alperen Gürses (0000-0001-6564-7428, Ahi Evran University, corresponding author), İsmail Ceylan (0000-0002-6465-0243, Ahi Evran University) |
| Year | 2026 |
| Volume | 26 |
| Issue | 1 |
| Pages | 285-285 |
| Publication date | 2026-01-23 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | BMC Medical Education (JOURNAL) |
| Journal identifiers | ISSN: 1472-6920 • E-ISSN: 1472-6920 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.1186/s12909-026-08656-3 |
| PMID | 41572253 |
| OpenAlex | W7125521684 |
| Language | EN |
| References cited | 24 |
BACKGROUND: Stability across repeated administrations is essential for educational use of large language models (LLMs), yet it is rarely quantified in non-English, curriculum-aligned anatomy contexts. METHODS: Eleven contemporary LLMs answered 100 Turkish, faculty-authored, curriculum-aligned anatomy multiple-choice questions from AYDEP, targeting the undergraduate Physiotherapy and Rehabilitation anatomy curriculum in three independent runs (≥ 12-hour intervals). Testing used developers' web interfaces in Turkey with browsing disabled and default generation settings (August-September 2025). Performance was summarized with a stability-aware 0-3 item score (number of correct responses across three runs) and predefined response-consistency classes. RESULTS: A subset of models achieved near-ceiling totals with high run-to-run stability, whereas others showed greater session-to-session variability (i.e., changes in the selected option across independent runs initiated as separate sessions under identical inputs). Several nominal differences among higher-performing systems did not remain significant after multiplicity control. Within-family updates produced selective, not universal, gains. Many models exhibited medians of 3 (IQR 3-3) on the 0-3 scale (ceiling effects), and lower means were accompanied by larger dispersion. Consistency profiles provided information beyond mean accuracy by distinguishing reliably correct from volatile behavior. In addition, we observed "consistent & wrong" patterns on a subset of items, where the same incorrect option was repeatedly selected across runs. CONCLUSION: In Turkish, curriculum-aligned anatomy items, contemporary LLMs can be both accurate and stable, but single-trial accuracy can mask volatility and stable systematic errors. Adoption decisions should prioritize stability-aware appraisal (including consistent-correct and consistent-wrong rates), with local validation on institutional item banks and periodic re-evaluation as models evolve. Extending this framework to multimodal anatomy and constructed-response tasks will further inform trustworthy, learner-facing use.
German · Turkish · Anatomy and Medical Technology · Artificial Intelligence in Healthcare and Education · Text Readability and Simplification
How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Performance of ChatGPT on USMLE
Bi̇r Öğrenme Yöneti̇m Si̇stemi̇ Olarak Aydep’i̇n Si̇stem Kabul Edi̇lebi̇li̇rli̇k Modeli̇ne Göre İncelenmesi̇
Twelve tips on creating and using custom GPTs to enhance health professions education
| Highly cited | No |
|---|