Assessing the performance of ChatGPT in medical ethical decision-making

A comparative study with USMLE-based scenarios

Bibliographic Data

ID21515191
AuthorsAli A Khan (0000-0002-4250-8828, Warren Alpert Medical School, Brown University, Providence, Rhode Island, USA, corresponding author), Ali R Khan (0000-0002-0760-8647, The University of Texas Medical Branch at Galveston, Galveston, Texas, USA, corresponding author), Saminah Munshi (Warren Alpert Medical School, Brown University, Providence, Rhode Island, USA), Hari Dandapani (0000-0002-4428-7838, Warren Alpert Medical School, Brown University, Providence, Rhode Island, USA), Mohamed Jimale (0000-0002-4998-1482, The University of Texas Medical Branch at Galveston, Galveston, Texas, USA), Franck Mbuntcha Bogni (Brown University), Hussain Khawaja (0000-0001-7134-6701, Warren Alpert Medical School, Brown University, Providence, Rhode Island, USA)
Year2025
Volume51
Issue10
Pages693-699
Publication date2025-10-01
Open AccessNo
TypeARTICLE
VenueJournal of Medical Ethics (JOURNAL)
Journal identifiersISSN: 0306-6800 • E-ISSN: 1473-4257
PublisherBMJ (PUBLISHER • GB)
DOI10.1136/jme-2024-110240
PMID39863417
OpenAlexW4406820257
LanguageEN
Citations received2
References cited16

INTRODUCTION: The integration of artificial intelligence (AI) into healthcare introduces innovative possibilities but raises ethical, legal and professional concerns. Assessing the performance of AI in core components of the United States Medical Licensing Examination (USMLE), such as communication skills, ethics, empathy and professionalism, is crucial. This study evaluates how well ChatGPT versions 3.5 and 4.0 handle complex medical scenarios using USMLE-Rx, AMBOSS and UWorld question banks, aiming to understand its ability to navigate patient interactions according to medical ethics and standards. METHODS: We compiled 273 questions from AMBOSS, USMLE-Rx and UWorld, focusing on communication, social sciences, healthcare policy and ethics. GPT-3.5 and GPT-4 were tasked with answering and justifying their choices in new chat sessions to minimise model interference. Responses were compared against question bank rationales and average student performance to evaluate AI effectiveness in medical ethical decision-making. RESULTS: GPT-3.5 answered 38.9% correctly in AMBOSS, 54.1% in USMLE-Rx and 57.4% in UWorld, with rationale accuracy rates of 83.3%, 90.0% and 87.0%, respectively. GPT-4 answered 75.9% correctly in AMBOSS, 64.9% in USMLE-Rx and 79.6% in UWorld, with rationale accuracy rates of 85.4%, 88.9%, and 98.8%, respectively. Both versions generally scored below average student performance, except GPT-4 in UWorld. CONCLUSION: ChatGPT, particularly version 4.0, shows potential in navigating ethical and interpersonal medical scenarios. However, human reasoning currently surpasses AI in average performance. Continued development and training of AI systems can enhance proficiency in these critical healthcare aspects

Empathy · Interpersonal communication · Medical education · Artificial Intelligence in Healthcare and Education · Computer Science · Ethics and Social Impacts of AI · Explainable Artificial Intelligence (XAI · Medicine · Psychology · Social Psychology

Tools

Open DOI