The answer may vary

Large language model response patterns challenge their use in test item analysis

Bibliographic Data

ID21447558
AuthorsLauren K Buhl (0000-0002-4305-2957, Department of Anesthesiology, Dartmouth Hitchcock Medical Center, corresponding author)
Year2025
Volume47
Issue11
Pages1761-1766
Publication date2025-11-02
Open AccessNo
TypeARTICLE
VenueMedical Teacher (JOURNAL)
Journal identifiersISSN: 0142-159X • E-ISSN: 1466-187X
PublisherInforma UK Limited (PUBLISHER • GB)
DOI10.1080/0142159x.2025.2497891
PMID40319392
OpenAlexW4410064563
LanguageEN
References cited20

INTRODUCTION: The validation of multiple-choice question (MCQ)-based assessments typically requires administration to a test population, which is resource-intensive and practically demanding. Large language models (LLMs) are a promising tool to aid in many aspects of assessment development, including the challenge of determining the psychometric properties of test items. This study investigated whether LLMs could predict the difficulty and point biserial indices of MCQs, potentially alleviating the need for preliminary analysis in a test population. METHODS: Sixty MCQs developed by subject matter experts in anesthesiology were presented one hundred times each to five different LLMs (ChatGPT-4o, o1-preview, Claude 3.5 Sonnet, Grok-2, and Llama 3.2) and to clinical fellows. Response patterns were analyzed, and difficulty indices (proportion of correct responses) and point biserial indices (item-test score correlation) were calculated. Spearman correlation coefficients were used to compare difficulty and point biserial indices between the LLMs and fellows. RESULTS: Marked differences in response patterns were observed among LLMs: ChatGPT-4o, o1-preview, and Grok-2 showed variable responses across trials, while Claude 3.5 Sonnet and Llama 3.2 gave consistent responses. The LLMs outperformed fellows with mean scores of 58% to 85% compared to 57% for the fellows. Three LLMs showed a weak correlation with fellow difficulty indices (r = 0.28-0.29), while the two highest scoring models showed no correlation. No LLM predicted the point biserial indices. DISCUSSION: These findings suggest LLMs have limited utility in predicting MCQ performance metrics. Notably, higher-scoring models showed less correlation with human performance, suggesting that as models become more powerful, their ability to predict human performance may decrease. Understanding the consistency of an LLM's response pattern is critical for both research methodology and practical applications in test development. Future work should focus on leveraging the language-processing capabilities of LLMs for overall assessment optimization (e.g., inter-item correlation) rather than predicting item characteristics

Item response theory · Natural language processing · Psychometrics · Test (biology) · Artificial Intelligence in Healthcare and Education · Clinical Psychology · Computer Science · Medical Education and Admissions · Psychology · Psychometric Methodologies and Testing

Tools

Open DOI