Feasibility and psychometric evaluation of large language model–generated radiology learning materials for board examination preparation

BACKGROUND: Large language models (LLMs) can generate structured educational content at scale, yet their role in postgraduate radiology training remains untested. METHODS: This pilot study evaluated the feasibility of deploying LLM-generated radiology board preparation materials, the psychometric properties of adapted survey instruments, and exploratory effect-size estimates for sample-size planning of a future multicentre trial. We conducted a prospective observational pilot among 41 radiology residents preparing for the 2026 Korean Radiology Board Examination. A governed seven-stage LLM pipeline generated 6,000 Anki flashcards, 833 infographic summaries, and presentation slides. Materials were distributed freely, and 24 completers self-selected into users (n = 15) and non-users (n = 9). Outcomes included enrollment yield, retention rate, voluntary adoption rate, internal consistency (Cronbach's alpha) of adapted cognitive load and satisfaction scales, and change scores in extraneous cognitive load (ECL) and self-efficacy (SE), compared using Mann-Whitney U tests with Cohen's d and 95% confidence intervals. RESULTS: Of 114 registered candidates, 44 were recruited (38.6%); 41 were enrolled (yield 93.2%); retention was 58.5% (24 of 41), below the prespecified 80% threshold; and voluntary adoption was 62.5%. The ECL scale showed acceptable reliability (alpha = 0.87 baseline, 0.80 post-examination). After reverse-coding one reverse-worded item (E3), the corrected three-item trust-in-AI alpha was 0.583 (bootstrap 95% CI 0.151 to 0.746), below the conventional 0.70 threshold. Between-group differences were non-significant for ECL (d = - 0.28; 95% CI - 1.50 to 0.59) and SE (d = 0.26; - 0.62 to 1.24). Among users, educational quality was rated 3.33 of 5.00, and 86.7% reported encountering no factual inaccuracies. A medium-effect scenario (d = 0.50) would require 160 participants. CONCLUSIONS: Retention (58.5%), the trust-scale alpha of 0.583, and the - 40 Net Promoter Score together indicate that the data-collection protocol, the AI-trust instrumentation, and the user-experience design all require modification before an efficacy trial is justified. Operational deployment was feasible at the levels of enrollment yield and voluntary adoption, but not retention. Adapted cognitive load and satisfaction scales reached acceptable reliability; the trust-in-AI and single-item self-efficacy outcomes did not meet conventional thresholds. Exploratory between-group point estimates were small and unstable across sensitivity strata; they are reported as inputs for sample-size planning and should not be interpreted as evidence for or against educational benefit.

Content validity · Educational measurement · Internal consistency · MEDLINE · Observational study · Psychometrics · Artificial Intelligence in Healthcare and Education · Innovations in Medical Education · Radiology practices and education