Word Predictability as a Measure of Second Language Proficiency

Bibliographic Data

ID22411262
AuthorsLangdon Holmes (0000-0003-4338-4609, Vanderbilt University), Scott Crossley (Vanderbilt University), Wesley Morris (0000-0001-6316-6479, Vanderbilt University), Joon Suh Choi (Vanderbilt University)
Year2026
Publication date2026-07-09
Open AccessYes
TypeARTICLE
VenueLanguage Learning (JOURNAL)
Journal identifiersISSN: 0023-8333 • E-ISSN: 1467-9922
PublisherWiley (PUBLISHER • GB)
DOI10.1111/lang.70037
OpenAlexW7167827078
LanguageEN
References cited74

This study introduces predictability BERT , a novel metric for assessing second language (L2) proficiency based on the predictability of word choices in learner language production. Using BERT (Devlin et al., 2019), we calculated the conditional probability of each word in a text given its surrounding context. We evaluated predictability BERT on two datasets: the Lexical Proficiency Corpus ( N = 480), containing analytic ratings of lexical proficiency, and TOEFL 11 ( N = 11,000; Blanchard et al., 2013), containing standardized language proficiency scores. Results show that predictability BERT correlated with ratings of collocation accuracy ( r = .80) and lexical proficiency ( r = .73). In a multilevel model, predictability BERT was the strongest predictor of language proficiency compared to conventional measures of lexical and phraseological sophistication, explaining 59% of variance in TOEFL scores. These findings suggest high‐proficiency L2 learners make more predictable word choices, supporting usage‐based theories emphasizing the role of statistical learning in L2 development.

Collocation (remote sensing) · Language proficiency · Lexical density · Metric (unit) · Predictability · Second language · Test of English as a Foreign Language · Word (group theory) · EFL/ESL Teaching and Learning · Language Development and Disorders · Second Language Acquisition and Learning

Tools

Open DOIOpen Access