Exploring the Potential of Conversational AI for Assessing Second Language Oral Proficiency

Bibliographic Data

ID7169351
AuthorsYasin Karatay (0000-0001-8297-5626, Cambridge University Press & Assessment Cambridge UK, corresponding author), Jing Xu (0000-0001-7414-9383, Cambridge University Press & Assessment Cambridge UK)
Year2025
Volume59
IssueS1
Publication date2025-12-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueTESOL Quarterly (JOURNAL)
Journal identifiersISSN: 0039-8322 • E-ISSN: 1545-7249
PublisherWiley (PUBLISHER • GB)
DOI10.1002/tesq.70003
OpenAlexW4411616805
LanguageEN
Citations received8
References cited40

Interactional Competence (IC) is an important subcomponent of oral proficiency, but many computer‐mediated oral English assessments fall short in assessing this construct mainly due to technological limitations. Spoken Dialogue Systems (SDSs) have shown promise in assessing L2 oral communication, yet further investigation is needed on their effectiveness in eliciting IC features in high‐stakes assessment contexts. This study is unique in that it analyzed both test takers' and an SDS's spoken discourse in human–computer interactions. Using an SDS to simulate an examiner in an IELTS Speaking task, the study explored how well the system mimicked human–human interaction and elicited IC features from test takers, focusing on the IC features documented in prior research. Thirty participants completed the SDS‐mediated test, with their performances rated by two trained raters. Semi‐structured interviews with 10 test takers were conducted following the assessment. The findings revealed that the SDS successfully elicited key IC features, which helped distinguish test takers at different proficiency levels, with reliable scoring across raters. Most test takers found the SDS competent, though some noted its limitations in nonverbal communication and conversational flow. These results suggest that SDSs have potential in oral proficiency assessments and provide valuable insights for refining SDS design to ensure reliable and valid assessments

Language proficiency · Linguistics · Mathematics education · Computer Science · Natural Language Processing Techniques · Philosophy · Psychology · Speech and dialogue systems · Subtitles and Audiovisual Media

Tools

Open DOIOpen Access