ChatGPT4o as an AI Peer Assessor in EFL Speaking Classrooms
Examining Scoring Reliability and Feedback Effectiveness
Bibliographic Data
| ID | 16911864 |
|---|---|
| Authors | Junfei Li (0000-0002-5184-1737, Shanghai Dianji University), Jinyan Huang (0000-0003-1641-6580, Jiangsu University, corresponding author), Thomas Sheeran, Thomas J Sheeran (Niagara University) |
| Year | 2025 |
| Volume | 15 |
| Issue | 3 |
| Publication date | 2025-01-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | SAGE Open (JOURNAL) |
| Journal identifiers | ISSN: 2158-2440 • E-ISSN: 2158-2440 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/21582440251369938 |
| OpenAlex | W4413841833 |
| Language | EN |
| Citations received | 1 |
| References cited | 48 |
This study investigated the role of ChatGPT4o as an AI peer assessor in English-as-a-foreign-language (EFL) speaking classrooms, with a focus on its scoring reliability and the effectiveness of its feedback. The research involved 40 first-year English major students from two parallel classes at a Chinese university. Twenty from one class served as speech sample providers; and the other 20 served as human peer assessors. In addition, ChatGPT4o served as an AI peer assessor. The study employed univariate and multivariate generalizability (G-) theory to compare the consistency and reliability of holistic and analytic scoring between ChatGPT4o and human peer assessors. The results demonstrated that ChatGPT4o provided significantly more consistent and reliable scores across domains such as accuracy, fluency, and complexity. Moreover, ChatGPT4o delivered more comprehensive and effective feedback, offering clear guidance for improvement. However, interviews with human peer assessors revealed concerns about ChatGPT4o’s limitations in capturing the subtle aspects of spoken language, such as emotion and cultural context, and the potential over-reliance on ChatGPT4o in EFL assessments. The findings suggested that while ChatGPT4o as an AI peer assessor can enhance the reliability and quality of peer assessments, its adoption should be carefully managed to complement, rather than replace, human judgment, ensuring a balanced approach in EFL speaking classrooms
Higher education · Mathematics education · Peer Assessment · Peer evaluation · Peer feedback · Artificial Intelligence in Healthcare and Education · Intelligent Tutoring Systems and Adaptive Learning · Online Learning and Analytics · Psychology
Generalizability Theory
Enhancing academic writing skills and motivation
Peer feedback
Collaborating with ChatGPT in argumentative writing classrooms
Using chatbots to scaffold EFL students’ argumentative writing
To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing
Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement
Impact of ChatGPT on learners in a L2 writing practicum
Using ChatGPT for second language writing
ChatGPT for good? On opportunities and challenges of large language models for education
Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings
AI technologies in the ESL/EFL writing classroom
Using automated corrective feedback tools in language learning
Chatbot-based learning of logical fallacies in EFL writing
The effects of peer assessment on learner autonomy
Evaluating peer feedback as a reliable and valid complementary aid to teacher feedback in EFL writing classrooms
The quality assurance of a national English writing assessment
The reliability, effectiveness, and benefits of peer assessment in college EFL speaking classrooms
The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability
An investigation of the validity of a speaking assessment for adolescent English language learners
Automated scoring and feedback systems
Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms
Does peer assessment promote student learning? A meta-analysis
Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing
Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices
Assessing Second Language Speaking Proficiency
From traditional to alternative feedback
Chinese pre-service English teachers’ beliefs about English as an international language (EIL)
The Power of Feedback
Developing sustainable feedback practices
| Citations per year | 1 |
|---|---|
| Citation span | 2026 - 2026 (1) |
| Highly cited | No |