Accuracy and fairness of generative AI in automated essay scoring
Comparing GPT-4o, feature-based models, and human raters
Bibliographic Data
| ID | 22420538 |
|---|---|
| Authors | Huang Yue (0000-0003-2175-9852, corresponding author), Corey Palermo (0000-0003-1921-5127), Joshua Wilson (0000-0002-7192-3510, University of Delaware) |
| Year | 2026 |
| Volume | 69 |
| Pages | 101047 |
| Publication date | 2026-07-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Assessing Writing (JOURNAL) |
| Journal identifiers | ISSN: 1075-2935 • E-ISSN: 1873-5916 |
| Publisher | Elsevier BV (PUBLISHER) |
| DOI | 10.1016/j.asw.2026.101047 |
| OpenAlex | W7153310222 |
| Language | EN |
| Citations received | 1 |
| References cited | 18 |
This study examines the accuracy and fairness of generative AI–based automated essay scoring (AES) in a developmentally emergent writing context, compared with a feature-based scoring engine and human ratings. Despite growing interest in generative AI for AES, limited research has examined model performance in early-grade writing, where linguistic and transcription skills are still developing. Using 1768 essays from Grades 3–4, we evaluated GPT-4o under three prompting strategies and a fine-tuned configuration. Human–machine agreement was assessed against human–human benchmarks at both holistic and trait levels using quadratic weighted kappa, exact and adjacent agreement rates, and classification metrics. Fairness was examined using overall score accuracy, overall score difference, and conditional score difference. Results indicate that carefully designed prompting improves GPT-4o’s scoring accuracy, approaching human and feature-based AES performance, while the fine-tuned model achieved the closest alignment to human–human agreement. Trait-level analyses revealed complementary strengths: the best-performing prompting approach aligned more closely on higher-level traits (e.g., organization and style), whereas the feature-based engine showed stronger alignment on development of ideas and all surface-level traits. Fairness analyses indicated that no model was entirely free of subgroup differences. These findings suggest that generative AI–based AES can approximate human scoring in early-grade writing when appropriately calibrated.
Automation · Generative grammar · Generative model · Online assessment · Authorship Attribution and Profiling · Sentiment Analysis and Opinion Mining · Topic Modeling
Exploring the potential of using an AI language model for automated essay scoring
Validating the Interpretations and Uses of Test Scores
Large Language Model and Traditional Machine Learning Scoring of Evolutionary Explanations
Toward an Understanding of the Distinct Nature of L2 Writing
An argument-based approach to validity
| Unique citing works | 1 |
|---|---|
| Citations per year | 1 |
| Citation span | 2026 - 2026 (1) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 1 |