A comparative analytical study of LLM-based automated grading in programming education

The integration of Large Language Models (LLMs) into programming education has opened new avenues for automated assessment, extending beyond traditional test-case-based evaluation. Despite the rapid proliferation of LLM-based grading research, the literature remains methodologically fragmented, making it difficult to establish robust evaluation practices and emerging design principles. Existing reviews provide broad conceptual perspectives on AI in education and automated assessment, but few offer a grading-specific, taxonomy-driven analytical framework. As research in this area rapidly expands across heterogeneous educational contexts, a structured comparative synthesis is both feasible and increasingly necessary. This paper presents a comparative analytical study of LLM-based automated grading systems in computer science education. Following a PRISMA-guided methodology, we systematically analyze a corpus of 17 peer-reviewed studies (2023–2026), including high-impact journals and leading computing education conferences. We develop an inductively derived taxonomy that characterizes existing systems across five dimensions: grading paradigm, role of the LLM, human involvement, evaluation strategy, and generalization scope. The reviewed literature is explicitly stratified by level of evidence to distinguish exploratory pilots from more mature empirical evaluations. Our analysis indicates that fully automated direct-grading approaches still dominate the current literature, while interest in hybrid and human-supervised configurations—in which LLMs increasingly function as triaging or assistance mechanisms—is clearly growing but not yet prevalent. Current studies rely predominantly on agreement-based evaluation metrics, particularly correlation-based comparisons with human graders, whereas cross-rubric validation and pedagogically grounded assessment measures remain limited. We adopt a structured review approach because the principal challenge in this emerging field lies not only in generating additional experiments, but in establishing a coherent framework through which heterogeneous findings can be interpreted and compared. Overall, this review structures the current exploratory landscape and offers actionable insights toward more robust, transparent, and educationally grounded AI-assisted grading frameworks

Automated method · Automation · Computer programming · Intelligent Tutoring Systems and Adaptive Learning · Online Learning and Analytics · Teaching and Learning Programming

• Generative AI tools and assessment

  Open Access•Benjamin Luke Moorhouse, Marie Alina Yeo et al.•Computers and Education Open•2023

  Add to list