Reliable but not rigorous
Evaluating ChatGPT's reliability, validity, and bias in automated academic grading
Bibliographic Data
| ID | 22206166 |
|---|---|
| Authors | Raed Awashreh (0000-0002-2252-0299, United Arab Emirates University), Hisham Al Ghunaimi (0000-0002-5494-2242, College of Applied Sciences, Nizwa), Said AlGhenaimi (0000-0002-6117-2645, College of Applied Sciences, Nizwa, corresponding author) |
| Year | 2026 |
| Volume | 13 |
| Pages | 102788 |
| Publication date | 2026-06-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Social Sciences & Humanities Open (JOURNAL) |
| Journal identifiers | ISSN: 2590-2911 • E-ISSN: 2590-2911 |
| Publisher | Elsevier BV (PUBLISHER) |
| DOI | 10.1016/j.ssaho.2026.102788 |
| OpenAlex | W7154481171 |
| Language | EN |
| References cited | 18 |
Purpose This study investigates whether ChatGPT can function as a reliable and fair evaluator of student academic work in higher education, with particular attention to grading reliability, evaluative validity, and potential bias across social science disciplines. Design/methodology/approach Using an experimental comparative design, the study analysed 61 undergraduate assignments (Political Science reflection papers and Public Administration reports) graded independently by ChatGPT and human instructors using identical analytic rubrics. Quantitative analyses (descriptive statistics, correlations, t-tests, and effect sizes) were combined with qualitative examination of AI-generated feedback to assess consistency, discrimination, and alignment with instructor judgment. Findings Results reveal that ChatGPT demonstrates high procedural reliability and strong rank-order correlation with instructors but systematically inflates grades and compresses score distributions. The AI disproportionately rewards structural coherence, linguistic fluency, and formatting while showing limited sensitivity to analytical depth, theoretical engagement, and originality, indicating a clear leniency bias and reduced evaluative validity. Originality/value The study empirically advances understanding of the reliability–validity paradox in AI-assisted assessment by demonstrating that grading consistency does not equate to pedagogical fairness, positioning leniency bias as a structural outcome of large language model design rather than random error. Practical implications ChatGPT is best suited for formative feedback and diagnostic support, not summative grading. Institutions should adopt calibrated human–AI hybrid assessment models and clear governance frameworks. Social implications Unregulated AI grading risks normalizing grade inflation and misrepresenting academic achievement, potentially undermining trust in educational credentials. Limitations The study is limited to two disciplines, one institution, and a single AI model, restricting generalizability
Automated method · Automation · Quality assurance · Academic integrity and plagiarism · AI in Service Interactions · Artificial Intelligence in Healthcare and Education
Interpreting Black-Box Models
Retracted Article
A systematic review of AI-powered collaborative learning in higher education
ChatGPT in the higher education
The influence of ChatGPT on student engagement
Exploring the use of artificial intelligence (AI) in the delivery of effective feedback
Challenging cheating in higher education
| Citation velocity | historical |
|---|---|
| Highly cited | No |