Reflective prompt engineering
A new strategy for automated short answer scoring in biology
Bibliographic Data
| ID | 21589504 |
|---|---|
| Authors | Moriah Ariely (0000-0001-6539-3544, Weizmann Institute of Science, corresponding author), Asaf Salman (0000-0001-5796-0158, Weizmann Institute of Science), Anat Yarden (0000-0002-3948-9400, Weizmann Institute of Science), Giora Alexandron (0000-0003-2676-6912, Weizmann Institute of Science) |
| Year | 2025 |
| Pages | 1-23 |
| Publication date | 2025-07-13 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | International Journal of Science Education, Part B (JOURNAL) |
| Journal identifiers | ISSN: 2154-8455 • E-ISSN: 2154-8463 |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/09500693.2025.2523571 |
| OpenAlex | W4412375575 |
| Language | EN |
| References cited | 33 |
The educational landscape is rapidly evolving, driven by Generative Artificial Intelligence (GenAI) advancements. As GenAI transforms educational practices, a key area of impact is developing automatic scoring systems to efficiently and accurately assess students’ problem-solving and explanation skills. This study introduces a novel prompt engineering strategy, Reflective Prompt Engineering (RPE), which applies iterative human-AI collaboration through discussion and reflection with a powerful LLM to enhance scoring performance. In RPE, human experts guide the AI by integrating its inferred criteria and language into subsequent prompts, enabling reflective alignment and improvement. We applied RPE to score open-ended biology items using analytic grading rubrics, evaluating its performance against two benchmarks: a BERT-based scoring model and multiple examples prompts with no discussion. Performance (accuracy and Cohen’s Kappa) was assessed in two scenarios: within-item and cross-items. RPE achieved excellent agreement with human experts (Cohen’s Kappa > 0.8) using only 40–60 examples and consistently outperformed the multiple examples strategy, achieving significantly higher Kappa values. Our findings highlight the potential of RPE to optimise AI alignment with human-designed tasks and open pathways for broader applications across diverse domains, advancing automated evaluation practices with enhanced precision and adaptability
Biology · Mathematics education · Science education · Advanced Text Analysis Techniques · Computer Science · Educational Assessment and Pedagogy · Psychology · Topic Modeling
An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers
Examining Science Education in ChatGPT
Pre-train, Prompt, and Predict
Using automated analysis to assess middle school students' competence with scientific argumentation
AI and formative assessment
Applying machine learning to automatically assess scientific models
Causal‐mechanical explanations in biology
Evaluation of construct-irrelevant variance yielded by machine and human scoring of a science teacher PCK constructed response assessment
The Power of Feedback
| Highly cited | No |
|---|