A Ranking Method for Evaluating Constructed Responses
Bibliographic Data
| ID | 20282255 |
|---|---|
| Authors | Yigal Attali (0000-0002-3907-247X, Educational Testing Service, Princeton, NJ, USA, corresponding author) |
| Year | 2014 |
| Volume | 74 |
| Issue | 5 |
| Pages | 795-808 |
| Publication date | 2014-10-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Educational and Psychological Measurement (JOURNAL) |
| Journal identifiers | ISSN: 0013-1644 • E-ISSN: 1552-3888 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0013164414527450 |
| OpenAlex | W1987799952 |
| Language | EN |
| Citations received | 4 |
| References cited | 11 |
This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and subsequent rankings by graders serve to update the prior scores of responses. Final response scores are determined by weighting the prior and ranking information. This approach allows for scaling comparative judgments on the basis of a single ranking, eliminates rater effects in scoring, and offers a conceptual framework for combining human and automated evaluation of constructed response tasks. To evaluate this approach, groups of graders evaluated responses to two tasks using either the ranking (with sets of 5 responses) or traditional rating approach. Results varied by task and the relative weighting of prior versus ranking information, but in general the ranking scores showed comparable generalizability (reliability) and validity coefficients
Generalizability theory · Machine learning · Rank (graph theory) · Ranking (information retrieval) · Rating scale · Reliability (semiconductor) · Set (abstract data type) · Statistics · Task (project management) · Weighting · Artificial Intelligence · Behavioral and Psychological Studies · Computer Science · Mathematics · Multi-Criteria Decision Making · Psychology · Software Engineering Research
Investigating the Influence of Training Difficulty on the Learning Outcomes of Medical Students
Rating procedures to evaluate generic critical online reasoning in an open Internet environment
From substitution to redefinition
Problematising the equivalence of the test results of performance-based critical thinking tests for undergraduate students
| Citations per year | 0.36 |
|---|---|
| Citation span | 2015 - 2026 (12) |
| Highly cited | No |