The reliability of setting grade boundaries using comparative judgement
Bibliographic Data
| ID | 11482502 |
|---|---|
| Authors | Tom Benton (0000-0003-4275-2684, Cambridge University Press, corresponding author), Gill Elliott (0000-0002-2369-4055, Cambridge University Press) |
| Year | 2016 |
| Volume | 31 |
| Issue | 3 |
| Pages | 352-376 |
| Publication date | 2016-05-26 |
| Open Access | No |
| Type | ARTICLE |
| Venue | Research Papers in Education (JOURNAL) |
| Journal identifiers | ISSN: 0267-1522 • E-ISSN: 1470-1146 |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/02671522.2015.1027723 |
| OpenAlex | W2008316232 |
| Language | EN |
| Citations received | 3 |
| References cited | 15 |
In recent years the use of expert judgement to set and maintain examination standards has been increasingly criticised in favour of approaches based on statistical modelling. This paper reviews existing research on this controversy and attempts to unify the evidence within a framework where expertise is utilised in the form of comparative judgement. Initially, the paper introduces a mathematical model for the way in which comparative judgement may operate. Data from existing studies of comparative judgement are then used to estimate suitable parameters for this model. Having derived a working mathematical model for the operation of expert judgement, the paper will then demonstrate that this model provides results that are broadly consistent with existing research, including both studies that are critical and studies that are supportive of the use of expert judgement. The model will then be applied to examine the required scale and design of expert-driven approaches to standard maintaining in order for these to have any chance of being as reliable as the currently favoured, purely statistical approaches. The paper also discusses the minimum conditions required for this approach to designing expert-driven approaches to be appropriate
Judgement · Management science · Operations research · Order (exchange · Political science · Reliability (semiconductor · Scale (ratio · Set (abstract data type · Statistical model · Computer Science · Engineering · Law · Student Assessment and Feedback · Artificial Intelligence
Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing
Exploring the practical feasibility of adaptive comparative judgment as a summative assessment method
Consistency Analysis of Assessment Boards in University Entrance Examinations in Spain
Test Equating, Scaling, and Linking
A law of comparative judgment.
Rank Analysis of Incomplete Block Designs
Setting and maintaining GCSE and GCE grading standards
Using modern test theory to maintain standards in public qualifications in England
Would the real gold standard please step forward
Evaluating the rank‐ordering method for standard maintaining
The Effects of Consistency of Performance on A Level Examiners' Judgements of Standards
Grade Awarding Judgements in Differentiated Examinations
Logit Models for Sets of Ranked Items
| Citations per year | 1 |
|---|---|
| Citation span | 2023 - 2025 (3) |
| Highly cited | No |