How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory Study
Bibliographic Data
| ID | 20286293 |
|---|---|
| Authors | Gregory M Hurtz (0000-0002-5061-9202), Norman R Hertz (Department of Consumer Affairs, Sacramento, California) |
| Year | 1999 |
| Volume | 59 |
| Issue | 6 |
| Pages | 885-897 |
| Publication date | 1999-12-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Educational and Psychological Measurement (JOURNAL) |
| Journal identifiers | ISSN: 0013-1644 • E-ISSN: 1552-3888 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/00131649921970233 |
| OpenAlex | W2078578716 |
| Language | EN |
| Citations received | 1 |
| References cited | 19 |
Although the Angoff method is perhaps the most highly regarded and widely used judgmental method for setting criterion-referenced cutoff scores, little research has been conducted to determine the optimal number of raters to employ for obtaining stable cutoff scores. In the present study, Angoff ratings obtained from eight different occupational licensing examinations were evaluated via generalizability theory to estimate the optimal number of raters. Results indicated that approximately 10 to 15 raters is an optimal target range, although fewer raters may sometimes be sufficient. It is recommended that 10 to 15 raters be sampled as a general rule, followed by analyses to determine the optimal number of raters for examinations in a particular profession using a particular procedural variant of the Angoff method
Cutoff · Developmental psychology · Generalizability theory · Inter-rater reliability · Range (aeronautics) · Rating scale · Statistics · Mathematics · Multi-Criteria Decision Making · Psychology · Psychometric Methodologies and Testing · Risk and Safety Analysis · Social Psychology
| Citations per year | 0.04 |
|---|---|
| Citation span | 2003 - 2003 (1) |
| Highly cited | No |