Using Guttman errors to explore rater fit in rater-mediated performance assessments
Bibliographic Data
| ID | 4563701 |
|---|---|
| Authors | Stefanie A Wind (0000-0002-1599-375X, University of Alabama, corresponding author) |
| Year | 2018 |
| Volume | 11 |
| Issue | 3 |
| Pages | 205979911881439 |
| Publication date | 2018-09-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Methodological Innovations (JOURNAL) |
| Journal identifiers | ISSN: 2059-7991 • E-ISSN: 2059-7991 |
| Publisher | SAGE Publications Inc (PUBLISHER) |
| DOI | 10.1177/2059799118814396 |
| OpenAlex | W2902150877 |
| Language | EN |
| Citations received | 1 |
| References cited | 29 |
Model-data fit indices for raters provide insight into the degree to which raters demonstrate psychometric properties defined as useful within a measurement framework. Fit statistics for raters are particularly relevant within frameworks based on invariant measurement, such as Rasch measurement theory and Mokken scale analysis. A simple approach to examining invariance is to examine assessment data for evidence of Guttman errors. I used real and simulated data to illustrate and explore a nonparametric procedure for evaluating rater errors based on Guttman errors and to examine the alignment between Guttman errors and other indices of rater fit. The results suggested that researchers and practitioners can use summaries of Guttman errors to identify raters who exhibit misfit. Furthermore, results from the comparisons between summaries of Guttman errors and parametric fit statistics suggested that both approaches detect similar problematic measurement characteristics. Specifically, raters who exhibit many Guttman errors tended to have higher-than-expected Outfit MSE statistics and lower-than-expected estimated slope statistics. I discuss implications of these results as they relate to research and practice for rater-mediated assessments
Econometrics · Guttman scale · Item response theory · Level of measurement · Parametric statistics · Psychometrics · Rasch model · Rating scale · Statistics · Computer Science · Mathematics · Multi-Criteria Decision Making · Psychology · Psychometric Methodologies and Testing · Reliability and Agreement in Measurement
Introduction to Nonparametric Item Response Theory
A Theory and Procedure of Scale Analysis
Applying the Rasch Model
Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory
Using the Many-Faceted Rasch Model to Evaluate Standard Setting Judgments
Adjacent-Categories Mokken Models for Rater-Mediated Assessments
A systematic review of methods for evaluating rating quality in language assessment
Cross-National Measure of Fear-Based Xenophobia
A rating formulation for ordered response categories
Standard Errors and Confidence Intervals for Scalability Coefficients in Mokken Scale Analysis Using Marginal Models
Examining the Internal Structure Evidence for the Performance Assessment for California Teachers
Weighted kappa
Using Mokken methods to develop robust cross-national scales
| Unique citing works | 1 |
|---|---|
| Citations per year | 0,14 |
| Citation span | 2019 - 2019 (1) |
| Citation velocity | historical |
| Highly cited | No |
| Citation types | Neutral: 1 |