Exploring Within-Rater Category Ordering
A Simulation Study Using Adjacent-Categories Mokken Scale Analysis
Bibliographic Data
| ID | 20281189 |
|---|---|
| Authors | Stefanie A Wind (0000-0002-1599-375X, The University of Alabama, Tuscaloosa, AL, USA, corresponding author), Randall E Schumacker (The University of Alabama, Tuscaloosa, AL, USA) |
| Year | 2018 |
| Volume | 78 |
| Issue | 5 |
| Pages | 887-904 |
| Publication date | 2018-10-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Educational and Psychological Measurement (JOURNAL) |
| Journal identifiers | ISSN: 0013-1644 • E-ISSN: 1552-3888 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0013164417724841 |
| PMID | 32655174 |
| OpenAlex | W2743090876 |
| Language | EN |
| Citations received | 1 |
| References cited | 16 |
The interpretation of ratings from educational performance assessments assumes that rating scale categories are ordered as expected (i.e., higher ratings correspond to higher levels of judged student achievement). However, this assumption must be verified empirically using measurement models that do not impose ordering constraints on the rating scale category thresholds, such as item response theory models based on adjacent-categories probabilities. This study considers the application of an adjacent-categories formulation of polytomous Mokken scale analysis (ac-MSA) models as a method for evaluating the degree to which rating scale categories are ordered as expected for individual raters in performance assessments. Using simulated data, this study builds on the preliminary application of ac-MSA models to rater-mediated performance assessments, in which a real data analysis suggested that these models can be used to identify disordered rating scale categories. The results suggested that ac-MSA models are sensitive to disordered categories within individual raters. Implications are discussed as they relate to research, theory, and practice for rater-mediated educational performance assessments
Cognitive psychology · Developmental psychology · Econometrics · Inter-rater reliability · Interpretation (philosophy) · Item response theory · Polytomous Rasch model · Psychometrics · Rating scale · Scale (ratio) · Statistics · Advanced Statistical Modeling Techniques · Clinical Psychology · Computer Science · Mathematics · Psychology · Psychometric Methodologies and Testing · Reliability and Agreement in Measurement
A Theory and Procedure of Scale Analysis
A Rasch Model for Partial Credit Scoring
The Rasch Rating Model and the Disordered Threshold Controversy
An Expanded Derivation of the Threshold Structure of the Polytomous Rasch Model That Dispels Any “Threshold Disorder Controversy”
Exploring Incomplete Rating Designs With Mokken Scale Analysis
Exploring Rating Quality in Rater-Mediated Assessments Using Mokken Scale Analysis
Adjacent-Categories Mokken Models for Rater-Mediated Assessments
A systematic review of methods for evaluating rating quality in language assessment
| Unique citing works | 1 |
|---|---|
| Citations per year | 0,14 |
| Citation span | 2019 - 2019 (1) |
| Citation velocity | historical |
| Highly cited | No |
| Citation types | Neutral: 1 |