Kappa and Rater Accuracy
Paradigms and Parameters
Bibliographic Data
| ID | 20286182 |
|---|---|
| Authors | Anthony J Conger (Purdue University, West Lafayette, IN, USA, corresponding author) |
| Year | 2017 |
| Volume | 77 |
| Issue | 6 |
| Pages | 1019-1047 |
| Publication date | 2017-12-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Educational and Psychological Measurement (JOURNAL) |
| Journal identifiers | ISSN: 0013-1644 • E-ISSN: 1552-3888 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0013164416663277 |
| PMID | 29795943 |
| OpenAlex | W2511585989 |
| Language | EN |
| Citations received | 5 |
| References cited | 21 |
Drawing parallels to classical test theory, this article clarifies the difference between rater accuracy and reliability and demonstrates how category marginal frequencies affect rater agreement and Cohen’s kappa (κ). Category assignment paradigms are developed: comparing raters to a standard (index) versus comparing two raters to one another (concordance), using both nonstochastic and stochastic category membership. Using a probability model to express category assignments in terms of rater accuracy and random error, it is shown that observed agreement (Po) depends only on rater accuracy and number of categories; however, expected agreement (Pe) and κ depend additionally on category frequencies. Moreover, category frequencies affect Pe and κ solely through the variance of the category proportions, regardless of the specific frequencies underlying the variance. Paradoxically, some judgment paradigms involving stochastic categories are shown to yield higher κ values than their nonstochastic counterparts. Using the stated probability model, assignments to categories were generated for 552 combinations of paradigms, rater and category parameters, category frequencies, and number of stimuli. Observed means and standard errors for Po, Pe, and κ were fully consistent with theory expectations. Guidelines for interpretation of rater accuracy and reliability are offered, along with a discussion of alternatives to the basic model
Affect (linguistics) · Algorithm · Cohen's kappa · Concordance · Econometrics · Inter-rater reliability · Interpretation (philosophy) · Kappa · Reliability (semiconductor) · Residual · Statistics · Variance (accounting) · Computer Science · Mathematics · Multi-Criteria Decision Making · Psychology · Psychometric Methodologies and Testing · Reliability and Agreement in Measurement
Kappa Coefficients for Missing Data
A Comparative Analysis of Influences on Choosing a Music Teaching Occupation
Government communication in times of crisis
Evaluation of studies in the field of social sustainability in housing
Emotion-dependent linguistic features of autobiographical memory of different specificity
The Dependability of behavioral measurements
Computing inter‐rater reliability and its variance in the presence of high agreement
The Measurement of Observer Agreement for Categorical Data
A Comparison of Three Indexes of Agreement Between Observers
A Ratio Test of Interrater Agreement With High Specificity
A Note on the G Index of Agreement
Testing the Difference of Correlated Agreement Coefficients for Statistical Significance
An Unbiased Estimate of Global Interrater Agreement
A Coefficient of Agreement for Nominal Scales
Estimating false alarms and missed events from interobserver agreement
Measures of response agreement for qualitative data
Large sample standard errors of kappa and weighted kappa
Diversity of decision-making models and the measurement of interrater agreement
| Citations per year | 0.71 |
|---|---|
| Citation span | 2019 - 2025 (7) |
| Highly cited | No |