When Nonresponse Mechanisms Change

Effects on Trends and Group Comparisons in International Large-Scale Assessments

Bibliographic Data

ID20282174
AuthorsKaroline A Sachse (0000-0001-6688-1267, Humboldt-Universität zu Berlin, Berlin, Germany, corresponding author), Nicole Mahler (0000-0003-1743-4748, Humboldt-Universität zu Berlin, Berlin, Germany), Steffi Pohl (0000-0002-5178-8171, Freie Universität Berlin, Berlin, Germany)
Year2019
Volume79
Issue4
Pages699-726
Publication date2019-08-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueEducational and Psychological Measurement (JOURNAL)
Journal identifiersISSN: 0013-1644 • E-ISSN: 1552-3888
PublisherSAGE Publications (PUBLISHER • US)
DOI10.1177/0013164419829196
PMID32655180
OpenAlexW2914848324
LanguageEN
Citations received6
References cited40

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms

Cartography · Econometrics · Geography · Mechanism (biology) · Missing data · Scale (ratio) · Statistics · Advanced Causal Inference Techniques · Mathematics · Psychometric Methodologies and Testing · Statistical Methods and Bayesian Inference

Tools

Open DOISci-HubOpen Access