Evaluating the Measurement Error of Interviewer Observed Paradata

As survey researchers have begun exploiting paradata—for example, for the correction of nonresponse bias—the quality of these data has come into question. Inaccurate information is likely to affect the resulting statistics and conclusions drawn from such data. This paper focuses on one type of paradata, observations made by interviewers during the data-collection process, and assesses the quality of these observations by examining their measurement error properties. The analysis uses the UK Census Nonresponse Link Study, which links interviewers' observations collected on six major UK surveys with Census data. Comparing five interviewer observations with self-reports from the Census, the accuracy of the observations for both respondents and nonrespondents to the surveys is evaluated. A multilevel modeling approach is used to explore under which conditions the interviewers' observations match the reports on the Census forms, accounting for the clustering of sample members within interviewers and areas. The analysis finds that the overall percent agreement between the observations and the Census is generally high, ranging from 87 to 98 percent. The type of housing structure and the final result code are significantly associated with measurement error. For four of the five observations, there is evidence that the interviewer significantly influences the level of measurement error, even after controlling for household, interviewer, and area characteristics. The results presented here will inform future analyses assessing the quality of interviewers' observations

Census · Data collection · Data quality · Econometrics · Interview · Non-response bias · Observational error · Population · Quality (philosophy · Sample (material · Sociology · Statistics · Computer Science · Health disparities and outcomes · Healthcare Policy and Management · Mathematics · Psychology · Survey Methodology and Nonresponse · Demography

• A formula to communicative success in the structured telephone interview

  Open Access•Anna Ipatova, Dmitry Rogozin•Sociological Journal•2014

  Incluir item

• Understanding ‘Don't know’ answers to survey questions - an International comparative analysis using interview paradata

  Kingsley Purdam, Joseph W Sakshaug et al.•Innovation The European Journal…•2020

  Incluir item

• Sources of Variance in the Accuracy of Interviewer Observations

  Open Access•Brady T West, Dan Li•Sociological Methods & Research•2019

  Incluir item

• Do Interviewer Assessments of Respondents’ Performance Accurately Reflect Response Behavior

  Open Access•Laura Jacobs, Geert Loosveldt et al.•CAM•2020

  Incluir item

• Factors Associated with Interviewers’ Evaluations of Respondents’ Performance in Telephone Interviews

  Open Access•Dana Garbarski, Jennifer Dykema et al.•Public Opinion Quarterly•2023

  Incluir item

• Which Is the Better Investment for Nonresponse Adjustment

  Jennifer Sinibaldi, Mark Trappmann et al.•Public Opinion Quarterly•2014

  Incluir item

• Do Interviewer Postsurvey Evaluations of Respondents’ Engagement Measure Who Respondents Are or What They Do? A Behavior Coding Study

  Antje Kirchner, Kristen Olson et al.•Public Opinion Quarterly•2017

  Incluir item

• Can Interviewers Effectively Rate the Likelihood of Cases to Cooperate

  Stephanie Eckman, Eckman et al.•Public Opinion Quarterly•2013

  Incluir item

• Factors Affecting the Accuracy of Interviewer Observations

  Brady T West, Frauke Kreuter•Public Opinion Quarterly•2013

  Incluir item

• Can Marketing Data Aid Survey Research? Examining Accuracy and Completeness in Consumer-File Data

  Josh Pasek, M Jones-Jang et al.•Public Opinion Quarterly•2014

  Incluir item

• Advances in Survey Methods for the Developing World

  Noam Lupu, Kristin Michelitch•Annual Review of Political Science•2018

  Incluir item

• La calidad de la encuesta en la sociedad digital

  Open Access•Ma Ángeles Cea D''Ancona, Angeles Cea De Ancona et al.•Revista Española de Investigaciones…•2025

  Incluir item

• A Practical Technique for Improving the Accuracy of Interviewer Observations of Respondent Characteristics

  Open Access•Brady T West, Frauke Kreuter•CAM•2015

  Incluir item

• Doorstep Interactions and Interviewer Effects on the Process Leading to Cooperation or Refusal

  Open Access•Gabriele B Durrant, Julia D’arrigo•Sociological Methods & Research•2014

  Incluir item

• Measurement Error in Nonlinear Models

  Raymond J Carroll, David Ruppert et al.•Measurement Error in Nonlinear…•2006

  Incluir item

• Measurement Error Models

  Open Access•Wayne A Fuller•Measurement Error Models•1987

  Incluir item

• Measuring Attitudes Cross-Nationally

  Roger Jowell, Caroline Roberts et al.•Measuring Attitudes Cross-Nationall…•2007

  Incluir item

• Responsive Design for Household Surveys

  Open Access•Robert M Groves, Steven G Heeringa•Journal of the Royal Statistical…•2006

  Incluir item

• Measuring the built environment

  Open Access•Weich, Elizabeth Burton et al.•Health & Place•2001

  Incluir item

• Effects of Interviewer Attitudes and Behaviors on Refusal in Household Surveys

  Gabriele B Durrant, Robert M Groves et al.•Public Opinion Quarterly•2010

  Incluir item