Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys

This article examines the use of probing techniques in web surveys to identify validity problems of items. Conventional cognitive interviewing is usually based on small sample sizes and thus precludes quantifying the findings in a meaningful way or testing small or special subpopulations characterized by their response behavior. This article investigates probing in web surveys as a supplementary way to look at item validity. Data come from a web survey in which respondents were asked to give reasons for selecting a response category for a closed question. The web study was conducted in Germany, with respondents drawn from online panels ( n = 1,023). The usefulness of the proposed approach is shown by revealing validity problems with a gender ideology item

Cognition · Cognitive interview · Data science · Developmental psychology · Ideology · Interview · Item response theory · Political science · Politics · Psychometrics · Sample (material · Sociology · Web survey · World Wide Web · Applied Psychology · Computer Science · Focus Groups and Qualitative Methods · Psychology · Social and Intergroup Psychology · Social Psychology · Survey Methodology and Nonresponse

• Improving measurements by survey infrastructures synergies

  Open Access•Vera Lomazzi•Quality & Quantity•2022

  Incluir item

• Assessing conceptual comparability of single-item survey instruments with a mixed-methods approach

  Open Access•Ranjit K Singh, Cornelia E Neuert et al.•Quality & Quantity•2024

  Incluir item

• Measures of gender role attitudes under revision

  Open Access•Jessica Gabriele Walter•Social Science Research•2018

  Incluir item

• Insights into sustainability

  Amanda D Manzanares, Lydia Horne et al.•International Journal of…•2026

  Incluir item

• Assessing Cross-National Equivalence of Measures of Xenophobia

  M Braun, Michael Brauns et al.•International Journal of Public…•2012

  Incluir item

• Automatic Classification of Open-Ended Questions

  Open Access•Matthias Schonlau, Hyukjun Gweon et al.•Social Science Computer Review•2019

  Incluir item

• Asking Probing Questions in Web Surveys

  Open Access•Dorothée Behr, Lars Kaczmirek et al.•Social Science Computer Review•2012

  Incluir item

• Question Order Effects in Cross-Cultural Web Probing

  Open Access•Patricia Hadler•Social Science Computer Review•2021

  Incluir item

• Probing in online mixed-device surveys

  Open Access•Vera Toepoel, Karlijn Mathon et al.•Bulletin of Sociological…•2021

  Incluir item

• A Critical Examination of the Reliability and Validity of a Gender Role Attitude Scale in Flanders (Belgium)

  Open Access•Myriam Halimi, Els Consuegra et al.•Sex Roles•2017

  Incluir item

• What does the general national pride item measure? Insights from web probing

  Open Access•Katharina Meitinger•International Journal of…•2018

  Incluir item

• The intricate geographies of gender ideologies in Germany

  Open Access•P Dirksmeier•Geoforum•2015

  Incluir item

• Attitudes Towards Gender Roles in Cross-Cultural Surveys

  Open Access•Andreea A Constantin, Andreea Constantin et al.•Social Indicators Research•2014

  Incluir item

• Tackling Traditional

  Keera Allendorf, E Short et al.•Sociology of Development•2025

  Incluir item

• The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys

  Open Access•Patricia Hadler•Sociological Methods & Research•2025

  Incluir item

• Do men and women really have different gender role attitudes? Experimental insight on gender-specific attitudes toward paid and unpaid work in Germany

  Open Access•Sabine Düval•Social Science Research•2023

  Incluir item

• Structural stability, quantitative change

  Open Access•Alice Barth, Miriam Trübner•Social Science Research•2018

  Incluir item

• Design, Evaluation, and Analysis of Questionnaires for Survey Research

  Willem E Saris, Irmtraud N Gallhofer•Design, Evaluation, and Analysis…•2007

  Incluir item

• Cognitive Interviewing

  Gordon Willis•Cognitive Interviewing•2005

  Incluir item

• A Review and Synthesis of the Measurement Invariance Literature

  Open Access•Robert J Vandenberg, Charles E Lance•Organizational Research Methods•2000

  Incluir item

• Analysing patriarchy capitalism and women's employment in Europe

  John Macinnes•Innovation The European Journal…•1998

  Incluir item

• Random Probes of GSS Questions

  Tom W Smith, Tom Smith•International Journal of Public…•1989

  Incluir item

• The Length of Responses to Open-Ended Questions

  Open Access•Martyn Denscombe•Social Science Computer Review•2007

  Incluir item

• Asking Probing Questions in Web Surveys

  Open Access•Dorothée Behr, Lars Kaczmirek et al.•Social Science Computer Review•2012

  Incluir item

• The Influence of Topic Interest and Interactive Probing on Responses to Open-Ended Questions in Web Surveys

  Open Access•Jennifer L Holland, Leah Melani Christian et al.•Social Science Computer Review•2008

  Incluir item

• Evaluation of Cognitive Interviewing Techniques

  Open Access•Gordon Willis, Gordon B Willis et al.•Bulletin of Sociological…•1997

  Incluir item

• Using Egalitarian Items to Measure Men’s and Women’s Family Roles

  Open Access•Michael Brauns, Michael Braun•Sex Roles•2008

  Incluir item

• Open-Ended Questions in Web Surveys

  Jolene D Smyth, Don A Dillman et al.•Public Opinion Quarterly•2009

  Incluir item

• Research Synthesis

  R Baker, Stephen J Blumberg et al.•Public Opinion Quarterly•2010

  Incluir item

• Sources of Error in Cognitive Interviews

  Frederick G Conrad, Johnny Blair•Public Opinion Quarterly•2009

  Incluir item

• Research Synthesis

  Phillip Beatty, P C Beatty et al.•Public Opinion Quarterly•2007

  Incluir item

• Survey Research

  Jon A Krosnick•Annual Review of Psychology•1999

  Incluir item

• The Random Probe

  Howard Schuman•American Sociological Review•1966

  Incluir item

• Respondent Debriefings Conducted by Experts

  Open Access•Elizabeth Nichols, Jennifer Hunter Childs•CAM•2009

  Incluir item