Combining Text Mining and Manual Thematic Analysis to Understand Participant Experiences With Surveys

Bibliographic Data

ID3019113
AuthorsAmelie Nordstrom (0009-0000-1378-6805), Aditya Maheshwari (0000-0002-0716-939X), Anil Maheshwari (0000-0002-1274-4598), Rheann Quenneville, Grace Shen-Tu (Alberta’s Tomorrow Project, Calgary, AB, Canada)
Year2025
Volume24
Publication date2025-09-01
Open AccessYes
TypeARTICLE
VenueInternational Journal of Qualitative Methods (JOURNAL)
Journal identifiersISSN: 1609-4069 • E-ISSN: 1609-4069
PublisherSAGE Publications Inc (PUBLISHER)
DOI10.1177/16094069251378860
OpenAlexW4414115824
LanguageEN
Citations received3
References cited32

Participant engagement in large-scale surveys is vital to health and social science research. Understanding the factors that affect participant engagement requires methods that can meaningfully interpret participant feedback. Open-ended survey questions can capture these kinds of nuanced responses, but can be challenging to analyze at scale. To address this challenge, this study combines computational text mining with a manual thematic analysis to examine over 15,000 open-text survey responses. These responses provided details about participants' experiences with longitudinal public health surveys administered by Alberta's Tomorrow Project. The text mining analysis consists of both sentiment analysis and topic modelling to identify broad patterns in participant experiences. This is combined with a thematic analysis of a purposive sample of 852 of these responses to validate the computational methods and identify themes not captured by the topic model. The findings suggest that many participants found the survey easy to complete and valued having the opportunity to participate. Challenges experienced included difficulty with recall-based questions, irrelevant response options, and program functionality issues. Providing customizable communication methods to adapt to individual preferences was also widely reported to be important by participants. We find that the conclusions from the analyses were largely consistent across the computational text mining and manual thematic analyses. However, the computational approaches overgeneralize or miscategorize certain responses, highlighting the benefits of incorporating both methods to analyse large volumes of qualitative survey responses

Data-Driven Disease Surveillance · Ethics in Clinical Research · Focus Groups and Qualitative Methods

Tools

Open DOIOpen Access