Prompt-imposed structure
A methodological evaluation of LLM-generated survey data for theory-based modeling
Bibliographic Data
| ID | 12595099 |
|---|---|
| Authors | Zhangwei Zheng (0009-0004-9813-4214, National University of Malaysia, corresponding author), B A M Hafizuddin-Syah (0000-0002-9738-9432, National University of Malaysia), Qin Lingda Tan (0009-0008-2435-4184, National University of Malaysia) |
| Year | 2026 |
| Pages | 1-18 |
| Publication date | 2026-02-04 |
| Open Access | No |
| Type | ARTICLE |
| Venue | International Journal of Social Research Methodology (JOURNAL) |
| Journal identifiers | ISSN: 1364-5579 • E-ISSN: 1464-5300 |
| Publisher | Taylor & Francis (PUBLISHER • GB) |
| DOI | 10.1080/13645579.2026.2625160 |
| OpenAlex | W7127946488 |
| Language | EN |
| References cited | 28 |
A key methodological concern in using large language models (LLMs) for survey research is whether their outputs are merely ‘prompt-valid’ rather than ‘theory-valid.’ This study empirically evaluates that distinction. Using a validated structural model linking perceived aging risk, attitudes toward private medical insurance, and insurance purchase intentions, it compares one human dataset with four GPT-4o datasets generated under progressively structured prompts. GPT-4o produces plausible Likert-scale responses and, when given path-informed instructions, recovers the expected mediation structure. However, without such cues, the model fails to reproduce the theory-consistent relationships that reliably emerge in human data. These findings indicate that structural coherence in LLM-generated responses primarily reflects the relational logic imposed by the prompt rather than autonomous inference. LLMs may support pretesting and instrument development, but their role in theory testing should remain supplementary, transparent, and methodologically constrained.
Data collection · Qualitative research · Research methodology · Survey data collection · Survey methodology · Survey research · demographic modeling and climate adaptation · Psychometric Methodologies and Testing · Statistical Methods and Bayesian Inference
Measuring the Hedonic and Utilitarian Dimensions of Consumer Attitude
The theory of planned behavior
When to use and how to report the results of PLS-SEM
A study of normative and informational social influences upon individual judgment
Division of labor and performance under cooperative and competitive conditions
| Highly cited | No |
|---|