Validating LLM-Generated Synthetic Survey Samples: A Use-Dependent Multi-Dimensional Framework for Social-Science Applications

Abstract

Large language models (LLMs) make it increasingly feasible to generate synthetic survey respondents, but their empirical usefulness cannot be judged by a single notion of accuracy. A synthetic sample may perform weakly at the respondent level while still preserving distributional or substantive relationships relevant to social-science analysis. We develop a use-dependent, multi-dimensional validation framework distinguishing predictive, distributional, and substantive validity, and illustrate it using Eurobarometer data on attitudes toward artificial intelligence. Four off-the-shelf LLMs generate responses conditional on observed respondent profiles, while supervised machine learning provides a predictive reference for interpreting their performance. The results reveal substantial differences across validation dimensions: some LLM-generated samples preserve meaningful marginal heterogeneity, demographic ordering, cross-country variation, and predictor structure despite imperfect respondent-level correspondence. Performance varies across models and outcomes, but the broader picture is cautiously constructive. Synthetic samples should therefore be evaluated against the empirical properties required by their intended use.