This paper introduces a cloud-native framework for evaluating how accurately and transparently large language models (LLMs) perform applied statistical analysis on complex, survey-based data. Most public benchmarks assess reasoning on text-based tasks and do not test a model’s ability to filter microdata, apply survey weights, or generate valid population estimates—core requirements in policy research. To address this gap, we developed and tested a structured evaluation process using 35 validated descriptive and causal questions based on the American Community Survey (ACS) Public Use Microdata Sample.

We compared model performance under two workflows: reasoning-only and code execution. In the code-execution setting, models generated and ran Python code against real data in a secure, serverless environment, enabling iterative refinement and reproducible analysis. Results show that data access and tool invocation substantially improve accuracy, particularly for descriptive tasks, while causal inference performance varies meaningfully across models. Findings underscore that trustworthy AI-driven analytics depend not only on model capability but also on metadata quality, prompt design, and controlled analytic workflows.