Calibrated Multilevel Regression with Poststratifiction for the Analysis of SMS Survey Data
Evidence | Publications

Calibrated Multilevel Regression with Poststratifiction for the Analysis of SMS Survey Data

Working Paper #66
Published: Jun 26, 2021
Publisher: Mathematica
Global Foundations Data Analytics Advanced Analytics Surveys and Data Collection Data Strategy
Download
Full Publication
Authors

Jonathan Gellar

Sarah Hughes

Constance Delannoy

Erin Lipman

Shirley Jeoffreys-Leach

Bobby Berkowitz

Grant J. Robertson

Key Findings
  • Estimates based on SMS surveys tend to be biased due to unobserved confounding and survey mode effects.
  • We introduce a procedure, Calibrated MRP, which produces efficient, unbiased estimates by incorporating a small amount of face-to-face data.
  • We illustrate the success of this procedure by applying it to a survey that measures financial inclusion in Uganda.
  • Calibrated MRP has the potential to greatly reduce the costs of data collection, especially in resource-poor countries.

Face-to-face (FTF) surveys have been the traditional method to gather nationally representative data and remain the dominant data collection mode in resource-poor countries. Conducting these surveys is expensive and time consuming. With the rapid expansion of mobile phone use, Short Message Service (SMS) presents an opportunity to conduct inexpensive, fast, and scalable surveys. However, using SMS surveys to obtain nationally representative estimates can lead to two types of bias: selection bias and survey mode effects. Standard adjustments to correct for nonrepresentative sampling can be insufficient to remove all biases. We introduce calibrated multilevel regression with poststratification (calibrated MRP), a procedure that corrects for residual bias by incorporating a relatively small sample of FTF data that is known to be unbiased. We apply this method to the problem of estimating financial inclusion (access to formal banking systems) in eight countries in Africa and Asia; we focus on Uganda in the interest of clarity. We find that our calibrated MRP approach is effective in replicating estimates from a larger and much more expensive F2F survey. This paper includes a description of our methods as well as results from the financial inclusion study and a discussion of limitations and future areas for research.

Forthcoming in the Journal of Survey Statistics and Methodology

DOI: 10.1093/jssam/smab033

Article: Calibrated Multilevel Regression with Poststratification for the Analysis of SMS Survey Data

How do you apply evidence?

Take our quick four-question survey to help us curate evidence and insights that serve you.

Take our survey