Calibrated Multilevel Regression with Poststratifiction for the Analysis of SMS Survey Data

Calibrated Multilevel Regression with Poststratifiction for the Analysis of SMS Survey Data

Working Paper #66
Published: Mar 26, 2021
Publisher: Mathematica

Constance Delannoy

Shirley Jeoffreys-Leach

Bobby Berkowitz

Grant J. Robertson

Key Findings
  • Estimates based on SMS surveys tend to be biased due to unobserved confounding and survey mode effects.
  • We introduce a procedure, Calibrated MRP, which produces efficient, unbiased estimates by incorporating a small amount of face-to-face data.
  • We illustrate the success of this procedure by applying it to a survey that measures financial inclusion in Uganda.
  • Calibrated MRP has the potential to greatly reduce the costs of data collection, especially in resource-poor countries.

Face-to-face (FTF) surveys have been the traditional method to gather nationally representative data and remain the dominant data collection mode in resource-poor countries. Conducting these surveys is expensive and time consuming. With the rapid expansion of mobile phone use, Short Message Service (SMS) presents an opportunity to conduct inexpensive, fast, and scalable surveys. However, using SMS surveys to obtain nationally representative estimates can lead to two types of bias: selection bias and survey mode effects. Standard adjustments to correct for nonrepresentative sampling can be insufficient to remove all biases. We introduce calibrated multilevel regression with poststratification (calibrated MRP), a procedure that corrects for residual bias by incorporating a relatively small sample of FTF data that is known to be unbiased. We apply this method to the problem of estimating financial inclusion (access to formal banking systems) in eight countries in Africa and Asia; we focus on Uganda in the interest of clarity. We find that our calibrated MRP approach is effective in replicating estimates from a larger and much more expensive F2F survey. This paper includes a description of our methods as well as results from the financial inclusion study and a discussion of limitations and future areas for research.

Follow the Evidence

Interested in the most current findings from Mathematica? Subscribe to our bi-weekly newsletter, Evidence & Insights, to stay up to date with the issues that matter to you.

Sign Me Up