Process and Implementation Evaluations

Although an impact evaluation can assess the extent to which an intervention resulted in changes in key outcomes, an implementation study provides us with the understanding of what was actually implemented and how; why the intervention changed outcomes or did not; and, if it did, how it might have done so. Implementation studies can provide important documentation of how a program was implemented for subsequent refinement and program scale-up, if successful.

Mathematica conducted a retrospective implementation study of the Total Sanitation and Sanitation Marketing (TSSM) intervention in Indonesia. TSSM was designed to strengthen the supply of and demand for sanitation and improve the enabling environment for sanitation interventions. Although an impact evaluation had been commissioned, our study sought to understand how the TSSM program was implemented in East Java, Indonesia, and reconciled some apparent discrepancies between program monitoring data and impact evaluation data involving program progress. The study used a mixed-methods approach that drew on both qualitative and quantitative data, including data gathered from a large cross-section of stakeholders at the national, provincial, district, subdistrict, village, and hamlet (subvillage) levels. The findings from the study provided important insights on reasons for program success in some places and not in others and will help to inform the impact analysis.