Evaluation Design and Implementation

Our strengths include designing and conducting studies of program implementation and effectiveness. We also integrate studies of operations and effects to generate actionable findings for decision makers, develop conceptual frameworks and logic models, and design demonstrations for new initiatives.

We help clients develop and implement designs that provide the most credible and actionable information within the time and resources available by addressing the following questions:

• What are the client's most pressing research questions concerning the program to be evaluated?
• What is the strongest evaluation design that could be used to answer the questions, minimizing bias and maximizing precision?
• What are the assumptions underlying the proposed approach, and how can we test them to ensure the findings will be robust and defensible?
• How do program effects vary across important subgroups of individuals or sites participating in the study? How are other stakeholders affected by the program being evaluated?
• If the program is found to have the desired effects, how will we identify the primary determinants of success? If the program does not have the desired effects, what barriers were most responsible? 
• How will the results from impact and implementation analyses be integrated to ensure the client has a clear and consistent set of answers to key questions?
• What are the possible challenges to the research, and how can we address them effectively?

• Pregnancy Assistance Fund Feasibility and Design Study and Federal Evaluation of Expectant and Parenting Youth. Expectant and parenting teens and young adults can face immense challenges, including interruptions in their education, difficulties in the workforce, and health and relationship problems. The study team designed three rigorous evaluations – two randomized controlled trials (RCTs) and one retrospective study based on a natural experiment – to evaluate innovative programs in California, Texas, and Washington, DC. The first of the three studies to be completed found large, positive impacts on the educational outcomes for parenting teens in DC. public schools. Drawing on the lessons learned in designing these studies, the team has also published a paper providing guidance to other researchers designing RCTs that randomize a small number of clusters.
• Comprehensive Primary Care Initiative. Improving primary care delivery is one of the the most pressing domestic health care needs. This study investigates the effects on Medicare expenditures, service use, and quality of care from providing nearly 500 practices (serving 2.5 million patients) with additional funding, ongoing data on their performance, and opportunities to learn from experts and peers. In addition to conventional analyses, the study uses Bayesian methods to produce more precise estimates and meaningful statements about the probability that the program is achieving its goals. In addition, it uses new methods to find associations between improvements in care delivery and improvements in key program outcomes. 
• SNAP Food Security. Estimating the effect of the nation's largest nutrition assistance program on food insecurity using household survey data has been challenging, because households that participate in SNAP can differ in systematic ways from households that do not. The study team used a creative design to measure changes in food security, including a cross-sectional analysis and a longitudinal comparison, to offer the most definitive evidence to date on the program.
• MasterCard Foundation Scholars Program Learning Partnership. Mathematica is designing and implementing a monitoring, evaluation, and learning results framework design and identifying key learning questions for this program that aims to engender social transformation by developing young leaders whose education and human capital skills contribute to the economic growth and social development.