Innovation and Support for Assessing Summative Evaluation Readiness: Lessons Learned
Lessons from the Field, OPRE Report # 2023-121
Download
Associated Project
Building Program Capacity to Support Youth at Risk of Homelessness (YARH): Phases I-III
Prepared for:
U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation
Clients
Key Findings
YARH convened two evidence-building meetings to support selection of an intervention ready for summative evaluation. Four factors guided discussions for recommending a YARH-2 intervention for the summative evaluation:
- Interest of the field in the intervention
- Readiness of the intervention for a summative evaluation
- Rigor of evidence that would result from the proposed design for the summative evaluation
- Likelihood of detecting statistically significant favorable impacts
Following these meetings, Mathematica recommended moving forward with a summative evaluation of Colorado’s Pathways to Success comprehensive service model.
Mathematica also led an evidence-building meeting focused on providing feedback to the YARH sites not participating in the summative evaluation and identifying opportunities to continue their evidence-building work.
YARH’s evidence-building process offers several lessons learned, including:
- Grantees and funders are eager to participate in summative evaluations to understand what works, for whom, and how.
- Successfully supporting grantees requires careful consideration of evaluation-focused activities while they are simultaneously providing services.
- Transparency about what information would inform decisions about site readiness to participate in a summative evaluation was vital to the success of the multiphase program and the grantees.
YARH has conducted a unique multiphase evidence-building process where 18 grantees received funding to develop a comprehensive service model designed to prevent homelessness among youth and young adults who have been involved in the child welfare system. Six of those grantees received additional funding in YARH-2 to conduct initial implementation and formative evaluations of the comprehensive service models designed in YARH-1. During YARH-3, one grantee was selected as the focus of a rigorous summative evaluation, including an impact and an implementation study. The remaining YARH-2 grantees received additional support to continue their evidence-building in YARH-3.
This brief describes the multiphase evidence-building process YARH undertook to select the Colorado Pathways to Success comprehensive service model for the YARH-3 summative evaluation and to support YARH-2 grantees not yet ready for summative evaluation in identifying lessons learned and possible next evidence-building steps. It should be of interest to evaluators, funders, child welfare agencies, and program developers and implementers.
More like this from Mathematica
How do you apply evidence?
Take our quick four-question survey to help us curate evidence and insights that serve you.
Take our survey