Innovation and Support for Assessing Summative Evaluation Readiness: Lessons Learned
U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation
YARH convened two evidence-building meetings to support selection of an intervention ready for summative evaluation. Four factors guided discussions for recommending a YARH-2 intervention for the summative evaluation:
- Interest of the field in the intervention
- Readiness of the intervention for a summative evaluation
- Rigor of evidence that would result from the proposed design for the summative evaluation
- Likelihood of detecting statistically significant favorable impacts
Following these meetings, Mathematica recommended moving forward with a summative evaluation of Colorado’s Pathways to Success comprehensive service model.
Mathematica also led an evidence-building meeting focused on providing feedback to the YARH sites not participating in the summative evaluation and identifying opportunities to continue their evidence-building work.
YARH’s evidence-building process offers several lessons learned, including:
- Grantees and funders are eager to participate in summative evaluations to understand what works, for whom, and how.
- Successfully supporting grantees requires careful consideration of evaluation-focused activities while they are simultaneously providing services.
- Transparency about what information would inform decisions about site readiness to participate in a summative evaluation was vital to the success of the multiphase program and the grantees.
YARH has conducted a unique multiphase evidence-building process where 18 grantees received funding to develop a comprehensive service model designed to prevent homelessness among youth and young adults who have been involved in the child welfare system. Six of those grantees received additional funding in YARH-2 to conduct initial implementation and formative evaluations of the comprehensive service models designed in YARH-1. During YARH-3, one grantee was selected as the focus of a rigorous summative evaluation, including an impact and an implementation study. The remaining YARH-2 grantees received additional support to continue their evidence-building in YARH-3.
This brief describes the multiphase evidence-building process YARH undertook to select the Colorado Pathways to Success comprehensive service model for the YARH-3 summative evaluation and to support YARH-2 grantees not yet ready for summative evaluation in identifying lessons learned and possible next evidence-building steps. It should be of interest to evaluators, funders, child welfare agencies, and program developers and implementers.