Building Evidence for a New Intervention or Program Takes Time
Download
Associated Project
Building Program Capacity to Support Youth at Risk of Homelessness (YARH): Phases I-III
Prepared for:
U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation
Key Findings
This brief uses YARH as an example of how evidence is built through the following steps:
- Design an intervention
- Implement, refine, and document the intervention
- Produce indications of outcomes
- Produce evidence of impacts
- Replicate and produce strong evidence of positive impacts
In step 1, YARH grantees designed new interventions based on evidence from a range of sources, such as administrative data, case record reviews, and focus groups with youth and young adults. In step 2, they implemented pilot versions of their interventions, conducted small-scale tests to examine intervention fidelity and intervention processes, and established continuous quality improvement (CQI) processes; they used this evidence to refine their interventions and document key elements in intervention manuals. In step 3, grantees conducted several formative evaluation activities to determine if staff implemented the revised interventions with fidelity and to produce indications of outcomes for their interventions. In step 4, ACF and its partners selected Colorado’s Pathways intervention for a federally-led rigorous summative evaluation to produce evidence of impacts for this intervention. Step 5 is not currently part of YARH, but the summative evaluation in step 4 sets the stage for potential replication and further study of Pathways’ impacts.
This brief uses the term evidence broadly and focuses on various types of evidence that, over time, can contribute to a solid understanding of a new program or intervention. Understanding the process of building evidence can help program staff and evaluators identify next steps for building evidence about the intervention while working toward a possible impact evaluation. Funders, grantees, and evaluators, along with community members and program participants, can help define, use, and create evidence over time.
Throughout the Youth At-Risk of Homelessness (YARH) grant program, the Children’s Bureau and Mathematica helped grantees develop and fine-tune comprehensive service models (referred to as interventions). The Children’s Bureau and Mathematica also helped grantees consider how they could contribute to the evidence base for other programs, including child welfare agencies interested in developing interventions to reduce homelessness among youth and young adults. Over a decade, YARH grantees engaged external evaluators, funders, local service providers, youth, and young adults to gather evidence. Efforts to build evidence culminated in the decision to fund a federal impact and implementation study of Colorado’s Pathways to Success (Pathways) intervention. This brief uses YARH as an example of how evidence is built in steps. Readers—including program staff, funders, and evaluators—can draw on the YARH examples to inform future evaluation efforts.
More like this from Mathematica
Efficiency Meets Impact.
That's Progress Together.
To solve their most pressing challenges, organizations turn to Mathematica for deeply integrated expertise. We bring together subject matter and policy experts, data scientists, methodologists, and technologists who work across topics and sectors to help our partners design, improve, and scale evidence-based solutions.
Work With Us