Identifying What Works to Help Job Seekers Succeed

Mathematica provided recommendations to the Administration for Children and Families (ACF) within the Department of Health and Human Services for enhancing their processes to identify and share information about evidence-based programs aimed at improving the employment and earnings of people with low incomes.

“Mathematica’s response to this request for comment recognizes and applauds ACF’s efforts to update the review protocol while highlighting opportunities for additional changes that could further improve the transparency of the Pathways evidence review process,” wrote Mathematica. “This work could enhance program effectiveness and advance ACF’s goal of providing decision makers with useful, accurate, and clear information to inform practical decision making and achieve the best possible outcomes.”

The recommendations were submitted in response to ACF’s request for comments on proposed revisions to existing standards and on new standards for including and rating the quality of program cost studies in the draft “Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0.” The draft provides a revised protocol for the Pathways to Work Evidence Clearinghouse, a publicly available resource designed to identify what works to help job seekers with low incomes succeed in the labor market.

Drawing on its experience operating multiple evidence review clearinghouses for the federal government and other partners, as well as conducting rigorous evaluations of employment programs, Mathematica provided several recommendations to ACF, including:

• Plan regular updates to search and screening procedures to keep pace with rapidly evolving technologies, including new uses of artificial intelligence.
• Update evidence standards and program effectiveness definitions to improve the rigor of study quality ratings and program effectiveness labels, and encourage researchers to report their findings in ways that benefit clearinghouse users.
• Clarify and enhance the new cost study standards so that they can be consistently applied to support reliable and rigorous study quality ratings.
• Implement procedures for reporting on the findings from cost studies that help ensure that users interpret and use the cost estimates correctly.

“Evidence-based information is essential for the efficient and effective allocation of federal resources to improve employment outcomes,” said James Mabli, senior vice president and managing director of the Nutrition, Health, and Human Services Division at Mathematica. “ACF’s proposed changes to the Pathways review protocol will provide critical insights into employment and training programs, equipping program leaders with the action-ready information they need for informed decision-making.”

Mathematica’s experience operating evidence review clearinghouses includes the What Works Clearinghouse for the U.S. Department of Education and the Home Visiting Evidence of Effectiveness for ACF’s Office of Planning, Research, and Evaluation. Its evaluations of employment programs include the Next Generation of Enhanced Employment Strategies (NextGen) and Employment Coaching projects.