Mathematica-mpr.com is now Mathematica.org. Please update your bookmarks. Learn more about this change.
Case Study: Evaluating the Effectiveness of Employment and Training Services to Improve Job Seekers' Success
This evaluation presents a tremendous opportunity for participating areas to document the importance of WIA-funded services to customers and to share our findings with Congress, the Office of Management and Budget, and, ultimately, the public. The results of this evaluation will be critical in shaping the future of the U.S. workforce investment system.
The evaluation is being conducted in 28 randomly selected Local Workforce Investment Areas representing 19 states.
About 35,000 job seekers were randomly assigned between November 2011 and April 2013.
Notes on Data Collection. The multi-year evaluation is examining the extent to which participation in the WIA Adult and Dislocated Worker programs improves job seekers’ success in the labor market. Job seekers were randomly assigned to one of three groups based on the level of services received. One group received all services available, and the other two groups received a reduced set of services. To minimize disruption to program procedures, the evaluation team worked closely with programs to customize study procedures for their local contexts and train their staff. During two rounds of implementation study site visits at each of the 28 local areas, the study team collected information about how local areas deliver services to job seekers and employers. Detailed cost information was also collected from each local area for the cost-benefit study. In addition, we interviewed state workforce agency staff to collect information about state-level policy and guidance. Follow-up telephone surveys at 15 and 30 months, after a job seeker enrolled in the study, are assessing program outcomes. Administrative data on study participants’ earnings and service receipt are also being collected.
Partners. Mathematica is leading the evaluation with the support of its partners Social Policy Research Associates, MDRC, and the Corporation for a Skilled Workforce.
The nation’s public workforce system provides employment and training services to job seekers through the Workforce Investment Act (WIA) Adult and Dislocated Worker programs. When enacting WIA in 1998, Congress required a rigorous evaluation using experimental methods. Mathematica’s study was designed to meet the gold standard for program evaluations—to provide rigorous and defensible estimates of the effectiveness of these programs nationwide.
The evaluation occurred in Local Workforce Investment Areas representative of local areas nationwide. Thirty local areas were randomly selected to participate in the study. Even though study participation was voluntary, Mathematica, working with its partners and Employment and Training Administration senior staff, persuaded 26 of the 30 randomly selected local areas to participate. Two local areas randomly selected as replacements also agreed to participate.
Random assignment ensures that estimates of program effectiveness are unbiased. The evaluation randomly assigned about 35,000 job seekers to three study groups: (1) a group that could receive all WIA-funded services and training for which they were eligible; (2) a group that could receive WIA counseling and other staff-intensive services, but not training; and (3) a group that could receive neither training nor staff-intensive services. By randomly assigning to these three groups, the evaluation can provide estimates of the overall effectiveness of the programs, as well as the effectiveness of training and the staff-intensive services separately.
Our approach ensures that study findings will not be open to two common criticisms. First, findings will be applicable to programs nationwide neutralizing criticism that participating areas were unusual. Second, random assignment of job seekers avoids a common critique of non-experimental studies—that the findings are biased because those who choose to receive services differ in important ways from those who do not receive services
A series of policy-focused briefs and a report discussing the implementation of the WIA Adult and Dislocated Worker programs will be available in 2015. Findings on the effectiveness of the program measured 15 months after job seekers enrolled will be available in 2016. Findings on the longer-term effectiveness of the programs and their benefits and costs will be available in 2017.
This case study is for informational purposes only. Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.
For 10 years, the Workforce Investment Act of 1998 (WIA) was the major source of public funding for employment and training services in the United States. WIA’s Adult and Dislocated Worker programs provide employment and training for low-income disadvantaged job seekers as well as workers laid off from their jobs. The Workforce Innovation and Opportunity Act of 2014 (WIOA) supercedes WIA and offers a similar set of services to job seekers as under WIA.
In 2008, the U.S. Department of Labor’s Employment and Training Administration commissioned Mathematica and its partners to conduct a national random assignment evaluation of these programs’ effectiveness.