Rapid-Cycle Evaluation

Rapid-cycle evaluation uses a rigorous, scientific approach to provide decision makers with timely and actionable evidence of whether operational changes improve program outcomes. Often, changes can be tested in a matter of months, and decision makers can have a high degree of confidence in the results. Rapid-cycle evaluation can also help avoid investments in changes that are unlikely to produce the desired results.

Rapid-cycle evaluation can be used to help determine.... 

Education Technologies

  • Whether to buy a new software application to help students learn how to read. School districts and principals routinely consider new educational digital tools or software packages, which have the potential to positively (or negatively) affect students’ learning outcomes. Oftentimes, however, anecdotal experience or customer recommendations are the only information available to support these decisions. Mathematica worked with the U.S. Department of Education, Office of Education Technology to develop a web-based, interactive toolkit that guides decisionmakers step-by-step through a low-cost, quick-turnaround evaluation to provide rigorous evidence on the effectiveness of such products.
  • Related projects: Rapid-Cycle Tech Evaluations Accelerate Decisionmaking, Evaluation of iReady for IDEA Public Schools, Evaluation of IREAD for Clarksdale Municipal School District

Employment

  • The most effective outreach strategies for increasing program engagement and uptake of employment services resources. A variety of government, private, and nonprofit organizations across the country offer reemployment services to veterans. However, many of these resources are underused while unemployment rates among veterans remain higher than unemployment rates among nonveteran adults (Bureau of Labor Statistics 2015). In partnership with the U.S. Department of Labor and local agencies, Mathematica designed and tested a series of personalized emails, informed by insights from behavioral science, to encourage stronger job-seeker engagement with reemployment services. Through a quick-turnaround study, we measured statistically significant increases in service take-up and program completion as a result.
  • Related project: Behavioral Interventions for Labor-Related Programs

Family Supports

  • Whether a new messaging approach, including proactive reminders, improves TANF participants’ timely submission of work activity participation hours. Many local Temporary Assistance for Needy Families (TANF) and other public assistance programs struggle with on-time and accurate submissions of required participation documentation. In Colorado, Mathematica is partnering with a county workforce center to try out different nudges—email and postcard reminders—in an effort to improve the rate of timely reporting by participants.
  • Related projects:  Project IMPROVE: Improving Program Outcomes Via Evidence-Based Technical Assistance, Welfare and Family Self-Sufficiency Research (Project AWESOME), Larimer County TANF TA

International

  • Whether behavior-change campaigns can alter attitudes towards child welfare in developing countries. A consortium of organizations—including Mathematica as the technical lead—is partnering with the U.S. Agency for International Development (USAID) to design and conduct rapid feedback tests of alternative implementation options to improve outcomes at their operating units around the globe. In Cambodia, the project is currently conducting two pilots to learn whether variations in community-based and online behavior-change campaigns can reduce support for residential care institutions for children. Future pilots will study tuberculosis care practices among health care practitioners in India and community-engagement to promote early childhood education in Tanzania.
  • Related project: Rapid Feedback Measurement, Evaluation, Research, and Learning (MERL)