Case Study: Mathematica's Abstinence Evaluation: Responding to a Changing Policy Climate
It is a pleasure to let you know that this exceptionally fine report has been selected for this high honor. Among the many factors leading to this recognition were the study's impact on the field of teen sexual behavior, national policy, and national legislation; the appropriate and thorough attention to political viability; and the extensive and adroit measures and designs needed to study the challenging questions at all—let alone longitudinally.
Mathematica was committed to involving a range of diverse stakeholders with varying views and interests throughout the evaluation. A 2010 article in the American Journal of Evaluation by Brandon et al. details how this stakeholder engagement proved vital to the success of the evaluation and offered lessons to the field from the evaluation’s best practices.
The evaluation, which involved four longitudinal survey waves, achieved an 80 percent response rate over time, and ensured strict confidentiality for approximately 2,000 teenage participants on a sensitive topic.
Mathematica's abstinence study team, led by Christopher Trenholm, vice president; director, NJ Health Research, Health Research Division, and executive vice president and chief operating officer Barbara Devaney, received the 2009 Outstanding Evaluation Award from the American Evaluation Association (AEA).
The study was extensively cited in the national and international media, including the Washington Post, USA Today, NPR, BBC, Newsweek, and ABC News. Peer-reviewed journals such as the Journal of Policy Analysis and Management also featured the evaluation. Years after its completion, it remains among the most popular downloads from the Mathematica website.
Notes on the Study Design. The multi-year evaluation used a random assignment design to estimate impacts on youth attitudes and behaviors. Youth were enrolled in the study sample over three consecutive school years, from fall 1999 through fall 2001. Four surveys were administered throughout the course of the evaluation, a baseline and three follow-up surveys. The final survey was given to 2,057 youth in 2005 and 2006, roughly four to six years after they began participating in the study. All told, 1,209 youth participated in one of the abstinence education programs and 848 were assigned to the control group. By the time the last follow-up survey was completed, youth had entered their mid- to late teens, permitting the researchers to reliably measure program impacts on teen sexual activity and other risk behaviors. The evaluation included an implementation and process analysis drawing on three data sources: (1) review of program documents and records; (2) interviews and focus groups with program staff, school staff, community leaders, parents, and program participants; and (3) on-site program observations.
Historically, sex education programs of any kind for teenagers have been a hot button policy issue. Until Mathematica’s multi-year evaluation there was little solid information available on the effectiveness of abstinence programs. In such a politically charged decision-making climate, policymakers need high quality, objective research that can scientifically inform the debate.
In 1998, in recognition of our commitment to evidence-based, high quality research, the U.S. Department of Health and Human Services, Office of the Secretary, Assistant Secretary for Planning and Evaluation commissioned Mathematica to conduct a congressionally mandated evaluation of the effectiveness of abstinence education programs. The resulting multi-year evaluation, which benefited from the advice and guidance of a panel of experts representing diverse perspectives, used an experimental design to estimate impacts on youth attitudes and behaviors and included an extensive implementation analysis. Eligible youth were randomly assigned to either the program group, which was offered abstinence-only education services, or the control group, which was not offered these services. Four rounds of follow-up data collection were completed to assess program impacts on attitudes and behaviors throughout the teen years.
Mathematica’s evaluation influenced the debate on the effectiveness of abstinence programs. More importantly, it led to major changes in federal policy aimed at reducing adolescent sexual activity and its associated risks, including a renewed focus on evidence-based program funding, deepened investment in programs for older teens, and expanded support to evaluations modeled after the rigorous study design adopted by Mathematica
Key findings from the evaluation included:
- Youth in the program group were no more likely than those in the control group to have abstained from sex and, among those who reported having had sex, they had similar numbers of sexual partners.
- Contrary to concerns raised by critics of abstinence education, youth in the program group were no more likely to have unprotected sex than youth in the control group.
- Targeting youth at young ages may not be sufficient.
This case study is for informational purposes only. Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.
In 1998, the U.S. Department of Health and Human Services, Office of the Secretary, Assistant Secretary for Planning and Evaluation commissioned Mathematica to conduct a congressionally mandated evaluation of the effectiveness of abstinence education programs.
Mathematica’s comprehensive nine-year abstinence education evaluation used the most rigorous, scientifically based approach to measure program impacts.