Education Policy Research

Reports

• "The Effectiveness of Mandatory-Random Student Drug Testing." Susanne James-Burdumy, Brian Goesling, John Deke, and Eric Einspruch, July 2010. Executive Summary. This report presents findings from an evaluation of the mandatory-random student drug testing (MRSDT) programs in 36 high schools from seven districts that received grants from the U.S. Department of Education’s Office of Safe and Drug-Free Schools in 2006.

• "Student Characteristics and Outcomes in Alternative and Neighborhood High Schools in Philadelphia." Hanley Chiang and Brian Gill, April 2010. This report presents descriptive analyses of the enrollment patterns, background characteristics, test scores, and graduation rates of students enrolled in three types of public high schools in Philadelphia: alternative disciplinary schools; alternative accelerated schools for students who struggled academically in regular public high schools; and regular, neighborhood-zoned high schools.

• "The Evaluation of Charter School Impacts." Philip Gleason, Melissa Clark, Christina Clark Tuttle, and Emily Dwoyer, May 2010. This report provides findings from the largest charter school study of its kind to date to use an experimental design. On average, charter middle schools that hold lotteries are neither more nor less successful than traditional public schools in improving student achievement, behavior, and school progress; however, impacts varied widely across schools.
• "Student Characteristics and Achievement in 22 KIPP Middle Schools." Christina Clark Tuttle, Bing-ru Teh, Ira Nichols-Barrer, Brian P. Gill, and Philip Gleason, June, 2010. This report presents the first findings from a multi-year evaluation of the Knowledge Is Power Program (KIPP), a network of charter schools for disadvantaged students. Using a matched comparison group design, results show that for the vast majority of KIPP schools in the evaluation, impacts on students’ state assessment scores in math and reading are positive, statistically significant, and educationally substantial.
• "College Experiences Survey: Methodological Summary." David DesRoches, John Hall, and Betsy Santos, December 2009. This report summarizes the College Experiences Survey data collection procedures and results. The survey was part of the College Student Attrition Project conducted by the Survey Research Center at Princeton University, an evaluation to better understand the factors associated with college persistence and attrition.
• "An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year Two Impact Report." Steven Glazerman and Allison Seifullah, May 2010. This report details second-year results from Mathematica’s study on Chicago Public Schools’ Teacher Advancement Program (TAP), a Teacher Incentive Fund grantee. The TAP model offers teachers performance pay, professional development, and advancement opportunities based in part on their students’ achievement growth.
• "Effectiveness of Selected Supplemental Reading Comprehension Interventions: Findings from Two Student Cohorts." Susanne James-Burdumy, John Deke, Julieta Lugo-Gil, Nancy Carey, Alan Hershey, Russell Gersten, Rebecca Newman-Gonchar, Joseph Dimino, Kelly Haymond, and Bonnie Faddis, May 2010. This study evaluated the effectiveness of four supplemental reading comprehension programs in helping disadvantaged fifth graders improve reading comprehension. The study, supported by the Institute of Education Sciences in the U.S. Department of Education and conducted over the course of two school years, found positive impacts for one of the four curricula.
• "Roads to Success: Estimated Impacts of an Education and Career Planning Program During Middle School." Duncan Chaplin, Martha Bleeker, and Kevin Booker, April 2010. This report assesses the impact of two years of exposure to the school and career planning program, Roads to Success, on eighth graders from high-poverty rural schools. The main findings are that the program had no measurable effect on desire to learn job skills or study/learning habits. However, exploratory analyses suggest positive impacts on talking to school staff about career/school plans and on gaining confidence in two areas: choosing careers and understanding what is required for career success. Estimated impacts on student behaviors were mixed.
• “Survey of Outcomes Measurement in Research on Character Education Programs.” Ann Person, Emily Moiduddin, Megan Hague-Angus, and Lizabeth Malone, December 2009. Through the Partnerships in Character Education Program, the federal government distributes about $25 million annually to state and local education agencies for character education programs. This report documents the constructs measured in studies of 36 randomly selected programs. It also includes a framework for systematically describing and assessing measures of character education outcomes and provides a resource for evaluators to help identify and select measures of outcomes for character education programs. Among student level outcomes, those most often measured were: academic content (measured for 14 programs), prosocial dispositions and interpersonal strengths (each measured for 11 programs), discipline issues and interpersonal competencies (each measured for 13 programs), and substance use and intrapersonal competencies (each measured for 11 programs).
• “Using State Tests in Education Experiments: A Discussion of the Issues.” Henry May, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason, November 2009. Securing data on students’ academic achievement is typically one of the most important and costly aspects of conducting education experiments. As state assessment programs have become practically universal and more uniform in terms of grades and subjects tested, the relative appeal of using state tests as a source of study outcome measures has grown. However, the variation in state assessments—in both content and proficiency standards—complicates decisions about whether a particular state test is suitable for research purposes and poses difficulties when planning to combine results across multiple states or grades. This paper aims to help researchers evaluate and make decisions about whether and how to use state test data in education experiments. It outlines the issues that researchers should consider, including how to evaluate the validity and reliability of state tests relative to study purposes; factors influencing the feasibility of collecting state test data; how to analyze state test scores; and whether to combine results based on different tests. It also highlights best practices to help inform ongoing and future experimental studies. Many of the issues discussed are also relevant for nonexperimental studies.
• New Leaders for New Schools, a nonprofit organization committed to training school principals, heads the Effective Practice Incentive Community (EPIC), an initiative that offers financial awards to effective educators. Two new reports present methods for identifying effective EPIC schools and teachers using value-added models:
• "Measuring School Effectiveness in Memphis: Year 2." Liz Potamites, Duncan Chaplin, Eric Isenberg, and Kevin Booker, October 2009. This report presents the methods used to identify effective schools in the Memphis City Schools during the second year of the project.
• "Measuring School and Teacher Effectiveness in the EPIC Charter School Consortium: Year 2." Liz Potamites, Kevin Booker, Duncan Chaplin, and Eric Isenberg. October 2009. This report presents the methods used to identify effective schools and teachers for a consortium of over 140 charter schools in 17 states and the District of Columbia during the second year of the project.

Journal Articles

• "An Experimental Evaluation of Four Elementary School Math Curricula." Roberto Agodini and Barbara Harris, Journal of Research on Educational Effectiveness, July 2010 (subscription required). This article examines the effectiveness of four distinct elementary school math curricula: Investigations in Number, Data, and Space^®; Math Expressions; Saxon Math^™; and Scott Foresman-Addison Wesley Mathematics (SFAW). An average-performing first-grader's percentile rank in math would be 9 to 12 points higher if the school used Math Expressions or Saxon^™, instead of Investigations^® or SFAW.
• "An Experimental Assessment of the Effects of Reading Software Products on Reading Test Scores." Mark Dynarski, Roberto Agodini, and Larissa Campuzano, Effective Education (subscription required), March 2010. This study evaluated the effectiveness of nine reading instructional software applications—five for first graders and four for fourth graders. Using standardized tests, classroom observation, and other school records data, findings indicated the software altered instructional activities but did not significantly impact test scores. An additional year of followup yielded mixed results, with measured effects being lower in the first grade and higher in the fourth grade.
• “How Accountability Pressure on Failing Schools Affects Student Achievement.” Hanley Chiang, Journal of Public Economics, October 2009. Although the threat of sanctions on low-performing schools can raise student test scores in the short run, the extent to which these test score improvements are due to schools' manipulation of the accountability system has remained uncertain. This article evaluates the relative importance of educational reforms and gaming behavior in generating test score gains by threatened schools. The author uses a regression discontinuity design to estimate medium-run effects on student test scores from having attended a threatened elementary school. Threat-induced math improvements from elementary school largely persist at least through the first one to two years of middle school, while evidence for persistence of reading improvements is less consistent. Sanction threats tend to raise school spending on instructional technology, curricular development, and teacher training.
• "Statistical Power for Random Assignment Evaluations of Education Programs." Peter Z. Schochet, Journal of Educational and Behavioral Statistics, March 2008. This article examines theoretical and empirical issues related to the statistical power of impact estimates for experimental evaluations of education programs. The author considers designs where random assignment is conducted at the school, classroom, or student level, and employs a unified analytic framework using statistical methods from the literature. Focusing on standardized test scores of elementary school students, this article discusses appropriate precision standards and, for each design, the required number of schools to achieve those standards using empirical values of intraclass correlations, regression R2 values, and other parameters. Clustering effects vary by design but are typically large. As a result, large school samples are required for education trials, and many evaluations will have sufficient power to detect precise impacts only for relatively large subgroups of sites.
• “After-School Program Effects on Behavior: Results from the 21st Century Community Learning Centers Program National Evaluation.” Susanne James-Burdumy, Mark Dynarski, and John Deke, Economic Inquiry, January 2008. This paper presents evidence on after-school programs’ effects on behavior from the national evaluation of the U.S. Department of Education’s 21st Century Community Learning Centers after-school program. Findings come from both of the study’s components: (1) an elementary-school component based on random assignment of 2,308 students in 12 school districts, and (2) a middle-school component based on a matched comparison design including 4,264 students in 32 districts. Key findings include higher levels of negative behavior for elementary students and some evidence of higher levels of negative behaviors for middle school students.
• "Academic Achievement and School Functioning Among Non-Incarcerated Youth Involved with the Juvenile Justice System." Jonathan D. Brown, Anne W. Riley, Christine M. Walrath, Philip J. Leaf, and Carmen Valdez, Journal of Education for Students Placed at Risk, January-March 2008. This article reports on the education problems of youth involved with the juvenile justice system but not incarcerated. More than half demonstrated deficits in academic functioning, with standard achievement scores as low as five standard deviations below the normative mean. Non-Caucasian youth and those who received special education services or lived in an urban area had lower achievement. These findings suggest that youth involved with the justice system but not incarcerated demonstrate problems in academic achievement similar to incarcerated youth and may benefit from targeted education interventions. 

Issue Briefs

• "Infusing Academics into Career and Technical Education." Joshua Haimson, James R. Stone, III, and Donna Pearson, Trends in Education Research, Issue Brief #3, December 2008. Integrating academic learning into career and technical education (CTE) classes can be challenging for educators and curriculum developers but can be aided by securing detailed feedback from CTE teachers. Drawing on a recent study, this issue brief identifies challenges developers faced in infusing more math into CTE curricula and notes that incorporating academic learning into CTE requires substantial time, effort, and other resources.

Other

• "Good Things Come in Small Packages: Lessons Learned from the Ohio High School Transformation Initiative." Sarah Jones, Monica Martinez, and Cindy Cai. In Saving America's High Schools, edited by Becky A. Smerdon and Kathryn M. Borman, November 2009. The American educational system is in a continuous state of reform, yet outcomes are less than satisfactory. This chapter reports on a small reform initiative in Ohio showing that, while small schools have benefits, overall, the structure needs to be rethought to maximize resources, meet all education requirements set by the state, while simultaneously providing students with diverse options.
• What Works Clearinghouse (WWC)
• Administered by Mathematica for the U.S. Department of Education's Institute of Education Sciences, the WWC produces a variety of reports that assess and summarize education research. WWC reports can help educators make important decisions about what curriculums to implement, what products to purchase, and what methods to use in their classrooms and schools.
  
  See the WWC's latest releases at whatworks.ed.gov and explore available Practice Guides, Intervention Reports, and Quick Reviews or take a guided tour of the site.