Case Study: Our Charter School Research: Providing an Objective Voice in the Debate
The report’s conclusion that 'not all charter schools are the same' won’t surprise many who follow the debate. What’s interesting is what the report tells us about which charter schools work better.
Mathematica’s evaluation was the first large-scale randomized trial of the effectiveness of charter schools in several states.
Mathematica’s charter school report was downloaded over 9,000 times and cited nearly 65 times in peer-reviewed research.
The study received widespread media coverage in outlets such as the Wall Street Journal and Education Week, and results have been included in peer-reviewed journals such as Economics of Education Review as well as a meta-analysis of the literature on The Effects of Charter Schools on Student Achievement (Betts and Tang 2011).
Notes on Data Collection. The evaluation drew from six sources of data: (1) a baseline survey of parents, (2) administrative records on student test scores and demographics, (3) a parent survey, (4) a student survey, (5) a principal survey, and (6) a survey of charter school authorizers. Charter middle schools that held admissions lotteries for the 2005-2006 and 2006-2007 school years were recruited to participate. The study sample included 2,330 students who were followed for two years after the lottery. So that other researchers can take advantage of the rich survey and test score data collected for this study, Mathematica made a restricted-use data file available through the U.S. Department of Education.
Charter schools are a central, yet controversial, component of today’s school reform movement. Although publicly financed, charters lack many of traditional public schools’ regulations involving staffing, curriculum, and budgeting.
In 2011-2012, more than 5,700 charter schools served 1.9 million students—about 4 percent of all public school students—in 40 states and the District of Columbia (Center for Education Reform, 2014).
Despite this growth and increasing policy emphasis on charter schools, sound research on the charter school puzzle has been scarce.
Mathematica’s study, the most comprehensive and geographically diverse using charter school lotteries to date, provided rigorous evidence for patterns suggested by existing research, which estimated negative or insignificant impacts for geographically diverse samples of charter schools, but positive impacts for charter schools in urban areas.
Mathematica set the bar for conducting future research on lottery-based charter school admissions by employing “best practices” and ensuring true random assignment. We have used this methodology in other studies of charter schools to improve quality and ensure objective results including our evaluations of the Knowledge Is Power Program, Charter Management Organizations, the Equity Project Charter School, and Expeditionary Learning Schools.
The study found that, overall, charter school attendance had negative, but not statistically significant, impacts on student achievement. Charter school attendance lowered student achievement in reading and math by between 0.06 and 0.08 standard deviations in the two school years immediately following the lottery. According to estimates from Hill et al. (2007), this translates to approximately one-quarter-year less instruction for students in charter schools per year. In addition:
- Study charter schools were more effective for lower income and lower achieving students and less effective for higher income and higher achieving students.
- Charter schools in large urban areas had positive impacts on students’ achievement in math; those outside these large urban areas had negative impacts on achievement
- These results contributed to the debate and suggested that policymakers might want to focus charter middle schools in poor urban areas, where they may be more effective in raising student achievement.
This case study is for informational purposes only. Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.
In 2004, the U.S. Department of Education’s Institute of Education Sciences commissioned Mathematica to conduct the first nationwide, lottery-based study of charter schools.
The lottery-based design compared outcomes for applicants admitted to the charter middle school through the lottery to outcomes for applicants participating in the lottery who did not get a winning lottery draw and so were not admitted.
To provide rigorous evidence of charter schools’ effectiveness, the study included a large and geographically diverse sample—36 charter middle schools in 15 states.
One limitation of other lottery-based studies of charter schools has been inadequate documentation and questions about the conduct of the lottery process, which undermined confidence in the results. Mathematica carefully monitored charter school admissions lotteries in person, helping to ensure that the lottery procedures, and students’ resulting admission status, were truly random, and that the study results were as rigorous as possible.