Case Study: Comparing the Predictive Validity of High-Stakes Standardized Tests

The Massachusetts Executive Office of Education was responsible for recruiting the study sample and administering the tests. The study sample consisted of 847 first-year college students who had graduated from Massachusetts high schools and then enrolled at one of 11 public in-state campuses that participated in the study. Students who volunteered to take part were randomly assigned to complete one component of either the MCAS or PARCC exams, ensuring that students taking the PARCC assessment were not systematically different from those taking the MCAS. Mathematica Policy Research analyzed the resulting data and explained results of the study to state policymakers.

The results of this study fueled an intense debate. Although some stakeholders believed the PARCC was a more effective tool than the MCAS at measuring students’ college readiness, the study showed no difference between the new test and the long-standing assessment. Mathematica’s study revealed that both exams succeeded in predicting college readiness.

Key findings include the following:

• MCAS and PARCC scores are comparable to Scholastic Assessment Test scores in terms of predicting college outcomes.
• The overall validity of scores on PARCC assessments in predicting college grades is similar to the validity of scores on the MCAS. In both math and English language arts, the correlation between PARCC scores and college outcomes was statistically indistinguishable from the correlation between MCAS scores and college outcomes.
• Meeting the PARCC standard for college readiness in math predicted a higher level of college performance than meeting the MCAS standard for math proficiency. Students meeting the PARCC math standard were also found to be less likely to need remediation than those meeting the standard for the MCAS.
• In English language arts, however, there were no statistically significant differences between the level of college performance predicted by the PARCC standard for college readiness and the MCAS proficiency standard.
• Because the underlying scores on the MCAS and PARCC assessments are equally predictive of college outcomes, the study revealed that Massachusetts policymakers had more than one way to align high school mathematics test standards with college readiness: either adopt the PARCC exam or continue using MCAS while simply setting a higher score threshold for college readiness. Either of these options would ensure that the state’s high school assessments provide better information about college readiness to students, parents, educators, and policymakers.
• Even though we cannot directly compare the assessments of other states with the PARCC, the study provides useful evidence for any state considering adopting PARCC assessments. Indeed, in many other states, the difference between existing proficiency standards and those of PARCC is likely to be substantially larger than in Massachusetts, where proficiency standards were already well above those of most states.

This case study is for informational purposes only. Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design  and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.