Guidelines for Multiple Testing in Impact Evaluations of Educational Interventions

Statistical procedures that correct for multiple testing typically result in hypothesis tests with reduced statistical power because adjustment methods reduce the likelihood of identifying real differences between contrasted groups. There is disagreement among researchers about the use of multiple testing procedures and the appropriate trade-off between type I error and statistical power (type II error). These guidelines were developed to handle multiple testing in education research. In addition, the report provides details on the nature of the multiple testing problem and the statistical solutions that have been proposed; the creation of composite outcomes measures; and the Bayesian hypothesis testing approach.