Bayesian research methods empower decision makers to discover what works by putting new research findings in context alongside an existing evidence base, which ultimately leads to stronger confidence in their conclusions. This approach also strengthens transparency, objectivity, and equity.
Using Bayesian methods to better understand what works
For decades, researchers have relied on statistical significance tests as the basis for answering questions about the effectiveness of various policy programs and interventions. Because researchers believed that statistically significant findings are unlikely to be due to random chance, statistically significant findings were much more likely to be published in journals, reported in the media, and used to make important real-world decisions.
But statistical significance does not actually mean what people believed it meant. In 2016, the American Statistical Association (ASA) released an unprecedented statement about the misuse of statistical significance, asserting that scientific conclusions and policy decisions should not be based only on whether a p-value—or a measure of statistical significance—passes a specific threshold. For example, negative side effects of a drug could be ignored because their p-value is a little larger than 0.05, which just misses the statistical significance threshold. An article in the scientific journal Nature, with over 800 signatories, argued for retiring statistical significance altogether. Statistical significance is too often easily misinterpreted to mean either “the intervention worked” or “the intervention did not work.” This artificially binary approach to interpreting study findings can mean that potential positive impacts that don't meet the threshold for being statistically significant are ignored.
BAyeSian Interpretation of Estimates (BASIE)
BASIE is a framework for using an evidence-based Bayesian approach to interpret traditional (non-Bayesian) impact estimates. BASIE, first introduced by Mathematica experts John Deke and Mariel Finucane, represents a substantial improvement over statistical significance. BASIE draws on prior evidence regarding the effectiveness of previously evaluated interventions to assess the probability that an intervention worked. In 2020, Deke and Finucane recorded a webinar on Bayesian Interpretation of Impact Estimates from Education Evaluations for the annual conference of the Society for Research on Educational Effectiveness. Focusing on education interventions, they provided step-by-step instructions, easy-to-use tools and code, essential methodological details for researchers who want to use BASIE to interpret estimates from impact studies, and other resources.
A Paradigm Shift to Bayesian Methods Could Improve Evidence-Based Decision Making
On the latest episode of On the Evidence, Mariel Finucane, John Deke, and Tim Day, of the Center for Medicare & Medicaid Innovation, discuss why there’s growing demand for alternative ways of assessing the impact of policies, and how a modernized, evidence-informed update to Bayes’s Rule can help decision makers assess whether a policy or program works and whether it would work for certain groups of interest.
Getting more accessible and actionable information to those who need it
Imagine you’re a leader in your state’s human services agency, or a legislator in charge of investing resources in social services. With a limited budget, you’re eager to focus on programs that have a track record of working for the people you’re trying to reach. There’s an existing body of research, but it’s hard to interpret because studies vary both in quality and conclusions. Applying a Bayesian approach, we’re able to distill the existing evidence into a bottom-line assessment of the probability that something works.
- We used a Bayesian model to assess how a health initiative known as Comprehensive Primary Care (CPC) impacted Medicare expenditures and health outcomes. The CPC initiative was a unique collaboration between the Centers for Medicare and Medicaid Services (CMS) and other private and public payers, including commercial insurers and Medicaid managed care plans, to help overcome barriers to better primary care. The research team compared the change in expenditures using both Bayesian and traditional models. Bayesian methods offer several benefits not found within traditional research methods, including more precise findings that incorporate prior evidence, adjusting for multiple comparisons, and framing results in plain language about the probability that the initiative would actually reduce Medicare expenditures.
- Our Employment Strategies for Low-Income Adults Evidence Review (ESER) used a Bayesian framework to help HHS understand interventions that work best to improve employment outcomes and determine which are most effective under specific conditions and for which types of workers. Under more traditional research methods, the ESER database included few instances of potentially informative combinations of strategies and populations, so researchers were less able to determine precise and compelling conclusions using this approach. The Bayesian approach strengthens analysis of combinations of study features—outcomes, strategies, and populations—that appear only rarely in the data. Using Bayesian methods, we found that several interventions were effective at improving low-income adults’ labor market outcomes, including 10 that were highly likely to improve labor market outcomes by at least 5 percent.
- Our research team used Bayesian estimates based on existing studies about the influence of education on civic engagement to measure the impact of a charter school program known as Democracy Prep on helping students become more active citizens (for example, visiting legislators, attending public meetings, and discussing literature on citizen engagement). We found a 98 percent probability that enrolling in Democracy Prep produced a positive impact on voter registration, and a 98 percent probability that enrolling produced a positive impact on voting in the 2016 election.
Bayesian methods better support more equitable evaluation
There is an urgent need to better understand the effect programs and policies have on equity outcomes. Incorporating more equitable evaluation principles into social policy research helps ensure that all community stakeholders—including those who operate and participate in programs—have the chance to contribute to and benefit from the evaluation. Bayesian methods make equitable evaluation more likely by examining a wider range of outcomes for a greater diversity of communities. They can solve the problem of multiple comparisons (that is, examining multiple outcomes in the same analysis) without imposing artificial limits on what we can learn. We can also support community stakeholders’ suggestions to look at a broader set of outcomes or subgroups than originally specified in a study’s design.
These methods can also strengthen subgroup analyses by using information from related subgroups to increase the precision of each subgroup impact estimate. They challenge traditional research methods that are heavily reliant on larger subgroup sample sizes, which can fail to represent certain groups.