Bayesian Interpretation of Cluster-Robust Subgroup Impact Estimates: The Best of Both Worlds

Policymakers are often interested in understanding the impact of an intervention on specific subgroups, not just an overall population. But analyzing subgroup impacts poses challenges. Subgroup estimates are noisier than whole population estimates due to smaller sample sizes. In addition, within the null hypothesis significance testing framework, the chance of a statistically significant impact estimate—where in truth the policy has no meaningful impact—grows rapidly with the number of subgroups considered. Hierarchical Bayesian models address these problems by using partial pooling of information between subgroups to make precise estimates of effects even in smaller subgroups and to correct for the multiple comparisons problem in a data driven way. However, Bayesian models can be computationally infeasible with large data. We propose a “best of both worlds” hybrid approach that combines the low computational cost of fitting non-Bayesian models with the interpretability and precision of Bayesian models. We use a health policy simulation to show that, compared to its non-Bayesian counterpart, this hybrid approach produces more precise estimates and more accurately estimates the probability of favorable subgroup impacts, both of which can lead to healthcare cost savings in a plausible policy scenario with small heterogeneous impacts.