Evaluation of heterogeneity and heterogeneity interval estimators in random-effects meta-analysis of the standardized mean difference in education and psychology.

Meta-analyses are conducted to synthesize the quantitative results of related studies. The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population. This distribution is often assumed to be normal with a mean and variance. The population variance, also called heterogeneity, can be estimated numerous ways. Research exists comparing subsets of heterogeneity estimators over limited conditions. Additionally, heterogeneity is a parameter estimated with uncertainty. Various methods exist for heterogeneity interval estimation, and similar to heterogeneity estimators, these evaluations are limited. The current simulation study examined the performance of Bayesian (with 5 prior specifications) and non-Bayesian estimators over conditions found after a review of meta-analyses of the standardized mean difference in education and psychology research. Three simulation conditions were varied: (a) number of effect sizes per meta-analysis, (b) true heterogeneity, and (c) sample size per effect size within each meta-analysis. Estimators were evaluated over average bias and means square error. Methods of interval estimation were then evaluated with the estimators found to operate optimally. Interval estimators were evaluated based on coverage probability, interval width, and coverage of the estimated value. Overall, the Paule and Mandel estimator, in conjunction with the Jackson method of interval estimation, is recommended if no knowledge exists with regards to the expected value of heterogeneity when synthesizing the standardized mean difference effect size. If heterogeneity is expected to be small (e.g., < .075), then REML with the profile likelihood interval estimator is recommended. Sensitivity analysis evaluating differences in substantive conclusions with a suite of heterogeneity estimators, such as Paule and Mandel, REML, and Hedges and Olkin, is recommended. (PsycInfo Database Record (c) 2020 APA, all rights reserved).