Spatiotemporal Forecasting of Opioid-Related Fatal Overdoses: Towards Best Practices for Modeling and Evaluation.

To inform public health interventions, researchers have developed models to forecast opioid-related overdose mortality. These efforts often have limited overlap in the models and datasets employed, presenting challenges to assessing progress in this field. Furthermore, common error-based performance metrics, such as root mean squared error (RMSE), cannot directly assess a key modeling purpose: the identification of priority areas for interventions. We recommend a new intervention-aware performance metric, Percentage of Best Possible Reach (%BPR). We compare metrics for many published models across two distinct geographic settings, Cook County, Illinois and Massachusetts, assuming the budget to intervene in 100 census tracts out of 1000s in each setting. The top-performing models based on RMSE recommend areas that do not always reach the most possible overdose events. In Massachusetts, the top models preferred by %BPR could have reached 18 additional fatal overdoses per year in 2020-2021 compared to models favored by RMSE. In Cook County, the different metrics select similar top-performing models, yet other models with similar RMSE can have significant variation in %BPR. We further find that simple models often perform as well as recently published ones. We release open code and data for others to build upon.