Accurate uncertainty quantification in Bayesian calibration accounting for correlated targets
SMDM 47th Annual Meeting
The Problem
Model calibration using a likelihood on multiple targets often assumes those targets are independent when constructing the overall goodness-of-fit measure. But calibration targets – such as survival, prevalence, and disease proportions over time – are inherently correlated. When this independence assumption is violated, calibration underestimates parameter uncertainty and, consequently, the uncertainty in model-predicted outcomes.
The Approach
We calibrated a Sick-Sicker cohort Markov model to three time-dependent targets – survival, disease prevalence, and the proportion Sick of the total Sick-or-Sicker cohort – using four approaches to compute the overall likelihood. First, using all target observations assuming independence. Second, selecting a subset of observations based on the effective sample size (ESS) of each target treated as a time series. Third, constructing a correlated likelihood assuming multivariate normality with a covariance matrix estimated from an autocorrelated error of order 1. Fourth, estimating sufficient statistics for each target and using those as calibration targets instead. All approaches used the IMIS algorithm to draw 1,000 parameter sets from the joint posterior distribution.
What We Found
4 Likelihood approaches compared
3 Correlated calibration targets
1,000 Posterior parameter sets via IMIS
Assuming independence underestimated parameter and model-predicted uncertainty. All three methods that account for correlation produced wider, more realistic uncertainty bounds, with the multivariate normal (MVN) approach producing the widest bounds – even wider than the target data itself. ESS and sufficient statistics produced similar, well-calibrated uncertainty bounds that fell within target uncertainty, but ESS requires deciding which observations to include, while sufficient statistics avoids that arbitrary choice entirely.
Why It Matters
Using sufficient statistics from the target data as calibration targets quantifies uncertainty more accurately – without incorrectly assuming independence, picking an arbitrary subset of observations, or assuming a specific correlation structure. This provides a principled, practical solution for any health policy model calibrated to time-series targets.