Using QALYs as an outcome for assessing global prediction accuracy in diabetes simulation models

Objectives: (1) To demonstrate the use of quality-adjusted life-years (QALYs) as an outcome measure for comparing performance between simulation models and identifying the most accurate model for economic evaluation and health technology assessment. QALYs relate directly to decision-making and combine mortality and diverse clinical events into a single measure using evidence-based weights that reflect population preferences. (2) To explore the usefulness of Q2 (Q-squared), the proportional reduction in error, as a model performance metric and compare it with other metrics: MSE; mean absolute error; bias (mean residual); and R2. Methods: We simulated all EXSCEL trial participants (n=14,729) using the UK Prospective Diabetes Study Outcomes Model software versions 1 (UKPDS-OM1) and 2 (UKPDS-OM2). The EXSCEL trial compared once-weekly exenatide with placebo (median 3.2 years’ follow-up). Default UKPDS-OM2 utilities were used to estimate undiscounted QALYs over the trial period based on the observed events and survival. These were compared with the QALYs predicted by UKPDS-OM1/2 for the same period. Results: UKPDS-OM2 predicted patients’ QALYs more accurately than UKPDS-OM1 (MSE: 0.210 vs. 0.253; Q2: 0.822 vs. 0.786). UKPDS-OM2 underestimated QALYs by an average of 0.127, vs. 0.150 for UKPDS-OM1. UKPDS-OM2 predictions were more accurate for mortality, myocardial infarction and stroke, while UKPDS-OM1 better predicted blindness and heart disease. Q2 facilitated comparisons between subgroups and (unlike R2) was lower for biased predictors. Conclusions: Q2 for QALYs was useful for comparing global prediction accuracy (across all clinical events) of diabetes models. It could be used for model registries, choosing between simulation models for economic evaluation and evaluating the impact of recalibration. Similar methods could be used in other disease areas.