Statistical Modelling 5 (2005), 95–118
The practical utility of incorporating model selection uncertainty into
prognostic models for survival data
Nicole Augustin
Department of Statistics,
University of Glasgow,
Glasgow
UK
Willi Sauerbrei
Institut für Medizinische Biometrie und Medizinische Informatik,
Universitätsklinikum Freiburg,
Stefan-Meier-Str. 26,
D–79104 Freiburg
Germany
eMail:
wfs@imbi.uni-freiburg.de
Martin Schumacher
Institut für Medizinische Biometrie und Medizinische Informatik,
Universitätsklinikum Freiburg,
Freiburg
Germany
Abstract:
Predictions of disease outcome in prognostic factor models are usually
based on one selected model. However, often several models fit the
data equally well, but these models might differ substantially in
terms of included explanatory variables and might lead to different
predictions for individual patients. For survival data, we discuss
two approaches to account for model selection uncertainty in two data
examples, with the main emphasis on variable selection in a
proportional hazard Cox model. The main aim of our investigation is
to establish the ways in which either of the two approaches is useful
in such prognostic models. The first approach is Bayesian model averaging
(BMA) adapted for the proportional hazard model, termed 'approx. BMA' here.
As a new approach, we propose a method which averages over a set of
possible models using weights estimated from bootstrap resampling as
proposed by Buckland et al., but in addition, we perform an initial
screening of variables based on the inclusion frequency of each variable
to reduce the set of variables and corresponding models. For some necessary
parameters of the procedure, investigations concerning sensible choices are
still required. The main objective of prognostic models is prediction, but
the interpretation of single effects is also important and models should
be general enough to ensure transportability to other clinical centres.
In the data examples, we compare predictions of our new approach with
approx. BMA, with 'conventional' predictions from one selected model and
with predictions from the full model. Confidence intervals are compared
in one example. Comparisons are based on the partial predictive score
and the Brier score. We conclude that the two model averaging methods
yield similar results and are especially useful when there is a high
number of potential prognostic factors, most likely some of them without
influence in a multivariable context. Although the method based on bootstrap
resampling lacks formal justification and requires some ad hoc decisions,
it has the additional positive effect of achieving model parsimony by
reducing the number of explanatory variables and dealing with correlated
variables in an automatic fashion.
Keywords:
BOOTSTRAP; MODEL AVERAGING; MODEL SELECTION UNCERTAINTY;
PROGNOSTIC FACTOR MODELS; SURVIVAL ANALYSIS
back