Search tips
Search criteria 


Logo of aapsjspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
AAPS J. 2005 September; 7(3): E523–E531.
Published online 2005 October 7. doi:  10.1208/aapsj070353
PMCID: PMC2751255

A new equivalence based metric for predictive check to qualify mixed-effects models


The main objective of any modeling exercise is to provide a rationale for effective decision making during drug development. The aim of the current simulation experiment was to evaluate the properties of predictive check as a covariate model qualification technique and, more importantly, to introduce and evaluate alternative criteria to qualify models.

Original concentration-time profiles (yod) were simulated using a 1-compartment model for an intravenous drug administered to 25 men and 25 women. The typical clearance for male subjects (TVCLm) was assumed to be 5-fold higher than that for female subjects (TVCLf). Fifty such trials under the same design were generated randomly. Predictive check was used as the model qualification tool to study predictive performance of true (males ≠ females) and false (males=females) models in the context of maximum likelihood estimation. For each yod, 200 replications were generated to study the properties of a discrepancy variable, a statistic that depends on the model, and a test statistic, a statistic that does not depend on the model. Several qualification criteria were evaluated in assessing predictive performance, such as, predictive p-value (Pp), probability of equivalence (peqv), and probability of rejecting the null hypothesis (data=model) using the Kolmogorov-Smirnov test (pks).

The Pp value was calculated using sum of squared errors as a discrepancy variable. For both of the models, the Pp values uniformly ranged between 0 and 1. The pattern of Pp values suggests that qualification of the false model is unlikely. For both of the models, the range of peqv is about 0.95 to 1.0 for concentration at 0.5 hours. However, this is not the case for the concentration at 4 hours, which is primarily dependent on the clearance. The false model (0.35 to 0.50) has poor predictive performance compared with the true model (0.65 to 0.80) using peqv. The pks suggests no difference in the distributions of replicated and original concentrations at all of the time points for both of the models.

Discrepancy variables cannot aid in rejecting false models, whereas the use of a test statistic can aid in rejecting false models. However, selection of an informative test statistic is challenging. As far as the qualification criteria are considered, the equivalence-based comparison of a test statistic is more informative than a significance-based comparison. No convincing evidence exists in the literature demonstrating the added advantages of predictive check as a routine model qualification tool over the existing tools, such as diagnostic plots or mechanistic reasoning. However, when a model is to be used for designing a trial, it should at least be able to regenerate the data used to build the model. In such cases, predictive check might offer insights into potential inconsistencies.

Keywords: model validation, model qualification, posterior predictive check, test statistic, discrepancy variable

Full Text

The Full Text of this article is available as a PDF (311K).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
1. Sheiner LB, Steimer JL. Pharmacokinetic/pharmacodynamic modeling in drug development. Annu Rev Pharmacol Toxicol. 2000;40:67–95. doi: 10.1146/annurev.pharmtox.40.1.67. [PubMed] [Cross Ref]
2. Gobburu JV, Marroum PJ. Utilisation of pharmacokinetic-pharmacodynamic modelling and simulation in regulatory decision-making. Clin Pharmacokinet. 2001;40:883–892. doi: 10.2165/00003088-200140120-00001. [PubMed] [Cross Ref]
3. Holford NHG, Hale M, Ko HS, Steimer JL, Sheiner LB. Simulation in Drug development: good practices.Center for Drug Development Science. Available at: Accessed January 1, 2004.
4. Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2001;28:231–252. doi: 10.1023/A:1011527125570. [PubMed] [Cross Ref]
5. FDA. Guidance: Population Pharmacokinetics.FDA guidance page. Available at: Accessed January 13, 2004.
6. Rubin D. Bayesianly justifiable and relevant frequency calculations for the applied statisitcian. Ann Stat. 1984;12:1151–1172. doi: 10.1214/aos/1176346785. [Cross Ref]
7. Gelman A, Carlin JB, Stern HS, Rubin DB. Model checking and sensitivity analysis. In: Gelman A, editor. Bayesian Data Analysis. London: Chapman & Hall; 1995. pp. 161–189.
8. Gelman A, Meng XL. Model checking and model improvement. In: Gilks WR, Richardson S, Spiegelhalter DJ, editors. Markov Chain and Monte Carlo in Practice. London: Chapman & Hall; 1996. pp. 189–202.
9. Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001;28:171–192. doi: 10.1023/A:1011555016423. [PubMed] [Cross Ref]
10. Gelman A, Meng XL, Stern HS. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica. 1996;6:733–807.
11. Guttman I. The use of a concept of a future observation in the goodness-of-fit problems. J R Statist Soc B. 1967;1:83–100.
12. Tiao GC, Xu D. Robustness of maximum likelihood estimates for multi-step predictions: the exponential smoothing case. Biometrika. 1993;80:623–641. doi: 10.1093/biomet/80.3.623. [Cross Ref]
13. Gelman A, Goegebeur Y, Tuerlinckx F, Van Mechelen I. Diagnostic checks for discrete data regression models using posterior predictive simulations. J Royal Stat Soc: C. 2000;49:247–268. doi: 10.1111/1467-9876.00190. [Cross Ref]
14. Duffull SB, Aarons L. Development of a sequential linked pharmacokinetic and pharmacodynamic simulation model for ivabradine in healthy volunteers. Eur J Pharm Sci. 2000;10:275–284. doi: 10.1016/S0928-0987(00)00085-3. [PubMed] [Cross Ref]
15. Duffull SB, Chabaud S, Nony P, Laveille C, Girard P, Aarons L. A pharmacokinetic simulation model for ivabradine in healthy volunteers. Eur J Pharm Sci. 2000;10:285–294. doi: 10.1016/S0928-0987(00)00086-5. [PubMed] [Cross Ref]
16. Friberg LE, Freijs A, Sandstrom M, Karlsson MO. Semiphysiological model for the time course of leukocytes after varying schedules of 5-fluorouracil in rats. J Pharmacol Exp Ther. 2000;295:734–740. [PubMed]

Articles from The AAPS Journal are provided here courtesy of American Association of Pharmaceutical Scientists