Home | About | Journals | Submit | Contact Us | Français |

**|**BMC Med Res Methodol**|**v.10; 2010**|**PMC2944306

Formats

Article sections

- Abstract
- Background
- Methods
- Results
- Discussion
- Conclusions
- Abbreviations
- Competing interests
- Authors' contributions
- Pre-publication history
- References

Authors

Related links

BMC Med Res Methodol. 2010; 10: 79.

Published online 2010 September 3. doi: 10.1186/1471-2288-10-79

PMCID: PMC2944306

Noémie Soullier,^{1,}^{2,}^{3} Elise de La Rochebrochard,^{1,}^{2,}^{3} and Jean Bouyer^{}^{1,}^{2,}^{3}

Noémie Soullier: moc.liamg@reilluos.eimeon; Elise de La Rochebrochard: rf.deni@ehcor; Jean Bouyer: rf.mresni@reyuob.naej

Received 2010 March 26; Accepted 2010 September 3.

Copyright ©2010 Soullier et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

This article has been cited by other articles in PMC.

In longitudinal cohort studies, subjects may be lost to follow-up at any time during the study. This leads to attrition and thus to a risk of inaccurate and biased estimations. The purpose of this paper is to show how multiple imputation can take advantage of all the information collected during follow-up in order to estimate the cumulative probability *P(E) *of an event *E*, when the first occurrence of this event is observed at *t *successive time points of a longitudinal study with attrition.

We compared the performance of multiple imputation with that of Kaplan-Meier estimation in several simulated attrition scenarios.

In missing-completely-at-random scenarios, the multiple imputation and Kaplan-Meier methods performed well in terms of bias (less than 1%) and coverage rate (range = [94.4%; 95.8%]). In missing-at-random scenarios, the Kaplan-Meier method was associated with a bias ranging from -5.1% to 7.0% and with a very poor coverage rate (as low as 0.2%). Multiple imputation performed much better in this situation (bias <2%, coverage rate >83.4%).

Multiple imputation shows promise for estimation of an occurrence rate in cohorts with attrition. This study is a first step towards defining appropriate use of multiple imputation in longitudinal studies.

Longitudinal studies are the most appropriate way of analysing the occurrence of an event *E *and of estimating its cumulative probability during follow-up *P(E)*. One of the most important drawbacks of such studies is that subjects may be lost to follow-up at any time point. These drop-outs accumulate over time and lead to attrition of the initial cohort, which has fewer and fewer participants as time goes by. Attrition can lead to biased and inaccurate estimations of *P(E) *[1], especially in classical complete-case analyses [2,3]. For example, in a UK cohort study of ocular outcome after premature birth, which involved 558 children born before 32 weeks of gestation in 1990-1991, abnormalities were more frequent among drop-outs, and complete-case analysis thus underestimated the prevalence of ocular abnormalities in the study population [4].

Survival analysis and Kaplan-Meier estimation are commonly used to examine time-to-event measurements [5,6]. This approach takes into account the fact that subjects are followed for different lengths of time. However, it assumes that censored patients (including patients lost to follow-up) would have the same probability of experiencing a subsequent event as non-censored patients. This assumption has been challenged in various fields [4,7,8]. One specific example, which motivated the present study, is the issue of *in vitro *fertilization (IVF) success rates [9]. In IVF cohorts, drop-out corresponds to treatment interruption, which is partly linked to a poor likelihood of success [10]. Couples who drop out therefore have a lower chance of success than couples who persevere [11]. The Kaplan-Meier approach tends to overestimate the IVF success rate, and alternative methods have recently been proposed: drop-outs are divided into two groups depending on the chances of success (poor/not poor) [12], or, equivalently, according to whether IVF treatment is interrupted for medical causes or not [13]. Couples with a poor chance of success will be considered as having a zero probability of subsequent success, whereas those with a good prognosis will have the same chance of subsequent success as those who persevere. This method makes it possible to take into account auxiliary information on the probability of success among drop-outs, through individual prognostic factors, considered in two groups. This idea may be developed by looking for ways of taking into account more of this auxiliary information. Multiple imputation is a good candidate approach in this setting.

Multiple imputation was developed thirty years ago [14] and is now used in observational as well as in randomized studies [15,16]. However, epidemiological applications have been limited [17]. Previous works have addressed the use of multiple imputation in longitudinal studies [18-20], but few focused on estimation of the occurrence rate at the end of the follow-up. This is a special issue, as it implies imputation of the missing covariates and outcomes. Hsu et al. developed a weighted Kaplan-Meier method to take into account dependant interval censoring in the estimation of a recurrence rate [21,22]. Our objective is similar. However, we preferred to use multiple imputation, which consists in replacing missing values with a set of plausible values, based on auxiliary information and which has the advantage of being routinely implemented and easily applicable. Moreover, IVF studies have a particular way of defining time, as each time point of the study is an IVF attempt. The length of time elapsing between the attempts is not considered. The method of Hsu et al. implied imputing midpoints of intervals and is therefore not appropriate here. Kaplan-Meier estimation was chosen for comparison with multiple imputation, because this method is often used to estimate cumulative success rates in IVF.

The purpose of this paper is to show how multiple imputation can take advantage of all information collected during follow-up. It focuses specifically on the estimation of a cumulative probability *P(E) *of an event *E*, when the first occurrence of this event is observed at *t *successive time points of a longitudinal study. The performance of multiple imputation was compared with that of the Kaplan-Meier approach by simulating several attrition scenarios.

Three types of missingness mechanisms can be distinguished: missing completely at random (MCAR), missing at random (MAR) and not missing at random (NMAR) [23]. If the probability of drop-out depends neither on missing data nor on observed data, then the data are MCAR. In this case attrition implies a loss of power, but no bias occurs in complete-case estimations. An example of MCAR situation in the context of IVF is the accidental loss of patient records. If, given the observed data, the probability of drop-out does not depend on missing data, then the data are MAR. Bias may arise but it can be minimized with appropriate methods, such as multiple imputation, that take into account available information explaining the missingness mechanism. As an example, suppose that records of patients older than 35 years are recorded separately (file A for patients younger than 35, and file B for patients older than 35). The situation is MAR if all the file A is available but a part of file B is accidentally lost. Finally, if the probability of drop-out depends on missing data or on other unobserved variables, then the data are NMAR. Bias due to attrition cannot be reduced in this case, and sensitivity analyses have to be conducted under various NMAR assumptions. An example is the situation in which patients are recorded differentially according to the outcome of the attempt: all patients with a delivery would be recorded, but only a part of those with pregnancy loss would. The present study focuses on MAR attrition.

Data were simulated using SAS Software (SAS Institute Inc. 2004. SAS/STAT^{® }9.1 User's Guide. Cary, NC: SAS Institute Inc.).

The simulation process took place in three steps detailed below: first, a complete cohort was simulated, then attrition was added and samples were randomly drawn, and finally the different methods of estimation were applied to the samples with attrition.

A cohort of 100 000 subjects with four follow-up time points was simulated. Simulated data consisted in a binary outcome *E *representing occurrence of the event, and a continuous covariate *X *predictive of both occurrence and drop-out. Measures for subject *i *at time point *t *were named *E _{it }*and

The covariate *X *was simulated Gaussian and the link between the successive time points was the following:

$$\begin{array}{l}{X}_{i\mathit{1}}~N(0,10)\\ \begin{array}{cc}{X}_{it}={\alpha}_{it}+{\beta}_{it}\times {X}_{i(t-\mathit{1})}+{\epsilon}_{it},& t=2,3,4,{\epsilon}_{it}~N(0,{s}_{it}^{2})\end{array}\end{array}$$

For simplicity, coefficients were taken as being equal to one: *α _{it }*=

The outcome *E _{it }*was drawn from a Bernoulli distribution with a probability of occurrence of the event equal to

Only the first occurrence of the event was of interest. Thus, the final endpoint *E _{i }*(occurrence/non-occurrence) was either non-occurrence of the event in subject

The final true occurrence rate *P(E) *represented the cumulative occurrence rate of *E _{i }*at the end of the follow-up, calculated in this complete cohort of 100 000 subjects.

The binary variable *D _{it }*indicates whether subject

The variable *D _{it }*(interrupt/continue) was drawn from a Bernoulli distribution with a probability of drop-out of

Probabilities of occurrence of the event and probabilities of drop-out at each time point, according to the subject category

The total drop-out rate represented the cumulative drop-out rate at the end of the follow-up, calculated in the cohort of 100 000 subjects with drop-outs.

The Kaplan-Meier method [5,6] was applied considering drop-out as censor. Subjects encountering four non-events were also censored at the end of the study. The event occurrence rate was estimated as [*1-S(t)*] at time *t *= 4, *S(t) *being the survival function.

Multiple imputation consists of replacing missing values with a set of plausible values, based on auxiliary information. In this way, all subjects can be included in the analysis. Multiple imputation was implemented with the SAS MI procedure (SAS Institute Inc. 2004. SAS/STAT^{® }9.1 User's Guide. Cary, NC: SAS Institute Inc.). Multiple imputation was performed at each time point (except the first, where all subjects were observed) for the subjects who failed and dropped out at a previous time point. For instance at time point 2, values were imputed for subjects i with *E _{i1 }*= 0 (ie subjects with failure at time point 1); at period 3, values were imputed for subjects i with

Table Table11 presents the eight scenarios that were tested and gives the final drop-out and true occurrence rates resulting from each of the eight scenarios. A scenario was determined by a probability of occurrence *p _{it }*and a probability of drop-out

In scenarios 2 and 3, the difference between the categories *C _{1 }*and

In scenarios 4 to 8, the probabilities of occurrence *p _{it }*and the probabilities of drop-out

For each scenario, a cohort of 100 000 subjects was created with corresponding *p _{it }*and

Results of the simulations for the 8 scenarios are displayed in Table Table22.

Estimation of the final occurrence rate of the event *P(E) *at the end of follow-up according to the estimation method in MCAR and MAR scenarios (500 replications of samples sized 2000)

As expected, in MCAR scenarios 1, 2 and 3, both methods (Kaplan-Meier and multiple imputation) were unbiased. Their coverage rates were close to the nominal level of 95% (range = [94.6%; 95.8%]).

In scenario 4, data were MAR but probabilities of occurrence of the event were equal in both categories of subjects. Both methods were as efficient as in MCAR scenarios: estimates were unbiased and coverage rates close to 95%. This was expected because both categories had the same probability of occurrence of the event (= 10%), so the difference in the probabilities of drop-out did not impact the estimation.

In the MAR scenarios 5 to 8, the amplitude of bias obtained with the Kaplan-Meier method was approximately 10% of *P(E) *(bias≈5%) and the coverage rate was very poor (less than 15%). The bias tended towards an underestimation of the true occurrence rate of the event *E *when subjects in the category *C _{2 }*with a higher probability of the event had a higher probability of drop-out (scenarios 5 and 6), and towards an overestimation when subjects in the category

In these MAR scenarios 5 to 8, multiple imputation yielded a very small bias (less than 2%) and a very good coverage rate (above 83%). The method seemed to be less efficient in scenarios 7 and 8 when the category *C _{2 }*with the higher probability of occurrence had the lower probability of drop-out.

To point up the importance of the difference between the estimations distributions in MAR scenarios, histograms of all estimates according to the method used are provided in Figure Figure11 for scenarios 3 (MCAR) and 5 (MAR).

Simulations were done with a smaller sample size of 500 (results not shown). This led to larger standard errors, that mechanically increased the coverage rate. In scenarios 5 to 8, the coverage rates ranged from 41.6% to 69.4% for the Kaplan-Meier estimate and from 88.4% to 94.8% for multiple imputation estimations. Bias was unchanged. Conclusions remained the same concerning the performance of the methods.

Multiple imputation has been developed to deal with missing data, which is a particular problem in longitudinal studies which suffer from attrition. However, little attention has been paid to how multiple imputation might be applied to studies with successive data collection points. The estimation of an occurrence rate in a cohort with attrition is a recurrent issue in reproductive health in general, and in IVF studies in particular. These studies have the characteristics of having discrete time points corresponding to attempts. Intervals between time points are not taken into account, thus leading to no distribution of the event times. Methods developed in previous publications thus cannot be applied here [25,26] We therefore used a simulation study to compare the performance of multiple imputation with that of the widely used Kaplan-Meier method, for the estimation of an occurrence rate in a cohort with attrition.

We found that in MAR scenarios, Kaplan-Meier estimation performed fairly well with respect to bias, but not with respect to the coverage rate. The direction of the biases was as expected, owing to the differences in the probabilities of occurrence of the event and of drop-out between the two subject categories. Indeed, when subjects dropping out tended to have a higher chance of experiencing the event, the Kaplan-Meier assumption that subjects dropping out have the same chance of occurrence of the event as other subjects led to an under-estimation, and vice-versa. The amplitude of bias was quite small. However, the Kaplan-Meier estimate was very precise around this biased estimation. This implies that the confidence interval rarely included the true value, and therefore that the coverage rate was poor. Thus, even if the bias is considered acceptable, one must be aware that the confidence interval will almost never contain the true value.

In contrast, multiple imputation was virtually bias-free and gave robust estimates in all the scenarios. This approach enables auxiliary information to be incorporated in the estimation. This makes it possible to take into account the fact that subjects' chances of experiencing the event may differ. This was the case in the simulated cohort in which two categories of subjects co-existed and were defined by their covariate values. Thus, multiple imputation succeeded in correcting the bias inherent in the Kaplan-Meier method. By replacing each missing value with several values, multiple imputation takes into account the uncertainty of the imputed value [23,27]. Therefore, the standard errors of multiple imputation were larger than those of the Kaplan-Meier estimation, resulting in larger confidence intervals and better coverage rates.

In comparison with the method of Hsu et al. [22], multiple imputation makes it possible to include the information contained in auxiliary variables without summarizing them in risk scores. Moreover, it gives a good estimation of variance, as it takes into account the uncertainty of the imputed values.

Multiple imputation was evaluated here under the assumption that the probability of drop-out immediately after time point *t *depended on the observed covariate at time point *t*. This type of attrition is likely to occur in most epidemiological cohorts. For example, a cohort of patients attending regular medical appointments would fit this attrition mechanism if the probability of dropping out after a given visit was likely to depend on the results of the same visit. This is the case in IVF, where failure of a procedure may indicate a poor chance of subsequent success and lead a couple to discontinue treatment. Multiple imputation at each time point is compatible with this attrition mechanism, as imputation at time point *t *is performed according to the observed or imputed covariate values at the same time point. This highlights the fact that data have to be investigated in order to apply a multiple imputation strategy that fits their pattern. The part of the auxiliary information that has to be included in the imputation model would be of particular interest: in some situations, baseline data could be sufficient, whereas in other situations, inclusion of intermediate measures could be advantageous. Imputation appears a preferable method than complete case analysis that assumes stronger hypotheses (MCAR data) and induces an important loss of statistical power.

The multiple imputation method could be generalized without theoretical problems to situations with a larger number of time intervals. However, it would rise practical difficulties such as a (much) longer computational time and a decreasing accuracy of the estimates, especially with limited sample size, due to categories of subjects with drop-out rates almost equal to 100% as the time goes by.

Data not missing at random were not tested in this work. In practice, implementations of multiple imputation in statistical software are made under the MAR assumption. Attrition was fully explained in the simulation study, as all the variables explaining attrition were taken into account. This may not be the case with observational data. Sensitivity analyses are necessary to determine whether the results are robust to deviations from this assumption.

The second difference between simulated and observational data is that, with simulated data, the model underlying the response and the covariates is known. The relations are then correctly used for imputation. With observational data,matters can be much more difficult. The relations used for imputation rarely reproduce exactly the relations underlying the data. With observational data, multiple imputation could lose precision and eventually generate bias. The consequences of misspecification of the imputation model were not explored here and require more research.

Although our findings have the limitations of any simulation study, they point to a new way of estimating an occurrence rate in cohorts with attrition, using auxiliary information collected during follow-up. The robust estimations of multiple imputation in this simulation study are very satisfactory and promising. Our results suggest that in an observational context, comparison of the results of Kaplan-Meier estimation and of multiple imputation could provide a measure of the impact of attrition on the estimation. If this impact appears small, further analyses could be made using the easier-to-implement Kaplan-Meier approach.

This study is a first step towards defining the appropriate use of multiple imputation in longitudinal studies.

IVF: in vitro fertilization; MAR: missing at random; NMAR: not missing at random

The authors declare that they have no competing interests.

NS designed and implemented the simulation study, conducted the analysis and drafted the article. ELR and JB helped to design the study and interpret the results and they critically revised the article. All the authors read and approved the final manuscript.

The pre-publication history for this paper can be accessed here:

This work was supported by the French Institute for Public Health Research (IReSP); and the Ile-de-France Region [doctoral grant to NS].

- Deeg DJ. Attrition in longitudinal population studies: does it affect the generalizability of the findings? J Clin Epidemiol. 2002;55(3):213–215. doi: 10.1016/S0895-4356(01)00472-3. [Cross Ref]
- Crawford SL, Tennstedt SL, McKinlay JB. A comparison of analytic methods for non-random missingness of outcome data. J Clin Epidemiol. 1995;48(2):209–219. doi: 10.1016/0895-4356(94)00124-9. [PubMed] [Cross Ref]
- Greenland S. Response and follow-up bias in cohort studies. Am J Epidemiol. 1977;106(3):184–187. [PubMed]
- Pennefather PM, Tin W, Clarke MP, Dutton J, Fritz S, Hey EN. Bias due to incomplete follow up in a cohort study. Br J Ophthalmol. 1999;83(6):643–645. doi: 10.1136/bjo.83.6.643. [PMC free article] [PubMed] [Cross Ref]
- Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. J Am Stat Assoc. 1958;53(282):457–481. doi: 10.2307/2281868. [Cross Ref]
- Altman DG. Practical statistics for medical research. London: Chapman & Hall; 1991.
- Van Beijsterveldt CE, van Boxtel MP, Bosma H, Houx PJ, Buntinx F, Jolles J. Predictors of attrition in a longitudinal cognitive aging study: the Maastricht Aging Study (MAAS) J Clin Epidemiol. 2002;55(3):216–223. doi: 10.1016/S0895-4356(01)00473-5. [PubMed] [Cross Ref]
- McLean RR, Hannan MT, Epstein BE, Bouxsein ML, Cupples LA, Murabito J, Kiel DP. Elderly cohort study subjects unable to return for follow-up have lower bone mass than those who can return. Am J Epidemiol. 2000;151(7):689–692. [PubMed]
- Daya S. Life table (survival) analysis to generate cumulative pregnancy rates in assisted reproduction: are we overestimating our success rates? Hum Reprod. 2005;20(5):1135–1143. doi: 10.1093/humrep/deh889. [PubMed] [Cross Ref]
- Olivius C, Friden B, Borg G, Bergh C. Why do couples discontinue in vitro fertilization treatment? A cohort study. Fertil Steril. 2004;81(2):258–261. doi: 10.1016/j.fertnstert.2003.06.029. [PubMed] [Cross Ref]
- Sharma V, Allgar V, Rajkhowa M. Factors influencing the cumulative conception rate and discontinuation of in vitro fertilization treatment for infertility. Fertil Steril. 2002;78(1):40–46. doi: 10.1016/S0015-0282(02)03160-6. [PubMed] [Cross Ref]
- Olivius K, Friden B, Lundin K, Bergh C. Cumulative probability of live birth after three in vitro fertilization/intracytoplasmic sperm injection cycles. Fertil Steril. 2002;77(3):505–510. doi: 10.1016/S0015-0282(01)03217-4. [PubMed] [Cross Ref]
- Stolwijk AM, Wetzels AM, Braat DD. Cumulative probability of achieving an ongoing pregnancy after in-vitro fertilization and intracytoplasmic sperm injection according to a woman's age, subfertility diagnosis and primary or secondary subfertility. Hum Reprod. 2000;15(1):203–209. doi: 10.1093/humrep/15.1.203. [PubMed] [Cross Ref]
- Rubin DB. Multiple imputations in sample surveys - a phenomenological Bayesian approach to nonresponse. Proc Survey Res Meth Sec (American Statistical Association) 1978. pp. 20–34.
- Barzi F, Woodward M. Imputations of missing values in practice: results from imputations of serum cholesterol in 28 cohort studies. Am J Epidemiol. 2004;160(1):34–45. doi: 10.1093/aje/kwh175. [PubMed] [Cross Ref]
- Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1(4):368–376. doi: 10.1191/1740774504cn032oa. [PubMed] [Cross Ref]
- Klebanoff MA, Cole SR. Use of multiple imputation in the epidemiologic literature. Am J Epidemiol. 2008;168(4):355–357. doi: 10.1093/aje/kwn071. [PMC free article] [PubMed] [Cross Ref]
- Kristman VL, Manno M, Côté P. Methods to account for attrition in longitudinal data: do they work? A simulation study. Eur J Epidemiol. 2005;20(8):657–662. doi: 10.1007/s10654-005-7919-7. [PubMed] [Cross Ref]
- Engels JM, Diehr P. Imputation of missing longitudinal data: a comparison of methods. J Clin Epidemiol. 2003;56(10):968–976. doi: 10.1016/S0895-4356(03)00170-7. [PubMed] [Cross Ref]
- Mazumdar S, Tang G, Houck PR, Dew MA, Begley AE, Scott J, Mulsant BH, Reynolds CF III. Statistical analysis of longitudinal psychiatric data with dropouts. J Psychiatr Res. 2007;41(12):1032–1041. doi: 10.1016/j.jpsychires.2006.09.007. [PMC free article] [PubMed] [Cross Ref]
- Hsu CH, Taylor JM, Murray S, Commenges D. Survival analysis using auxiliary variables via non-parametric multiple imputation. Stat Med. 2006;25(20):3503–3517. doi: 10.1002/sim.2452. [PubMed] [Cross Ref]
- Hsu C-H, Long Q, Alberts D. Estimation of colorectal adenoma recurrence with dependent censoring. BMC Med Res Methodol. 2009;9(1):66. doi: 10.1186/1471-2288-9-66. [PMC free article] [PubMed] [Cross Ref]
- Little RJ, Rubin DB. Statistical Analysis with Missing Data. 2. Hoboken, NJ: John Wiley & Sons Inc; 2002.
- Rubin DB. Multiple imputation after 18+ years. J Am Stat Assoc. 1996;91(434):473–489. doi: 10.2307/2291635. [Cross Ref]
- Taylor JMG, Murray S, Hsu C-H. Survival estimation and testing via multiple imputation. Statistics & Probability Letters. 2002;58(3):221–232.
- Jan B, Shah S, Shah S, Fazli Qadir M. Weighted Kaplan Meier estimation of survival function in heavy censoring. Pak J Statist. 2005;21(1):55–63.
- Schafer JL. Analysis of Incomplete Multivariate Data. Vol. 72. Boca Raton, FL: Chapman & Hall/CRC; 1997.

Articles from BMC Medical Research Methodology are provided here courtesy of **BioMed Central**

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |