Search tips
Search criteria 


Logo of amjepidLink to Publisher's site
Am J Epidemiol. 2012 September 15; 176(6): 527–533.
Published online 2012 August 31. doi:  10.1093/aje/kws313
PMCID: PMC3447603

Effectiveness of Adjuvanted Influenza Vaccination in Elderly Subjects in Northern Italy


Although vaccination against influenza is recommended for elderly and high-risk patients in many countries, efficacy in the elderly has been suboptimal. The MF59 adjuvanted trivalent inactivated vaccine (ATIV) was developed to increase the immune response of elderly subjects to influenza vaccination, but its effectiveness has not yet been well documented. This prospective, observational study evaluated the relative effectiveness of ATIV versus nonadjuvanted trivalent inactivated vaccine (TIV) in individuals at least 65 years of age in Lombardy, northern Italy. Hospitalizations for influenza or pneumonia (International Classification of Diseases, Ninth Revision, Clinical Modification, codes 480–487) during the 2006–2007, 2007–2008, and 2008–2009 influenza seasons were identified from administrative databases. Stratified and regression analyses, including the propensity score to adjust for confounding, as well as generalized estimating equations to account for repeated vaccination, were used. Overall, 107,661 records were evaluated, contributing 170,988 person-seasons of observation. Since ATIV is preferentially recommended for more frail individuals, subjects vaccinated with ATIV were older and had more functional impairment and comorbidities. In the primary analysis, risk of hospitalization for influenza or pneumonia was 25% lower for ATIV relative to TIV (relative risk = 0.75, 95% confidence interval: 0.57, 0.98). To the extent that there is residual bias, ATIV is likely to be even more protective than this result suggests.

Keywords: adjuvanted influenza vaccine, elderly, influenza, pneumonia

Globally, influenza infection is a major cause of morbidity and mortality. In many countries, vaccination to prevent influenza and its complications is recommended for “high-risk” groups, such as the elderly, patients with chronic conditions, and institutionalized populations. Nevertheless, vaccination of the elderly remains controversial. Since there is a gradual decline in immune competence with age (13), immunogenicity and hence vaccine effectiveness in the elderly are suboptimal. Assessment of effectiveness is complicated by the degree of match between vaccine and virus, which varies from year to year with mismatches leading to lower vaccine efficacy (4, 5). Observational studies comparing vaccinated with unvaccinated cohorts have generally found that conventional influenza vaccines are modestly effective in preventing hospitalization and mortality during the influenza season (6). Controversies stem from the scarcity of data from randomized studies and the potential for bias in observational studies due to differences in the baseline health status between those who are vaccinated and those who are not (7, 8). It was therefore recognized that the added value of a field study would include only vaccinated persons in a head-to-head comparison, thus substantially reducing the potential for confounding.

MF59 adjuvanted trivalent inactivated vaccine (ATIV) (Fluad; Novartis Vaccines and Diagnostics, Siena, Italy) has been registered in Europe since 1997 for vaccination of persons 65 years of age or older. Clinical trials have shown that MF59 adjuvanted vaccines are more immunogenic than conventional nonadjuvanted vaccines and provide better immunogenicity against drifted strains that are different from the virus strains included in the vaccine (9). Nevertheless, improved effectiveness in preventing influenza and related complications has not been well evaluated in the elderly (10).

The Italian National Health Care System provided an opportunity to evaluate comparative influenza vaccine effectiveness in a field setting where medical data are accessible and have been used for pharmacoepidemiologic studies (11, 12). Italian guidelines on the prevention and control of influenza provide free access to vaccines for high-risk persons, with adjuvanted vaccines generally preferentially recommended for more frail high-risk individuals (13). We conducted a prospective, population-based cohort study, known as the “Lombardy Influenza Vaccine Effectiveness” (LIVE) Study, in northern Italy, of ATIV versus trivalent inactivated vaccine (TIV) effectiveness during 3 consecutive influenza seasons starting in 2006–2007. Here, we report the primary objective, which was to assess the relative risk of hospitalizations for influenza or pneumonia during the influenza season in ATIV versus TIV vaccinees. The secondary objectives, including an assessment of safety and other secondary outcomes, will be reported separately.


Residents in the provinces of Cremona, Bergamo, Mantova, Lecco, and Pavia in the northern Italian region of Lombardy, who were at least 65 years of age and sought influenza vaccination at local health authorities’ district offices or participating general practitioners were eligible for enrollment into this prospective cohort study during each of the 3 vaccination seasons, 2006–2007, 2007–2008, and 2008–2009. We excluded residents who were in the hospital, nursing homes, or rehabilitation centers in the 30 days preceding immunization, as well as those receiving home care or who were allergic to influenza vaccines.

Eligible subjects were informed by the vaccinators about the study and asked for their consent to participate; all those who accepted were administered a brief questionnaire to record basic demographic data and information on potential confounders, including smoking status, conditions potentially affecting immune response, functional status (as assessed through self-reported answers to questions about physical capabilities), presence of children in the household, and receipt of an influenza vaccine the previous year. Then they were administered either ATIV or the conventional nonadjuvanted trivalent subunit vaccine (Agrippal; Novartis Vaccines and Diagnostics), according to local, regional, and national influenza vaccination policy recommendations. There was no attempt at random assignment of vaccines. Information on the type of administered vaccine was recorded for each participant by the vaccinators, along with the previously collected information.

Both vaccines contained the recommended virus strains for the respective influenza season in the Northern Hemisphere. Vaccine doses were donated by the manufacturer and delivered to the local health authorities, who then distributed them to their district offices or participating general practitioners.

For each participant, residence status was confirmed through record linkage with administrative databases; all linkage failures were excluded from the study. Additionally, the presence of chronic disease or other relevant routinely collected medical history information was ascertained through record linkage with databases containing data on hospitalizations (discharge diagnoses), outpatient drug prescriptions (active ingredient and estimated duration of treatment), receipt of ambulatory care with specialist, and certified exemption from copayment of health-care costs.

The primary outcome was defined as a hospitalization for influenza or pneumonia occurring during a defined period in the influenza season (and in any case at least 3 weeks after vaccination), recorded in the hospital administrative database (discharge diagnosis with International Classification of Diseases, Ninth Revision, Clinical Modification, codes 480–487). Influenza season was defined on the basis of a nationwide surveillance network (“Influnet”) that monitors virologically confirmed influenza occurrence in Italy annually. The network includes 1,000 general practitioners and family physicians, and it provides weekly incidence data, stratified by age and region (14). We pooled data over the 3 influenza seasons, such that our elementary data record was a “person-season” at risk. As many people were included for more than one of the 3 years of observation, we used generalized estimating equations (15, 16) to take account of the correlation induced by measuring the experience of the same people for more than one influenza season.

Our case definition did not include positive laboratory confirmation of influenza virus. Therefore, in order to increase the specificity of the identification of cases hospitalized for influenza-related conditions, we defined 3 different time windows during the influenza season in which hospitalizations were counted. The broadest time window corresponded to the entire influenza season, as determined from Influnet. The narrowest time window corresponded to the period of adjacent weeks, around the peak influenza occurrence, having an influenza rate that exceeded 1 case per 1,000 person-weeks (17). An intermediate time window was defined in the same way but with a threshold of 0.5 case per 1,000 person-weeks. Although the broader windows capture more cases, they are less specific for influenza-related cases. Accordingly, we defined a priori our primary analysis to be based upon the narrowest window as it provided the greatest specificity and hence the least bias. Also, in order to estimate the amount of potential misclassification of the discharge diagnosis, a sample of hospital discharge records was validated, and the diagnosis was compared with the actual hospital discharge diagnoses.

Since adjuvanted vaccine was preferentially recommended for high-risk frail individuals at many sites, it was known a priori that analysis of study outcomes would have to take this source of potential bias into account. To assess and control for confounding, we used stratification coupled with Mantel-Haenszel summary estimates of a pooled effect measure. Variables assessed as potential confounders included age, gender, influenza season, local health authorities and vaccine provider, functional status, smoking, recent infectious disease, transfusion, intestinal disorder, self-reported flu symptoms, cumulative length of stay in the hospital and cumulative number of drug prescriptions (both in the 5 years preceding the vaccination), infectious disease, and chronic conditions, such as chronic obstructive pulmonary disease (COPD), kidney disease, diabetes, cardiovascular disease, peripheral vascular disease, cancer, and history of hospitalization for pneumonia, influenza, or emphysema. Because of the number of potential confounders, we were not able to control for all confounders simultaneously in a single stratified analysis; we therefore also conducted a multivariate analysis that used a propensity score as a summary confounding score. For the propensity score model, we used a logistic regression to estimate the probability of receiving ATIV versus TIV. Variables that were included in this model included age, sex, influenza season, community and provider, cumulative length of stay in the hospital, and cumulative number of drug prescriptions in the 5 years preceding the vaccination, physical impairment, smoking, presence of children in the home, recent transfusion, recent intestinal disorder, recent self-reported flu symptoms, recent infectious disease, history of hospitalization for pneumonia, influenza or emphysema, COPD, diabetes, cardiovascular disease, chronic kidney disease, peripheral vascular disease, and cancer.

To avoid including in the propensity score model nonconfounding predictors of exposure, which would not reduce confounding but would decrease precision, we fit a preliminary logistic model predicting hospitalization with influenza-like illness that included all these covariates, along with study vaccine, to determine the strength of relation of each variable with the study outcome. As a second stage, we then created the propensity score model using those predictors from the preliminary outcome model that had a relative risk of at least 1.4: age, sex, influenza season, community and provider, physical impairment, cumulative length of stay in the hospital and cumulative number of drug prescriptions in the 5 years preceding the vaccination, history of hospitalization for pneumonia, influenza or emphysema, COPD, chronic kidney disease, diabetes, recent infectious disease, and recent transfusion (18). From this model, we computed the propensity to receive ATIV for each person-season of observation and added that to the data as an additional, derived variable. To improve comparability of the 2 vaccine groups, we controlled for the propensity score in multivariate models, but first we excluded all outlier observations (“trimming”), defined as those below the lower 2.5% of the tail of the TIV observations (3,355 observations) and above the upper 2.5% tail of the ATIV observations (2,894 observations). These tails are outside the primary area of overlap of the propensity scores, and they increase residual confounding in any type of analysis (19).

In the multivariate analyses, we used generalized estimating equations to account for the inclusion of people in more than one season. Our final multivariate analysis was based on doubly robust estimation, in which the strongest confounders and the propensity score based on all confounders were included in the logistic model; this model in principle should provide the best control of confounding achievable with these data (20).

We conducted an additional analysis to compare the risk of hospitalization before each influenza season in the 2 vaccine groups. We therefore identified the events that occurred in the period May–September in people subsequently vaccinated and enrolled in the study. This was done under the assumption that, if our model in the primary analysis had completely adjusted for confounding, with the application of the same doubly robust technique outside the influenza season (i.e., without influenza activity and independently of the vaccine effect), we would estimate no risk difference between the 2 cohorts; any increased risk would represent the magnitude of the residual bias.

Statistical analysis was performed by using SAS, version 9.1, software (SAS Institute, Inc., Cary, North Carolina). We used multiple (i.e., 5) imputations (Proc MI and Proc MIANALYZE) to handle missing values (21, 22).

The study protocol was submitted to and approved by the ethics committees of the participating local health authorities.


A total of 107,661 people fulfilled the study eligibility criteria. The unit of our analysis was person-season, or equivalently, the number of vaccinations provided. One person could contribute up to 3 separate influenza seasons of risk for influenza-related outcomes, each of the 3 having either vaccine assignment. Of the total of 107,661 participants, 43,667 were included for more than 1 year; of these, 23,484 received at least one vaccination of either type. Overall, 88,449 ATIV and 82,539 TIV were administered for a total of 170,988 person-seasons; after the data trimming based on the person-season distribution, we included a total of 164,254 person-seasons in the primary analysis.

As expected, given the observational nature of the study and the regional recommendations on the preferential use of adjuvanted influenza vaccines in high-risk patients, the 2 vaccine groups showed some imbalance at baseline with respect to age, functional limitations, and prevalence of chronic conditions, with the recipients of ATIV showing more functional impairment and comorbidities compared with the recipients of TIV (Table 1).

Table 1.
Distribution of Selected Characteristics of Interest by Vaccine Type in the Study Population, Lombardy, Italy, 2006–2009a

Figure 1 shows the influenza incidence in Lombardy by week during the 3 study years. Seasons 2006–2007 and 2008–2009 were mainly A/H3N2 epidemics, whereas season 2007–2008 showed mainly circulation of A/H1N1 and B viruses. There were a total mismatch for the A/H1N1-like strain and a partial mismatch for the A/H3N2-like strain in the 2007–2008 season, as well as a partial mismatch for B strains for all 3 years. Given the strains circulating, the mismatches which occurred in 2007–2008 were the only ones that could have affected the vaccine effectiveness appreciably.

Figure 1.
Time windows defined by the influenza incidence/1,000 person-weeks in Lombardy, Italy, 2006–2009. The intermediate time window includes adjacent weeks above the threshold of 0.5 case/1,000 person-weeks each year. Bars represent narrow time windows ...

The primary analysis used the narrowest definition of the time window for influenza-related events, which should have the greatest specificity for the outcomes of interest, corresponding to calendar weeks 4–7 inclusive in 2006–2007 and weeks 52–54 and 1–7 for the subsequent 2 influenza seasons. During these periods, there were 114 hospitalizations for influenza and pneumonia among the 84,665 person-seasons at risk for the ATIV group (0.135%), compared with 111 among 79,589 for the TIV group (0.139%). The crude risk ratio was 0.97 (95% confidence interval (CI): 0.74, 1.25).

These crude comparisons are confounded by the various factors that are unbalanced between the 2 vaccine groups (refer to Table 2 for the distribution of cases and person-seasons by age). We controlled for confounding by local health authorities-provider and age using simultaneous stratification by these 2 variables, resulting in a Mantel-Haenszel risk ratio estimate of 0.87 (95% CI: 0.67, 1.13). We then extended this stratification to include sex, history of lung-related hospitalization, level of functional impairment, and season; the corresponding Mantel-Haenszel summary risk ratio was 0.80 (95% CI: 0.61, 1.06). These analyses indicated that there was substantial confounding in the crude data, but with control of the main confounders, the adjuvanted vaccine group seemed to have about a 20% lower risk of hospitalizations for influenza and pneumonia.

Table 2.
Hospitalizations for Influenza and Pneumonia (Cases) and Person-Seasons at Risk by Age and Vaccine Type, Within the Narrowest Time Window in the Study Population, Lombardy, Italy, 2006–2009

We then used the propensity score as a summary confounder in a multivariate logistic model based on the trimmed data, which included the strongest confounders along with the propensity score. From this model, we estimated a risk ratio of 0.75 (95% CI: 0.57, 0.98) for ATIV relative to TIV, an association slightly stronger but very close to the results of the stratified analyses (Table 3).

Table 3.
Primary Analysis Results of the Multivariate Model (Narrowest Influenza Time Window) in the Study Population, Lombardy, Italy, 2006–2009

We repeated the above analyses using the intermediate and broader time windows. These analyses have a less specific outcome than the analysis using the narrow time window for influenza-related hospitalizations, as they include a relatively greater proportion of background hospitalizations as the influenza epidemic began or waned. Thus, one would expect that there would be more cases, but that the associations found would be biased. As expected, the number of cases increased from 225 for the analysis using the narrow time window to 374 for the intermediate and 735 for the broadest time window; the risk ratio estimates were 0.75 for the narrow time window, 0.83 for the intermediate window, and 0.88 for the broadest window (Table 4), thus suggesting that misclassification of outcome in our study leads to a reduction of the estimated effect. In the validation of hospital records in a subset of cases, we confirmed in 99.4% of the sample the concordance between diagnoses recorded in the medical charts and the diagnosis code in the database.

Table 4.
Risk Ratio Estimate of ATIV Versus TIV by Time Window During the Influenza Season in the Study Population, Lombardy, Italy, 2006–2009

Moreover, the risk of hospitalization in the ATIV group before each influenza season (May–September)—after propensity score estimation, trimming, and adjustment for confounding in a doubly robust multivariate model—was 17% higher than that in the TIV control group (risk ratio = 1.17, 95% CI: 0.96, 1.43).


This is the first large-scale study of the comparative effectiveness of ATIV versus TIV. The study population of 107,661, contributing 170,988 person-seasons of observation, is considerably larger than could have been readily enrolled into a randomized clinical trial.

We were able to link study participants to available administrative data, combining both self-reported information through questionnaire and vaccine status at the time of vaccination with retrospective data on hospitalizations, drug prescriptions, and other outcomes from administrative databases (extended to several years prior to enrollment). By conducting the study over 3 years, we reduced the effect of year-to-year variation in the antigenic “match” between the vaccine and circulating influenza strains, a problem that hampers the interpretation of findings in any study based on only one influenza season.

The estimated 25% reduction in the risk of an influenza- or pneumonia-related hospitalization is an underestimate. Given the lack of laboratory confirmation for the presence of influenza infection among our cases, even the most stringent time window that we used, and from which this estimate comes, is susceptible to imperfect identification of relevant cases. We are not able to determine whether the resulting misclassification of outcome is differential or nondifferential, since in our study subjects receiving ATIV were more frail and therefore may have had more baseline hospitalizations. Our risk ratio estimate is in any case closer to the null that it would be if the identification of influenza-related hospitalizations were perfect.

Methodological research has shown that, although controlling confounding by propensity scores performs only about as well as more traditional methods (23), calculating propensity scores does provide the ability to identify and exclude outliers from the 2 study groups who have propensity scores outside the range, or the central distribution, of the other group. This “trimming” improves the validity of any analysis by restricting the study to comparable observations. In this study, we used trimming by propensity score as an initial step, followed by 2 methods to control confounding, stratification on the individual covariates and doubly robust multivariate modeling.

We found that more stringent control of confounding resulted in a stronger association in the direction of lower risk of hospitalization among ATIV recipients, which was expected given the observed imbalance in relevant covariates between the 2 vaccine cohorts.

Such imbalance also explains the 17% higher risk of hospitalization in the ATIV group before each influenza season, in a time when we would have expected the adjusted risk of hospitalization to be the same in the 2 groups. The combined evidence that more frail people have been enrolled in the ATIV cohort compared with the TIV cohort and that our analysis has not removed all the confounding leads to the conclusion that we have likely underestimated the effectiveness of ATIV.

Some observational studies of influenza vaccine efficacy in preventing deaths have been encumbered by selection bias, raising controversies about the validity of the observational approach in assessing influenza vaccine effectiveness (2431). In these studies, bias most likely reflects a “healthy-vaccinee” effect, in which those who are at high risk of a near-term death are less likely to be vaccinated. Such confounding bias should not be a problem in this study, which considered hospitalization for influenza or pneumonia instead of all-cause mortality and involved a head-to-head comparison of 2 influenza vaccines.

In conclusion, we found that vaccination with ATIV reduced the risk of hospitalization for influenza or pneumonia in the elderly during the peak of the influenza season by 25% relative to vaccination with TIV. Residual bias indicates that this value is likely to be an underestimate. The routine use of ATIV in the elderly would provide important clinical benefit over the traditional nonadjuvanted vaccines.


Author affiliations: Azienda Sanitaria Locale della provincia di Cremona, Cremona, Italy (Salvatore Mannino, Marco Villa, Liana Boldori); Azienda Ospedaliera “Ospedali Riuniti Villa Sofia-Cervello,” Palermo, Italy (Salvatore Mannino); Istituto di Ricerche Farmacologiche “Mario Negri,” Milano, Italy (Giovanni Apolone); Azienda Ospedaliera “Arcispedale Santa Maria Nuova,” Reggio Emilia, Italy (Giovanni Apolone); Department of Epidemiology, School of Public Health, University of Washington, Seattle, Washington (Noel Weiss); Novartis Vaccines and Diagnostics, Siena, Italy (Nicola Groth); Azienda Sanitaria Locale della provincia di Pavia, Parvia, Italy (Ivana Aquino); Azienda Sanitaria Locale della provincia di Mantova, Mantova, Italy (Fausta Caramaschi); Azienda Sanitaria Locale della provincia di Lecco, Lecco, Italy (Antonio Gattinoni); Azienda Sanitaria Locale della provincia di Bergamo, Bergamo, Italy (Giancarlo Malchiodi); RTI-Health Solutions, Research Triangle Park, North Carolina (Kenneth Rothman); Department of Epidemiology, School of Public Health, Boston University, Boston, Massachusetts (Kenneth Rothman); and Section of Preventive Medicine, Department of Medicine, Boston University, Boston, Massachusetts (Kenneth Rothman).

The study was partially funded by Novartis Vaccines and Diagnostics, who supplied both study vaccines and assisted with some administrative and organizational aspects of the study. Dr. Groth, a full-time employee of Novartis, contributed to study design, the literature search, data interpretation, and writing of the manuscript and, thus, is a coauthor of this work.

The authors are grateful to local health authorities personnel and general practitioners who—with their work—made it possible to conduct this study.

S. M. has received reimbursement for travel expenses related to study meetings or presentations. He was also a consultant to Chiron Vaccines (now part of Novartis Vaccines and Diagnostics). M. V., a local health authority employee of Cremona, received no payment from Novartis but has received reimbursement for travel expenses related to study meetings or presentations. G. A. has received consulting fees by Novartis Vaccines and Diagnostics for this study and reimbursement for travel expenses related to study meetings or presentations. He has been a consultant to GlaxoSmithKline and Sanofi-Aventis. N. W. has received consulting fees by Novartis Vaccines and Diagnostics for this study and reimbursement for travel expenses related to study meetings or presentations. N. G. is an employee of Novartis Vaccines and Diagnostics, with stock options benefits. K. R. has received no consulting fees from Novartis Vaccines and Diagnostics; his institution has received fees from Novartis Vaccines and Diagnostics for his participation in this study, as well as reimbursement for travel expenses related to study meetings or presentations. The other authors have no conflicting interests to report.


1. Armstrong BG, Mangtani P, Fletcher A, et al. Effect of influenza vaccination on excess deaths occurring during periods of high circulation of influenza: cohort study in elderly people. BMJ. 2004;329(7467):660. doi:10.1136/bmj.38198.594109.AE. [PMC free article] [PubMed]
2. Goodwin K, Viboud C, Simonsen L. Antibody response to influenza vaccination in the elderly: a quantitative review. Vaccine. 2006;24(8):1159–1169. [PubMed]
3. McElhaney JE. The unmet need in the elderly: designing new influenza vaccines for older adults. Vaccine. 2005;23(suppl 1):S10–S25. [PubMed]
4. Ansaldi F, Canepa P, Parodi V, et al. Adjuvanted seasonal influenza vaccines and perpetual viral metamorphosis: the importance of cross-protection. Vaccine. 2009;27(25-26):3345–3348. [PubMed]
5. Carrat F, Flahault A. Influenza vaccine: the challenge of antigenic drift. Vaccine. 2007;25(39-40):6852–6862. [PubMed]
6. Jefferson T, Di Pietrantonj C, Al-Ansary LA, et al. Vaccines for preventing influenza in the elderly. Cochrane Database Syst Rev. 2010;(2):CD004876. [PubMed]
7. Nelson JC, Jackson ML, Weiss NS, et al. New strategies are needed to improve the accuracy of influenza vaccine effectiveness estimates among seniors. J Clin Epidemiol. 2009;62(7):687–694. [PubMed]
8. Simonsen L, Viboud C, Taylor RJ, et al. Influenza vaccination and mortality benefits: new insights, new opportunities. Vaccine. 2009;27(45):6300–6304. [PubMed]
9. Durando P, Icardi G, Ansaldi F. MF59-adjuvanted vaccine: a safe and useful tool to enhance and broaden protection against seasonal influenza viruses in subjects at risk. Expert Opin Biol Ther. 2010;10(4):639–651. [PubMed]
10. Hannoun C, Megas F, Piercy J. Immunogenicity and protective efficacy of influenza vaccination. Virus Res. 2004;103(1-2):133–138. [PubMed]
11. Corrao G, Zambon A, Nicotra F, et al. Issues concerning the use of hormone replacement therapy and risk of fracture: a population-based, nested case-control study. Br J Clin Pharmacol. 2008;65(1):123–129. [PMC free article] [PubMed]
12. Sturkenboom MC, Romano F, Simon G, et al. The iatrogenic costs of NSAID therapy: a population study. Arthritis Rheum. 2002;47(2):132–140. [PubMed]
13. CIRCOLARE del Ministero della Salute, Rome, Italy—02 August 2007. Prevenzione e controllo dell'influenza: raccomandazioni per la stagione 2007–2008. (In Italian). ( ). (Accessed March 21, 2012)
14. Instituto Superiore Di Sanita, Rome, Italy. Influnet: Sorveglianza Epidemiologica. (In Italian). ( ). (Accessed March 14, 2012)
15. Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13–22.
16. Zeger SL, Liang K-Y, Albert PS. Models for longitudinal data: a generalized estimating equation approach. Biometrics. 1988;44(4):1049–1060. [PubMed]
17. Nordin J, Mullooly J, Poblete S, et al. Influenza vaccine effectiveness in preventing hospitalizations and deaths in persons 65 years or older in Minnesota, New York, and Oregon: data from 3 health plans. J Infect Dis. 2001;184(6):665–670. [PubMed]
18. Brookhart MA, Schneeweiss S, Rothman KJ, et al. Variable selection for propensity score models. Am J Epidemiol. 2006;163(12):1149–1156. [PMC free article] [PubMed]
19. Stürmer T, Rothman KJ, Avorn J, et al. Treatment effects in the presence of unmeasured confounding: dealing with observations in the tails of the propensity score distribution—a simulation study. Am J Epidemiol. 2010;172(7):843–854. [PMC free article] [PubMed]
20. Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biometrics. 2005;61(4):962–973. [PubMed]
21. Rubin DB. Multiple imputation after 18+ years. J Am Stat Assoc. 1996;91(434):473–489.
22. Schafer JL. Analysis of Incomplete Multivariate Data. New York, NY: Chapman and Hall; 1997.
23. Stürmer T, Joshi M, Glynn RJ, et al. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol. 2006;59(5):437–447. [PMC free article] [PubMed]
24. Groenwold RH, Hoes AW, Nichol KL, et al. Quantifying the potential role of unmeasured confounders: the example of influenza vaccination. Int J Epidemiol. 2008;37(6):1422–1429. [PubMed]
25. Hak E, Verheij TJ, Grobbee DE, et al. Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications. J Epidemiol Community Health. 2002;56(12):951–955. [PMC free article] [PubMed]
26. Jackson LA, Jackson ML, Nelson JC, et al. Evidence of bias in estimates of influenza vaccine effectiveness in seniors. Int J Epidemiol. 2006;35(2):337–344. [PubMed]
27. Jackson LA, Nelson JC, Benson P, et al. Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors. Int J Epidemiol. 2006;35(2):345–352. [PubMed]
28. Nichol KL. Challenges in evaluating influenza vaccine effectiveness and the mortality benefits controversy. Vaccine. 2009;27(45):6305–6311. [PubMed]
29. Nichol KL, Nordin JD, Nelson DB, et al. Effectiveness of influenza vaccine in the community-dwelling elderly. N Engl J Med. 2007;357(14):1373–1381. [PubMed]
30. Simonsen L, Reichert TA, Viboud C, et al. Impact of influenza vaccination on seasonal mortality in the US elderly population. Arch Intern Med. 2005;165(3):265–272. [PubMed]
31. Simonsen L, Taylor RJ, Viboud C, et al. Mortality benefits of influenza vaccination in elderly people: an ongoing controversy. Lancet Infect Dis. 2007;7(10):658–666. [PubMed]

Articles from American Journal of Epidemiology are provided here courtesy of Oxford University Press