In this large population-based study, we applied a ‘ratio-of-ratios’ modeling approach to reduce the influence of difficult-to-measure individual-level confounders on the association between vaccination and three outcomes among older community-dwelling Ontario residents. The unmeasured confounders of greatest concern include physical frailty and dementia, which are incompletely captured in administrative health records and death certificates, but are likely associated with high mortality risks and low vaccination rates. During weeks of moderate-to-high influenza activity, influenza vaccination was associated with a (non-significant) 22% reduction in influenza-associated deaths (i.e. those deaths in the vaccinated population that would exceed an expected value in the absence of influenza circulation). Excess deaths occurring within 30 days of a pneumonia/influenza hospitalization, and excess pneumonia/influenza hospitalizations were significantly reduced, by 25% and 19%, respectively. As expected, no benefit from influenza vaccination was observed for UTI hospitalizations. Despite demonstrating a moderate level of VE for the three primary outcomes, the predicted mean annual numbers of events prevented in Ontario were small (139 all-cause deaths, 51 deaths in the 30 days following a pneumonia/influenza hospitalization, and 235 pneumonia/influenza hospitalizations) because the proportions of these deaths and hospitalizations that were associated with influenza activity were small. It is important to highlight that the protective effects of large-scale vaccination campaigns might be greater than these estimates because of indirect protective effects of such campaigns (i.e., herd-immunity) 
. However, the quantification of indirect effects is extremely difficult for diseases that cannot be definitely diagnosed without specific laboratory testing, including influenza infections 
. Thus, we have focused on the direct and more conservative benefits of influenza vaccination in this study.
We estimated the effectiveness of influenza vaccination for prevention of outcomes that: 1) occurred during weeks when specific laboratory testing for influenza revealed that influenza viruses were circulating at pre-specified levels in Ontario; and 2) were above a seasonally adjusted baseline of counts for each outcome. These outcomes are in contrast to those used in many other cohort studies, in which all events occurring during periods of any influenza virus circulation were used as outcomes measures. The importance of this difference is easily demonstrated: because only 6.0% of all deaths among Ontarians aged ≥65 years occurred during weeks when influenza circulated were above a seasonal baseline (and thus were categorized as influenza-associated), a VE of 22% for this outcome represented a 1.6% reduction in deaths among vaccinated individuals. This 1.6% reduction can be interpreted as a population-based estimate of vaccine effects on all deaths occurring during periods of influenza virus circulation. A meta-analysis of cohort studies calculated a VE of 47% for the prevention of all-cause mortality among community dwelling elderly during influenza seasons 
. Studies by Jackson et al
. and Mangtani et al
., demonstrated the bias inherent in cohort analyses by detecting putative vaccine benefits for non-specific outcomes when influenza activity is nil 
. Thus, we confirm with data from the most populous province in Canada that VE estimates not accounting for individual-level baseline risks for mortality are unrealistically optimistic. Because greater proportions of 30-day pneumonia/influenza deaths and pneumonia/influenza hospitalizations were attributed to influenza, greater percentages of these more specific events were averted by influenza vaccination, and thus the VE estimates of 25% and 19%, respectively, against these outcomes are likely more robust.
Our results can be compared directly with those from a few other studies that explicitly sought to adjust for unmeasured confounding, including healthy vaccinee effects. Using medical chart review to collect data on covariates not traditionally available in administrative data, Jackson et al
. estimated a VE of 8% (95% CI, −10%–23%) against community-acquired pneumonia during influenza seasons 
, consistent with our estimate of 19% VE (CI 4%–31%) and a 4.1% reduction of all pneumonia/influenza hospitalizations occurring during periods of influenza virus circulation among vaccinated individuals. The Armstrong et al
. study estimate of VE for influenza-associated all-cause mortality was 85% during 1996–2000. The cohort size of ~25,000 in that study meant that its 95% CI of 13%–100% covered essentially the entire possible range of vaccine benefits; thus their CI does include our point estimate 
. Fireman et al
. estimated that influenza vaccination was associated with a 47% reduction (95% CI not provided) in influenza-associated (as opposed to all) deaths between 1996 and 2005 in a single U.S. managed care plan 
. The same group also reported a 28% reduction (95% CI not provided) in influenza-associated pneumonia/influenza hospitalizations 
, which is similar to our VE estimate for pneumonia/influenza hospitalizations. Another recent study used an instrumental variable approach to estimate influenza VE. The use of an instrumental variable to adjust for unmeasured confounding is common in econometric analyses of observational data 
. In that Ontario study conducted during the 2000-2009 influenza seasons, VE among adults aged ≥65 years was 6% (95% CI, 0%–16%) for all-cause deaths and 14% (95% CI, 8%–21%) for a composite outcome of a pneumonia/influenza hospitalization or death 
; these results are comparable to our results on the reduction in deaths due to all-cause deaths or pneumonia/influenza hospitalization. The instrumental variable method depends on finding a covariate closely associated with vaccination, but unrelated to outcome; it is often difficult to find a good instrument. Hence, an advantage of the regression methods we used is that they are more broadly applicable and likely more generalizable for use in other populations.
This study has a number of strengths. We studied influenza vaccine effectiveness during 15 influenza seasons, representing ~21 million person-years of observation. We included temperature in our analyses, a variable that both affects winter mortality among older persons and is associated with influenza circulation, in temperate regions 
, which may have improved the precision of our VE estimates. By further developing methods pioneered by Armstrong et al
., and using three outcomes of varying specificity, we estimated the effectiveness of influenza vaccination in preventing serious influenza-associated events in individuals aged ≥65 years to a greater level of precision than previously. We also carefully specified whether a specific VE estimate applied to all events occurring during an influenza season, or to excess events occurring during periods of a specific level of influenza activity (e.g., when 5% of specimens submitted for influenza testing were positive). Finally, our analyses use a generalized linear model framework, and therefore are easy to implement using standard statistical analysis software packages.
Our study also has a number of limitations. First, although most Ontario residents aged ≥65 years receive influenza vaccination in physician offices, some are vaccinated in settings where billing claims are not submitted (e.g., clinics organized by public health departments). Billing claims were found to be 75% sensitive and 90% specific compared with self-report of influenza vaccination in one study 
. Misclassification resulting from use of billing data would bias our results towards the null as the unvaccinated group's risk would be falsely lowered because of the inclusion of misclassified vaccinated individuals. Second, cause-specific mortality data were not available. We used excess mortality within the 30 days following a pneumonia/influenza hospitalization to provide an outcome more specific for influenza than all-cause deaths. Third, measures of influenza virus circulation are a key data element in our analyses and the influenza surveillance data we used were potentially susceptible to ascertainment and testing biases over time. However, there were no major changes in data collection or laboratory methods during the study, and the weekly proportion of tests positive for influenza is a robust measure of viral activity 
. Fourth, because only a small number (2%) of influenza tests were positive for influenza B viruses, we could not provide a specific estimate of VE for influenza B-associated events, so VE estimates could be made only for influenza A-related outcomes. Finally, in common with all population-based retrospective cohort studies, we did not have data on laboratory-confirmed influenza infections from a per-protocol prospective testing scheme. It is unlikely that such data will ever be collected on a community- or province-wide scale because of the obvious logistical and resource requirements.
Based on our results and those from other studies, influenza vaccines that are more effective in preventing serious complications of influenza infections are clearly needed, particularly for older persons. Several strategies offering potentially more effective vaccines are being pursued. For example, a high-dose inactivated vaccine was licensed recently in the United States based on superior immunogenicity data 
. It will be important to assess whether new influenza vaccines prevent more serious albeit rare complications of influenza infections than the decades-old standard inactivated vaccines. Large observational studies using bias-reducing methods likely represent the only possible option to study the relative effectiveness of new versus standard influenza vaccines for the serious outcomes of greatest interest, including mortality. In addition, the methods we used may also be suitable for evaluating other large-scale public health interventions in populations in which unmeasured individual-level characteristics like frailty and dementia, for example, may be important confounders.