“That influenza vaccine prevents mortality from influenza, particularly among the aged and chronically ill, is based upon inference. It is presumed that vaccine protection demonstrated in studies among younger persons is similar among the aged and chronically ill, the group at particular risk of death should they acquire the disease. It is further assumed that such protection against clinical disease serves to protect them also against mortality associated with epidemic influenza. No studies, however, have yet been reported which measure the efficacy of the vaccine in prevention of influenza-associated mortality.” [3]

Consequently, the recommendation to immunize elderly people against influenza became “grandfathered” into practice in North America, precluding placebo-controlled trials for ethical reasons. Only three randomized controlled trials (RCTs) have since assessed the efficacy of split trivalent inactivated influenza vaccine (TIV) in preventing laboratory-confirmed influenza in the elderly [4]–[6]. None was conducted in North America, none assessed vaccine protection against serious outcomes, most involved young/low-risk elderly, and all used serologic outcomes, which, as recently suggested, may overestimate influenza vaccine effectiveness (IVE) [7]. Two trials assessed serologically-confirmed influenza illness in community-dwelling elderly showing significant vaccine protection of 58% (95% confidence interval [CI] 26–77%;Netherlands,1991–92; [4]) and 65% (16–85%;Bangkok, 1998–99; [6]). One assessed protection among nursing home residents with a non-significant vaccine efficacy of 50% (−26–80%;Russia,1996–97; [5]).

The remaining evidence for the benefit of TIV in the elderly comes entirely from observational studies [8]–[31]. Fedson et al were among the first to report influenza vaccine effectiveness (IVE) against hospitalization and death in the elderly using the linked immunization and health outcome administrative databases for the population of Manitoba, Canada [9]. Fedson et al reported IVE against hospital admission for pneumonia of 37–39% and against all-cause mortality of 27–30% during the 1982–83 and 1985–86 seasons. Subsequent observational studies showed similarly impressive estimates of vaccine protection. In a 2002 meta-analysis, Vu et al [20] summarized IVE against pneumonia and influenza hospitalizations across nine studies as 33% (95%CI 19–47%) and against all-cause mortality across four studies as 50% (95%CI 45–56%). In most studies, adjustment for underlying health conditions further increased estimates of IVE [10], [19], [21], [23], [25], [28]—an effect believed to be the result of accounting for confounding by indication whereby higher-risk patients may be more likely to receive vaccine [32]. Estimates were interpreted as reliable because the same high level of vaccine protection was not measured outside periods of influenza circulation, notably in the summer [13], [16], [21], [22], [24], [28], [29].

More recently a Cochrane meta-analysis summarized similar estimates of IVE but questioned the quality and interpretation of available data [31]. Others have expressed similar skepticism in particular because substantial increase in TIV coverage among the elderly from <15% to >65% over the past four decades has not been accompanied by the kind of decrease in influenza-related mortality implied by such vaccine protection [33]. IVE against all-cause mortality of 50% is counter-intuitive given that the fraction of winter excess mortality attributed to influenza has never exceeded 10% over those four decades [33], [34]. Rather than under-estimating IVE through confounding by indication, observational designs may have instead substantially over-estimated IVE through healthy user (or healthy vaccinee) bias whereby elderly patients with poorer prognosis may be less likely to receive vaccine compared to the healthy [34], [35].

Jackson et al illustrated this effect in estimates for IVE against hospitalization and death that were paradoxically higher pre-season than during the period of peak influenza activity [36]. In follow-up case-control and cohort analyses, including detailed chart review, Jackson and others were able to adjust for specific markers of functional status and frailty and reported marked attenuation of vaccine benefit [37], [38]. Most influenza deaths were accrued in a small but special subgroup of under-immunized and incapacitated elderly experiencing acute decline.

We defined immunization status based on vaccine receipt by the first two weeks of November for a given year (the period by which 90% of those immunized had received vaccine). Based on that consistent categorization, we found that the ultimately non-immunized group had higher hospitalization and mortality rates than the immunized before the period of influenza circulation. Given the impossibility of a true vaccine effect during the fall period prior to vaccine distribution, multiple analytical methods were evaluated based on the degree by which they could reduce this obvious positive bias. As in other studies, adjustment by standard regression and propensity score matching only exacerbated bias, substantially increasing IVE estimates against hospitalization and death before and during the influenza period.

Simonsen et al have suggested three criteria to identify residual bias in observational studies of IVE [33]. The first criterion specifies that a null vaccine effect should be observed during pre-influenza periods. As already discussed, this was not shown in our study. Secondly, IVE should be least pronounced during seasons when vaccine components are poorly-matched to circulating strains. As shown in Figures 2a/2b, we found no obvious correlation between vaccine match and IVE estimates. Lastly, IVE estimates should be most pronounced for more specific outcomes (e.g., pneumonia and influenza hospitalization) and less so for non-specific outcomes (e.g., all-cause mortality). In our study, as in others, the reverse was true.

In combination, this framework signals substantial bias in our estimates of IVE derived using classical methods and a well-established administrative database similar to that typically used by others reporting substantial vaccine protection. Further stratification based on prior influenza immunization revealed at least one obvious indicator of this bias: rates of hospitalization and death varied with change in immunization habit relative to the preceding two years. We showed high rates of hospitalization and death among those forgoing immunization after having twice consecutively received vaccine—an excess that was greatest even before the influenza period began—highlighting a nuanced form of healthy user bias. These individuals may represent the “healthy user recently turned unhealthy”, whose acute failure to receive vaccine is a marker for imminent decline, thereby contributing to hospitalization and death counts in the unimmunized and spuriously inflating estimates of vaccine protection. Baxter et al have also recently shown that forgoing immunization predicts death in those who had received vaccine in the previous five years but predicts survival in patients who had never before received vaccine [46]. We were able to show these same effects based on change from just two years of influenza immunization habit, suggesting that alteration in vaccine behavior is a robust predictor of death, albeit one which could not, through multivariable analysis, correct the bias described in this paper. In the opposite direction, seeking vaccine after forgoing for several years may reflect a change in health/risk status resulting in disproportionate contribution to hospitalization within the immunized group—a classic form of confounding by indication spuriously lowering IVE estimates. These forms of bias influence IVE estimates in opposite directions and may differentially affect hospitalization and death as serious outcome indicators, adding to the complexity of analysis based on administrative data sets.

Because of the numerous limitations outlined in this paper, we resisted citing specific IVE estimates in the discussion of our findings. Overall, compared to estimates reported by Fedson et al two decades ago in the same population and with similar methods, our aggregate estimates for vaccine protection during the influenza period are slightly higher against all-cause mortality but lower against hospitalization [9]. We have presented analyses using a specific hospitalization outcome (pneumonia, influenza, or acute respiratory disease listed only as most responsible admission diagnosis) which may not have captured all hospitalizations attributable to influenza; in preliminary analyses, however, less specific hospitalization outcomes were also explored (e.g., all-cause hospitalization), and trends in IVE across periods and methods were similar to those using the more specific definition. In this study, we applied the unconventional approach of assigning immunization status during the fall periods—before the campaign started—based on a future exposure, with the intent of illustrating baseline differences between those eventually immunized and those never immunized in a given season. This approach would have classified those who died between the fall analysis period and the availability of vaccine as unimmunized. Although these individuals did not have the opportunity to receive vaccine in the subsequent season, any effect of such misclassification on rates and IVE estimates would be small: of the 408 persons who were classified as unimmunized and hospitalized during the fall across the six study seasons, only 12 (3%) died before the start of the pre-influenza period.

While administrative data sets are recognized as efficient methods to estimate IVE, the validity of estimates derived in that way appears highly questionable. Our study illustrates the profound non-comparability of immunized and non-immunized individuals which is not corrected, but rather is exacerbated, by adjustment for standard confounders. Rather than continuing to assert potentially misleading evidence for protection, we see a few possible directions for research enhancement. More detailed exploration of covariates associated with acute decline is clearly needed. One example, which we were unable to explore in this study, may be a measure of suddenly stopping long-term medication for a chronic condition. Instrumental variables as a proxy for frailty could also be used [47], however, a reliable proxy not associated with the outcome has been challenging to identify in studies of IVE [48]. Case-control or nested case-control studies such as those conducted by Jackson et al have been successful in addressing functional status and frailty but such studies are labor-intensive and less amenable to annual repeat [36], [37]. Other methods have also been proposed [46], [49]–[51], but as long as original RCT evidence for vaccine protection in the elderly remains scant, it will be impossible to compare findings from observational studies against a reliable gold standard or to interpret true vaccine benefit. It may therefore be time to reopen the discussion for a properly designed RCT, with appropriate antiviral treatment, data monitoring, oversight, and stopping rules. Given that placebo-controlled trials may be ethically controversial in the elderly, randomization to standard versus enhanced (e.g., adjuvanted) formulations may be more acceptable.