This study returned to the administrative databases that were among the first to show substantial reduction in serious influenza outcomes among immunized elderly. In revisiting the Manitoba database, we exposed similar evidence for bias that others have found, with the most pronounced but implausible effects (i.e., differences between immunized and non-immunized groups) observed in the elderly prior to influenza circulation or even vaccine distribution. We showed that change in immunization habit relative to the preceding two years may be a readily accessible and recognizable marker for this bias.
We defined immunization status based on vaccine receipt by the first two weeks of November for a given year (the period by which 90% of those immunized had received vaccine). Based on that consistent categorization, we found that the ultimately non-immunized group had higher hospitalization and mortality rates than the immunized before the period of influenza circulation. Given the impossibility of a true vaccine effect during the fall period prior to vaccine distribution, multiple analytical methods were evaluated based on the degree by which they could reduce this obvious positive bias. As in other studies, adjustment by standard regression and propensity score matching only exacerbated bias, substantially increasing IVE estimates against hospitalization and death before and during the influenza period.
Simonsen et al have suggested three criteria to identify residual bias in observational studies of IVE 
. The first criterion specifies that a null vaccine effect should be observed during pre-influenza periods. As already discussed, this was not shown in our study. Secondly, IVE should be least pronounced during seasons when vaccine components are poorly-matched to circulating strains. As shown in , we found no obvious correlation between vaccine match and IVE estimates. Lastly, IVE estimates should be most pronounced for more specific outcomes (e.g., pneumonia and influenza hospitalization) and less so for non-specific outcomes (e.g., all-cause mortality). In our study, as in others, the reverse was true.
In combination, this framework signals substantial bias in our estimates of IVE derived using classical methods and a well-established administrative database similar to that typically used by others reporting substantial vaccine protection. Further stratification based on prior influenza immunization revealed at least one obvious indicator of this bias: rates of hospitalization and death varied with change in immunization habit relative to the preceding two years. We showed high rates of hospitalization and death among those forgoing immunization after having twice consecutively received vaccine—an excess that was greatest even before the influenza period began—highlighting a nuanced form of healthy user bias. These individuals may represent the “healthy user recently turned unhealthy”, whose acute failure to receive vaccine is a marker for imminent decline, thereby contributing to hospitalization and death counts in the unimmunized and spuriously inflating estimates of vaccine protection. Baxter et al have also recently shown that forgoing immunization predicts death in those who had received vaccine in the previous five years but predicts survival in patients who had never before received vaccine 
. We were able to show these same effects based on change from just two years of influenza immunization habit, suggesting that alteration in vaccine behavior is a robust predictor of death, albeit one which could not, through multivariable analysis, correct the bias described in this paper. In the opposite direction, seeking vaccine after forgoing for several years may reflect a change in health/risk status resulting in disproportionate contribution to hospitalization within the immunized group—a classic form of confounding by indication spuriously lowering IVE estimates. These forms of bias influence IVE estimates in opposite directions and may differentially affect hospitalization and death as serious outcome indicators, adding to the complexity of analysis based on administrative data sets.
Because of the numerous limitations outlined in this paper, we resisted citing specific IVE estimates in the discussion of our findings. Overall, compared to estimates reported by Fedson et al two decades ago in the same population and with similar methods, our aggregate estimates for vaccine protection during the influenza period are slightly higher against all-cause mortality but lower against hospitalization 
. We have presented analyses using a specific hospitalization outcome (pneumonia, influenza, or acute respiratory disease listed only as most responsible admission diagnosis) which may not have captured all hospitalizations attributable to influenza; in preliminary analyses, however, less specific hospitalization outcomes were also explored (e.g., all-cause hospitalization), and trends in IVE across periods and methods were similar to those using the more specific definition. In this study, we applied the unconventional approach of assigning immunization status during the fall periods—before
the campaign started—based on a future exposure, with the intent of illustrating baseline differences between those eventually immunized and those never immunized in a given season. This approach would have classified those who died between the fall analysis period and the availability of vaccine as un
immunized. Although these individuals did not have the opportunity to receive vaccine in the subsequent season, any effect of such misclassification on rates and IVE estimates would be small: of the 408 persons who were classified as unimmunized and hospitalized during the fall across the six study seasons, only 12 (3%) died before the start of the pre-influenza period.
While administrative data sets are recognized as efficient methods to estimate IVE, the validity of estimates derived in that way appears highly questionable. Our study illustrates the profound non-comparability of immunized and non-immunized individuals which is not corrected, but rather is exacerbated, by adjustment for standard confounders. Rather than continuing to assert potentially misleading evidence for protection, we see a few possible directions for research enhancement. More detailed exploration of covariates associated with acute decline is clearly needed. One example, which we were unable to explore in this study, may be a measure of suddenly stopping long-term medication for a chronic condition. Instrumental variables as a proxy for frailty could also be used 
, however, a reliable proxy not associated with the outcome has been challenging to identify in studies of IVE 
. Case-control or nested case-control studies such as those conducted by Jackson et al have been successful in addressing functional status and frailty but such studies are labor-intensive and less amenable to annual repeat 
. Other methods have also been proposed 
, but as long as original RCT evidence for vaccine protection in the elderly remains scant, it will be impossible to compare findings from observational studies against a reliable gold standard or to interpret true vaccine benefit. It may therefore be time to reopen the discussion for a properly designed RCT, with appropriate antiviral treatment, data monitoring, oversight, and stopping rules. Given that placebo-controlled trials may be ethically controversial in the elderly, randomization to standard versus enhanced (e.g., adjuvanted) formulations may be more acceptable.
Until then, our findings add to the growing uncertainty about whether current influenza vaccines provide needed protection to the elderly. Insomuch as the elderly suffer the most severe consequences of influenza infection, resolving whether current vaccine options offer benefit and otherwise advocating for improved approaches should be priorities for the influenza immunization program.