The progression from data to evidence to evidence-based decisions, described up to now and portrayed in , is an idealization. In reality, the decision-making process is not so simple. Evidence is neither always perfect nor available when required. Even if it were, sources other than surveillance and epidemiologic data should and do influence decision making.
When confident estimates of the absolute and relative severity, transmissibility, and other key parameters of a novel influenza strain are unavailable, historical experience of pandemics can provide valuable evidence for planning.120
All 3 influenza pandemics that occurred in the twentieth century—in 1918, 1957, and 1968—shared a set of features: greater burden of severe disease in younger people than seen during seasonal flu; persistence of this pattern into subsequent years;121
and transmission outside of normal flu season in temperate regions. These common characteristics informed public health planning during the 2009 pandemic, especially during the early stages.120
Also valuable was knowledge of the differences among the pandemics: major variation in overall severity, variation in the impact on the elderly, and increased severity in the fall wave versus the spring wave (which occurred only in 1918). Awareness of these differences expanded the range of possibilities for which to plan. Even as evidence accumulated on overall severity of the 2009 pandemic and as estimates of the case-fatality ratio and other measures of severity declined, historical considerations justified planning for the contingency that severity could drastically increase.
Decisions to react aggressively to the pandemic were made when data could not reliably estimate per-case severity, so the possibilities consistent with the early data ranged from an outcome considerably milder than a typical flu season to one comparable to a severe flu season, or worse. In the face of this uncertainty, developed countries had to decide whether to invest billions of dollars in vaccine procurement. Such investments are cost-effective in average influenza seasons. Thus, the expectation they would be so even in a mild pandemic was justified—all the more so if severity were higher than a typical flu season or if the virulence of the virus changed.
Even if decision makers had known the pandemic strain's severity in 2009 was lower than envisioned in pandemic planning scenarios, we suspect vaccines would have been procured and that this would have been a sound decision. Suppose, for example, data had been available in April 2009 to definitively show a case-fatality ratio of well below 1 in 10,000 in all groups—a ratio below the lower end of current estimates for the 2009 pandemic virus in developed countries38,44,45,67
—but that the virus was transmissible from person to person and was spreading widely. Would it have been responsible for public health authorities to defer vaccine procurement, given the historical precedent for a mild infection to turn virulent in a matter of months?112–114
Arguably, historical experience would trump contemporary evidence from surveillance and other sources and call for an investment in prevention, regardless of the estimated severity of the disease at the time of the decision.
Public opinion affects policy decisions about pandemic response in at least 2 ways. First, since policymakers are ultimately responsible to the population, they must take into account a number of factors besides the (uncertain) projected public health benefits of a decision. A policy may receive little public support, even if it most efficiently uses resources to solve a public health problem, while another decision with little immediate benefit may be judged desirable.
Second, public opinion may constrain the range of policy options available to decision makers because these policies rely on voluntary decisions made by individuals. The use of an adjuvant-containing vaccine, for example, might be a prudent public health decision for maximizing the number of available doses, but public opposition might have made such a decision impractical in the U.S. even if regulatory concerns had been met. Political consequences aside, public opposition could also reduce uptake of the vaccine, potentially leading to counterproductive outcomes.
As evidence accumulates, it may be desirable to change policies; however, some factors may restrict such changes. First, implementing decisions and disseminating recommendations takes time—particularly when policies require hiring and training staff for “surge” operations—and such delays can render a policy change ineffective. Second, even if a policy could be changed, its benefit must be weighed against any potential undesirable effects, such as confusing the public, clinicians, or other recipients with a revised public health message. In extreme cases, officials can even lose their credibility if guidance is perceived as being inconsistent.
Limitations in how analysts and decision makers process, evaluate, and prioritize information also hinder the incorporation of surveillance evidence into decision making. Although the 2009 experience was not as severe as other scenarios considered in pandemic planning exercises, the response was nevertheless an extreme escalation in public health agency activity, with decisions made under conditions of stress, fatigue, and time pressure, as well as with limited information.
Such conditions make cognitive errors more likely.122,123
At the Symposium, decision makers discussed the difficulties of authenticating and balancing conflicting information and in prioritizing the many decisions required. They also reported that certain forms of data were a distraction. For example, following the very early stages of the 2009 pandemic, case counts were an unreliable indicator of infectious spread because of inconsistencies in testing and reporting.1
Spatial variation in case confirmation further challenged data interpretation. During the spring-summer wave, Milwaukee and the state of Wisconsin devoted more effort to case testing and confirmation than most other jurisdictions. However, the differences in their efforts were not known to all data recipients, creating confusion about how much of the geographic variation in reported pandemic flu activity was real and how much of it was due to differences in ascertainment.
All users of the data confronted these challenges, but they were particularly acute for decision makers, such as elected officials, who lacked direct access to the primary data. Generally, as new data are gathered, it is difficult to determine what biases exist and how much they distort the evidence. For example, in the first week of May 2009, an approximate 40-fold difference existed in the ratio of deaths to cases in data from the U.S. (about 0.1%) and Mexico (about 4%). While both figures were biased, it was unclear which (if either) accurately reflected case severity.
We have discussed the importance of mathematical and statistical modeling in surveillance and epidemiology data processing. Some of these techniques are unfamiliar to many public health officials, and the outputs of these models depend strongly on the quality of their inputs, which are often uncertain in a pandemic. Consequently, the greater the sophistication or complexity of a method, the more difficult it may be for decision makers to understand the strengths and limitations of the evidence provided.
Given the time constraints and other pressures just described, the usual scientific checks and balances of peer review, replication, and debate can be compressed into a very short time period for findings that are presented rapidly after their generation. Thus, in such a setting, there is an added responsibility for those who present decision makers with results of complex analyses to highlight the limitations and assumptions of their models, as well as to identify particularly robust predictions. As important, modelers need data to calibrate models and estimate their parameters, but they may lack an understanding of the biases and limitations of data collected in an emerging epidemic and about the possible changes over time in the ascertainment of cases. An additional responsibility in the 3-way interaction among public health agencies that gather data, decision makers, and modelers (some individuals may have more than one of these roles) is to ensure that these limitations are understood and accounted for in the process of modeling or other “processing” of the data.
It should also be noted that seasonal influenza epidemics occurring outside of pandemics offer opportunities to develop, disseminate, and explain new methods to the planners who will rely on them during an emergency. A complementary strategy, used in several places in 2009, is to “embed” mathematical modelers and statisticians with skills in the analysis of epidemic data within public health agencies during a pandemic, to facilitate rapid exchange of data, questions, and analyses.