|Home | About | Journals | Submit | Contact Us | Français|
Mathematical and simulation models are increasingly used to plan for and evaluate health sector responses to disasters, yet no clear consensus exists regarding best practices for the design, conduct, and reporting of such models. We examined a large selection of published health sector disaster response models to generate a set of best practice guidelines for such models.
We reviewed a spectrum of published disaster response models addressing public health or healthcare delivery, focusing in particular on the type of disaster and response decisions considered, decision makers targeted, choice of outcomes evaluated, modeling methodology, and reporting format. We developed initial recommendations for best practices for creating and reporting such models and refined these guidelines after soliciting feedback from response modeling experts and from members of the Society for Medical Decision Making.
We propose six recommendations for model construction and reporting, inspired by the most exemplary models: Health sector disaster response models should address real-world problems; be designed for maximum usability by response planners; strike the appropriate balance between simplicity and complexity; include appropriate outcomes, which extend beyond those considered in traditional cost-effectiveness analyses; and be designed to evaluate the many uncertainties inherent in disaster response. Finally, good model reporting is particularly critical for disaster response models.
Quantitative models are critical tools for planning effective health sector responses to disasters. The recommendations we propose can increase the applicability and interpretability of future models, thereby improving strategic, tactical, and operational aspects of preparedness planning and response.
Disasters such as the 2001 U.S. terrorist attacks, Hurricane Katrina, and the 2008 Chengdu earthquake have highlighted the need for improved public health and medical disaster response capabilities at local, regional, national, and international levels. Disasters are commonly defined as events whose consequences exceed the capability of local or regional medical and public health systems to provide necessary responses in a timely manner (Figure 1). We use the term “health sector disaster response” to encompass all activities of medical and public health systems affected by a disaster, including treatment of existing casualties and prevention and treatment of future casualties. Major unresolved issues remain at all levels of disaster response decision making, including long-range strategic planning, tactical response planning, and real-time operational support.* For example, how should the logistical systems for response to various types of public health disasters be organized? How much hospital-based surge capacity might be needed for potential mass casualty events? What is the most efficient and effective way to rapidly dispense medications or vaccines to large numbers of individuals?
Increasingly, simulation and mathematical modeling are being used to explore the capabilities, costs, and consequences of public health and medical disaster response plans. Members of the Society for Medical Decision Making and its journal have been active in this nascent field, publishing a variety of models including those that address the epidemiological and medical consequences of bioterrorism-related illness and pandemic influenza, logistical challenges during disaster responses, and the cost-effectiveness of specific disaster response strategies.1–5 Recently, the U.S. Department of Health and Human Services and the Centers for Disease Control and Prevention created initiatives to support modeling projects of relevance to their missions. However, the field of health sector disaster response modeling lacks a coherent thematological or methodological organization. Such organization could assist modelers in understanding and addressing the requirements of end users, facilitate consistent and systematic reporting of model inputs and results, and promote the development of standardized methodological approaches -- just as the Report from the Panel on Cost Effectiveness in Health and Medicine provides guidance for the field of cost-effectiveness analysis.6
This paper describes the range of models that have been developed for public health and medical responses to disasters and proposes a set of recommendations for developing and reporting the results of these models—something that has been lacking in the literature. This paper addresses three questions: 1) What kinds of health sector disaster response models have been developed? We describe exemplary and representative articles from the literature on applied decision support modeling for health sector disaster response, paying particular attention to the ways in which these articles differ from classic cost-effectiveness studies. 2) What methodologies have been and might be used for such models? We discuss the use of a range of modeling techniques for a variety of decision problems. 3) What are best practices for modeling and reporting of health sector disaster responses? We provide recommendations that we hope will serve as guides for the development of models focused on public health and medical responses to disasters; for funders of relevant modeling efforts; and for journals interested in publishing the findings of these models.
We sought to identify best practices in modeling health sector disaster responses and the reporting of such models. We focused on models that can support informed decision making about responses to public health disasters. We did not perform a systematic review; instead, we sought to identify a range of articles that used a variety of modeling methodologies to address a broad range of disasters and response decisions. The focus of our analysis was on modeling methodologies, model quality, and the reporting of model analysis and results—but not on the specific planning insights from the models. Thus, we do not present detailed results of analyses performed with the included models.
We searched for published models from relevant sources including MEDLINE, International Abstracts in Operations Research, content experts, and the authors of included models. We also searched the tables of contents of relevant journals in such diverse fields as economics, industrial engineering, operations research, nuclear technology, and risk analysis (e.g., Operations Research, Management Science, Risk Analysis, European Journal of Operational Research, Interfaces, Journal of Intelligent Transportation Systems, Nuclear Technology, Decision Analysis). From recent U.S. National Preparedness Guidelines,7 we identified a variety of disasters that could have a significant effect on public health.
We identified 567 potentially relevant articles, reviewed 120, and included 67 articles describing 55 models. The models we selected for inclusion are representative of the literature and are not intended to be exhaustive; however, they directly inform our recommendations. We included only articles that describe a quantitative model designed to support informed decision making about public health or medical responses to disasters. We excluded highly stylized mathematical models that have little immediate practical application (e.g., those focused on the mathematics of infectious disease progression), non-peer reviewed models and those presented in abstracts only, and articles not written in English.
From each included article, we collected information about the purpose of the model, type of disaster considered, aspect of response the model was designed to address, modeling approach, model setting, decision makers included in the model, decisions modeled, and model outputs. We also collected information about the quality of each model and its reporting based in part on established criteria for the evaluation of healthcare models (e.g., whether all relevant model inputs and their sources were described, whether all assumptions of the model were explained).6
As a team, we developed criteria for characterizing high quality modeling and model reporting and then developed a set of recommendations addressing the design, conduct, and reporting of such models. During these detailed discussions, we drafted and then revised the recommendations in an iterative manner. There were no significant disagreements among the writing team regarding model classification or resulting recommendations.
We sought critical feedback from content experts and members of the Society for Medical Decision Making (SMDM) on drafts of this position paper. We presented our results at the 2008 SMDM Annual Meeting in Philadelphia and the 2008 INFORMS conference in Washington, DC, and posted a draft of this paper on the SMDM website for additional comments. The final paper incorporates these comments.
Table 1 describes the included studies and outlines the purpose of each model, the disaster considered, the model design, and outputs. Included models address a variety of decision makers (e.g., first responders, hospital officials, planners, public health officials), geographic settings (e.g., local, regional, national), decisions modeled (e.g., dispensing, inventory/stockpiling, supply chain network design, prevention or mitigation of disaster effects, treatment, health care work force staffing, transportation), and outcomes evaluated (e.g., costs, morbidity, mortality, quality-adjusted life years, hospital utilization measures, logistical outcomes), and use a range of modeling methodologies. The quality of modeling methodology and model reporting was variable (Figure 2). The following sections provide an overview of response decisions modeled, outcomes evaluated, and modeling methodologies used in these selected papers.
We identified models that address the following broad categories of response decisions: treatment and control of disease outbreaks, response to natural disasters, response to manmade disasters, hospital planning, and disaster response logistics. These categories are not mutually exclusive (for example, a model examining response to a natural disaster could focus on logistical issues). Also, models that focus on treatment and control of disease outbreaks are, of necessity, significantly different from models that focus on responses to natural and manmade disasters; however, all of these disasters require public health and medical responses.
A variety of models have addressed preparation for and response to noncommunicable diseases such as anthrax or communicable diseases such as pandemic influenza, bioterrorist smallpox, and SARS. These models assess the impact of a variety of mitigation strategies, including vaccination, prophylaxis and treatment with antimicrobials, social distancing measures (e.g., masks, school closures), quarantine, travel restrictions, and vaccine development. Several notable models of disease outbreaks have been validated with available data from relevant prior outbreaks in humans3, 8, 9 and animal experiments.10, 11 In one study, researchers from three different groups compared the results of their pandemic influenza simulation models.12 They noted that, “the results need to be viewed more as helping structure thinking about pandemic planning, rather than being predictive of the precise effectiveness of different policies.”12
Despite the frequency and potentially catastrophic health consequences of natural disasters (e.g., in 2007, 406 natural disasters killed 17,000 people and affected another 201 million people13; and in 2004, the Boxing Day tsunami killed more than 200,000 people14), remarkably few published models have focused on the public health and medical responses to such events. Several of the included models focused on earthquakes,15, 16 including one that is geographically customizable and allows the user to estimate the impact of health facility damage, rescue time, and out-of-region transportation on overall mortality.15 One study of evacuation timing in response to the threat of a hurricane presented receiver operating characteristic curves for four coastal U.S. cities comparing the probability of correct landfall prediction against the probability of a false alarm.17
Relatively few models have addressed response to manmade disasters such as conventional warfare, radiologic and chemical accidents, acts of terrorism, and large-scale industrial accidents. Some manmade disasters such as war and famine have a slow onset, and may require sustained responses in which relief demands may have occasional spikes.18, 19 One model of famine in Sudan used data from the ongoing crisis there to determine the optimal reorder points and order quantities for relief supplies.18 The authors used simulation to evaluate the interdependence of critical response variables, and reported key insights that humanitarian agencies can use to improve disaster response.19 A study of a hypothetical airport accident used historical data and interviews with responders to simulate the relationships among transport time, triage classification, and patient survival rates.20
Several models have evaluated public health responses to nuclear incidents.21, 22 In general, these models evaluate tradeoffs among different accident mitigation objectives (e.g., minimize adverse health effects, minimize costs). One study of competing plans for distribution of potassium iodide after the release of radioactive iodine from a nuclear accident or terrorism showed how the preferred distribution plan changes as a function of the weights associated with different objectives.22
Hospital planning models for public health disasters have estimated demand for health care, evaluated triage strategies, planned hospital capacity, and developed appropriate protocols for decontamination and the use of personal protective equipment. Some of these models have been made publicly available in easy-to-use spreadsheets for use by planners for estimating demand for both inpatient and outpatient hospital services,4, 23 and have been used to estimate response capability in a variety of specific settings.24–26
Several models investigate hospital capacity during disasters and, importantly, evaluate potential response bottlenecks at key sites (e.g., emergency departments, operating rooms) and for key resources (e.g., specialized staff). 27–33 One model has been used to evaluate a hospital’s disaster plan, train decision makers, and assist in managing real situations by identifying bottlenecks and evaluating a variety of response strategies.27–29 One model evaluates tradeoffs in hospital strategies that involve altering the standard of care and increasing ICU surge capacity.30, 31 Another model estimates expected overcrowding of emergency departments due to adverse events from rapid vaccination campaigns.33
Logistical decisions are integral to many aspects of disaster response. These include decisions about procurement, transport, stockpiling, and maintenance of needed supplies; mass vaccination, prophylaxis and treatment; transportation and evacuation; and assignment of response personnel.
For many disasters, needed supplies must be rapidly dispensed to large populations. Several models evaluate the location, design, and operation of mass dispensing centers and other supply facilities (e.g., staffing, facility layout).3, 9 One publicly available model has been widely utilized by public health officials to plan dispensing of antibiotics, vaccines, and other disaster supplies in response to anthrax bioterrorism.3, 9, 34, 35 Another model, a decision support system that evaluates facility layout and staffing allocations for emergency mass dispensing, was tested in a field exercise conducted in Georgia.34, 35
Dispensing antimicrobials, vaccines, and other materials during a large-scale disaster response places enormous strain on local health sector infrastructure; conversely, the limits of that infrastructure may influence the course of the disaster. Thus, it is critical that models examining such scenarios evaluate the feasibility of modeled interventions. One pandemic influenza planning model considered the impact of vaccinating 10 million U.S. residents per week over the course of a multi-month pandemic response.36 An unresolved issue is how or whether such vaccination would be feasible given the public health and medical infrastructure, vaccine supply chains, and other aspects of current response logistics in the U.S.
Other important logistical decisions relate to the design and management of the disaster response supply chain including, for example, the location of supply centers, disaster recovery centers, and dispensing sites.37–39 Two notable models incorporate multiple components of the anthrax response supply chain: one considers local level components (e.g., dispensing points, dispensing processes, staffing and traffic management plans)40; the other considers national, regional, and local components.5, 41 Integrated models such as these can help identify the critical elements needed for efficient performance of a disaster response supply chain.
A variety of routing and transportation problems arise in response logistics planning. Models have been developed to determine the allocation of relief supplies and scheduling and routing of vehicles for the “last mile” distribution of supplies in response to a disaster,42 evaluate operational routing and loading decisions for helicopter dispatch in the aftermath of a disaster,43 optimize the transportation of triaged casualties to local hospitals for treatment,44 determine efficient ways to transport patients to hospitals, and model the evacuation of a populated area in anticipation of a disaster.20, 45, 46 Some studies investigate alternative routing of supplies after disaster-related damage to local infrastructure (e.g., bridge destruction after earthquakes), notably using real-world data to inform model inputs.44, 47, 48 A critical factor that can affect disaster response logistics is the behavior of the public and first responders. Several included models have considered such behavioral factors: for example, radiologic exposure after a dirty bomb and timeliness of care to patients as a function of use of cumbersome personal protective equipment;49, 50 contamination risk after chemical terrorism as a function of the behavior of the public and first responders;51 and costs and benefits of alternative communication strategies during an anthrax response as a function of the behavior of the public in response to public health instructions.52
The included models considered a wide variety of outcomes. For example, in addition to disease incidence and prevalence,53 the included models of infectious disease response considered outcomes such as hospitalizations,54 outpatient visits,53 deaths,54, 55 number of doses of drugs or vaccines required,55 and chance that a pandemic is averted.56 Hospital planning models typically evaluate the demand for various hospital services28, 29, 57 and patient waiting times,16 and often attempt to identify potential bottlenecks in the response.30, 31
Some analyses include costs and attempt to measure the cost-effectiveness of various planning and response strategies. For example, researchers have evaluated the cost-effectiveness (or cost-benefit) of strategies for preparation for and response to influenza pandemics,57–61 anthrax attacks,5, 52, 62, 63 and SARS.64 One study estimated the financial burden of incorrect evacuation decisions in response to hurricanes.17 Comparing estimated health outcomes and costs associated with different disaster response strategies may promote the effective and efficient use of limited public health resources.65
A number of the included response models did not consider intervention costs, sometimes as a result of direct proscription of such analyses by funding sources. In one analysis of the effectiveness of several response strategies for reducing inhalational anthrax mortality (e.g., pre-attack vaccination, antibiotic prophylaxis), the authors noted that their funders discouraged them from considering costs.66–68 They concluded that they had “mixed success” influencing policymakers, partly because one of their key findings was perceived as “too expensive and logistically difficult to achieve.”68
A variety of modeling methodologies have been applied to analyze disaster response decisions, with considerable variation in the type and complexity of mathematical and analytical techniques used. These include statistical analyses,32, 69 simple spreadsheet calculations,4, 54, 58 Markov models,17, 63, 70 epidemiological models,4, 53–55, 58, 61, 71, 72 supply chain and inventory management models,5, 18, 19, 41 facility location models,38, 39 and vehicle routing and network flow techniques.43, 44, 46–48 Some of the models are quite simple, and consider a relatively small number of entities (e.g., groups of individuals categorized by disease stage, or patients in a single emergency room),32, 55, 61, 71 whereas others simulate complex systems with thousands or even millions of individuals.36, 73
Because of the inherent uncertainty and complexity of disasters and disaster response, many researchers have used computer simulation to model potential outcomes of a disaster under different scenarios: for example, to determine staffing requirements at dispensing centers3 and plan dispensing strategies34, 35 after a bioterror attack, to determine effective routes and assignments for evacuating a population,46 and to evaluate strategies for transporting of victims of an airport disaster to hospitals.20, 45 One large-scale agent-based simulation models all individuals in a dynamically interacting urban population after a disaster such as that caused by biological, radiological, or chemical hazards.73 Such a model can be used to investigate a variety of logistical and operational decisions, but must be interpreted in light of the large number of input assumptions required.
Some researchers have made extensive efforts when developing models to incorporate input from disaster response planners, and to build models that planners can use. For example, one spreadsheet model for planning mass dispensing and vaccination clinics was developed in collaboration with county health officials74 and made available as a tool that emergency response planners can use to automatically generate spreadsheets customized for their own needs.75, 76
Based on our evaluation of the included models, we propose six recommendations for best practices for modeling and reporting of public health and medical responses to disasters. Clearly, models for any purpose, disaster responses or otherwise, should be rigorous in their design, execution, and reporting. The leaders in the field of modeling health sector responses to disasters come from a wide array of disciplines (e.g., clinical medicine, public health, medical informatics, operations research). Typically, their analyses follow established guidelines for best practices in modeling and model reporting from their relevant disciplines (e.g., 6, 77). The following recommendations identify specific additional features needed for modeling health sector responses to disasters that are not always included in other standards, and represent an effort to enunciate best practices for this new field of research and action.
Health sector disaster response models differ from many models in that they are designed to support ongoing planning endeavors. These range from long-term strategic decisions to immediate-term operational decisions. Thus, they might be used at one point for pre-event planning and later for decision making during an actual response. Moreover, in contrast to published reports of cost-effectiveness analyses that provide insights into a decision problem, disaster response models often need to be used for ongoing decision making by planners who must customize models for their own needs. To complicate matters, these planners typically do not have any particular modeling expertise—a fact that has critical implications for the design and reporting of such models.
A critical aspect of any disaster response model is that it be useful for addressing the practical planning problems faced by clinicians, first responders, and public health officials. Thus, such models should be able to realistically assess the health effects of disasters and the response capabilities of the public health and health care systems. To do so, these models should be designed to address realistic scenarios with inputs based on real-world data. Where data are not available, other related data from related disasters, drills and tabletop exercises, or expert opinion could be incorporated. For such models to be useful, their results should be validated to the extent possible—for example, by comparing model outputs to observed data from relevant disasters. Disaster response models should evaluate the feasibility (e.g., implementation logistics) of alternative response strategies proposed. We advocate the early involvement of key stakeholders as part of an iterative process to support the design of realistic and useful models.
Some highly stylized mathematical models can provide critical insights into disaster response planning; however, for many disaster response planning endeavors, ongoing decision making by end users is needed. In this case, they must be designed so that they (or appropriate end-user versions of them) can be readily used by response planners. Health sector disaster response models can play an important role in ongoing decision making, in local settings by response professionals (e.g., public health officials, physicians, emergency responders, hospital management, military planners, vaccine and pharmaceutical manufacturers, among many others) who may not be analytically trained but have considerable ground-level expertise. Thus, the input data and assumptions of models need to be clear so that planners can evaluate the relevance of model results. Models must provide results quickly and allow users to easily perform sensitivity analyses on different response scenarios. Good models will also allow users to evaluate tradeoffs in strategies (e.g., evaluating the tradeoffs between investment in local stockpiles versus the development of local dispensing capacity, or downgrading the standard of care in hospitals versus enhancing hospital surge capacity for the critically ill). Finally, the usefulness of such models is greatly enhanced when they are publicly available for use by planners, who can customize them for their own situations.
The appropriate level of complexity is difficult to define a priori; it is a function of the planning decision being addressed and the desired level of detail of the results. For example, models designed to consider dispensing center operations may need to provide detailed results, whereas models designed to evaluate broader strategic decisions such as comparisons of treatment options may reasonably provide less detailed results. Good models will be complex enough to provide credible insights into the planning problems they were designed to address while remaining as simple and tractable as possible, thus supporting the use of such models for decision making by planners.
Unlike traditional cost-effectiveness analyses, outcomes other than morbidity, mortality, and costs may also be important in the design or evaluation of a disaster response plan. Critical outcomes of disaster response may include resource utilization (e.g., hospital capacity usage, emergency department closure rates, and transportation network congestion) and other logistical measures (e.g., bottlenecks, evacuating timing); and measures related to behavior of responders and the public (e.g., patient adherence to prophylaxis and treatment, patient self-referral patterns). Unfortunately, few well-validated metrics exist for many of these key outcomes (this is an important area of active research). It may also be important to consider longer term outcomes such as the capacity for sustained response and long-term health effects (e.g., mental health effects) since plans for sustained responses may differ from those for rapid responses. Costs may not be particularly relevant for every decision that is analyzed. Modelers should carefully consider the appropriate outcome measures to include, as well as means for validating those metrics.
Public health disasters are rare events that may unfold in unpredictable ways. Thus, disaster response models must be designed to allow for comprehensive sensitivity analyses of uncertain model parameters and uncertain event scenarios. A notable feature of all preparedness activity is that plans must be made and paid for now in order to improve response to a wide range of possible future events, most of which will never occur. The cost-effectiveness of a disaster response plan is often critically dependent on the probability that the disaster occurs as well as its possible magnitude. One way to evaluate the effectiveness of a specific response plan under a variety of potential disaster conditions is with scenario planning.78 Scenario planning can highlight critical aspects of disaster response plans (e.g., bottlenecks) and can assist in designing response plans with adequate capacity and redundancy. As appropriate, good response models will incorporate the uncertainties associated with human and behavioral factors. For example, it may be important to evaluate a variety of assumptions about the behavior of both the public and first responders during an event (e.g., willingness of first responders to come to work in a dangerous environment, selection of treatment facilities by the public, adherence to recommended treatment/prophylaxis, behaviors in response to evacuation orders).
Given the rarity of most disasters, the uncertainties involved in both the events and their response, and the considerable societal importance placed on effective disaster response, detailed reporting of modeling methodologies, assumptions, and data sources is essential. Good model reporting will enable planners to customize analyses for their own purposes and compare the results of different models. A flow chart describing the event sequence may be helpful for planners who are interested in implementing the model. Good reporting of disaster models also includes the presentation of sensitivity analyses that identify key uncertainties, and the development of policy statements around these results. Additionally, reports of disaster response models should include specific insights for planners, describe how the model could be applied to response scenarios other than those specifically considered, and identify model limitations and key needs for future data collection.
Recent large-scale disasters in the U.S. and internationally have underscored the need for effective and efficient public health and medical responses. Partly as a result of these events, the field of disaster response modeling has emerged. The growing literature of disaster response modeling includes a number of exemplary models which we present in this paper. This field differs in key ways from traditional cost-effectiveness and healthcare modeling, leading to specialized requirements for model design and reporting. These informed our six recommendations: disaster response models should 1) be designed to address real-world disaster response problems, 2) be made available (in some form) for use by planners, 3) strike an appropriate balance between computational complexity and usability, 4) evaluate relevant disaster response outcomes, which often extend beyond those considered in traditional cost-effectiveness analyses, 5) explore critical uncertainties, and 6) be presented in sufficient detail that their results can be fully interpreted.
These recommendations are intended to increase the usefulness of disaster response models. Some of these recommendations are similar to those in established standards for modeling in some areas. For example, in medical cost-effectiveness analysis, it is standard practice to explore the effects of critical uncertainties on the results of the analysis. However, this is not necessarily the case in other modeling disciplines such as logistics or operations research. Other recommendations, such as making models available for use by planners, are not widely included in any published guidelines for public health or medical modeling.
The nascent field of modeling public health responses to disasters is reminiscent of the state of medical cost-effectiveness modeling in the 1970s and 1980s: It is highly heterogeneous in terms of methodologic approaches, outcomes evaluated, and quality of presentation. This has led to different approaches in framing and analyzing the research questions. As a consequence, it is typically not possible to compare results across models—even across models that evaluate related problems (such as treatment strategies for inhalational anthrax). Just as standard guidelines have facilitated the comparison of medical cost-effectiveness models by encouraging modelers to adhere to a set of principles (e.g., the adoption of the societal perspective when appropriate, discounting both costs and benefits, etc.),6 we hope that our recommendations will similarly help to standardize approaches adopted by modelers of public health responses to disasters.
Quantitative models are essential to planning effective health sector responses to disasters. The recommendations we propose should increase the applicability and interpretability of future models, thereby improving strategic, tactical, and operational aspects of preparedness planning and response.
Grant Support: This project received no funding support.
We thank the members of the Society for Medical Decision Making for their thoughtful comments on an earlier draft of this position paper. We thank Wei Xiong and Helen-Ann Brown from Weill Medical College of Cornell University and Mary Hill-Harmon and Peter Highnam from the Biomedical Advanced Research and Development Authority of the U.S. Department of Health and Human Services for their assistance with our literature review. The authors received no funding support for this project. Margaret Brandeau is supported by grant R01-DA15612 from the National Institute on Drug Abuse.
*Decision making can be considered strategic, tactical, or operational based on the spatial and temporal range of the decisions. For example, state officials make strategic decisions during the development of pandemic influenza response plans; county health officials make tactical decisions when procuring influenza response stockpiles based on those plans; and, during an actual response to an influenza pandemic, local health officials face the operational decisions of determining how to utilize staff and resources to most effectively respond to the emergency.