PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of plosonePLoS OneView this ArticleSubmit to PLoSGet E-mail AlertsContact UsPublic Library of Science (PLoS)
 
PLoS One. 2011; 6(9): e24043.
Published online 2011 September 6. doi:  10.1371/journal.pone.0024043
PMCID: PMC3167826

Dynamic Health Policies for Controlling the Spread of Emerging Infections: Influenza as an Example

Reza Yaesoubi1,2,* and Ted Cohen1,2,3
Mike B. Gravenor, Editor

Abstract

The recent appearance and spread of novel infectious pathogens provide motivation for using models as tools to guide public health decision-making. Here we describe a modeling approach for developing dynamic health policies that allow for adaptive decision-making as new data become available during an epidemic. In contrast to static health policies which have generally been selected by comparing the performance of a limited number of pre-determined sequences of interventions within simulation or mathematical models, dynamic health policies produce “real-time” recommendations for the choice of the best current intervention based on the observable state of the epidemic. Using cumulative real-time data for disease spread coupled with current information about resource availability, these policies provide recommendations for interventions that optimally utilize available resources to preserve the overall health of the population. We illustrate the design and implementation of a dynamic health policy for the control of a novel strain of influenza, where we assume that two types of intervention may be available during the epidemic: (1) vaccines and antiviral drugs, and (2) transmission reducing measures, such as social distancing or mask use, that may be turned “on” or “off” repeatedly during the course of epidemic. In this example, the optimal dynamic health policy maximizes the overall population's health during the epidemic by specifying at any point of time, based on observable conditions, (1) the number of individuals to vaccinate if vaccines are available, and (2) whether the transmission-reducing intervention should be either employed or removed.

Introduction

The recent appearance of novel human pathogens such as H1N1 and H5N1 influenza, and SARS has stimulated efforts to develop methods to determine health policies that allow for the rapid modification of public health interventions in response to varying epidemiological situations and changing availability of information [1], [2]. In this paper, we examine the potential utility of dynamic health policies for controlling the spread of an emerging infectious disease. Dynamic health policies make real-time recommendations, in response to changing disease characteristics (e.g. infectivity, antimicrobial resistance levels), population characteristics (e.g. disease prevalence, proportion of individuals that are immune), and resource constraints (e.g. vaccines, antimicrobial drugs, personnel, and budget).

Most existing approaches for identifying optimal strategies for infectious disease control use simulation or mathematical models of disease spread to compare the performance of a limited number of pre-determined health policies. A number of these studies aim to identify optimal health policies for vaccine allocation before the start of an epidemic without explicitly considering interventions which can be employed during the epidemic [3][6]. A larger number of studies investigate the effect of both initial immunization and the use of controlling interventions during epidemics, such as use of antiviral for treatment, case isolation, school closure, and internal travel and border restrictions [7][13].

Although these approaches can provide insight into which baseline strategies may best reduce the impact of epidemics, they are not generally structured to assist real-time decision making through the dynamic change of health recommendations as new data become available over the course of epidemic. In this paper, we focus on developing optimal dynamic health policies for controlling an emerging human pathogen. These policies allow decision-makers to use cumulative real-time data from the epidemic and current information about resource availability to guide their selection of possible interventions at any particular point in time.

We use a simplified model of influenza spread to illustrate the development and the employment of dynamic health policies. Control of influenza epidemics may involve: (1) reducing the susceptibility of uninfected individuals either before or during the epidemic (through vaccination or antiviral prophylaxis), (2) reducing contact rates in the population (through social distancing such as isolation of diagnosed cases, quarantine of households of diagnosed cases, closing of schools), and (3) reducing the infectiousness of infected individuals (through treatment or isolation).

While vaccines provide protection from seasonal influenza, the emergence of a novel strain would likely mean the absence (or shortage) of an effective vaccine for at least the first several months of the epidemic [14], [15]. During this period, control strategies would largely rely on social distancing and potentially on stockpiles of antiviral drugs used for treatment and/or prophylaxis. Therefore, efforts to control an emerging influenza epidemic would be bounded by (1) the availability of effective vaccines and antiviral drugs, and/or (2) the availability of money and resources for vaccine procurement, diagnosis and treatment of new cases, and the actual implementation of transmission reducing interventions during the epidemic. Under such resource constraints, we define the optimality of a dynamic health policy as the efficient use of available resources (e.g. budget and medical system capacity) to maximize the overall health of the population (e.g. minimizing the number of deaths or hospitalizations, or maximizing other measures such as quality-adjusted life years).

In this paper, we define several broad types of interventions for controlling the spread of an emerging influenza virus. We use these crudely classified interventions and a simplified model of influenza transmission to illustrate the use of a dynamic optimization methodology (namely, Markov Decision Process [16]) to specify dynamic health policies. Here, rather than focusing on developing a comprehensive model of influenza spread, we demonstrate how these dynamic policies allow real-time decision making under different resource constraints. We also discuss how future research in this area can help make the implementation of these policies possible.

Method

In this section, we first discuss the set of possible interventions that can be used for controlling an influenza epidemic. We describe the effect of these interventions on disease spread as well as on the overall health-related and monetary consequences of the epidemic. We then define dynamic health policies for controlling the epidemic and characterize their optimality. We finally propose a methodology to identify the optimal dynamic health policies for controlling the epidemic.

A Model for Controlling the Spread of Influenza

Influenza Epidemic State and Decision Sets

An influenza epidemic is usually described by a SIR (Susceptible-Infective-Recovered) model, in which the individuals recovered from infection are assumed to acquire permanent immunity to that viral strain [7], [17]. Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e001.jpg denote the number of susceptibles, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e002.jpg denote the number of infectives, and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e003.jpg denote the number of recovered at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e004.jpg. Since influenza epidemics usually last for several months and the number of deaths is generally small relative to population size, it is reasonable to assume that the population size does not change over the course of epidemic. For a population of a fixed size An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e005.jpg, the state of the disease spread at any given time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e006.jpg can be identified by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e007.jpg. Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e008.jpg denote the state space defined as An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e009.jpg.

Decisions are made at points of time referred to as decision epochs. It is more convenient to assume that the decisions are made at discrete points of time, rather than continuously over time; hence, we assume that the set of decision epochs, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e010.jpg, is discrete; that is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e011.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e012.jpg denotes the decision horizon length. We classify the possible interventions to control the spread of influenza into two categories: (1) “irreversible” interventions such as vaccination employed either before or during the epidemic which reduce the number of susceptibles, and (2) “reversible” interventions which can be turned on and off during the course of epidemic to reduce the transmission of infection to susceptibles, such as hygienic interventions, social distancing, and treatment. We implement these two types of decisions in our model as follows:

  1. Vaccination: At any decision epoch, conditional on the availability of vaccine, the decision maker will specify the number of susceptibles to vaccinate. If effective vaccines are abundant and vaccination has zero cost, this decision is trivial: vaccinate all susceptibles. However, throughout this paper we assume that vaccine, if available, is acquired at a price. We denote this decision by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e013.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e014.jpg is the set of possible values for the decision variable An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e015.jpg. For cases where vaccine availability exceeds need, we may assume that An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e016.jpg. For simplicity, we assume that vaccination at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e017.jpg results in immunization by the next decision epoch. Hence, a decision to vaccinate An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e018.jpg portion of susceptibles at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e019.jpg results in the reduction of susceptibles to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e020.jpg by the next decision epoch.
  2. Transmission-reducing intervention: These interventions may be either employed or lifted over the course of epidemic to reduce the transmission of infection to remaining susceptibles. These measures will include social distancing (e.g. school or public place closure), hygienic interventions (e.g. mask use), and treatment or isolation of cases. Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e021.jpg denote the set of such interventions, where 0 represents “no intervention.” We denote the transmission-reducing decision made at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e022.jpg by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e023.jpg.

We categorize “treatment” as a transmission-reducing intervention which can be turned on or off during the epidemic. Of course, when sufficient antivirals are available, all new cases receive appropriate treatment, in which case “treatment” will not be included as a decision in the model. However, under conditions of antiviral limitation, one may include “treatment” as an intervention which can be turned on or off during the epidemic; this situation may occur when the use of antiviral must be prioritized among population subgroups [3], [4].

To control the epidemic, a policy maker will continue to make decisions until the prevalence of the disease is sufficiently low. The stochastic nature of transmission (which is especially important during emergence or eradication), prevents accurate identification of the time when the disease will be eradicated; hence, we consider an infinite decision horizon: An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e024.jpg [16]. Although the decision horizon is infinity, we assume the decision making process stops when there are no more infectious individuals in the population.

Rewards and Transition Probabilities

As the result of vaccination employed at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e025.jpg, the number of susceptibles is reduced from An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e026.jpg to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e027.jpg by the next decision epoch, and the policy maker receives a reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e028.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e029.jpg is the unit price of vaccine.

As the result of a transmission-reducing intervention employed at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e030.jpg, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e031.jpg, the spread of disease at the next decision epoch is determined by the probability distribution An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e032.jpg, and the policy maker receives a reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e033.jpg.

The reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e034.jpg can be characterized in several ways and the choice of reward structure should reflect the policy maker's set of priorities. For example, if the policy maker wants to minimize the total number of individuals infected over the course of epidemic, then reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e035.jpg can be simply defined as An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e036.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e037.jpg is the number of new infections during the period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e038.jpg. However, efforts to control epidemics may be bounded by the availability of medical resources, such as vaccines, medical personnel, and antiviral drugs, and monetary resources for vaccine procurement, diagnosis and treatment, and implementation of interventions. In these situations, where both health-related outcomes and the resource consumption level are essential for determining the optimality of a health policy, a more comprehensive reward function is needed. A common approach for defining optimality in these situations is to assume that the policy maker's objective is to maximize the population's net monetary benefit [18]. To characterize the reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e039.jpg accordingly, we must define several additional parameters:

  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e040.jpg: policy maker's willingness-to-pay (WTP) for health.
  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e041.jpg: cost incurred for each incident infection (this may include diagnosis, treatment and other indirect costs).
  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e042.jpg: cost of implementing the intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e043.jpg for period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e044.jpg; we assume An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e045.jpg.
  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e046.jpg: expected costs incurred during the period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e047.jpg if the disease spread at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e048.jpg is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e049.jpg and the policy maker implements the intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e050.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e051.jpg; An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e052.jpg.
  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e053.jpg: loss in health (quantified by quality-adjusted life years) due to infections.
  • An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e054.jpg: expected loss in population's health during the period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e055.jpg if the disease spread at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e056.jpg is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e057.jpg and the policy maker uses intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e058.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e059.jpg; An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e060.jpg.

Now, the reward An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e061.jpg, defined as the expected net monetary benefit during the period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e062.jpg if the disease spread at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e063.jpg is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e064.jpg and the policy maker chooses intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e065.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e066.jpg, is calculated by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e067.jpg.

Decision Rules, Health Policies and Optimality

A decision rule prescribes an action for each state for a specified decision epoch. For decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e068.jpg, a decision rule is a function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e069.jpg, which specifies the number An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e070.jpg of susceptibles to vaccinate given the initial disease state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e071.jpg. For decisions that must be made during the course of the epidemic (An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e072.jpg), we focus on a Markovian decision rule because this is the most convenient to implement and evaluate. Such decision rules are functions An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e073.jpg, which for each state of the disease spread, assign a transmission-reducing intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e074.jpg and a proportion An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e075.jpg of susceptibles to vaccinate. This decision rule is said to be Markovian (memoryless) because it depends on previous disease states and previously utilized interventions only through the current state of the disease spread.

A policy specifies a decision rule to be used at all decision epochs. In other words, a policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e076.jpg is a sequence of decision rules An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e077.jpg. We call a policy stationary if An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e078.jpg for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e079.jpg; that is, the policy prescribes the same decision for the given state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e080.jpg regardless of the time period in which this state is reached. In this paper, we are only interested in characterizing stationary health policies since they are the most feasible to implement in practice and their optimality can be proven [16].

Assuming that the influenza spread is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e081.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e082.jpg, the expected total discounted reward induced by policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e083.jpg over the course of epidemic is calculated as:

equation image
(1)

where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e085.jpg is the discount factor to account for the time value of the future rewards.

Now, assuming that at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e086.jpg, the state of the disease spread is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e087.jpg, the expected total reward induced by policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e088.jpg is:

equation image
(2)

Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e090.jpg denote the set of all possible stationary policies An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e091.jpg. We say that a policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e092.jpg is optimal whenever:

equation image
(3)

Define An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e094.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e095.jpg is calculated by Eq. (1) for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e096.jpg.

Implicit in the definition of stationary policies is the assumption that model parameters are known and do not change over time. This assumption allows the existence of an optimal stationary policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e097.jpg, where the function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e098.jpg is time independent. Undoubtedly, this assumption may be violated for emergent influenza epidemics, and the model parameters may need to be updated as new data accrues over time. We return to this issue in later sections and discuss how stationary policies can be determined in this setting. But, for clarity of presentation, we first assume that the parameters of the influenza model are constant and estimable from the initial spread of the epidemic.

A Markov Decision Process Formulation for Influenza Spread

There are several dynamic optimization methodologies that can be used to find the policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e099.jpg defined in inequality (3). The most appropriate optimization method depends on the complexity of the underlying epidemic model, the observability of the epidemic state, and the desired level of computational efficiency. As a rule of thumb, finding the exact optimal policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e100.jpg becomes more challenging (and sometimes impossible) as (1) the complexity of epidemic model increases or (2) the uncertainty around the true state of the disease spread arises. Several methodologies can help identify the approximate optimal policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e101.jpg even when the epidemic model is relatively complex and only some probabilistic knowledge about the current state of the epidemic can be obtained. We discuss the challenges of optimal decision-making under uncertainty at further length in the Discussion section.

If the disease dynamics can be modeled by a discrete-time Markov chain and the state of the epidemic is observable over the course of epidemic, then the stationary optimal health policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e102.jpg in inequality (3) can be efficiently obtained through Markov Decision Process (MDP) [16]. In our illustrative example of a novel influenza epidemic, we make several simplifying assumptions about the spread of influenza to be able to use MDP. We enumerate these assumptions in the following subsections; we then discuss how these simplifying assumptions can be relaxed in further work.

We describe epidemic influenza with a SIR model, in which individuals acquire permanent immunity through infection or vaccination. We do not consider the possibility of changes in the population size due to birth, immigration, or death in order to simplify the analysis. Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e103.jpg denote the time interval between two consecutive decision epochs. We assume that individuals become infected only through contact with other infectious members of the population, and that contacts during the interval An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e104.jpg occur according to a homogenous Poisson process, with rate An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e105.jpg.

We assume that a susceptible who is infected during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e106.jpg becomes infectious and symptomatic at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e107.jpg and interacts with the rest of the population during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e108.jpg; the individual is then removed from the population (or recovered) at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e109.jpg. As we will describe later, these simplifying assumptions allow the use of Markov decision process to identify the optimal decision at each decision epoch. In the Discussion section, we explain how these assumptions can be relaxed.

Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e110.jpg denote probability that a susceptible person becomes infected upon contact with an infectious individual and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e111.jpg denote probability that the next interaction of a random susceptible person is with an infectious person. When social distancing has not been used and mixing is homogenous, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e112.jpg is equal to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e113.jpg. Variables An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e114.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e115.jpg can be respectively modified by “hygienic interventions” (reducing the chance of transmission given contact between infectious and susceptible individuals) and “social distancing” (reducing the likelihood of contact between susceptible and infectious individuals). Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e116.jpg denote overall probability that a susceptible person becomes infected. This probability is calculated in [19] as:

equation image
(4)

Hence, given the state of the disease spread, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e118.jpg, the number of new infections during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e119.jpg, denoted by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e120.jpg, will have a binomial distribution with number of trials An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e121.jpg and the probability of success An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e122.jpg:

equation image
(5)

The transition probabilities of the Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e124.jpg can then be calculated by (refer to Text S1 for detailed steps):

equation image
(6)

We consider two types of interventions: (1) vaccination and (2) transmission-reducing interventions. First, we assume that no vaccines will become available any time during the epidemic; therefore, the health policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e126.jpg only specifies the optimal transmission-reducing intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e127.jpg to implement at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e128.jpg. We will show later how optimal vaccination decisions can be made once vaccine becomes available. Also, for simplicity, we assume that only one transmission-reducing intervention is available; hence An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e129.jpg. Selecting intervention 1 at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e130.jpg, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e131.jpg, reduces probability An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e132.jpg in Eq. (4) to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e133.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e134.jpg is the fractional reduction in the infection transmission rate.

Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e135.jpg denote the probability that the influenza epidemic will be in state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e136.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e137.jpg, given that that the state is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e138.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e139.jpg and the policy maker implements intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e140.jpg at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e141.jpg. This probability is calculated by transition probability (6).

Since we first assume that no vaccine is available during the course of the epidemic (we relax this assumption below), the policy maker can only select decisions from set An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e142.jpg. Given this assumption, the optimal solution to the policy maker's problem during the course of epidemic (i.e. maximizing Eq. (1)) is obtained by solving the following set of recursive equations [16]:

equation image
(7)

By definition, function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e144.jpg returns the expected total discounted reward induced by the optimal policy over the remaining course of epidemic if the current state of epidemic is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e145.jpg. Therefore, having found the function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e146.jpg, by solving the set of equations (7), we can then determine the optimal transmission-reducing decision for a given state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e147.jpg by:

equation image
(8)

Results

To illustrate the use of the proposed methodology, we consider the case of an influenza outbreak in an English boarding school reported in [20] and recently used by [2] and [21]. The population consisted of An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e149.jpg students and the infection was believed to be introduced by one student returning from Asia. The situation satisfies many requirements of a simple SIR model, particularly since no specific intervention was employed during the outbreak.

For a population of size An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e150.jpg, the transition probability matrix of the Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e151.jpg is of size An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e152.jpg, which causes computational problems for our effort to identify optimal health policies. To overcome this computational difficulty, several effective alternative solutions have been proposed in the literature of dynamic optimization; the reader is referred to [22] for comprehensive discussion. One approach to reduce the state space of the Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e153.jpg is “state aggregation”, in which the Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e154.jpg is approximated by the Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e155.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e156.jpg, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e157.jpg, is the proportion of population in class An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e158.jpg at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e159.jpg. Detailed steps for how one can make these approximations are provided in the attached Text S1. Note that although we consider a relatively small population here, however, the approximation method briefly described above can also be used for larger populations.

The influenza spread model described previously has two parameters, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e160.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e161.jpg, which should be estimated from the data. Using maximum likelihood estimation, we estimate An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e162.jpg per day and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e163.jpg. Figure 1 shows each day's expected number of new infections calculated by the model versus the observed data. Note that in Figure 1, the observations on days 1 and 2 are not shown. The reason is as follows: since in the approximate Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e164.jpg, the states of influenza spread are aggregated, the model is not able to accurately capture the state of the epidemic when the number of infections is very low. For the observations presented in Figure 1, the number of new cases in days 1 and 2 are respectively 1 and 6, which are too low to be captured by the approximate Markov chain An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e165.jpg. For detailed discussion, refer to Text S1 and [19].

Figure 1
Observed new cases in the English boarding school versus the model's predictions.

To determine optimal dynamic health policies for this population, we use the following arbitrary settings. We consider one transmission-reducing intervention which reduces the rate of infection transmission by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e167.jpg, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e168.jpg, and costs An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e169.jpg $.,2000 per day. One such transmission-reducing intervention might be “having all students wash their hands twice a day”. The vaccine price is set to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e170.jpg; each incident infection costs An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e171.jpg to diagnose and treat and results in a health loss of 0.00342 QALY (assuming that the treatment period lasts 5 days during which the health quality of the patient is reduced by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e172.jpg: An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e173.jpg).

Optimal policies during the epidemic are obtained by solving the set of equations (7) for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e174.jpg and then using Eq. (8) to find the optimal decision An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e175.jpg for each state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e176.jpg. We use a policy iteration algorithm [16] with a discount factor An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e177.jpg to solve the set of equations (7).

Figure 2 displays the optimal health policies for WTP for health equal to $.,25000/QALY. The conditions for recommending the use of the transmission-reducing intervention are presented as the grey regions within the triangle in the upper part of each figure. For example, the transmission-reducing intervention should be implemented if the state of the disease is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e178.jpg, and should not be used when the state of the disease is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e179.jpg. Text S1 includes additional health polices corresponding to different levels of WTP.

Figure 2
Optimal stationary health policy for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e180.jpg$.,25000/QALY.

As discussed previously, we must also identify the optimal number of susceptibles to vaccinate when an effective vaccine becomes available. Let us assume that the entire population is susceptible before the start of epidemic, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e184.jpg, among which An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e185.jpg susceptibles become infectious by time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e186.jpg. If we assume that effective vaccines become available at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e187.jpg at a market price An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e188.jpg, then the policy maker determines the number the suscpetibles to immunize, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e189.jpg, by solving the following optimization problem:

equation image
(9)

in which, the variable An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e191.jpg only takes integer values over the interval An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e192.jpg.

The rectangle in bottom of Figure 2, labeled as “% of susceptibles to immunize at An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e193.jpg,” represents the optimal health policies for immunization before the start of epidemic; again, if vaccine is not yet available, it may not be possible to achieve these levels of immunization. These recommendations are obtained by solving problem (9).

Employing Dynamic Health Policies to Control Influenza Spread

In this section, we discuss how the optimal dynamic health policy for using the transmission-reducing intervention and vaccinating additional susceptible individuals in decision epochs is determined as an epidemic progresses and new data become available.

Table 1 shows the observable information that accrues during an epidemic. Before the start of epidemic, at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e194.jpg, the policy maker obtains an estimate for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e195.jpg, denoted by An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e196.jpg. For a novel strain of influenza, we assume that the entire population is susceptible, hence An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e197.jpg for these situations. The policy maker also obtains a prior distribution for the number of susceptibles who may become infectious by time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e198.jpg. Let the number of infectives at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e199.jpg be randomly distributed according to probability mass function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e200.jpg with support An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e201.jpg. The policy maker can then use the following optimization problem to find the number of susceptibles to immunize at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e202.jpg:

equation image
(10)

in which, the variable An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e204.jpg only takes integer values over the interval An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e205.jpg.

Table 1
Observed information over the course of an epidemic.

Using observation of the number of new infections occurring during the epidemic, we update our knowledge on the state of the epidemic for the next decision epochs as follows. At the beginning of period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e221.jpg, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e222.jpg new infections are observed which implies that the number of susceptibles at this time is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e223.jpg. At the end of each period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e224.jpg, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e225.jpg new infections are observed. This observation implies that (1) the number of susceptibles at the beginning of period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e226.jpg is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e227.jpg and (2) the number of infectives at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e228.jpg is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e229.jpg, since we assume that all infectives at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e230.jpg are removed or recovered by the next decision epoch (see Table 1).

Now, knowing that the epidemic is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e231.jpg and assuming that no vaccine is available at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e232.jpg, the optimal transmission-reducing decisions for period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e233.jpg is obtained by solving Eq. (8) (which can also be summarized in form of the policy shown in Figure 2).

Now assume that at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e234.jpg, vaccines for the epidemic strain of influenza either become available for the first time or that depleted stocks of vaccine have now been replenished. Let An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e235.jpg denote the expected reward if influenza spread is in state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e236.jpg at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e237.jpg, the transmission-reducing intervention An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e238.jpg is implemented during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e239.jpg, and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e240.jpg susceptibles are vaccinated by the decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e241.jpg. Then, function An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e242.jpg is calculated as:

equation image
(11)

where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e244.jpg. The optimal transmission-reducing intervention (An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e245.jpg) and the number of susceptibles to vaccinate (An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e246.jpg) during the period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e247.jpg is then determined by solving the following problem:

equation image
(12)

where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e249.jpg is given by Eq. (11) and the variable An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e250.jpg only takes integer values over the interval An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e251.jpg.

As briefly pointed out before, to determine the stationary health policy An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e252.jpg, we assumed that the parameters of the influenza spread model are all known and do not change over time. It is, however, more realistic to assume that as new data become available over the course of epidemics, policy makers also update the parameters of the underlying transmission model. Table 2 outlines a procedure describing how dynamic health policies can be employed when policy maker is using the real-time data to also update the parameters of the influenza spread model.

Table 2
Determining Dynamic Health Policies Using Real-Time Data.

Optimal Dynamic Health Policies under Resource Constraint

The optimal health policies developed in the previous section do not consider the availability of resources (e.g. vaccines, budget, and antiviral for treatment) while making health recommendations. For example, the policy presented in Figure 2, recommends vaccinating An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e269.jpg of the remaining susceptibles when the initial proportion of susceptibles is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e270.jpg; this recommendation does not account for the number of vaccines that are actually available. Similarly, the recommendations for turning on and off the transmission-reducing intervention do not account for the budget and resources necessary to initiate or terminate these recommendations. The framework proposed here can be expanded to incorporate different forms of resource constraints while constructing dynamic health policies. In this section, we discuss policy development under three types of resource limitations.

Let us first consider the case where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e271.jpg vaccines are available at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e272.jpg, and the policy maker must decide how many susceptibles to vaccinate using these An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e273.jpg vaccines. For now, let us assume that no additional vaccines will become available during the epidemic and any unused vaccine by time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e274.jpg will be lost (e.g. shipped to other communities). The policy maker can now use Figure 3 to select the WTP for which a feasible vaccination recommendation can be implemented. For a given initial proportion of susceptibles An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e275.jpg, Figure 3 identifies the number of vaccines required for each value of WTP (An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e276.jpg) through solving the optimization problem (10). As an example, for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e277.jpg, if the policy maker's sole objective is to minimize cost (i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e278.jpg) 352 vaccines are required and for any WTP An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e279.jpg $.,25000/QALY, 367 vaccines should be used. Note that for An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e280.jpg, vaccinating less than 352 susceptibles results in a monetary loss, and vaccinating more than 367 susceptibles does not increase the population's expected net monetary benefit for any An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e281.jpg $.,25000/QALY.

Figure 3
Number of Vaccines Required.

Now to demonstrate how a vaccine supply constraint can be accounted for in constructing an optimal health policy, let us assume that for our population, at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e282.jpg (before the start of epidemic), there are An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e283.jpg vaccines available (the same approach can be followed for all other decision epochs). When An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e284.jpg vaccines are available, then according to Figure 3, for any initial proportion of susceptibles An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e285.jpg any health policy with An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e286.jpg $.,25000/QALY can be used to optimally allocate all of the available vaccines. For any initial proportion of susceptibles An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e287.jpg, all 550 vaccines should be used.

Now we consider a more complex scenario of vaccine limitation where vaccines become available in varying quantities over several decision epochs. Let us assume that at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e288.jpg, the policy maker knows with certainty that during the following An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e289.jpg decision epochs An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e290.jpg, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e291.jpg vaccines will become available. Like before, we assume that any vaccine unused during a period will be lost at the end of the period. Given epidemic state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e292.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e293.jpg available vaccines at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e294.jpg, the policy maker must now decide how many susceptibles to vaccinate and also whether to implement the transmission-reducing intervention. To illustrate how these decisions can be made at each decision epoch, let us assume that An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e295.jpg (cases with An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e296.jpg are solved in a similar fashion).

The optimal allocation is determined iteratively as follows. We start by finding the optimal decisions for the final decision epoch for which vaccines are supplied (i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e297.jpg in this example). If at decision epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e298.jpg, the epidemic is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e299.jpg with An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e300.jpg vaccines available, the optimal recommendations for vaccination and transmission-reducing intervention are determined by solving the problem:

equation image
(13)

where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e302.jpg is calculated by Eq. 11. Next, we step backward in time and determine the optimal decisions for epoch An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e303.jpg. If at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e304.jpg, the epidemic is at state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e305.jpg with An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e306.jpg vaccines available, the optimal decisions at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e307.jpg is determined by:

equation image
(14)

where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e309.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e310.jpg is already calculated in problem (13).

Now to employ this new dynamic health policy that was generated under assumptions of known vaccine constraints An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e311.jpg, we take the following steps (note that we assume here that no vaccine is available at time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e312.jpg):

  • Step 1: At time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e313.jpg, for the observed state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e314.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e315.jpg available vaccines, use problem (14) to find the optimal number of susceptibles to vaccinate and the transmission-reducing intervention to implement.
  • Step 2: Update the epidemic state using the observed number of new cases during period 1, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e316.jpg (see Table 1).
  • Step 3: At time An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e317.jpg, for the observed state An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e318.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e319.jpg available vaccines, use problem (13) to find the optimal number of susceptibles to vaccinate and the transmission-reducing intervention to implement.
  • Step 4: For the remaining decision epochs An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e320.jpg, use Table 1 to update the epidemic state and then use problem (8) to find the optimal transmission-reducing intervention to employ.

Finally, we discuss how budget constraints can be accounted for while generating dynamic health policies. From a methodological perspective, ensuring that budgetary limitations are not exceeded while attempting to control an epidemic is more challenging primarily due to the stochastic nature of disease spread that leads to high variance for the expected costs incurred. However, if we assume that the policy maker is mainly interested in keeping the expected cost incurred during epidemic lower than a constant budget threshold, Figure 4 can be used to select the health policy which satisfies such a constraint.

Figure 4
Expected Budget Required During Epidemic.

Suppose that for our population, after vaccination phase at An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e321.jpg, the proportion of susceptibles at An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e322.jpg is reduced to An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e323.jpg, and that the policy maker sets the budget threshold at An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e324.jpg$.,50000. Then, Figure 4 indicates that for WTP An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e325.jpg$.,100000/QALY the expected cost incurred during epidemic remains below An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e326.jpg$.,50000. Therefore, the policy maker should use the health policy generated by setting the willingness-to-pay An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e327.jpg$.,100000/QALY to guide decision making during the epidemic.

Evaluating the Effect of Dynamic Health Policies in Controlling the Spread of Influenza

To study the effect of employing the dynamic health policies, we built a simulation model for influenza spread in the population described above. Figure 5 displays the expected number of new infections during each period when An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e328.jpg and policies corresponding to different WTP are employed, given the realistic assumption that no effective vaccine was available before the start of epidemic. If vaccines were available in sufficient quantities to employ the and the vaccine recommendations in Figure 5, the epidemic would be averted and there would be no need to employ dynamic policies. The incidence of disease for these policies in each period were statistically different with each other at 1000 simulation runs. As shown in Figure 5, as the willingness-to-pay for health increases, the expected number of individuals infected during the epidemic is reduced since the policies corresponding to the higher willingness-to-pay tend to be more aggressive in implementing the transmission-reducing intervention.

Figure 5
Effect of dynamic health policies on controlling the spread of influenza when no vaccine is available.

Evaluating the Assumption of the Observability of New Cases

In previous section, we assumed that the policy maker is able to accurately measure the number of new infections occurring in each period. This is a strong assumption that will be violated for many infectious diseases (including influenza) where diagnosis is difficult and for which the number of reported cases is an underestimate for the actual number of infections. To examine the sensitivity of the performance of the generated dynamic health policies to the assumption that all cases are observed, we assume that the policy maker's observation of the number of cases during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e329.jpg is only a portion of the true number of cases; that is An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e330.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e331.jpg is the percent error in identifying the number of new cases during period An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e332.jpg.

If the error term An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e333.jpg is a constant in each period, i.e. An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e334.jpg, and known to the policy maker, the number of susceptibles and the number of infectives in Table 1 can be calculated, respectively, as An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e335.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e336.jpg. Hence, the constant observation error An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e337.jpg can be easily corrected in order to preserve the optimality of dynamic health policies under this error setting.

Now let us assume that the error term An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e338.jpg has the form An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e339.jpg, where An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e340.jpg is a constant known to the policy maker and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e341.jpg is a normally distributed noise with mean zero and standard deviation An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e342.jpg. The effect of the constant term An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e343.jpg can be corrected like before by using equations An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e344.jpg and An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e345.jpg to update the current information about the epidemic state in Table 1. The effect of the random noise cannot be corrected; yet, the impact of this noise on the performance of the dynamic health policies can be investigated through simulation. Figure 6 shows the effect of uncertainty around the number of observed cases on the capability of dynamic health policies in controlling the epidemic in our population, for different WTP for health. We find that as the standard deviation of noise, An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e346.jpg, increases, the total number of individuals affected by the epidemic also increases; however, this increase is linear with a low rate which implies that for our population the performance of the generated dynamic health policies is not highly sensitive to the assumption of observability of new cases.

Figure 6
Expected total number of new infections versus the standard deviation of the error in observing the number of new cases.

If this simulation analysis shows significant sensitivity of the performance of dynamic health policies to the assumption of observability of new cases, or the policy maker believes that the probability distribution of noise An external file that holds a picture, illustration, etc.
Object name is pone.0024043.e347.jpg varies over time (for instance, if vigilance of testing diagnosis increases or decreases over time), then more advanced optimization tools can be employed. This issue is briefly discussed in the Discussion section.

Discussion

The emergence of novel human pathogens (e.g. H1N1 and H5N1 influenza, SARS) and their devastating health and financial consequences on affected populations have highlighted the need for developing methods which allow real-time selection of health interventions to control the epidemic while effective vaccines are not available or not present in sufficient quantities to prevent disease spread. We refer to policies informed by such methods as dynamic health policies which are intended to allow for real-time recommendations to be made in response to changing disease and population characteristics as well as the availability of resources.

In contrast to most existing approaches for identifying optimal strategies for infectious disease control which use simulation or mathematical models of disease spread to compare the performance of a limited number of pre-determined health policies, we proposed the use of “dynamic programming” [23] to characterize and identify optimal dynamic health policies. We demonstrated how a Markov decision process [16] can be employed to find optimal dynamic health policies for a simple model of influenza epidemic, in which two types of interventions may be available during the epidemic to control the influenza spread: (1) vaccination, and (2) a transmission-reducing intervention, such as social distancing. The generated dynamic health policies help the policy maker to determine (1) how to allocate vaccines when they become available, and (2) whether the transmission-reducing intervention, such as school closure, should be employed or lifted given the number of susceptibles and infectives at any point of time.

While we used a discrete-time Markov decision process, a number of other methodologies have also been proposed and developed to determine or approximate optimal dynamic health policies for controlling emerging epidemic. For instance, Lefevre [24] used a continuous-time Markov decision model, Merl et al. [2] developed a statistical framework and Ludkovski and Niemi [21] developed a simulation-based model for dynamic determination of optimal policies for emerging epidemics. Undoubtedly, comparing the effectiveness of these methodologies when employed in real practice merits a separate research study.

The influenza model proposed in this paper is very simple and not intended to realistically model disease spread or be used directly to guide the selection of interventions. We present it only for illustration of our proposed approach for dynamic decision making. The model makes several simplifying assumptions that were required for using an MDP to generate the optimal health policies: first, we require that the number of new infections during each period is observable by the policy maker, and second, we assume that an infectious individual interacts with the rest of the population only during the next period and then is effectively removed (treated or isolated) from the population. Relaxing these two assumptions will mean the state of the epidemic is unobservable; yet, a probability belief about the epidemic state can often be generated as new data become available. If we relax these assumptions, we can use generalized discrete-time Markov models for infectious diseases proposed in [19] and use partially observable Markov decision process (POMDP) [25] to characterize optimal health policies. However, as the number of states required to model disease spread increases, MDP and POMDP rapidly lose their efficiency. In these cases, approximate dynamic programming [22] using a simulation model may be employed to identify optimal dynamic health policies. Accordingly, the framework we propose here can potentially be extended to inform decision-making for control of a pathogen with a more complex natural history such as tuberculosis or to design interventions that consider distinct responses targeted toward different risk groups. Such extensions are attractive topics for future research.

As a final note, for the successful implement of the dynamic health policies in practice, the mathematical or simulation model of the disease spread along with the optimization technique used for finding the dynamic health policies must be coupled with an surveillance system that can supply data to estimate the parameters of the underlying model and to provide knowledge on the state of the epidemic. Although dynamic optimization techniques are capable of handling noisy observations, inaccuracies in the surveillance and reporting system may result in suboptimal policies, further underscoring the tremendous importance of public health surveillance in defining responses to epidemics.

Supporting Information

Text S1

Calculating the Transition Probabilities for the Markov Model of Influenza Spread. Approximating the State Space for Influenza Spread. Aditional Information for the Illustrative Example.

(PDF)

Acknowledgments

The authors would like to thank Marc Lipsitch for his comments and suggestions on this work.

Footnotes

Competing Interests: The authors have declared that no competing interests exist.

Funding: The work is supported by National Institutes of Health grants DP2OD006663 and U54GM088558. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the Director of the National Institute of Health and the National Institute of General Medical Sciences of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1. Wallinga J, van Boven M, Lipsitch M. Optimizing infectious disease interventions during an emerging epidemic. Proceedings of the National Academy of Sciences. 2010;107:923–928. [PubMed]
2. Merl D, Johnson L, Gramacy R, Mangel M. A statistical framework for the adaptive management of epidemiological interventions. PloS One. 2009;4:e5087. [PMC free article] [PubMed]
3. Dushoff J, Plotkin J, Viboud C, Simonsen L, Miller M, et al. Vaccinating to protect a vulnerable subpopulation. PLoS Med. 2007;4:e174. [PMC free article] [PubMed]
4. Goldstein E, Apolloni A, Lewis B, Miller J, Macauley M, et al. Distribution of vaccine/antivirals and the ‘least spread line’ in a stratified population. Journal of the Royal Society Interface. 2010;7:755–764. [PMC free article] [PubMed]
5. Patel R, Longini I, Halloran ME. Finding optimal vaccination strategies for pandemic inuenza using genetic algorithms. Journal of Theoretical Biology. 2005;234:201–212. [PubMed]
6. Tanner M, Sattenspiel L, Ntaimo L. Finding optimal vaccination strategies under parameter uncertainty using stochastic programming. Mathematical Biosciences. 2008;215:144–151. [PubMed]
7. Arinaminpathy N, McLean A. Antiviral treatment for the control of pandemic inuenza: some logistical constraints. Journal of the Royal Society Interface. 2008;5:545–553. [PMC free article] [PubMed]
8. Dimitrov N, Goll S, Meyers L, Pourbohloul B, Hupert N. Optimizing tactics for use of the US antiviral strategic national stockpile for pandemic (H1N1) Inuenza, 2009. PLoS Curr Inuenza 2009 [PMC free article] [PubMed]
9. Ferguson N, Cummings D, Fraser C, Cajka J, Cooley P, et al. Strategies for mitigating an inuenza pandemic. Nature. 2006;442:448–452. [PubMed]
10. Ferguson N, Cummings D, Cauchemez S, Fraser C, Riley S, et al. Strategies for containing an emerging inuenza pandemic in Southeast Asia. Nature. 2005;437:209–214. [PubMed]
11. Halloran M, Ferguson N, Eubank S, Longini I, Cummings D, et al. Modeling targeted layered containment of an inuenza pandemic in the United States. Proceedings of the National Academy of Sciences. 2008;105:4639–4644. [PubMed]
12. Flahault A, Vergu E, Coudeville L, Grais R. Strategies for containing a global inuenza pandemic. Vaccine. 2006;24:6751–6755. [PubMed]
13. Longini I, Jr, Nizam A, Xu S, Ungchusak K, Hanshaoworakul W, et al. Containing pandemic inuenza at the source. Science. 2005;309:1083–1087. [PubMed]
14. Fedson D. Pandemic inuenza and the global vaccine supply. Clinical Infectious Diseases. 2003;36:1552–1561. [PubMed]
15. Webby R, Webster R. Are we ready for pandemic inuenza? Science. 2003;302:1519–1522. [PubMed]
16. Puterman ML. Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York, NY: John Wiley & Sons, Inc; 1994.
17. Grassly NC, Fraser C. Mathematical models of infectious disease transmission. Nature Reviews Microbiology. 2008;6:477–487. [PubMed]
18. Stinnett AA, Mullahy J. Net Health Benefits: A New Framework for the Analysis of Uncer- tainty in Cost-Effectiveness Analysis. Medical Decision Making. 1998;18:S68–S80. [PubMed]
19. Yaesoubi R, Cohen T. Generalized Markov models of infectious disease spread: A novel framework for developing dynamic health policies. European Journal of Operational Research In Press [PMC free article] [PubMed]
20. Anonymous. Inuenza in a boarding school. British Medical Journal. 1978:587. [PubMed]
21. Ludkovski M, Niemi J. Optimal dynamic policies for inuenza management. Statistical Communications in Infectious Diseases. 2010;2
22. Powell W. Approximate Dynamic Programming: Solving the curses of dimensionality. Wiley- Interscience 2007
23. Winston WL. Operations Research: Applications and Algorithms. 2003. Duxbury Press, fourth edition.
24. Lefevre C. Optimal control of a birth and death epidemic process. Operations Research. 1981;29:971–982. [PubMed]
25. Sondik E. The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations Research. 1978;26:282–304.

Articles from PLoS ONE are provided here courtesy of Public Library of Science