PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Vaccine. Author manuscript; available in PMC 2014 June 10.
Published in final edited form as:
PMCID: PMC3763189
NIHMSID: NIHMS466456

Removing the Regional Level from the Niger Vaccine Supply Chain

Tina-Marie Assi, PhD,1,2 Shawn T. Brown, PhD,3,4 Souleymane Kone, MS,5 Bryan A. Norman, PhD,6 Ali Djibo, MD,7 Diana L. Connor, MPH,1,2 Angela R. Wateska, MPH,1,2 Jayant Rajgopal, PhD,6 Rachel B. Slayton, PhD,1,2 and Bruce Y. Lee, MD, MBA1,2

Abstract

Objective

Since many of the world’s vaccine supply chains contain multiple levels, the question remains of whether removing a level could bring efficiencies.

Methods

We utilized HERMES to generate a detailed discrete-event simulation model of Niger’s vaccine supply chain and compare the current four-tier (central, regional, district and integrated health center levels) with a modified three-tier structure (removing the regional level). Different scenarios explored various accompanying shipping policies and frequencies.

Findings

Removing the regional level and implementing a collection-based shipping policy from the district stores increases vaccine availability from a mean of 70% to 100% when districts could collect vaccines at least weekly. Alternatively, implementing a delivery-based shipping policy from the central store monthly in three-route and eight-route scenarios only increases vaccine availability to 87%. Restricting central-to district vaccine shipments to a quarterly schedule for three-route and eight-route scenarios reduces vaccine availability to 49%. The collection-based shipping policy from district stores reduces supply chain logistics cost per dose administered from US$0.14 at baseline to US$0.13 after removing the regional level.

Conclusion

Removing the regional level from Niger’s vaccine supply chain can substantially improve vaccine availability as long as certain concomitant adjustments to shipping policies and frequencies are implemented.

Keywords: Vaccine Supply Chain, Niger, Immunization, Vaccine Delivery, Vaccine Distribution

INTRODUCTION

Since many of the world’s vaccine supply chains contain multiple levels which can lead to logistical bottlenecks, can removing a level bring efficiencies? A vaccine supply chain is the series of steps required to get vaccines from their manufacturers to their target populations for administration. An efficient vaccine supply chain is essential to making the necessary vaccines available at the immunization locations to the population.

Our previous studies showed that the Niger supply chain, which consists of four levels, faces some challenges in getting vaccines to the immunization locations(13). Vaccines are delivered to the central store bi-annually and subsequently flow through seven functioning regional stores, forty-two district stores, and over six-hundred health clinics for administration. The question remains: are all of these levels necessary, or can simplification through removing a level improve operational efficiency? Therefore, in collaboration with our World Health Organization (WHO) and Niger partners, our vaccine logistics modeling team constructed a detailed, discrete-event simulation model of Niger’s national vaccine supply chain to test the effects of removal of the regional level from the national vaccine supply chain and evaluate the impacts of various shipping policies on vaccine delivery.

METHODS

Model Description

Our team constructed a model utilizing the HERMES (Highly Extensible Resource for Modeling Supply Chains) program to represent the vaccine supply chain in Niger. HERMES is a software program developed in the programming language Python, using resources provided by the SimPy package(15). HERMES can rapidly create detailed discrete event simulation models of any vaccine supply chain. The resulting model simulates the operational policies, storage and administering facilities, transport procedures and equipment in a vaccine supply chain, while also accounting for stochastic variations in the system (e.g., the child arrival rate at a given administration facility). The model for Niger included all World Health Organization (WHO) Expanded Program on Immunization (EPI) vaccines (Table 1)(6)

Table 1
Average vaccine availability following the removal of the regional level from the vaccine supply chain in Niger under varying supply chain structures and shipping policies

Niger Vaccine Supply Chain

The supply chain consists of four levels, as illustrated in Figure 1. Vaccines are delivered to regional stores every six months. They are then delivered to eight regional stores quarterly. District stores will then collect vaccines from regional stores monthly, and Integrated Health Centers (IHC’s) will collect vaccines from districts monthly. Vaccine administration occurs at IHC’s every weekday. Most facilities are located in the south of Niger, close to population settlements. Transport distances and hours traveled between each origin and destination are also specified in the model. Descriptions of the data sources, population distribution, and model structure are detailed in previous publications(1, 3, 5).Child arrival is modeled using a combination of the target population density within a catchment area, and frequency specified by the WHO EPI vaccination calendar(1, 3, 5).

Figure 1
Schematic of existing and alternative vaccine supply chain frameworks in Niger

Supply Chain Performance and Cost Measures

HERMES generates a number of performance and cost measures. For this study, we focused on three key measures. The following formula calculated the vaccine availability (the percentage of children arriving at an IHC who are able to receive their recommended vaccine because there is sufficient vaccine on-hand) for each simulation for each vaccine type at each IHC:

equation M1
[1]

The following formulae calculated the capacity utilization (i.e., the percentage of net space available for transport or storage that is used) for each device:

equation M2
[2]

equation M3
[3]

Data used to compute storage, transport, and personnel costs came from the 2005 Niger Comprehensive Multi-Year Plan (cMYP) and the Cold Chain Equipment Inventory(710). All storage and transport costs account for operating and maintenance fees, and utilities. A rate of 3% adjusted costs to 2011 United States dollars (US$) (11).

The total logistics operating costs for the vaccine supply chain can be given by the following formula:

equation M4
[4]

where:

  • Costlaborpersonnel Costeach employee
  • Coststoragestorage unitCostper storage unit
  • Costtransporttransport routes Costper transport route
  • Costbuildingbuildings Costper building

The following expressions define the unit cost for each of the categories:

  • Unit Labor Costs
    • Costeach employee=Costemployee’s annual salary and benefits* % of time dedicated to vaccine logistics
  • Unit Storage Costs
    • Costper storage device unit=Coststorage unit energy usage + Coststorage unit maintenance + Coststorage depreciation
  • Unit Transport Costs
    • Costper km=Costvehicle maintenance per km + Costvehicle depreciation per km + Costfuel per km
    • Costfuel per km=Costfuel per liter / fuel efficiency of vehiclekm per liter
  • Unit Building Costs
    • Costper building=(Costdepreciation + Costannual utilities) * % of building utilized for logistics

Changes in delivery frequency affect per diem (included in transport) but it did not affect personnel costs as it was assumed that current storage location-based employees could handle the increased shipments without having to hire more personnel.

The geographic locations and distances of central, regional, district, and IHC supply chain facilities were initially established using geographic coordinates from the WHO Cold Chain Equipment Inventory, United Nations Food and Agriculture Organization (FAO)(12), and supplemented with data points found by utilizing GeoNames Geographical Database(13) and Google Earth(14). Unknown locations were assigned a point based on reasonable approximation travel distance between two locations was based on the straight-line distance and multiplied by a factor to convert it to an approximate road travel distance.

Simulated Scenarios

Each scenario was simulated over a one-year time horizon. We compared the performance and cost measures for different configurations of the Niger supply chain (Figure 1) to those for the baseline structure of Niger’s current supply chain. Corresponding to the four configurations shown, we examined the following different types of shipping policies, respectively:

  • Baseline: central store delivers vaccines in three shipping loops to seven regional stores quarterly using a combination of cold trucks and 4×4 trucks, from which each district procures its own vaccines using 4×4 trucks.
  • Shipping policy 2: district stores collect vaccines directly from the central store using a 4×4 truck as needed and when vaccines at the central store are available on monthly, bi-biweekly or weekly frequencies.
  • Shipping policy 3: the central store distributes vaccines directly to the district stores in three cold trucks along three shipping loops on a monthly or quarterly shipping frequency.
  • Shipping policy 4: the central store distributes vaccines directly to the district stores in eight cold trucks along eight shipping loops on a monthly or quarterly shipping frequency.

Vaccine shipments between the district and IHC stores in all scenarios occurred as necessary with a maximum shipping frequency of once per week, utilizing two vaccine carriers per trip. The shipping loops in policies 3 and 4 mimicked the original routing network in the baseline scenario, which are defined based on existing administrative boundaries and local knowledge of road conditions and accessibility. Table 3 details the storage and transport capacities for each scenario, by level and device.

Table 3
Costs following the removal of the regional level from the vaccine supply chain in Niger under varying supply chain structures and shipping policies

RESULTS

Vaccine Availability

Table 1 shows vaccine availabilities by vaccine type at the IHC level, for each simulated scenario. The differences seen among antigens are due to different dosing regimens, schedules, and open vial wastes. At baseline, the average vaccine availability for all EPI vaccines was 70%. Implementing policy 2, in which the district stores bypass the regional level to collect vaccines directly from the central store increased vaccine availability from an average of 70% at baseline to 84% when vaccines were collected monthly, 97% when vaccines were collected bi-weekly, and 100% when vaccines were collected weekly. Bypassing the regional stores, in turn avoiding the bottlenecks and inefficiencies at those locations, can increase vaccine availability by 14% without additional storage capacity or increased vaccine collection frequency, amounting to 1,388,353 more doses of vaccine available for immunization. And by increasing the frequency of trips vaccine availability can increase up to 30%. This suggests the there is a large bottleneck in transport and creative alterations of transport could be very beneficial to supply chains.

Similarly, implementing either shipping policy 3 or 4 in which the central store delivered vaccines in 3 or 8 shipping loops directly to district stores monthly increased vaccine availability from 70% at baseline to 87%. Both collection-based and delivery-based shipping policies outperformed the current shipping policy if vaccines could be shipped at least monthly.

However, irrespective of the number of shipping loops from the central store, restricting the shipping frequency from the central store to the next level in the supply chain to the present quarterly schedule decreased vaccine availability from 70% (baseline configuration) to 50% for both the 3-loop and 8-loop shipping frameworks. This suggests that while vaccine availability can be improved by removing the regional level, this also requires a shipping frequency from the Central Store to the Districts of at least once a month.

Vaccine Transport and Storage Utilization

Table 2 shows transport capacity utilizations by level and device for the baseline four-tier structure and the three-tier structure with different shipping policies. When the simulation model indicates transport utilization in excess of 100%, it means that the transport routes required more space than was available to deliver the necessary vaccines (e.g., a value of 200% would imply that the space required is twice what is currently available). Therefore, portions of vaccine orders would go unfulfilled until subsequent deliveries could deliver the remainder of the order resulting in delays and decreased vaccine availability at IHCs. This in turn, would mean more missed vaccination opportunities. Table 2 shows that in many cases, there are instances where multiple transport vehicles or multiple trips per vehicle would be required.

Table 2
Average transport and storage net capacity utilization following the removal of the regional level from the vaccine supply chain in Niger under varying supply chain structures and shipping policies

Table 2 also shows how removing the regional level and changing shipping policies affected average storage capacity utilization at the central, regional, district and IHC levels. The central level storage capacity utilization remains at 100% throughout all scenarios. While the IHC storage capacity is largely sufficient across scenarios, the capacities at central, regional and district stores are insufficient, particularly when vaccines are delivered or collected from the central store quarterly. These policies require significant additional capacity at some district locations to accommodate larger shipments that supply vaccines over longer periods. Moreover, quarterly shipments from the central store result in larger shipment volumes and therefore, greater numbers of overfilled vehicles, which reduce the number of required vaccines actually delivered to IHC stores. Increasing the transport frequency means that a greater number of smaller volume shipments will be collected by districts. Because the shipments are smaller at each delivery, they can more easily be stored in the district level transport and storage devices. This also means that there are fewer shipments requested that would need to be overfilled or bottlenecks at higher levels, and that there is a greater supply of vaccine to districts over a quarter. When the same transport device is used to collect a quarter’s worth of vaccine at one time, the vaccine availability is much lower because the transport device cannot accommodate that many vaccines. While increasing the transport frequency removes these shipping bottlenecks between the central and district levels, vaccine availability remains limited, though to a lesser degree, by storage constraints at the district level. While increasing the transport frequency removes these shipping bottlenecks between the central and district levels, vaccine availability remains limited, though to a lesser degree, by storage constraints at the district level.

Vaccine Supply Chain Costs

Table 3 shows the resulting costs from each of the experimental scenario. At baseline, the total costs, including vaccine transport (US$446,538), storage (US$403,671), building facility (US$700,375) and personnel (US$46,865) costs, amounted to US$1,597,449 per year, which translated to US$0.14 per dose delivered to IHCs over a year. Removing the regional level eliminated an entire level of personnel, storage, and building costs, which more than compensated for the increased Central to District transport costs due to longer travel distances. The lowest total costs scenarios were those in which the central store delivered vaccines quarterly on either 3 or 8 loop frameworks. However, scenarios in which the central store delivered vaccines to districts monthly on 3 or 8 loop shipping frameworks produced the lowest logistics cost per dose administered but significantly fewer overall doses reached the IHC level and resulted in significantly lower vaccine availabilities.

DISCUSSION

Results from HERMES suggest that removing the regional level from Niger’s vaccine supply chain can improve vaccine availability if either districts collect vaccines directly from the central store or if cold trucks deliver vaccines from the central store to districts monthly. Policies in which vaccines are shipped quarterly from the central store negatively impact vaccine availability because vaccines do not flow through the supply chain rapidly enough. The resulting shipping delays and constraints cause stock to collect at higher levels. Without also increasing capacity downstream, vaccine delivery is continually limited by storage and transportation bottlenecks. Additionally, without vehicles regularly depleting stock from the central store, the sizes of bi-annual shipments into 11 Niamey may exceed cold room capacity, thereby constraining overall vaccine availability from the very top of the supply chain.

Removing the regional level may also reduce supply chain logistics costs. While removing the regional level resulted in lower costs per dose administered, relative cost savings depended on the total number of doses delivered, implemented routing network (i.e., collection-based versus delivery-based from the central store) and shipping frequencies. While shipping policy 2 did not provide significant cost savings compared to shipping policies 3 and 4, it provided higher vaccine availability if the shipments could be made at least monthly. Our experiment showed that removing the regional level needs to be paired with delivery frequencies enough to supply adequate vaccine supply. This may require more meticulous planning and organization.

As each delivery, collection, shelving and un-shelving procedure carries a risk of vaccine breakage, mishandling, or temperature exposure, fewer supply chain levels can result in reduced risks of vaccine wastage. Moreover, while having fewer levels may increase travel distance and time between sites and ultimately transportation costs (i.e., fuel costs, vehicle maintenance and depreciation), it can also result in cost savings from reductions in total annual expenditures on storage facilities, human resources and cold chain equipment, which may outweigh the cost increases of longer transportation distances.

Streamlining vaccine supply chains can also simplify requisition and distribution logistics. Delivery-based supply chain policies (i.e., fixed volume of vaccines delivered on a predetermined schedule) depend on accurate vaccine demand forecasts at the IHC level. Without knowing how many children will arrive at an IHC, delivery-based policies risk under-supplying or over-supplying locations that experience unexpectedly high or low arrival rates. Vaccine shipments distributed through collection-based shipping policies (i.e., variable volume of vaccines collected if and when vaccines are needed) can be more closely matched to actual consumption. Streamlining the vaccine supply chain can also simplify distribution logistics in emergency situations, by reducing the number of steps required in delivering vaccines from the central store to administering sites in a shortened timeframe.

Nevertheless, important considerations remain in determining whether streamlining vaccine supply chains is both programmatically-effective and cost-effective. For instance, depending on the implemented shipping policy, removing the regional level can lead to increases in recurring or capital costs of transportation or to increases in storage or transport utilization in some locations which may reduce the supply chain’s ability to handle unanticipated surges in vaccine demand (e.g., due to truck breakdown, or in response to disease outbreak) or future vaccine introductions. These may require increased investments in capital resources (i.e., additional cold storage equipment or transport vehicles) and human resources (i.e., EPI logisticians, managers, nurses, drivers, etc.). Longer shipping distances may also increase risks of vaccine wastage during transportation.

Our study illustrates the usefulness of models in determining effects of decisions not immediately apparent. Modeling major supply chain structures changes prior to implementation can determine potential impact, benefits, challenges and needed modifications. Such an approach can save substantial time, effort, and resources involved with trial and error. Once a modeling study has helped establish and define a plan the next step could be progressive testing and implementation in the field. Our near future plan is to provide HERMES to in-country decision-makers so that they may construct and run simulation models of their own supply chains.

Models have widely been used by decision-makers in other industries including meteorology(17), manufacturing(18), transportation(19), aerospace(20), and finance(21), and sports and rehabilitation(22). Their use to date in public health, however, has not been as extensive(2325). While models have assisted responses to health-care associated infections and infectious disease transmission such as the 2009 H1N1 influenza pandemic, much of their potential remains untapped(2630).

Limitations

Models are simplified representations of real life and cannot account for every potential factor, event or outcome(3133). The model is based on data collected up until 2011 and may not represent changes that may occur in the future. Our experiments assumed current vaccine procurement, production and forecasting policies. Future studies can explore effects of varying these parameters in HERMES. Actual demand may vary from our estimated demand, which was drawn from cross sectional census data, although sensitivity analyses have demonstrated the effects of altering demand and indicated that the results are not sensitive to minor variations in demand. No new cold storage equipment was introduced into the system across scenarios, and our analysis did not include all EPI cost components such as costs for disease surveillance and supplementary vaccination campaigns. Constructing our model involved substantial data collection from a wide variety of sources including records and interviews at different locations. As a result, parameter values may vary in accuracy and reliability, although sensitivity analyses demonstrated that model outcomes are robust under a wide variety of circumstances.

CONCLUSIONS

Removing the regional level from the Niger vaccine supply chain may improve supply chain performance and cost if there are accompanying appropriate changes in shipping policies (i.e., collection-based shipments from the district to central level or monthly shipments from the central to district stores). This exploration suggests that efficiencies can be gained from reducing the number of levels in a supply chain. Future explorations may want to determine if similar findings apply to other countries’ vaccine supply chains.

HIGHLIGHTS

  • Simulation model of Niger vaccine supply chain logistics.
  • Removing a level can improve functions and reduce missed vaccination opportunities.
  • The impact of removing a level depends heavily on the shipping policy implemented.
  • Single changes in the supply chain can have complex dynamic effects.
  • Modeling can help elucidate dynamic interactions and effects, and guide policy.

ACKNOWLEDGMENTS

The HERMES Project team consists of (in alphabetical order): Tina-Marie Assi, PhD, Shawn T. Brown, PhD (Technical Lead), Brigid E. Cakouros, MPH, Sheng-I Chen, PhD, Diana L. Connor, MPH (Co-Coordinator), Erin G. Claypool, PhD, Leila A. Haidari, MPH, Veena Karir, PharmD, Bruce Y. Lee, MD, MBA (Scientific Lead), Jim Leonard, Leslie E. Mueller, MPH, Bryan A. Norman, PhD, Proma Paul, MHS, Jayant Rajgopal, PhD, Michelle M. Schmitz, BA, Rachel B. Slayton, PhD, Angela R. Wateska, MPH (Co-Coordinator), Joel S. Welling, PhD, and Yu-Ting Weng, MS. For further questions regarding HERMES, please contact B. Lee, MD MBA (BYL1/at/pitt.edu) or S. Brown, PhD (stbrown/at/psc.edu). This study was supported by the Vaccine Modeling Initiative (VMI), funded by the Bill and Melinda Gates Foundation and the National Institute of General Medical Sciences Models of Infectious Disease Agent Study (MIDAS) grant 1U54GM088491-0109. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. We would also like to acknowledge the important contributions of the following individuals to our study: Carol Levin (PATH), Mercy Mvundura (PATH), Dr. Amadou Garba (Schistosomiasis Control Initiative), Dr. Tiekoura Coulibaly (EPI Officer, WHO Niger), Dr. Abdou M. Chitou (EPI Administrator, UNICEF Niger), and Mr. Harou Moussa (EPI Office, WHO Niger).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

1. Lee B, Assi T, Rajgopal J, et al. Impact of introducing the pneumococcal and rotavirus vaccines into the routine immunization program in Niger. American Journal of Public Health. 2011 In press. [PMC free article] [PubMed]
2. Lee BY, Cakouros BE, Assi TM, Connor DL, Welling J, Kone S, et al. The impact of making vaccines thermostable in Niger's vaccine supply chain. Vaccine. 2012 Aug 17;30(38):5637–5643. [PMC free article] [PubMed]
3. Assi TM, Brown ST, Djibo A, Norman BA, Rajgopal J, Welling JS, et al. Impact of changing the measles vaccine vial size on Niger's vaccine supply chain: a computational model. BMC Public Health. 2011;11:425. [PMC free article] [PubMed]
4. IBM. C and C++ Compilers. 2010
5. Lee BY, Assi TM, Rookkapan K, Connor DL, Rajgopal J, Sornsrivichai V, et al. Replacing the measles ten-dose vaccine presentation with the single-dose presentation in Thailand. Vaccine. 2011 May 12;29(21):3811–3817. [PMC free article] [PubMed]
6. WHO. WHO prequalified vaccines: Filterable search for prequalified vaccines with product details. Geneva: 2011. [updated 2011; cited 2012 01 October]. Available from: http://www.who.int/immunization_standards/vaccine_quality/PQ_vaccine_list_en/en/index.html.
7. Brenzel L, Wolfson L, Fox-Rushby J, Miller M, Halsey N. Vaccine-Preventable Diseases. In: Jamison D, editor. Disease Control Priorities in Developing Countries. 2 ed. Washington, DC: The World Bank Group; 2006. pp. 389–411.
8. Khaleghian P. Immunization Financing and Sustainability: A Review of the Literature. Bethesday, Maryland: Partnerships for Health Reform; 2001. Contract No.: Document Number|.
9. WHO. Cost Analysis in Primary Health Care: A Training Manual for Programme Managers. Geneva, Switzerland: WHO; 1999. Contract No.: Document Number|.
10. WHO. Immunization Financing: Niger Comprehensive Multi-year Plan 2007–2010. Geneve: WHO; 2011. [updated 2011 12 May 2011; cited]. 2011:[Available from: http://www.who.int/immunization_financing/countries/cmyp/niger/en/index.html.
11. Gold MR. Cost-effectiveness in health and medicine. New York: Oxford University Press; 1996.
12. Food and Agriculture Organization of the United Nations. [cited 2010 06 January]; Available from: http://www.fao.org/.
13. GeoNames Geographic Database. [updated 2010; cited 2010 06 January];2010 Available from: http://www.geonames.org/.
14. Google Earth (Version 5.1.3533.1731) [Software] Mountain View, CA: Google Inc.; 2009. [updated 2009; cited]. Available from: http://www.google.com/earth/index.html.
15. PATH., WHO. Outsourcing the vaccine supply chian and logistics system to the private sector: The W western Cape Experience in South Africa. Seattle, WA: PATH, World Health Organization; 2011. Contract No.: Document Number|.
16. PATH. Tunisia: Demonstrating Innovative Supply Chain Solutions. Seattle, WA: PATH; 2012. Contract No.: Document Number|.
17. Klingberg J, Danielsson H, Simpson D, Pleijel H. Comparison of modelled and measured ozone concentrations and meteorology for a site in south-west Sweden: implications for ozone uptake calculations. Environ Pollut. 2008 Sep;155(1):99–111. [PubMed]
18. Lee JH, Kim CO. Multi-agent systems applications in manufacturing systems and supply chain management: a review paper. London, Royaume-Uni: Taylor & Amp: Francis; 2008.
19. Borndorfer R, Lobel A, Weider S. A bundle method for integrated multi-depot vehicle and duty scheduling in public transit. Computer-aided Systems in Public Transport. 2008;600(Part 1):3–21.
20. Moormann D, Mosterman JP, Looye G. Object-oriented computational model building of aircraft flight dynamics and systems. Paris, FRANCE: Elsevier; 1999.
21. Fouque J-P, Papanicolaou G, Sircar K. Financial Modeling in a Fast Mean-Reverting Stochastic Volatility Environment. Asia-Pacific Financial Markets. 1999;6(1):37–48.
22. Neptune RR. Computer modeling and simulation of human movement. Applications in sport and rehabilitation. Phys Med Rehabil Clin N Am. 2000 May;11(2):417–434. viii. [PubMed]
23. Trochim WM, Cabrera DA, Milstein B, Gallagher RS, Leischow SJ. Practical challenges of systems thinking and modeling in public health. Am J Public Health. 2006 Mar;96(3):538–546. [PubMed]
24. Leischow SJ, Milstein B. Systems thinking and modeling for public health practice. Am J Public Health. 2006 Mar;96(3):403–405. [PubMed]
25. Epstein JM. Generative social science: Studies in agent-based computational modeling. Princeton, NJ: Princeton University Press; 2006.
26. Cooley P, Lee BY, Brown S, Cajka J, Chasteen B, Ganapathi L, et al. Protecting health care workers: a pandemic simulation based on Allegheny County. Influenza Other Respi Viruses. 2010 Mar;4(2):61–72. [PMC free article] [PubMed]
27. Lee BY, Brown ST, Cooley P, Grefenstette JJ, Zimmerman RK, Zimmer SM, et al. Vaccination deep into a pandemic wave potential mechanisms for a "third wave" and the impact of vaccination. Am J Prev Med. 2010 Nov;39(5):e21–e29. [PMC free article] [PubMed]
28. Lee BY, Brown ST, Cooley P, Potter MA, Wheaton WD, Voorhees RE, et al. Simulating school closure strategies to mitigate an influenza epidemic. J Public Health Manag Pract. 2010 May-Jun;16(3):252–261. [PMC free article] [PubMed]
29. Lee BY, Brown ST, Cooley PC, Zimmerman RK, Wheaton WD, Zimmer SM, et al. A computer simulation of employee vaccination to mitigate an influenza epidemic. Am J Prev Med. 2010 Mar;38(3):247–257. [PMC free article] [PubMed]
30. Lee BY, Brown ST, Korch GW, Cooley PC, Zimmerman RK, Wheaton WD, et al. A computer simulation of vaccine prioritization, allocation, and rationing during the 2009 H1N1 influenza pandemic. Vaccine. 2010 Jul 12;28(31):4875–4879. [PMC free article] [PubMed]
31. Lee BY, Biggerstaff BJ. Screening the United States blood supply for West Nile Virus: a question of blood, dollars, and sense. PLoS medicine. 2006 Feb;3(2):e99. [PMC free article] [PubMed]
32. Lee BY. Digital decision making: computer models and antibiotic prescribing in the twenty-first century. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America. 2008 Apr 15;46(8):1139–1141. [PubMed]
33. Lee BY, Wiringa AE. The 2009 H1N1 influenza pandemic: a case study of how modeling can assist all stages of vaccine decision-making. Human vaccines. 2011 Jan 1;7(1):115–119. [PMC free article] [PubMed]