PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
WMJ. Author manuscript; available in PMC 2010 August 1.
Published in final edited form as:
WMJ. 2009 August; 108(5): 256–258.
PMCID: PMC2893396
NIHMSID: NIHMS185883

Syndromic Surveillance: Early Results from the MARISSA Project

Abstract

Context

The Madison Area Review of Systems Syndromic Surveillance Application (MARISSA) project is the ongoing development of a syndromic surveillance system at the University of Wisconsin Hospital and Clinics Emergency Department. Readily available electronic health data will be used to provide timely identification of increased respiratory illness activity in the Madison area.

Objective

To develop novel data sets and statistical methods for syndromic surveillance.

Design

Time series of daily counts of influenza-like illness (ILI) and fever in the University of Wisconsin Hospital and Clinics Emergency Department from June 13, 2007 to June 11, 2008.

Main Outcome Measures

Operating characteristics of limited baseline aberration detection methods with varying lengths of baseline periods were evaluated by simulation.

Results

All methods had false detection rates at or below the nominal levels. True detection rates were substantially higher for methods based on longer baseline periods.

Conclusion

The MARISSA project has adopted a limited baseline aberration detection method with a baseline period of 21 days and a nominal false alarm rate of once every 365 days for ongoing syndromic surveillance. Future work will explore the utility of novel data sources such as review of systems and vital signs for syndromic surveillance.

INTRODUCTION

The Madison Area Review of Systems Syndromic Surveillance Application (MARISSA) project is the ongoing development of a syndromic surveillance system at the University of Wisconsin Hospital and Clinics Emergency Department. Disease surveillance is critical for reducing morbidity and mortality associated with infectious disease outbreaks. Traditional surveillance methods rely on accumulated cases of reportable diseases or spontaneous reporting by clinicians or laboratories. The relatively new field of syndromic surveillance uses readily available electronic health data—eg school absenteeism, over-the-counter pharmaceutical sales, ambulance dispatch data, chief complaint, or discharge diagnosis—to provide more timely identification of disease activity.1-2 Within these data sets, markers of diseases of potential concern can be grouped into broad syndromes such as respiratory or gastrointestinal illness. For example, chief complaints of cough or shortness of breath may be grouped as a respiratory illness syndrome.

Regardless of the data source, abrupt increases in activity within a particular syndrome may signify potential acute outbreaks of disease that would trigger investigation and control measures. Aberrations are defined as a comparison of the observed count to expected historical variability for a similar time period. In the absence of extensive, multi-year historical activity data, limited baseline aberration detection methods use data from the recent past to define expected historical variability.3 In this report, we evaluate the operating characteristics of modified versions of the C1 limited baseline aberration detection method with varying lengths of the baseline period. We also provide an illustration of surveillance using a novel limited baseline aberration detection method with a 21-day baseline period that is currently being used in the MARISSA project.

METHODS

Data Sources

Daily counts of 2 syndromes, fever (recorded temperature above 100.4° F) and influenza-like illnesses (ILI) (ICD-9 discharge code), in the Emergency Department of the University of Wisconsin Hospital and Clinics from June 13, 2007 to June 11, 2008 were obtained from electronic medical records.

Limited Baseline Aberration Detection Methods

We evaluated the operating characteristics of modified versions of the C1 limited baseline aberration detection method with varying lengths of the baseline period.3-4 Daily counts were square-root transformed to stabilize variance and better approximate normality. Potential baseline periods consisted of the 7-28 days immediately prior to the current day. Alert thresholds were established using Student's t-distribution to achieve a desired nominal false detection rate. To estimate false detection rates, we generated 10,000 simulated daily counts (independent Poisson or negative binomial random variables) representative of the number of cases of ILI (mean 6.02, standard deviation [SD] 9.52) and fever (mean 2.34, SD 3.37) observed in our data to represent the in-control (non-aberration) process. To estimate true detection rates, we added an aberration, a Poisson random variable with mean 5, to each of the last 333 days of the observed processes or 10,000 simulations from the in-control process.

For the MARISSA project, we have adopted a baseline period of 21 days and set a nominal false alarm rate of once every 365 days so that values that exceed the baseline mean by 3.19 SDs are flagged. We present surveillance results using the MARISSA system for fever and ILI from June 13, 2007 to June 11, 2008.

RESULTS

Operating characteristics for limited baseline aberration detection methods with baseline periods of 7, 14, 21, and 28 days are presented in Table 1. All methods were well-calibrated, eg false detection rates were at or below the nominal levels. For both real and simulated datasets, true detection rates were substantially higher for methods based on longer baseline periods (Table 2).

Table 1
ICD-9 Diagnostic Codes Used to Define Influenza-Like-Illness (ILI)
Table 2
False Detection and True Detection Rates for Limited Baseline Aberration Detection Methods Based on Actual and Simulated Data Representative of Daily Counts of Fever and ILI in UW Hospital and Clinics Emergency Department, June 2007-2008

Daily counts of fever and ILI are displayed in Figure 1, along with the alert thresholds based on a false detection rate of once per 365 days. During the 344 days of active surveillance, there were no alerts for fever and 1 alert for ILI. The alert, with a recurrence interval of 741 days, occurred on March 8, 2008, with 21 ILI cases (baseline mean 10.6).

Figure 1
Daily counts of fever and influenza-like illnesses (ILI) in the Emergency Department of the University of Wisconsin Hospital and Clinics from June 13, 2007 to June 11, 2008. Solid line indicates alert threshold based on a nominal false detection rate ...

DISCUSSION

Our simulation studies suggest that the use of thresholds based on the t distribution for aberration detection controls the false detection rate, and the use of longer baseline periods for the estimation of means and SDs is associated with substantial improvements in performance of limited baseline aberration detection methods. The latter finding is in agreement with earlier studies with uncalibrated or empirically calibrated procedures.5-6 Based on these findings, the MARISSA system currently uses a baseline period of 21 days. Future development of the MARISSA surveillance system will investigate the possibility of further modifying the detection method to use 2 different baseline periods: a shorter period for estimation of the baseline mean and a longer period for estimation of the baseline SD.

As part of the development of the MARISSA system, we will also explore the utility of novel data sources. Chief complaint and discharge diagnosis may not fully capture available information on diseases of potential concern. For example, the reported chief complaint of a patient with pneumonia may be fatigue, with cough and fever as secondary complaints. Fatigue might not be included in the respiratory syndrome definition, and this encounter would therefore be unidentified. Electronic medical records allow storage of more comprehensive data from clinic visits, including review of systems and vital signs. Review of systems is a comprehensive survey of illness by organ system that avoids the limited selection of a single chief complaint or discharge diagnosis. It is thus a logical data source for acute disease surveillance and is potentially a more sensitive method for capturing complaints such as shortness of breath, cough, or wheezing. Vital signs provide a similar objective and comprehensive assessment of illness. Data gathered during the 2008-2009 and 2009-2010 influenza seasons will be used to evaluate the performance of review of systems and vital signs data compared to ICD-9 discharge diagnosis and chief complaint data as markers of underlying syndromes of interest.

Acknowledgments

Funding/Support: Support for this manuscript was provided by the Department of Medicine Research Committee.

Footnotes

Financial Disclosures: None declared.

REFERENCES

1. Persell DJ, Robinson CH. Detection and early identification in bioterrorism events. Fam Community Health. 31(1):4–16. [PubMed]
2. Travers D, Barnett C, Ising A, Waller A. Timeliness of emergency department diagnoses for syndromic surveillance. AMIA Annu Symp Proc. 2006:769–773. [PMC free article] [PubMed]
3. Centers for Disease Control and Prevention. Syndromic surveillance for bioterrorism following the attacks on the World Trade Center–New York City, 2001. MMWR. 2002;51:13–15. [PubMed]
4. Hutwagner LC, Thompson WW, Seeman GM, Treadwell T. A simulation model for assessing aberration methods used in public health systems used in public health surveillance for systems with limited baselines. Statin Med. 2005;24:543–550. [PubMed]
5. Xing J, Burkom H, Copeland J, Bloom S, Hutwagner L, Tokars J. Performance characteristics of control chart detection methods. Adv Dis Surveillance. 2007;4:122.
6. Chang H, Chen J, Cochrane DG, et al. A pilot study of aberration detection algorithms with simulated data. Adv Dis Surveillance. 2007;4:240.