Search tips
Search criteria 


Logo of amjepidLink to Publisher's site
Am J Epidemiol. 2010 January 15; 171(2): 177–188.
Published online 2009 December 4. doi:  10.1093/aje/kwp345
PMCID: PMC2878099

Near Real-Time Surveillance for Influenza Vaccine Safety: Proof-of-Concept in the Vaccine Safety Datalink Project


The emergence of pandemic H1N1 influenza in 2009 has prompted public health responses, including production and licensure of new influenza A (H1N1) 2009 monovalent vaccines. Safety monitoring is a critical component of vaccination programs. As proof-of-concept, the authors mimicked near real-time prospective surveillance for prespecified neurologic and allergic adverse events among enrollees in 8 medical care organizations (the Vaccine Safety Datalink Project) who received seasonal trivalent inactivated influenza vaccine during the 2005/06–2007/08 influenza seasons. In self-controlled case series analysis, the risk of adverse events in a prespecified exposure period following vaccination was compared with the risk in 1 control period for the same individual either before or after vaccination. In difference-in-difference analysis, the relative risk in exposed versus control periods each season was compared with the relative risk in previous seasons since 2000/01. The authors used Poisson-based analysis to compare the risk of Guillian-Barré syndrome following vaccination in each season with that in previous seasons. Maximized sequential probability ratio tests were used to adjust for repeated analyses on weekly data. With administration of 1,195,552 doses to children under age 18 years and 4,773,956 doses to adults, no elevated risk of adverse events was identified. Near real-time surveillance for selected adverse events can be implemented prospectively to rapidly assess seasonal and pandemic influenza vaccine safety.

Keywords: cohort studies, influenza, human, influenza vaccines, managed care programs, population surveillance, safety, vaccines

In April 2009, the first cases of pandemic H1N1 influenza were confirmed in several countries (1). The World Health Organization declared a pandemic in June 2009 (2), and influenza A (H1N1) 2009 monovalent vaccines were licensed in September 2009 (3). The Advisory Committee on Immunization Practices recommended initial target groups for these vaccines (4), recommended annual seasonal influenza vaccination for approximately 83% of the US population (5) (255 million children and adults), and may continue to incrementally expand the recommendations (6). Given rapid deployment of 2009 H1N1 vaccines and broad recommendations for seasonal influenza vaccines, monitoring vaccine safety in near real-time is critical (7, 8).

Current systems for evaluating vaccine safety have limitations. First, passive reporting systems, such as the Vaccine Adverse Event Reporting System, allow identification of potential adverse events but tend to underreport numbers of adverse events, lack denominator data, and are not designed to distinguish coincidental associations from potentially true causal relations (9). Second, traditional postlicensure studies can rigorously evaluate potential associations between vaccines and adverse events, but they do not monitor safety in near real-time, since they often require substantial time to design and implement, resulting in delays in the ability to detect possible adverse events. If analyses are conducted after the influenza season is over, potential problems will be identified only after all doses from that season's vaccine have already been administered.

In several recent studies, investigators have addressed this limitation of timeliness by comparing adverse event risks after administration of a vaccine of interest with risks following administration of a comparison vaccine or a preventive care visit in near real-time, using weekly data and a Poisson-based maximized sequential probability ratio test to adjust for repeated analyses (1013). However, selecting a comparison group can be particularly problematic for influenza vaccine, because of confounding by indication (14); influenza vaccine recipients are often systematically different from nonvaccinees (e.g., by functional status limitations, such as requiring assistance to bathe) in ways that may influence the incidence of adverse events and for which adjustment may not be possible using solely automated data (15). In addition, vaccination recommendations have expanded over time (5), resulting in changing characteristics of vaccinated and unvaccinated populations. Furthermore, vaccinees may be misclassified as unvaccinated in electronic data, particularly if vaccines are administered in nontraditional settings (e.g., workplaces) (16). Self-controlled case series (SCCS) (17, 18) and difference-in-difference (DID) (19) approaches compare risk and control periods within vaccinees, minimizing these potential biases and controlling for known confounders (e.g., age and sex) as well as unmeasured fixed covariates (e.g., race, ethnicity, socioeconomic status, and underlying disease).

In previous postlicensure studies, investigators have compared risk and control periods within persons receiving influenza vaccine (2022) or have conducted sequential analyses to monitor adverse events following administration of vaccines other than influenza vaccine (1013). In this report, we unite these methods by applying maximized sequential probability ratio testing to comparisons of risk and control periods within vaccinees. As proof-of-concept in preparation for pandemic and seasonal influenza vaccine safety monitoring, our objective was to mimic prospective surveillance for prespecified adverse events following seasonal influenza vaccine administration, using binomial-based (SCCS and DID) and Poisson-based analytic approaches.


Study design

We used 3 approaches to conduct safety surveillance for the first seasonal dose of trivalent inactivated influenza vaccine (TIV). The SCCS approach (17, 18) was used to investigate whether a predefined period following TIV administration in a particular season was riskier than a control period. The SCCS approach may be limited by the “healthy vaccinee” effect (23), since some patients may delay or forgo routine influenza vaccination shortly after experiencing an illness classifiable as an adverse event. To minimize this effect, the control period for each adverse event category excluded the 14 days prior to vaccination. Unlike another case-only method, the case-crossover design (24), SCCS does not require the assumption that the underlying probability of vaccination is the same in all intervals (25), which is unlikely to hold for seasonally administered vaccines such as influenza.

The DID approach was used to investigate whether TIV in a particular season was riskier than TIV in cumulative previous seasons, using control periods for both current and previous seasons. It is an extension of the SCCS approach, first applied in economics research (19). The DID approach adjusts for the healthy vaccinee effect, assuming that an adverse event's effect on subsequent vaccination is the same in the current season as in previous seasons. This assumption may not always be valid—for example, if a medical care organization changes its policy for vaccinating after the occurrence of a particular adverse event between seasons. In addition, if the DID approach compares the relative risk of adverse events following pandemic influenza vaccine with the relative risk following TIV, the appropriateness of the comparison may be limited if the populations receiving pandemic vaccine and TIV are substantially different (e.g., according to underlying risk conditions). This approach differs from simply comparing risk periods in the current season with those in previous seasons without control periods; this would adjust for the healthy vaccinee effect but could be biased by secular trends in risk unrelated to vaccination (26).

Finally, a Poisson-based approach was used to compare the risk of Guillian-Barré syndrome following TIV in the current season with that in previous seasons. This analytic approach has been previously described (27) and (unlike SCCS and DID) previously applied for near real-time surveillance within the Vaccine Safety Datalink (VSD) Project (1013).

We classified instances of adverse events as occurring within a risk period following TIV, when association with vaccination was plausible, or occurring within a control period before vaccination (“control period 1”) or after the risk period following vaccination (“control period 2”), when the adverse event was more plausibly attributable to causes other than vaccination. Only vaccinees with an adverse event were included in analysis, and only if an adverse event occurred in either the risk period or the control period but not both.

Study population

Established in 1990 to monitor and evaluate vaccine safety, the VSD Project collects medical care and vaccination data on over 9 million members annually enrolled in 8 participating medical care organizations (28, 29). Staff at each VSD site prepare data files containing demographic and medical information on the members, such as age, sex, enrollment, vaccinations, and International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes assigned to medical encounters occurring in outpatient, emergency department, and hospital settings.

Children aged 6 months–17 years and adults aged ≥18 years were eligible if they 1) received TIV during the weeks of September 1 through April 30 in the 2005/06, 2006/07, or 2007/08 influenza seasons, 2) experienced a predefined adverse event during either the risk period or the control period in any season, and 3) were enrolled at the time of vaccination in a medical care organization participating in the VSD Project. Uniformly formatted files with immunization and diagnosis data are updated weekly and include information on VSD site, age, sex, week of TIV administration, adverse event type according to ICD-9 codes, number of days between TIV administration and the adverse event, and medical setting in which the adverse event was diagnosed.

The 8 medical care organizations within the VSD Project are: Group Health Cooperative, Seattle, Washington; Harvard Pilgrim Health Care, Boston, Massachusetts; HealthPartners Research Foundation, Minneapolis, Minnesota; Kaiser Permanente of Colorado, Denver, Colorado; Kaiser Permanente of Northern California, Oakland, California; Kaiser Permanente of Southern California, Pasadena, California; Marshfield Clinic Research Foundation; Marshfield, Wisconsin; and Northwest Kaiser Permanente, Portland, Oregon. The Centers for Disease Control and Prevention and the institutional review board at each site provided approval and agreed that informed consent was not required.

Adverse events

The adverse events evaluated included seizures (30), meningoencephalitis (3133), Bell's palsy (34, 35), other cranial nerve disorders, demyelinating disease (32, 36, 37), disorders of the peripheral nervous system and neuropathy (32, 38), ataxia (39), anaphylaxis (40, 41), allergic reaction other than anaphylaxis (including angioneurotic edema and urticaria) (42), and Guillian-Barré syndrome (4345) (Table 1). These neurologic (46) and allergic (47) adverse events were selected because they are clinically well-defined, are serious, and either had already been observed in published studies or passive surveillance or were biologically plausible as a consequence of TIV vaccination (e.g., anaphylaxis via immunoglobulin E-mediated hypersensitivity) (48).

Table 1.
Definitions of Adverse Events Used for Influenza Vaccine Safety Surveillance, Vaccine Safety Datalink Project, 2005/06–2007/08

The length of the risk period (Table 1) was adverse event-specific and depended on the timing of adverse events relative to receipt of vaccine in published studies (30, 34, 49) and biologic plausibility. The day of vaccination was included in the risk period only for those adverse events for which a same-day diagnosis was biologically plausible (e.g., anaphylaxis) and, for improved specificity, only for adverse events diagnosed in inpatient or emergency department settings. Certain conditions (e.g., urticaria) that are listed as diagnoses in the outpatient setting are often chronic diagnoses present before vaccination (50).

For most adverse event categories, the control period was selected to occur before the risk period to facilitate timely analyses. However, control periods were selected to occur after the postvaccination risk period (control period 2) for 3 adverse event categories, because 1) a recent prior adverse event might preclude vaccination (i.e., allergic reaction and anaphylaxis) or 2) a person might have an underlying condition that is also an indication for vaccination (i.e., seizure disorder).

For most adverse event categories, an adverse event was included in analysis only if it was the first event of its type to occur in a year, irrespective of vaccination timing (Table 1). This restriction ensured that 1) the specificity of adverse event definitions was enhanced, since follow-up visits for preexisting conditions were not interpreted as new-onset events, and 2) multiple events of the same type could not be counted during a single observation period. For adverse events that could plausibly have more than 1 new-onset episode in a year (e.g., seizure), the event was included in analysis if it was the first event in a shorter time period.

Statistical analysis

In the SCCS approach, the null hypothesis assumes that the risk of an adverse event's occurrence on a day during the exposed risk period following the first TIV dose in a season is the same as the risk of its occurrence on a day during the unexposed control period. For the current season under surveillance, c represents the count of adverse events in the control period, d represents the count of adverse events in the postvaccination risk period (Figure 1, part A), and r represents the ratio of c to d expected under the null hypothesis, based on the ratio of the number of days in the control period to that in the postvaccination period. The estimate of the relative risk is dr/c, which changes as data accumulate weekly. As Kulldorff et al. (27) previously described, the likelihood under the null hypothesis is L0 = (1/[r + 1])d × (r/[r + 1])c, and the likelihood under the alternative hypothesis is L1 = (d/[c + d])d × (c/[c + d])c. The log-likelihood ratio is then ln(L1/L0) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(1/[r + 1]) – c ln(r/[r + 1]).

Figure 1.
Risk and control periods for an example potential adverse event proximal to vaccination in the 2007/08 influenza season. The cumulative number of events occurring during each period, summed across previous seasons, is represented by a and b. The number ...

In the DID approach, the null hypothesis is that the relative risk of an adverse event's occurrence in the risk period versus the control period during the current influenza season is no greater than the same relative risk across cumulative previous seasons. The vaccine in previous seasons was assumed to have an acceptable risk profile (5). The estimate of the relative risk from previous seasons was based on at least 5 seasons and was treated as known without error (51); however, DID analyses were conducted only for adverse events with 25 or more cases in the risk and control periods in previous seasons. Accumulated from vaccinated cases across previous seasons, a represents the number of adverse events in the control period and b represents the number of adverse events in the postvaccination risk period (Figure 1, part B). For DID, the estimate of the relative risk ratio is (d/c)/(b/a). The likelihood is L0 = (b/[a + b])d × (a/[a + b])c, while L1 is identical to that of the SCCS approach. Therefore, the log-likelihood ratio is ln(L1/L0) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(b/[a + b]) – c ln(a/[a + b]).

For Guillian-Barré syndrome, we used a Poisson-based approach (27) to compare the risk following TIV in the current season with that in previous seasons (relative risk = [d/doses in current season]/[b/doses in previous seasons]). SCCS and DID analyses were not conducted for this adverse event; since the occurrence of Guillian-Barré syndrome within 6 weeks of TIV is a precaution for receiving future TIV (5), fewer cases of Guillian-Barré syndrome may occur among vaccinees during a control period prior to vaccination. This can bias SCCS analysis, and there may be too few events to adjust for this bias using DID analysis. In contrast, a postvaccination control period matching the length of a 42-day risk period would not elapse until 84 days after vaccination, long after most vaccinations are administered.

Prospective weekly surveillance was mimicked using sequential analysis (10), with the strong assumption of complete data accrual without delay. To account for repeated weekly looks at the data, we applied the maximized sequential probability ratio test, using a binomial probability model for all adverse events except Guillian-Barré syndrome (27). The null hypothesis was rejected if the log-likelihood ratio reached an upper bound (the “critical value” of the log-likelihood ratio). The null hypothesis was not rejected if the total number of adverse events reached a prespecified “upper limit,” or if surveillance ended without reaching this upper limit and without the log-likelihood ratio reaching the critical value.

For each adverse event, specific to each age group and season, the critical value of the log-likelihood ratio was dictated by the user-specified upper limit of expected adverse events and α-level. Upper limits were selected on the basis of the approximate number of adverse events expected under the null hypothesis, assuming that the risk in the current season was no greater than the risk in cumulative previous seasons. For example, upper limits for 2005/06 were determined using the risk of adverse events in the VSD population during risk and control periods around influenza vaccination, accumulated across the 2000/01–2004/05 influenza seasons; upper limits for the following season, 2006/07, were determined using data accumulated across 2000/01–2005/06, etc. If upper limits were reached before all cases were accumulated for the season, the power would be lower than what would have been possible with a higher upper limit. Therefore, upper limits were chosen such that they would not usually be reached. The upper limits were identical between the SCCS and DID analyses.

For very rare adverse events (e.g., anaphylaxis), surveillance was conducted for all ages together. For less rare adverse events (e.g., seizures), surveillance was conducted for children and adults separately. Because many adverse events and age groups were considered, α = 0.01 was selected. We used 1-tailed tests, looking only for risks of vaccination rather than protective effects. Surveillance ended after the risk and control periods passed for April vaccinees; if the upper limit had not been reached by that point, the cumulative α value remained below 0.01, indicating a conservative test.

Statistical power for adverse events in the 2007/08 season was calculated using previously described simulations (27) and was determined by the ratio of days in the control period to days in the risk period, the stringent selection of α = 0.01, and predefined upper limits. Since surveillance could end before upper limits were reached, statistical power could only be approximated, assuming that surveillance lasted at least halfway to the upper limit (Table 2). For the Poisson-based Guillian-Barré syndrome analysis in 2007/08, the background incidence following TIV was assumed from previous seasons to be 8.5 per million doses. With the upper limit equal to 20 and α = 0.05, the power was 45% to detect a relative risk of 1.5 and 92% to detect a relative risk of 2.0 (27).

Table 2.
Statistical Power of the Binomial-based Maximized Sequential Probability Ratio Test for Different Adverse Events and True Relative Risks, Vaccine Safety Datalink Project, 2007/08

Critical values of the log-likelihood ratio, above which the null hypothesis would be rejected, were obtained for Poisson and SCCS analyses for r = 1 from published tables (27). Critical values for SCCS analysis for r ≠ 1 and for DID analysis (accounting for the ratio of cases in historical control periods to cases in historical risk periods) were obtained from the same program used to generate the published tables (27).

Analyses were conducted using SAS, version 9.1.3 (SAS Institute Inc., Cary, North Carolina). Sample input data, SAS code, and output are provided in the Web Appendix, which is posted on the Journal’s Web site (


The number of TIV doses administered to children and adults in the VSD population increased each season, consistent with increased utilization following expanded recommendations for TIV use (5254). In 2005/06, 317,108 first TIV doses were administered to children aged 6 months–17 years, and 1,432,218 were administered to adults aged ≥18 years. In 2006/07, the first dose counts in these age groups were 415,446 and 1,598,880, respectively, and in 2007/08, the dose counts were 462,998 and 1,742,858. Vaccine administration was temporally concentrated; each season, more than 85% of vaccines were administered in October and November.

Fifteen age-specific adverse event categories were analyzed for each of 3 seasons, for a total of 45 sequential analyses each using SCCS and DID approaches and 3 additional sequential analyses using the Poisson-based approach for Guillian-Barré syndrome. Of these, the number of adverse events reached the prespecified upper limit in 1 analysis in all ages (Guillian-Barré syndrome in 2006/07) and in 5 analyses in adults (seizure in 2005/06, other cranial nerve disorder in 2005/06, and disorder of the peripheral nervous system in all 3 seasons).

The critical value of the log-likelihood ratio was not reached in any analysis, so no associations between vaccination and adverse events warranting further investigation were detected. Summarized results as of each season's final surveillance week are shown in Table 3. The DID analyses, which adjusted for the relative risk in previous seasons, yielded relative risk ratio estimates that did not always correspond to the SCCS results in terms of magnitude and direction of effect, as would be expected. The relative risk and relative risk ratio estimates obtained using both approaches were not statistically significant for any adverse event. The Poisson-based approach for Guillian-Barré syndrome showed that the risk after vaccination each season was not significantly greater than in previous seasons.

Table 3.
Results of Sequential Testing for Influenza Vaccine Safety as of Each Season's Final Surveillance Week, Vaccine Safety Datalink Project, 2005/06–2007/08

Weekly sequential analysis results for 1 representative outcome, seizure in adults, are highlighted for both the SCCS (Figure 2, part A) and DID (Figure 2, part B) approaches. The relative risk or relative risk ratio had a rapid increase early in the season with few doses administered, but as the season progressed and more doses were given, the relative risk or relative risk ratio stabilized and quickly trended towards 1. During surveillance, the log-likelihood ratio peaked at only 12% of the critical value for SCCS and at only 35% of the critical value for DID. In other words, the SCCS approach showed that vaccinated cases had no statistically significant elevated risk of seizure in the risk period versus the control period in 2005/06, 2006/07, or 2007/08. The DID approach showed that this relative risk was not significantly greater in 2005/06 than in 2000/01–2004/05, not significantly greater in 2006/07 than in 2000/01–2005/06, and not significantly greater in 2007/08 than in 2000/01–2006/07.

Figure 2.
Weekly sequential analysis results for seizures among trivalent inactivated influenza vaccine recipients aged ≥18 years in the 2007/08 influenza season, Vaccine Safety Datalink Project. A) Self-controlled case series approach; B) difference-in-difference ...


Near real-time surveillance for selected adverse events following seasonal or pandemic influenza vaccine is possible in systems such as the VSD Project, where data are updated at least weekly. Implementing SCCS and DID approaches in sequential testing allows for a timely alternative to more traditional systems for evaluating influenza vaccine safety, while adjusting for important confounders. Previous randomized controlled trials (49, 55, 56), passive reporting systems (30, 57), and postlicensure safety studies (20, 21) have supported the overall safety of TIV. This multisite exercise was consistent with previous studies in showing no evidence of elevated risk following TIV for any of 10 predefined adverse event categories. Poisson-based analyses indicated that in each of 3 recent seasons, the risk of Guillian-Barré syndrome following vaccination was not significantly different from that in previous seasons. SCCS analyses indicated that the risk of adverse events during the risk period was not significantly different from that during the control period. DID analyses indicated that the relative risk in each season was not significantly different from the relative risk across cumulative previous seasons. SCCS and DID analyses compared risk periods with control periods within the same vaccinees each season, thus controlling for potential biases. Using binomial-based maximized sequential probability ratio testing, power was adequate for most adverse event categories to detect relative risks in the range of 2.0 and very good to detect high relative risks in the range of 5.0; more modest risk elevations may not have been detectable using these methods. However, if a modest elevation in risk were undetected for a very rare adverse event, the population attributable risk for vaccination would be very small. The lack of any safety signals can increase public confidence in influenza vaccine safety.

Using more than 1 analytic approach for surveillance may enhance the rigor of a safety assessment. A signal based on either method would warrant additional investigation for assessment of its validity. If both approaches were to produce a signal for the same adverse event, evidence for a safety concern requiring further evaluation would be strengthened. The relative risks and relative risk ratios produced by SCCS and DID were different, which was expected because they are calculated differently, but neither approach generated a safety signal. The application of repeated confidence intervals to maximized sequential probability ratio testing is under development (58).

At least 4 potential limitations affected the analyses. First, timely detection of a safety problem depends partially on the lag between adverse event occurrence and electronic reporting. This lag is typically 2 weeks for outpatient and emergency department data, but it ranges from 1 day to several weeks for hospitalizations, depending on the data source. The delay in ascertaining hospitalizations could be reduced with additional resources by using health plans that ascertain hospitalizations via electronic records (not claims) or by monitoring preadjudicated claims. Near real-time surveillance is most useful for adverse event categories requiring observation of only a brief risk period of less than 2–3 weeks, so that a signal could occur within a month after vaccines are given. In such a situation, surveillance may identify a safety problem before most vaccines for a season are administered, and warnings may be issued to prevent additional exposure. In contrast, using sequential analyses for adverse events with a lengthy risk window or a postvaccination control period sacrifices both power and simplicity with little potential gain in timeliness compared with a final analysis each spring.

Second, the VSD Project uses electronic data collected largely for administrative purposes. Some vaccinations are missed (16), as are telephone encounters and adverse events that are not medically attended. The sensitivity, specificity, and positive and negative predictive values of ICD-9 codes vary by adverse event and setting (5961) and have not been thoroughly characterized. Third, binomial-based maximized sequential probability ratio testing had limited power for timely detection of a modest elevation in risk for very rare adverse events (e.g., anaphylaxis). To minimize false-negative findings, power could be improved as the VSD population expands and by investing in larger surveillance systems—for example, the US Food and Drug Administration's Sentinel Initiative, which aims to use electronic data from 100 million people by 2012 (62). Using a historical comparison group and a Poisson probability model is another alternative for rare adverse events and may have better statistical power (27). Finally, although the SCCS and DID approaches adjusted for some confounders, they did not adjust for seasonality of vaccination and adverse event risk. A case-centered approach (63) accounting for the timing of vaccination in the overall population could be used to adjust for seasonal effects.

To our knowledge, this is the first application of both SCCS and DID approaches to the sequential monitoring of adverse events. In addition to influenza vaccines, these approaches may be applicable to monitoring the safety of other vaccines for which finding an appropriate comparison group is challenging and the healthy vaccinee effect may be substantial. These approaches also may be useful in drug safety surveillance (e.g., for influenza antiviral medications) and for other data sources (e.g., Medicare or other insurers) (62, 64, 65). New adverse events of potential concern (e.g., those detected through the Vaccine Adverse Event Reporting System) can be quickly added to surveillance, and subgroup analyses (e.g., for manufacturer-specific vaccine products) can be readily undertaken, although power may be limited. If a safety signal is generated in the future, additional investigations will distinguish true positives from false positives, including evaluation of the temporal association between vaccination and adverse events, high-risk subgroup analyses, and medical chart review (12, 66).

Near real-time safety surveillance for seasonal and pandemic influenza vaccines is possible, and prospective analyses for seasonal influenza vaccines are being conducted, starting with the 2008/09 season. With a well-defined population and the ability to access medical records for review, the VSD Project will monitor the safety of influenza A (H1N1) 2009 monovalent vaccines (67), using binomial- (SCCS and DID) and Poisson-based approaches.

Supplementary Material



Author affiliations: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts (Sharon K. Greene, Martin Kulldorff, Rong Li, Ruihua Yin, Tracy A. Lieu, Grace M. Lee); Kaiser Permanente Vaccine Study Center, Oakland, California (Edwin M. Lewis, Bruce H. Fireman, Roger Baxter); Immunization Safety Office, Centers for Disease Control and Prevention, Atlanta, Georgia (Eric S. Weintraub, Karen R. Broder); HealthPartners Research Foundation, Minneapolis, Minnesota (James D. Nordin); Kaiser Permanente of Colorado, Denver, Colorado (Jason M. Glanz); Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California (Steven J. Jacobsen); and Division of Infectious Diseases, Department of Medicine and Department of Laboratory Medicine, Children's Hospital Boston, Boston, Massachusetts (Grace M. Lee).

This work was supported by a subcontract with America's Health Insurance Plans under contract 200-2002-00732 from the Centers for Disease Control and Prevention and by the Agency for Healthcare Research and Quality, US Department of Health and Human Services (grant 5 K08 HS013908-04 to G. M. L.).

For their valuable contributions to conception, data acquisition, and technical support of this work, the authors acknowledge Irene M. Shui and Drs. Richard Platt, Jeffrey S. Brown, Edward Belongia, Allison L. Naleway, Lisa A. Jackson, James G. Donahue, Robert L. Davis, and Margarette Kolczak (deceased).

Preliminary results of this work were presented as a platform session (abstract G-769) at the 48th Annual Interscience Conference on Antimicrobial Agents and Chemotherapy and the Infectious Diseases Society of America 46th Annual Meeting in Washington, DC, on October 25, 2008.

The findings and conclusions in this report are those of the authors and do not necessarily represent the views or policies of the Centers for Disease Control and Prevention.

Edwin M. Lewis reports having received grant support from Sanofi Pasteur, Chiron, MedImmune, Merck, and Wyeth. Dr. James D. Nordin reports having received grant support from Sanofi Pasteur. Dr. Roger Baxter reports having received grant support from Sanofi Pasteur, Novartis, GSK, and MedImmune. No other authors have any potential or real conflicts of interest to declare.



International Classification of Diseases, Ninth Revision
self-controlled case series
trivalent inactivated influenza vaccine
Vaccine Safety Datalink


1. Update: novel influenza A (H1N1) virus infections—worldwide, May 6, 2009. MMWR Morb Mortal Wkly Rep. 2009;58(17):453–458. [PubMed]
2. Collin N, de Radiguès X. Vaccine production capacity for seasonal and pandemic (H1N1) 2009 influenza. World Health Organization H1N1 Vaccine Task Force. Vaccine. 2009;27(38):5184–5186. [PubMed]
3. Center for Biologics Evaluation and Research, US Food and Drug Administration. Influenza A (H1N1) 2009 Monovalent. Rockville, MD: Center for Biologics Evaluation and Research; 2009. ( (Accessed November 10, 2009)
4. Use of influenza A (H1N1) 2009 monovalent vaccine: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2009. MMWR Recomm Rep. 2009;58(RR-10):1–8. [PubMed]
5. Fiore AE, Shay DK, Broder K, et al. Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2009. MMWR Recomm Rep. 2009;58(RR-8):1–52. [PubMed]
6. Schwartz B, Hinman A, Abramson J, et al. Universal influenza vaccination in the United States: are we ready? Report of a meeting. J Infect Dis. 2006;194(suppl 2):S147–S154. [PubMed]
7. Iskander J, Broder K. Monitoring the safety of annual and pandemic influenza vaccines: lessons from the US experience. Expert Rev Vaccines. 2008;7(1):75–82. [PubMed]
8. US Department of Health and Human Services. HHS Pandemic Influenza Plan. Supplement 6. Vaccine Distribution and Use. Washington, DC: US Department of Health and Human Services; 2005. Vaccine monitoring and data collection. (Section S6-III.B3) ( (Accessed May 11, 2009)
9. Zhou W, Pool V, Iskander JK, et al. Surveillance for safety after immunization: Vaccine Adverse Event Reporting System (VAERS)—United States, 1991–2001. MMWR Surveill Summ. 2003;52(1):1–24. [PubMed]
10. Lieu TA, Kulldorff M, Davis RL, et al. Real-time vaccine safety surveillance for the early detection of adverse events. Med Care. 2007;45(10 suppl 2):S89–S95. [PubMed]
11. Yih WK, Nordin JD, Kulldorff M, et al. An assessment of the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (Tdap) vaccine, using active surveillance for adverse events in the Vaccine Safety Datalink. Vaccine. 2009;27(32):4257–4262. [PubMed]
12. Update: recommendations from the Advisory Committee on Immunization Practices (ACIP) regarding administration of combination MMRV vaccine. MMWR Morb Mortal Wkly Rep. 2008;57(10):258–260. [PubMed]
13. Belongia EA, Irving S, Shui IM, et al. Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent rotavirus vaccine (RotaTeq) Pediatr Infect Dis J. 2009 Nov 10 [Epub ahead of print] [PubMed]
14. Hak E, Verheij TJ, Grobbee DE, et al. Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications. J Epidemiol Community Health. 2002;56(12):951–955. [PMC free article] [PubMed]
15. Jackson ML, Nelson JC, Weiss NS, et al. Influenza vaccination and risk of community-acquired pneumonia in immunocompetent elderly people: a population-based, nested case-control study. Lancet. 2008;372(9636):398–405. [PubMed]
16. Greene SK, Shi P, Dutta-Linn MM, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med. 2009;37(6):552–555. [PubMed]
17. Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics. 1995;51(1):228–235. [PubMed]
18. Whitaker HJ, Farrington CP, Spiessens B, et al. Tutorial in biostatistics: the self-controlled case series method. Stat Med. 2006;25(10):1768–1797. [PubMed]
19. Meyer BD. Natural and quasi-experiments in economics. J Bus Econ Stat. 1995;13(2):151–161.
20. France EK, Glanz JM, Xu S, et al. Safety of the trivalent inactivated influenza vaccine among children: a population-based study. Arch Pediatr Adolesc Med. 2004;158(11):1031–1036. [PubMed]
21. Hambidge SJ, Glanz JM, France EK, et al. Safety of trivalent inactivated influenza vaccine in children 6 to 23 months old. JAMA. 2006;296(16):1990–1997. [PubMed]
22. Stowe J, Andrews N, Wise L, et al. Investigation of the temporal association of Guillain-Barré syndrome with influenza vaccine and influenzalike illness using the United Kingdom General Practice Research Database. Am J Epidemiol. 2009;169(3):382–388. [PubMed]
23. Virtanen M, Peltola H, Paunio M, et al. Day-to-day reactogenicity and the healthy vaccinee effect of measles-mumps-rubella vaccination [electronic article] Pediatrics. 2000;106(5):E62. [PubMed]
24. Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. 1991;133(2):144–153. [PubMed]
25. Farrington CP. Control without separate controls: evaluation of vaccine safety using case-only methods. Vaccine. 2004;22(15-16):2064–2070. [PubMed]
26. Davis R, Lewis N, Kulldorff M, et al. Rapid cycle analysis of vaccine safety data [abstract] Presented at the Vaccine Safety Datalink Annual Meeting, Redondo Beach, California, April 20, 2005.
27. Kulldorff M, Davis RL, Kolczak M, et al. A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance. (Working paper) Boston, MA: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute; 2009. (
28. Chen RT, DeStefano F, Davis RL, et al. The Vaccine Safety Datalink: immunization research in health maintenance organizations in the USA. Bull World Health Organ. 2000;78(2):186–194. [PubMed]
29. DeStefano F. The Vaccine Safety Datalink Project. Pharmacoepidemiol Drug Saf. 2001;10(5):403–406. [PubMed]
30. McMahon AW, Iskander J, Haber P, et al. Adverse events after inactivated influenza vaccination among children less than 2 years of age: analysis of reports from the Vaccine Adverse Event Reporting System, 1990–2003. Pediatrics. 2005;115(2):453–460. [PubMed]
31. Gross WL, Ravens KG, Hansen HW. Meningoencephalitis syndrome following influenza vaccination. J Neurol. 1978;217(3):219–222. [PubMed]
32. Haber P. “Immunization Safety Review: Influenza Vaccine and Possible Neurological Complications,”. Influenza vaccine and neurological adverse events: VAERS 7/1990–1/2003 [abstract]. Presented at the meeting. Washington, DC, March 13, 2003. Washington, DC: Institute of Medicine; 2003. ( (Accessed October 10, 2008)
33. Nakamura N, Nokura K, Zettsu T, et al. Neurologic complications associated with influenza vaccination: two adult cases. Intern Med. 2003;42(2):191–194. [PubMed]
34. Zhou W, Pool V, DeStefano F, et al. A potential signal of Bell's palsy after parenteral inactivated influenza vaccines: reports to the Vaccine Adverse Event Reporting System (VAERS)—United States, 1991–2001. Pharmacoepidemiol Drug Saf. 2004;13(8):505–510. [PubMed]
35. Chou CH, Liou WP, Hu KI, et al. Bell's palsy associated with influenza vaccination: two case reports. Vaccine. 2007;25(15):2839–2841. [PubMed]
36. Brostoff JM, Beitverda Y, Birns J. Post-influenza vaccine chronic inflammatory demyelinating polyneuropathy. Age Ageing. 2008;37(2):229–230. [PubMed]
37. Hull TP, Bates JH. Optic neuritis after influenza vaccination. Am J Ophthalmol. 1997;124(5):703–704. [PubMed]
38. Hull JH, Mead SH, Foster OJ, et al. Severe vasculitic neuropathy following influenza vaccination. J Neurol Neurosurg Psychiatry. 2004;75(10):1507–1508. [PMC free article] [PubMed]
39. Saito H, Yanagisawa T. Acute cerebellar ataxia after influenza vaccination with recurrence and marked cerebellar atrophy. Tohoku J Exp Med. 1989;158(1):95–103. [PubMed]
40. Bohlke K, Davis RL, Marcy SM, et al. Risk of anaphylaxis after vaccination of children and adolescents. Pediatrics. 2003;112(4):815–820. [PubMed]
41. Coop CA, Balanon SK, White KM, et al. Anaphylaxis from the influenza virus vaccine. Int Arch Allergy Immunol. 2008;146(1):85–88. [PubMed]
42. Horowitz L. Urticaria subsequent to administration of influenza vaccine. South Med J. 1979;72(10):1334–1335. [PubMed]
43. Juurlink DN, Stukel TA, Kwong J, et al. Guillain-Barré syndrome after influenza vaccination in adults: a population-based study. Arch Intern Med. 2006;166(20):2217–2221. [PubMed]
44. Schonberger LB, Bregman DJ, Sullivan-Bolyai JZ, et al. Guillain-Barré syndrome following vaccination in the National Influenza Immunization Program, United States, 1976–1977. Am J Epidemiol. 1979;110(2):105–123. [PubMed]
45. Haber P, DeStefano F, Angulo FJ, et al. Guillain-Barré syndrome following influenza vaccination. JAMA. 2004;292(20):2478–2481. [PubMed]
46. Piyasirisilp S, Hemachudha T. Neurological adverse events associated with vaccination. Curr Opin Neurol. 2002;15(3):333–338. [PubMed]
47. Kalaboka S, Annesi-Maesano I. The complex link between immunization against childhood diseases and allergy. Expert Rev Vaccines. 2007;6(4):635–643. [PubMed]
48. Moylett EH, Hanson IC. Mechanistic actions of the risks and adverse events associated with vaccine administration. J Allergy Clin Immunol. 2004;114(5):1010–1020. [PubMed]
49. Nichol KL, Margolis KL, Lind A, et al. Side effects associated with influenza vaccination in healthy working adults. A randomized, placebo-controlled trial. Arch Intern Med. 1996;156(14):1546–1550. [PubMed]
50. Moore KM, Duddy A, Lee GM, et al. Outpatient urticaria diagnosis codes have limited predictive value for same-day influenza vaccine adverse event detection. J Clin Epidemiol. 2009 Nov 2 [Epub ahead of print] [PubMed]
51. Li L, Kulldorff M. A conditional maximized sequential probability ratio test for pharmacovigilance. Stat Med. In press. [PubMed]
52. Harper SA, Fukuda K, Uyeki TM, et al. Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP) MMWR Recomm Rep. 2005;54(RR-8):1–40. [PubMed]
53. Smith NM, Bresee JS, Shay DK, et al. Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP) MMWR Recomm Rep. 2006;55(RR-10):1–42. [PubMed]
54. Fiore AE, Shay DK, Haber P, et al. Prevention and control of influenza. Recommendations of the Advisory Committee on Immunization Practices (ACIP) MMWR Recomm Rep. 2007;56(RR-6):1–54. [PubMed]
55. Barry DW, Mayner RE, Hochstein HD, et al. Comparative trial of influenza vaccines. II. Adverse reactions in children and adults. Am J Epidemiol. 1976;104(1):47–59. [PubMed]
56. Govaert TM, Dinant GJ, Aretz K, et al. Adverse reactions to influenza vaccine in elderly people: randomised double blind placebo controlled trial. BMJ. 1993;307(6910):988–990. [PMC free article] [PubMed]
57. Vellozzi C, Burwen DR, Dobardzic A, et al. Safety of trivalent inactivated influenza vaccines in adults: background for pandemic influenza vaccine safety monitoring. Vaccine. 2009;27(15):2114–2120. [PubMed]
58. Mullooly J, Crane B. MaxSPRT-based repeated interval estimates of event rate ratios for independent concurrent comparison populations [abstract] Presented at the Vaccine Safety Datalink Annual Meeting, Atlanta, Georgia, March 18–19, 2009.
59. Mullooly J, Drew L, DeStefano F, et al. Quality assessments of HMO diagnosis databases used to monitor childhood vaccine safety. Methods Inf Med. 2004;43(2):163–170. [PubMed]
60. Mullooly JP, Donahue JG, DeStefano F, et al. Predictive value of ICD-9-CM codes used in vaccine safety research. Methods Inf Med. 2008;47(4):328–335. [PubMed]
61. Shui IM, Shi P, Dutta-Linn MM, et al. Predictive value of seizure ICD-9 codes for vaccine safety research. Vaccine. 2009;27(39):5307–5312. [PubMed]
62. Platt R, Wilson M, Chan KA, et al. The new Sentinel Network—improving the evidence of medical-product safety. N Engl J Med. 2009;361(7):645–647. [PubMed]
63. Fireman B, Lee J, Lewis N, et al. Influenza vaccination and mortality: differentiating vaccine effects from bias. Am J Epidemiol. 2009;170(5):650–656. [PMC free article] [PubMed]
64. Burwen DR, La Voie L, Braun MM, et al. Evaluating adverse events after vaccination in the Medicare population. Pharmacoepidemiol Drug Saf. 2007;16(7):753–761. [PubMed]
65. Brown JS, Moore KM, Braun MM, et al. Active influenza vaccine safety surveillance: potential within a healthcare claims environment. Med Care. 2009 Sep 25. [Epub ahead of print] [PubMed]
66. Yih K, Kulldorff M. Three years of rapid cycle analysis—what we've learned and what remains to be done [abstract] Presented at the Vaccine Safety Datalink Annual Meeting, Atlanta, Georgia, March 18–19, 2009.
67. DeStefano F, Tokars J. H1N1 vaccine safety monitoring: beyond background rates. Lancet. 2009 Oct 31 [Epub ahead of print] [PubMed]

Articles from American Journal of Epidemiology are provided here courtesy of Oxford University Press