To study vaccine AEs associated with a specific vaccine, two types of immunization population denominators can be used. One is the total number of people immunized for one single vaccine in a region during a given period. Another type of population denominator is the total number of people who reported AE cases to VAERS for all vaccines in a region during a given period. Like other VAERS bioinformatics studies, our approach uses the second immunization population denominator 
. Although the VAERS spontaneous reporting system lacks a true control (i.e., people randomized to receive a placebo), our bioinformatics method analyzed AEs associated with a vaccine using all other vaccines as a quasi-control group for comparison 
In our study, we did not directly compare TIV vs. LAIV. Essentially, we tested all the AE case reports associated with TIV or LAIV independently against the whole VAERS database. Then the results of significantly enriched AEs in each group were identified using our combinatorial bioinformatics analysis pipeline. Our CODAE pipeline contains three methods for detection of true AE signals: PRR, Chi-square, and filtering based on the number of reports. After the significance of individual AEs associated with TIV or LAIV was identified, we compared the TIV and LAIV-enriched AEs through two ways. The first one is qualitative (i.e., presence or absence) comparison between the two AE lists. The second method is through quantitative comparison, i.e., comparing the ratios of TIV (or LAIV)-associated AE case number over the total number of the same AE in the whole VAERS database. In the end, we used ontology-based methods to classify and compare significantly enriched AEs in each group.
We hypothesized that the AE differences in the two sets of recipients (TIV vs. LAIV) emerged from different immune-response pathways induced by each type of vaccine. Our study suggests that the combinatorial CODAE bioinformatics approach can overcome the complex challenges in public post-vaccination event record data. The strategy of this study resolves the issue of high-noise data, especially when these data contain high-value hidden knowledge that can be evaluated by robust statistical tests. It is crucial to identify background information, as some AEs are common to many vaccines. Because the number of reports in VAERS database is large (616,215 cases, 75 vaccines), background information is not sensitive to minor change or adjustment such as removing reports from one or two vaccines from the studied sample set. One novel feature of our combinatorial workflow (summarized in ) is its application in comparing two cohort sets of AEs. Another novelty of our approach is the use of the OAE for categorization of identified AEs. While this combinatorial workflow was applied specifically to VAERS data, the concept can also be adapted and applied to other questions in the Translational Informatics domain. Furthermore, the preliminary result in the form of flat list (i.e., the simple text file of records) may be informative at an individual AE level. However, it is difficult to examine the flat list to identify the underlying biological systems when the system is composed of multiple interactions among multiple participating AEs. It is challenging to draw any connections between biological processes while the significant individual AE terms scatter across various different biological functions and systems. Examining these AEs based on their score rankings along with reorganizing results by their semantic similarity and functional relevance leads to a better representation of data that can overcome this issue. Analysis of alignment of semantic similarity to a reference structured controlled vocabulary is discussed below.
The results indicated that out of >37,000 TIV-associated AEs, 48 met the threshold for inclusion in the analysis, while of roughly 3,700 LAIV AEs, 68 met the threshold. Although this seems counter intuitive or surprising, TIV-associated AEs include two severe AEs (GBS and paralysis). Many other TIV-associated AEs are also related to neurological and muscular disorders, which can be considered as mild symptoms that can be further progressed to more severe symptoms including GBS and paralysis. GBS is classified as a syndrome that has indication of multiple symptoms. On the other hand, LAIV vaccination appears to induce many mild symptoms. No severe LAIV AEs that pass our thresholds has been detected.
The age patterns associated with the reports of GBS and other GBS-related disorders between patients immunized with TIV and LAIV are quite different (). For TIV-associated GBS cases, the age range of 45–79 (34 years) has a high peak of >
1 per 1,000 TIV AE case reports. The highest rate is approximately 2 cases per 1,000 TIV AE case reports at the age range of 55–69. For LAIV, the age ranges of 5–9 has slightly higher than 1 per 1,000 LAIV AE case reports, and the age range of 20–24 is associated with approximately 2 per 1,000 LAIV AE case reports. It appears that TIV is associated with GBS in a longer period of time and primarily occurs in adult and senior age, and LAIV-associated GBS primarily occurs in young age. Based on these observations, to better prevent GBS, it might be a good strategy to use TIV for young age patients and LAIV for adults and seniors.
Incidents of serious AEs are not always easy to detect in terms of population statistics as they may require a long period of observation. Therefore, the detection and confirmation of such incidents can be inconclusive or take a long time. Examples of time-consuming observations of vaccine post-marketing AEs include GBS after 1976 Swine Flu to 2009 A/H1N1 influenza vaccine campaigns 
, anthrax vaccine adverse events (VAEs) studied from 1990 to 2007 
, and 1990–2007 measles vaccine adverse effects studied in the Ivory Coast 
. Although the 1976 incidence of GBS following Swine Flu vaccination was detected in real time, debate and discussion of the incidence remained inconclusive.
One interesting finding from this study was the occurrence of GBS in TIV recipients. There have been many controversial results with regard to the post influenza vaccination incidents, whether or not influenza vaccines induce GBS in the recipients. When considering specific subgroups of influenza vaccines (TIV versus LAIV), our analysis suggests that compared to LAIV, TIV is more strongly correlated with GBS (). Haber et al. concluded that the occurrence of GBS in influenza vaccine recipients was merely temporal association, and the causal association was not implicated with any solid evidence 
. Furthermore, Haber et al. had challenged the study of Souayah et al. 
that used the VAERS dataset by pointing out the VAERS limitation due to lack of standardized case follow-up. Haber et al. also argued that influenza vaccine-associated GBS incidence should be determined by influenza season rather than calendar year 
. After a careful examination of the data, we found that pooling the entire VAERS dataset with our methodology could overcome the issues of omitted data or reporting intervals. Whether or not the reporting interval was based on the season or calendar year, overall incidence rate was not dependent on any one particular year or season. The number of post-influenza-vaccine GBS confirmed by neurologists in VAERS (1995–2003) as investigated by Haber et al. was observed to be 82%. This observation, when combined with additional data that became available in the later years, was still statistically significant as shown in a larger dataset such as the dataset used in this study. Souayah et al.'s study in 2009 remained firm in their conclusion of influenza vaccine-associated GBS with significant incidence rate 
. Evans et al. also associated GBS and rare adverse events with influenza vaccine by conducting a comparative study of the novel influenza (swine flu) prepandemic data in 2009 to 1976 National influenza Immunization Program data 
. Our study found that Souayah's and Evans' GBS association to influenza vaccines held true only when considering TIV, not LAIV. Furthermore, in a recent study by Moro et al., severe adverse events including GBS were implicated in the TIV high-dose recipients 
. To the best of our knowledge, our systematic comparative study is the first to suggest that severe adverse events included GBS are more likely to be associated with trivalent (killed) inactivated influenza vaccine (TIV), but not live attenuated influenza vaccine (LAIV) or monovalent inactivated influenza vaccine.
As indicated in our meta-analysis, six out of 19 influenza vaccine-associated GBS reports show increased incidence rate of GBS (
) (). In all these five studies that concluded the association between GBS and influenza vaccines, trivalent inactivated influenza vaccine (TIV) vaccination, instead of monovalent inactivated influenza vaccine (MIV) administration, was used. This phenomenon suggests no detected association of GBS to MIV vaccination. It is likely that the mix of different inactivated influenza strains in the TIV may increase the chance of obtaining GBS. However, these studies were usually based on individual case investigation with a relatively small cohort. Further investigation on the subject of monovalent versus trivalent inactivated influenza vaccines as the trigger of post-immunization GBS is required before conclusion can be made.
Even though the safety of TIVs is generally accepted at the population level, our analysis points towards LAIV as an alternative immunization that is less likely for the recipient to develop severe AEs such as GBS or paralysis. However, although the number of reported SAE cases associated with LAIV is very small that GBS and paralysis were not statistically enriched in LAIV group, the occurrences of LAIV-associated SAEs should still be investigated carefully. While GBS and paralysis (as categorized to be severe adverse events) were statistically enriched in the TIV group, for further study, the weighted-AE scoring method should be applied in future studies to properly address the issue of SAEs. All SAEs should automatically rank high in the significance of AE for both cohorts (TIV and LAIV).
Utilizing data from Vaccine Safety Datalink project, Lee et al. conducted a weekly sequential analysis of potential influenza vaccine adverse events from 9.2 million members in eight U.S. medical care organizations from November 2009 to April 2010 
. Both trivalent and monovalent seasonal killed and live attenuated influenza vaccines over a long observation period were examined. In total, 15 cases of GBS from 1,345,663 monovalent killed influenza vaccine (MIV)-vaccinated individuals were identified following MIV administration, 23 out of 2,741,150 cases after TIV, and zero out of 157,838 cases after LAIV. This study found that the GBS incidence after TIV administration was slightly lower than that after MIV administration. The incidence rates of GBS after both trivalent and monovalent killed influenza vaccines was approximately 1 in 100,000, which was not considered as statistically significant signals for GBS 
. This study has also been included in our meta-analysis (). The result of this study does not change our meta-analysis conclusion that GBS was associated with TIV rather than MIV vaccination. Interestingly, zero cases of GBS were identified in LAIV-vaccinated patients, supporting our conclusion that LAIV may be safer than TIV in terms of induction of GBS vaccine adverse events. However, the sample size (157,838) of LAIV-vaccinated patients is relatively low compared to MIV- or TIV-vaccinated patients in their study. Our conclusion was drawn by combinatorial statistical analysis of VAERS case report data. Using all available VAERS case report data, our combinatorial method compares TIV- or LAIV-associated vaccine adverse events (VAEs) with VAEs associated with other vaccines. Our results suggest statistically significant association between TIV and GBS and paralysis, while LAIV shows no statistical evidence of correlation to GBS. Further studies to verify our conclusion of the higher safety of LAIV over TIV in terms of GBS induction are required.
One novel finding from our combinatorial bioinformatics analysis of influenza vaccine adverse events is that beside the relatively higher reporting rate of GBS and paralysis severe adverse event (SAE) cases associated with TIV than LAIV, TIV vaccination was also associated with a set of other mild neurological and muscular adverse events (e.g., paraesthesia, hyporeflexia, musculoskeletal pain, and neuralgia) (). Although these symptoms are not considered SAEs, it is likely that these symptoms are signals of potential future GBS and paralysis SAEs due to their common neurological and muscular roots. The non-severe symptoms in some healthier patients may suggest signs of severe symptoms in other weaker patients. We hypothesize that the molecular interaction networks underlying these neurological and muscular adverse events, whether it is severe (i.e., GBS and paralysis) or non-severe (i.e., paraesthesia and hyporeflexia), are the same or similar at least at the early stage of these adverse events. By exploring these interaction networks, we can potentially identify the mechanisms of severe VAEs.
Further analysis of TIV- and LAIV-induced AEs by reorganizing into an ontological structure with reference to other community-accepted ontologies reveals certain challenges that need to be properly addressed. We have clustered AEs of each group of vaccines to COSTART (1995) (the foundation vocabulary that MedDRA was built upon) with an embedded hierarchical structure available on BioPortal (http://bioportal.bioontology.org/visualize/40390
). We found that the COSTART hierarchical structure might not be a suitable term reorganization reference as the COSTART/MedDRA structure lacked a specific definition on which the aspect of this hierarchical tree was based. It was not clear if the hierarchy was defined by biological processes, or anatomy of the body. Hypothetically, COSTART/MedDRA is a comprehensive dictionary of adverse event descriptors; however, it was not created for the purpose of computation and the structure organization may not be fully equipped for ontological machine processing. Many concepts in COSTART listed synonyms that were not true synonyms. For example, in COSTART, sinus headache was defined to be synonymous to headache, and infection upper respiratory was defined to be synonymous to infection. Many examples of this kind of synonym error occur throughout the COSTART hierarchy. Another major issue in using COSTART as an ontological reference was that COSTART contained duplicate classes that caused ambiguity in many situations. For example, ear disorder was a child under a parent class of the same class name ear disorder, hemorrhage was a child of parent class haemorrhage [same word], and hypotension was a child of parent class shock syndrome which was, in turn, a sibling class of another concept that also has class identifier of hypotension. In an improvement of COSTART that results in MedDRA (version 12, released on 03/01/2009), terms are reorganized in a more comprehensive hierarchical structure, but further issues of one asserted class falling under multiple asserted parent nodes, or ambiguous synonym listings remain problematic. Examples of these classes can be found in class properties of Migraine, Migraine headache, Sinus headache, and other AEs throughout MedDRA (Figure S3
We then explored another clinical ontology of SNOMED Clinical Terms (Version 07/31/2010) to find an alternative for AE term reorganization for the purpose of recognizing AEs based on biological relevance (Figure S4
). We found that, while SNOMED CT was thoroughly defined with the most detailed information of anatomical and physiological description, this ontology still may not be the best alternative for such purposes. The comprehensive organization of terms in SNOMED CT resulted in a structure that did not provide an apparent clustering for term recognition based on biological process, because classes at the individual AE level (leaf nodes) were scattered across the ontology due to the nature of very detailed parental subclasses.
From the investigation of MedDRA and SNOMED CT as a reference controlled vocabulary, neither suited the purpose for such reference. Although MedDRA may not be the best nomenclature system for the purpose of AE reporting, it has been referenced in VAERS for many years. Therefore, to be able to mine for discovery within VAERS records, we must find methods to process and interpret VAERS data in an efficient way that discovers the knowledge embedded within it. Also, sometimes, MedDRA terms that are reported in VAERS fall into many semantic types in SNOMED CT, namely Body structure, Clinical finding, Procedure, Special concept, or Qualifier value. Such terms are within the same semantic type and there are also sub-structures that may further divide MedDRA terms into many separate groups. One example scenario that appeared in this situation is how Edema was described and categorized on the two ontologies. In SNOMED, Edema is a Clinical finding while Edema of pharynx is a child of Disorder characterized by edema, Disorder characterized by edema is a subclass of Disease, while Disease is a Clinical finding. This, in turn, resulted in Disease that was a sibling of class Edema while containing a child of a child of class Edema of pharynx. This separation by different semantic types occurred frequently in SNOMED-CT.
In this study, we have shown that the application of OAE can provide insights into underlying processes that may be overlooked or hard to detect without efficient prior-knowledge structural hierarchy ( and Figure S2
). Our combinatorial bioinformatics approach integrates different biostatistics analysis methods with ontology-based AE term classification. This method can also be modified to answer adverse event questions in different areas, for example, drug adverse events. It is noted that current study only used a part of the OAE features. Different from MedDRA and SNOMED-CT that represent adverse event outcomes, the OAE targets the representation of the whole process starting from the medical intervention (e.g.
, vaccination) and ends with the discovery of the adverse event outcomes. Therefore, OAE provides a platform to examine in detail all the variables that can affect the results, such as patient age and sex, vaccination dose and route, and time interval between vaccination and the outcome of symptoms. Different from the original version of the Adverse Event Ontology (AEO) 
, the term “adverse event” defined in current OAE does not assume causal relation between an adverse event outcome and a medical intervention. The causal relation is now defined in an OAE term “causal adverse event”. One major task of the OAE research is to identify or predict the causal association based on various datasets including clinical adverse event reports.
Gleaning and cleaning real-world clinical data with this approach also introduces a novel hypothesis generator tool to aid translational informatics as the results are supported by statistical evaluation and validation of the findings. The method is designed to be discovery-driven rather than the traditional research hypothesis-driven approach. Two possible hypotheses that are derived from this post-vaccination adverse event investigation could be: (1) Hypothesis 1: TIV induces the occurrence of GBS that may be explained by the trigger in behavioral & neurological processes due to their possible shared gene interaction networks, and (2) Hypothesis 2: LAIV is more likely to trigger respiratory inflammatory response than TIV due to its mode of administration.
Conclusions from this study speak to the interest of personalized medicine of individuals. As a recent study by Liang et al. indicates the occurrence of GBS as below the background rate of severe adverse event induced by influenza vaccine 
, those observations were made on the whole population of influenza vaccine recipients including the majority who did not develop any major post-vaccination complication. Our study, in contrast, focuses on the sub-population of those whose cases have been submitted to VAERS as having a post-vaccination complication. This difference in the focused population group may lead to the hypothesis as to which molecular or genetic variation of the person can cause the occurrences of influenza vaccine-induced severe AEs.