|Home | About | Journals | Submit | Contact Us | Français|
Adverse drug events (ADE) cause considerable harm to patients, and consequently their detection is critical for patient safety. The US Food and Drug Administration maintains an adverse event reporting system (AERS) to facilitate the detection of ADE in drugs. Various data mining approaches have been developed that use AERS to detect signals identifying associations between drugs and ADE. The signals must then be monitored further by domain experts, which is a time-consuming task.
To develop a new methodology that combines existing data mining algorithms with chemical information by analysis of molecular fingerprints to enhance initial ADE signals generated from AERS, and to provide a decision support mechanism to facilitate the identification of novel adverse events.
The method achieved a significant improvement in precision in identifying known ADE, and a more than twofold signal enhancement when applied to the ADE rhabdomyolysis. The simplicity of the method assists in highlighting the etiology of the ADE by identifying structurally similar drugs. A set of drugs with strong evidence from both AERS and molecular fingerprint-based modeling is constructed for further analysis.
The results demonstrate that the proposed methodology could be used as a pharmacovigilance decision support tool to facilitate ADE detection.
The US Food and Drug Administration's (FDA) adverse event reporting system (AERS)1 is currently used in the USA as the main source for drug safety surveillance. The FDA receives reports of suspected adverse drug events (ADE),2 which are then entered into the AERS database and used to discover new ADE. Despite its relative success, AERS has some limitations: the system captures only approximately 10% of events,3 the number of patients taking any drug is uncertain, and therefore the number of patients actually at risk is unknown. In addition, there are data quality issues preventing its effective use.4 The limitations imply that many serious ADE go undetected. It is now widely recognized that only through the integration of several information sources can a more effective surveillance system be achieved.5 6
Medicinal chemistry has exploited the idea that similar molecules from the point of view of chemical structure have similar biological properties.7–9 Molecular fingerprints can be used to evaluate molecular similarity and identify molecules structurally similar to those with a certain property.10 From the ADE perspective, computer-aided methods based on molecular structures are useful for pharmaceutical companies working in drug design to predict toxicological effects leading to the selection of new candidates without possible undesirable effects.11 However, computer-aided structural studies intended to predict preclinical ADE have important limitations due to the complexity of modeling poly-pharmacology systems with multiple mechanisms of action.11 Nevertheless, the chemical aspect for ADE discovery provided by structure–activity relationship (SAR) models could be very useful in combination with different types of sources, such as AERS, in the detection of post-marketed ADE. The potential of SAR models to detect post-marketed ADE was studied by the FDA (informatics and computational safety analysis),12 which used AERS as a basis to construct SAR models. We hypothesize that computer-aided structural studies, and in particular molecular fingerprint-based modeling (MFBM), can be used to analyze novel ADE for which evidential support is not yet conclusive, to enrich subsets of suspected drugs generated by other sources likely to be interesting for further study, and to explain the molecular mechanisms responsible for ADE.
In this article, we present a new ADE discovery paradigm that uses chemical information via MFBM to enhance initial ADE signals generated from another source, such as AERS, and provide a decision support mechanism to facilitate the identification of novel ADE. The basic idea is to assess the likelihood of an ADE for a candidate drug based on its structural similarity to a set of drugs known to cause the event in which the candidate drug has been determined by another source. Our study is applied to the identification of drugs reported in AERS that were found to be associated with the serious ADE rhabdomyolysis, an important ADE characterized by skeletal muscle breakdown and the release of intracellular constituents into the bloodstream. Different studies have been published analyzing adverse events from the point of view of AERS analysis or computer-aided structural studies.11 13 14 However, to the best of our knowledge, no publication has combined the two methods. In this paper we describe the combination of AERS analysis and MFBM, and show that it reduces the number of false positives and yields improved results to identify adverse events in drugs better than either method by itself.
The study of the relationships between molecular structure and biological activity has acquired important relevance in medicinal chemistry.15 The basic method of developing quantitative molecular similarity models based on molecular fingerprints consists of three steps.
In the first step the structures of a set of molecules must be entered into the computer and prepared for analysis. It is very important to collect a representative dataset. Once the collection is completed, a set of cleaning rules should be applied to ensure that the molecular structures are in a form suitable for the next modeling step (see the Methods section for more details).16
The second step is the calculation of the molecular fingerprints associated with the molecules.17 Although there are different types of fingerprints, the essential idea is to represent molecules through a bit vector that codifies the existence of particular structural features or functional groups. As an example, using BIT_MACCS fingerprints, some substructures encountered in the molecule CH3-CH2-O-C(O)-NH2 are bit 23-NC(O)O group, bit 84-NH2 (amine group), bit 114-CH3-CH2-A (ethyl group), bit 123-OCO (ester group).16 18
The final step in the process is the calculation of the structural similarity of the collected molecules on the basis of their molecular fingerprints. There are different measurements to compare the similarity between molecular fingerprints. The Tanimoto coefficient (TC), which is one of the most widely applied, can span values between 0 and 1, where 0 means ‘maximum dissimilarity’ and 1 means ‘maximum similarity’.17 The TC between two molecular fingerprints A and B is defined as:
where NA is the number of structural features present in fingerprint A (eg, number of bits with a value 1), NB is the number of features present in fingerprint B and NAB is the number of features present in common to both fingerprints A and B.
The DrugBank database combines detailed chemical, biological and pharmacological drug information with protein target data. The database contains approved drugs as well as biotech, nutraceuticals and experimental drugs.19 Importantly for our study, it includes a field that specifies the structures (smile code) of each molecule.
There are different spontaneous reporting system databases that have been created to provide postmarketing drug safety information, such as AERS,1 the European Medicines Agency20 or the WHO international database.21 Pharmacoepidemiology databases, such as the general practice research databases in the UK,22 or new initiatives such as the observational medical outcomes partnership23 or the sentinel project5 also provide important information for detecting new ADE and improving drug safety. Researchers have also started to analyze clinical data in electronic health records as new sources of data to detect adverse events.24 In addition, multiple approaches have been developed to detect adverse event signals in these databases, most based on disproportionality measures, such as the relative reporting ratio (RRR) or reporting odds ratio (ROR).24 A more complex method is the gamma Poisson shrinker (GPS) algorithm, which is used in this study.14 25
The GPS algorithm was developed by DuMouchel,25 and is the signal generation algorithm endorsed and used by the FDA to generate ADE signals in AERS.26 GPS is based on a Bayesian modeling framework that attempts to account for the uncertainty in the RRR disproportionality measure (a ratio of the observed prevalence of a drug–event pair to its expected prevalence under the assumption of independence) associated with small counts, by ‘shrinking’ RRR towards the baseline case of no association (value of one), by an amount that is proportional to the variability of the RR statistic. The result of this shrinkage is a reduction of spurious associations when there are not enough data to support it. ADE signals are qualified by GPS using a measure called empirical Bayes geometric mean (EBGM), which is essentially an estimate of the posterior expected value of RRR.14
An overview of the proposed method is depicted in figure 1. Initially, a training dataset composed of 196 drugs reported to cause rhabdomyolysis was compiled using several reliable resources. This dataset was subjected to a series of preparation steps and then used to calculate molecular fingerprints, which are binary vectors that codify the presence/non-presence of different structural features in the molecule.16 Two different sets of candidate drugs (1162 approved DrugBank drugs and 59 drugs selected from AERS by the GPS algorithm) were subjected to the same process as the training dataset. The TC was then used to compare the similarity of the fingerprints between the candidate drugs and the training dataset drugs. A TC cut-off value of 0.80 or more signifies that the molecules are highly similar.
As mentioned above, a training dataset of 196 molecules known to be associated with the ADE rhabdomyolysis was gathered from the literature (hereafter ‘rhabdomyolysis training dataset’) and used to construct the MFBM. The list of the compounds along with corresponding references are given in supplementary table S1, available online only. The molecules were collected from different review articles, individual case reports and reliable websites, such as http://drugs.com. The molecular operating environment (MOE) software was used to represent the structures of all the molecules in this study,16 which were subsequently preprocessed with the module wash in the MOE software. A set of cleaning rules in the module wash were used. More specifically, group I metals in simple salts were disconnected from each molecule so that only the active ingredient was retained (ie, the largest molecular fragment). Also, when using the module wash, the protonation state was considered neutral (the least charge-bearing form of the molecule) and explicit hydrogens were added.
Three evaluation studies were performed. The objective of the studies was to measure performance of the MFBM by itself, of the AERS signal detection GPS method by itself and of the combined method. The precision of the three methods was calculated as the ratio of the true positives versus all the positive cases defined as true positive and false positive. In binary classification, precision is a measure of the exactness in defining the proportion of true positives within all the positive cases selected by the model. Other metrics provided in the study are sensitivity and specificity. Sensitivity measures the proportion of actual positives that are correctly classified by the model and specificity measures the proportion of actual negative cases correctly identified.
The first study measured the performance of the MFBM for detecting drugs that may cause rhabdomyolysis by using the 1162 approved drugs in the DrugBank database. The DrugBank database version 2.5 was downloaded19 and molecular fingerprints for all the drugs in DrugBank were computed using the MOE software, in the same fashion described in the previous section. Proteins were not taken into account in this study. As described in the previous section, molecular fingerprints between drugs in DrugBank and drugs in the rhabdomyolysis training dataset were compared using the TC. A matrix file containing the similarity between drugs based on TC was calculated using the fingerprint cluster module and the sim_matrix2txt.svl script in MOE.16 The similarity score of the drugs being evaluated by the model is defined as the maximum pairwise TC obtained for the drug against each drug in the rhabdomyolysis training dataset. The set of drugs with a high TC value (TC ≥0.85) were analyzed in detail, and the literature was searched to see if there were any rhabdomyolysis-related reports for those drugs. The results were also compared with a subset of drugs, which was the same size as the set of highly similar drugs, and was selected randomly from the DrugBank in order to compare the highly similar set with the random set.
The second evaluation study measured the performance of signal detection using the GPS algorithm by itself as applied to the AERS database. To generate initial rhabdomyolysis ADE signals, 3 years of AERS reports were processed, corresponding to reports from the years 2007 to 2009 and amounting to 1 310 334 individual reports. Demographic information, necessary for removing duplicate reports, was extracted from AERS and linked to each report. Duplicate reports were identified and removed using several heuristics described in a related earlier study that involved searching for reports with an exact match of drugs, adverse events and demographic data.13 Of the remaining reports only those that were reported by healthcare providers were selected. In addition, the study was limited only to drugs reported as a ‘primary’ or ‘secondary’ suspect. To reduce drug naming redundancy and strengthen the signals, drug names were automatically mapped to UMLS codes using a natural language system MedLEE,27 and the corresponding generic names were then obtained using RXNORM.28 As an example, the drug Avandia, which is a brand name, was mapped to the UMLS code C0875967 by MedLEE, and then mapped to the generic rosiglitazone (UMLS code C0289313) using RXNORM. A more detailed mapping process has previously been described.13 29 The overall preprocessing step resulted in a reduced set of 431 430 individual reports, including 14 737 unique drugs and 12 017 unique events.
After preprocessing the AERS reports, ADE signals were generated using the GPS algorithm, which was implemented exactly as specified in the original paper of DuMouchel,25 including stratification by age, gender and year, and using the recommended seeding parameters. Finally, rhabdomyolysis ADE signals were identified by selecting all signals having the MedDRA30 term ‘rhabdomyolysis’ specified as the event, and signals having an EBGM of 2 or greater, a suggested threshold,26 and having a prevalence of three or greater (ie, at least three reports associated with the signal). The GPS algorithm takes into account corrections for multiple hypothesis testing by shrinking the RRR to the baseline case of no association.
The evaluation was based on measuring the performance of the GPS algorithm applied to AERS to identify known ADE according to case reports in the literature and Micromedex.31
The drug-ADE signals generated using the GPS method in the second study were considered as candidate drugs, and were structurally compared with the drugs in the training dataset of the MFBM. Molecular fingerprints and TC calculations were computed for the AERS candidate drugs using the same methods as described in previous sections.
All the 1162 approved drugs from DrugBank were analyzed and the number of those drugs that were highly similar to drugs in the rhabdomyolysis training dataset (eg, the TC similarity measure was over 0.85) was 127. The TC similarity measures for this subset of drugs in the DrugBank database are provided in supplementary tables S2 and S3, available online only. When the literature was searched, articles indicating that these drugs could be the cause of rhabdomyolysis were found for 24 of the drugs (see table 1 and supplementary table S3, available online only). However, it is possible that some of the remaining 103 drugs could actually cause rhabdomyolysis, although currently there are no published reports concerning them. The results were compared with another subset of the same size, amounting to 127 different drugs selected randomly from DrugBank. Information to establish a relationship between them and the adverse effect rhabdomyolysis was also searched for in the literature, and only six of the 127 drugs were found to be the cause of rhabdomyolysis (see supplementary table S4, available online only). The estimated sensitivity of the model in the prediction of known ADE is 0.44 and the specificity is 0.91. The precision of the model is 0.19 (true positive/(true positive+false positive)). However, as some of the false positives could be true positives, the precision may actually be higher (the receiver operating characteristic (ROC) curve is reported in supplementary figure S1, available online only).
An initial set of 115 drugs associated with rhabdomyolysis was identified using the GPS algorithm on the AERS reports. Out of these 115 drugs, 56 were described in the literature to cause rhabdomyolysis and were also part of the set of 196 drugs used to construct our MFBM. Among the remaining 59 drugs, a bibliographic search based on Micromedex31 and case reports revealed that 13 of the drugs can cause rhabdomyolysis as an ADE (see table 2). No information relating to the other drugs as the cause of rhabdomyolysis was available (see supplementary table S5, available online only). The overall precision of AERS in this study, considering all the 115 drugs is 0.60 (true positive/(true positive+false positive)). If we take into account only the 59 drugs that were not included in the rhabdomyolysis training dataset, the precision is 0.22 (see table 1). Although AERS showed excellent results to detect drugs implicated in rhabdomyolysis, it is still possible to improve ADE detection by combining AERS with MFBM as we explain in the next section.
Out of the 115 drugs selected in AERS, 56 were already included in the rhabdomyolysis training dataset and thus were not used for the evaluation. Of the other remaining 59 drugs, 13 drugs were confirmed to cause rhabdomyolysis as an ADE, and no information was found for 46 of the other drugs reporting them as the cause of rhabdomyolysis, which were therefore considered negative cases. Based on the TC, the MFBM highlighted nine of the confirmed 13 drugs as structurally similar to a drug rhabdomyolysis training dataset, and therefore potentially a rhabdomyolysis-related ADE. A p value for the probability that the method identified these nine drugs by chance was 0.004, ie, highly unlikely. In addition, a random method would have selected only approximately four confirmed drugs, whereas the MFBM selected nine, thus achieving a greater than twofold AERS signal enrichment. The results showed that the application of the MFBM substantially reduces the number of false positives identified using AERS alone (see table 1). The ratio true positives/false positives for the set of drugs with TC of 0.80 or greater and selected by the combined model is higher than the ratio provided by the methods used individually (see table 1). Figure 2 shows the resulting ROC curve for the classification of 59 AERS test signals as true/false ADE using the MFBM. The area under the ROC curve is 0.7 (1 indicates perfect classifier and 0.5 indicates random classifier). Using a cut-off value of 0.80 for the TC, the sensitivity obtained was 0.69 when the specificity was 0.76, with an F-measure of 0.72. True positives and false positives are shown in table 1. Comparison of similarity distributions of the set of candidate rhabdomyolysis ADE signals and a random set of drugs related to other ADE showed a statistically significant difference (a Kolmogorov–Smirnov test p value of 0.006), indicating that MFBM detects a different structural behavior in both sets of drugs (see figure 3). Table 2 shows the TC for these 13 drugs as well as the most similar drug in the rhabdomyolysis training dataset.
We also performed a qualitative analysis of the predictions when using the combined method, and found interesting examples.
Several examples show that the combination of the AERS GPS analysis and MFBM can detect drugs that belong to different pharmacological classes but are still structurally related. An example of a drug found by our methodology is tramadol, which is an analgesic (pain reliever) used in treating moderate to severe pain. Based on AERS reports between 2007 and 2009, tramadol (brand name Ixprim) was determined to be a candidate for the ADE rhabdomyolysis with an EBGM measure of association25 of 4.4. According to our MFBM, tramadol was found to be similar to the drug venlafaxine with a TC of 0.93 (see tables 2 and and3).3). Venlafaxine is an antidepressant confirmed to cause rhabdomyolysis as a rare ADE.31 32 Different case reports were found in the literature35 36 to confirm these findings, indicating that tramadol may induce rhabdomyolysis as a rare ADE, through several mechanisms, including prolonged immobilization (due to central nervous system depression) and neuromuscular excitability (as serotonin syndrome). In Micromedex,31 an increased creatine kinase level is an adverse event associated with tramadol, which could be a clear indication of the potential for rhabdomyolysis.
Another interesting drug able to induce rhabdomyolysis selected by the model is gabapentin, a GABA analogue used to relieve pain, especially neuropathic pain and for the treatment of depressive disorders. It is structurally similar to aminocaproic acid (one of the 196 drugs in our rhabdomyolysis training dataset), a fibrinolytic inhibitor analog of the amino acid lysine used to treat excessive bleeding (see tables 2 and and33).
Another interesting case for further analysis is fusidic acid, described in the literature as an agent that can interact with statins and produce rhabdomyolysis.51 52 No rhabdomyolysis causal reports were found in which fusidic acid was administered to the patients as the only drug. However, its structure with a tetracyclic ring system is similar to carbenoxolone, a known drug able to produce hypokalemia and rhabdomyolysis (see table 3 and supplementary table S5, available online only).
Lower values for the TC can also be considered. As the TC value decreases, the possibility of finding different classes increases, which is more interesting than finding similar drugs in the same class, but also the dissimilarity and the risk of incorrect predictions is higher. Considering TC greater than 0.75 some cases of misoprostol similar to carbenoxolone and ranolazine similar to aripiprazole can be found. Although some reports were found in the literature, causality cannot be established for the adverse event (see supplementary information, available online only).
Many but not all of the predictions combining AERS GPS analysis and MFBM correspond to drugs belonging to the same pharmacological class, which can still be very useful for researchers without a strong background in pharmacology. Tiapride, a selective dopamine-2 receptor antagonist used as an antipsychotic and also as a treatment for movement disorders related to dopamine hyperactivity, is a suspected drug according to FDA reports. The MFBM shows that it is similar to amisulpride (see tables 2 and and3).3). Both drugs are implicated in rhabdomyolysis due to malignant neuroleptic syndrome.31 44 53 The system is thus capable of predicting the possible mechanism of action of the drug causing the adverse event.
The model can also detect the different enantiomers (stereoisomers that are mirror images of each other with the same bidimensional formula but with different three-dimensional orientations of their atoms) that compose a racemic mixture of a chiral drug (with asymmetric carbon atoms). This is the case of the pump inhibitors omeprazole and esomeprazole (see table 2), the racemic mixture and the S-enantiomer, both are captured by the model as being similar to pantoprazole (one of the 196 drugs in our rhabdomyolysis training dataset) and described in the literature as drugs able to induce rhabdomyolysis.31 A possible explanation for the adverse effect is the inhibition of potassium hydrogen ATPase, responsible for intracellular pH equilibrium, and leading to acidic intracellular conditions that degrade the cells.41
Another example is telbivudine, which is a nucleoside analog used to treat hepatitis B infection. Its structure is similar to zidovudine, the first antiretroviral drug approved for HIV therapy (see tables 2 and and3).3). There are different possible mechanisms by which these drugs could cause muscle injuries, such as mitochondrial DNA depletion, mitochondrial dysfunction, reduced levels of l-carnitine, or apoptosis.54 There are also different drugs, such as the benzodiazepines oxazepam, lormetazepam and alprazolam, with a strong signal according to AERS and MFBM, but no reports were found in the literature reporting these drugs as the cause of the adverse event (see supplementary table S5, available online only). However, it is possible that some of these drugs could actually cause rhabdomyolysis, although there are not yet any publications concerning them. Fosamprenavir, similar to darunavir (a drug in our rhabdomyolysis training dataset), could be another interesting case for further analysis.
The main goal of the study was to evaluate an approach that combines ADE evidence from two different sources: the FDA's spontaneous reporting system (AERS) and MFBM, and to demonstrate that the two sources combined together improve the precision of ADE detection. In the first study, the performance of the MFBM is assessed in an independent dataset (DrugBank). In the second study the GPS algorithm is applied to AERS. We showed that the GPS algorithm is an excellent method to generate sets of drugs highly associated with rhabdomyolysis. However, within the sets of drugs generated by GPS, the score provided by the algorithm includes many false positives, because there are confounding drugs with good scores that are not causally related to the ADE, and therefore that set could be improved by filtering out some of the confounders. The application of the fingerprint model rationalizes the AERS signals selected with the GPS score and provides a smaller set of candidates with better enrichment factors. Although the fingerprint model performs similarly in an independent dataset, the fingerprint model applied to the GPS candidates offers a better true positive/false positive ratio due to the higher concentration of rhabdomyolysis drugs provided by the GPS algorithm. The combined method cannot increase the number of true positives but considerably reduces the number of false positives detected by AERS analysis. When we ignored drugs already in our rhabdomyolysis training dataset, we found a twofold precision improvement. The simplicity of the model assists in highlighting the etiology of the ADE by identifying structurally similar drugs, for which information is available and can be used to help understand possible causes, such as the mechanism of action. Although we used the AERS data in this study to test the method for the ADE rhabdomyolysis, the method could also apply to the use of electronic health records as an initial source of ADE detection as well as to ADEs other than rhabdomyolysis.
This system is not designed to replace the existing pharmacovigilance methods used to evaluate the importance of the signals, ie, study of the potential relevance of the signal or the biological plausibility, but to enhance the existing methods, providing additional information to make decisions. The simplicity of the model allows the researcher to detect the drug in the training dataset for a given ADE that is most similar to the AERS ADE drug candidate, which is useful for examining the reports and the available information to decide the importance of the signal.
When the MFBM was applied to a large set of drugs in the DrugBank database19 (see supplementary tables S2–S4, available online only), the results were similar to those previously reported.11 The predictability of fingerprint-based models by themselves is limited due to the complexity associated with the modeling of complex human clinical adverse events. The performance of our fingerprint-based model was probably affected by high molecular variability, and by the large number of potential targets and biological mechanisms associated with the clinical adverse event rhabdomyolysis. Nevertheless, the model is very useful for generating sets of drugs with good enrichment factors.
The performance of the model is highly dependent on the suitable construction of a training dataset for a given ADE, requiring a heterogeneity representation of the different structural classes of drugs highly related to the adverse event. In addition, training datasets can be updated continuously in order to enhance performance whenever a new compound is identified as causing an event. Although BIT_MACCS fingerprints have been shown to be successful in representing molecular structure,55 56 alternative methods could be explored that use different structural representations for molecules. Different types of models that use complex data analysis methods, such as neural networks or support vector machines, could be developed, but these methodologies would increase the complexity of the process.
When evaluating performance, we considered drugs that were not yet known to cause rhabdomyolysis as false positives. However, it is possible that some of these drugs actually do cause rhabdomyolysis but that the association has not yet been discovered. Therefore, it is possible that the true false positive rate of this method is lower than we determined.
This study demonstrates the importance and the usefulness of the proposed method, which incorporates molecular structural information with data mining methods based on AERS to improve the initial signals generated from AERS. The proposed method led to the efficient prioritization of the ADE candidates selected from AERS so that the data could be monitored further to evaluate the importance of the generated signal. A set of drugs with strong evidence from both AERS and MFBM has also been constructed for further analysis. The results we obtained confirm that the molecular fingerprint-based structural similarity model could be used as a powerful pharmacovigilance decision support tool to facilitate early ADE detection.
The authors would like to thank Krystl Haerian, MD, for her help with the rhabdomyolysis dataset construction.
Funding: This work was supported by grants R01 LM010016 (CF), R01 LM010016-0S1 (CF), R01 LM010016-0S2 (CF), R01 LM008635 (CF) and 1R01LM010140-01 (RR), from the National Library of Medicine and by the Intramural Research Program of the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) of the National Institutes of Health.
Competing interests: None.
Provenance and peer review: Not commissioned; externally peer reviewed.