This systematic review on electronic ADE detection revealed some key limitations in the current literature: (1) most studies could not properly assess rule accuracy because they did not utilize a gold standard or did not apply the gold standard to all patients; (2) the accuracy of detection rules varied widely because of inconsistent event definitions and methodologies used for derivation and validation; (3) extremely few studies considered the underlying ADE prevalence when choosing which rules to derive or use; and (4) the majority of rules did not detect specific ADE types.
Currently, there is no systematic approach to validating ADE detection methods. An appropriate comparative gold standard is not routinely used, and if used, differs across studies resulting in varying measures of accuracy. Of the 24 studies in our systematic review that assessed rule accuracy, slightly less than half had appropriately used a gold standard and were thus able to measure rule sensitivity and specificity. The accuracy reported from these few studies was generally low but displayed a wide range, making it difficult to draw conclusions about the overall effectiveness of electronic ADE detection. The variability in evaluations we observed is likely, in part, caused by the expense of such efforts combined with the absence of funding. It can be challenging to obtain research funds to perform this work. As a result, most institutions will implement these rules with only partial evaluations, at best.
Difficulties making comparisons across studies are also compounded by non-standardized event definitions and detection methods.7
The included studies varied considerably in the information system(s) used and the rule criterion. We found that 40% of all studies only used a single information system. Single information systems restrict the number of criteria that can be used in electronic detection rules, which can reduce rule accuracy. System infrastructure which can support the linkage of multiple hospital information systems can positively influence the specificity of detection rules.2
Early electronic detection studies were limited by non-integrated information systems, and hence they derived less specific rules that were restricted to a single clinical criterion such as the prescription of an antidote or an abnormal laboratory result.13
In our systematic review, the detection rules in 58% of the studies were based on a single criterion. For example, rules such as ‘serum potassium less than 3.0 mmol/l or more than 6 mmol/l’13
or ‘patient receiving diphenhydramine’35
have high false positive rates because their occurrence is most commonly not related to an ADE. Whereas a rule such as ‘receiving ranitidine AND platelet count has fallen to <50% of previous value’ would be more discriminate identifying ranitidine-induced thrombocytopenia.4
Detection rules are only as good as the data they use and frequently ADE information is non-specific as a result of the incomplete documentation in the medical record. For example, if a surgical patient develops respiratory compromise in the hospital, he may have similar treatments and tests, irrespective of the cause. Whether the respiratory compromise is due to an excess post-operative intravenous rate or diastolic dysfunction, the patient will still likely undergo testing with a chest x-ray, cardiac enzymes, and an electrocardiogram and be treated with oxygen, diuretics, and morphine. Since most current electronic records do not record diagnoses but do capture tests and treatments, the two cases would look the same to an electronic ADE detection system even though in one case it is an ADE and the other it is not.
The same issue can lead to insensitive rules as well. Some ADEs may not have an associated laboratory test or treatment. In these cases, the adverse event will require documentation of an ADE-related diagnosis in the electronic record. Since these are sometimes not stored electronically, detection of such ADEs will be impossible.
We also examined how studies derived or utilized detection rules. Very few studies (10%) derived or used rules that were defined either by clinical need or the underlying ADE prevalence. A number of studies used or developed rules for particular ADEs only because they were possible or the required source data systems were available. These studies did not necessarily detect the ADEs that they should be detecting and cannot help us identify what the most prevalent and important ADEs are. Further, few studies (17%) derived or used electronic rules that detected specific ADE types. Attempting to identify all ADEs is an extremely complex task and greater success may be achieved by focusing on key ADEs.
In addition to the issues with the current literature described above, it was noted that there are few studies from hospitals outside of the USA. Thirty-one (65%) of the 48 studies in our systematic review were American. There is a need to study electronic triggers using patients in other countries, including Canada, for two reasons. First, other countries have different healthcare systems (eg, Canada has universal healthcare), so hospital utilization patterns and in-hospital medication use may be different. This could lead to different patterns of ADEs and consequently differences in how they are captured using electronic means. In addition, structural differences exist in information systems. For example, discharge abstracts in Canada are currently based on ICD-10, whereas in the USA they are based on ICD-9. Furthermore, hospital discharge abstracts from Canadian hospitals indicate whether or not recorded diagnoses are hospital complications. These differences in coding diagnoses affect the type of rule criteria that can be used in each country to detect ADEs.
This review characterized existing reports on the use of electronic detection of ADEs and assessed their accuracy. The main limitations of this study are that it is purely descriptive and the studies are very heterogeneous. In addition, our search strategy may not have located all relevant studies as a result of the exclusions applied. Nonetheless, our review did highlight some important methodological limitations in the existing published studies of electronic ADE detection. We identified three main limitations: (1) most studies did not properly assess rule accuracy; (2) extremely few studies considered the underlying ADE prevalence when choosing which rules to derive or use; and (3) the majority of rules did not detect specific ADE types. These limitations need to be addressed to realize the full potential of electronic detection of ADEs. Future research should also focus on the identification of rule characteristics which predict benefit. It is notable that we are unable to make recommendations on which information systems are most likely to be beneficial in the development of electronic alerts. This is a result of the relative lack of well performed studies and the poor performance of most rules. In the development of new alerts, investigators and system developers should pay more attention to the relative prevalence of specific ADEs and focus on those which are most common and serious.
We suggest a more rigorous approach to rule development and reporting. As a basis for future advances, there is a need for the industry to develop and adopt universal standards in ADE classification. There is existing work in this area, however, international consensus would be helpful. Even in the absence of this there are some general guidelines which should be followed for future publications reporting on the accuracy of electronic ADE rules. First, they need to identify the motivation for rule development, including whether it is informed by the prevalence and severity of the underlying ADE in a specific patient population, and whether the rule is to be used for detecting ADEs for quality control or alerting providers for modifying clinical care. Second, any report on the rule needs to explicitly specify the components comprising the ADE, that is, the medication, the population at risk, and the outcome. Third, the report needs to link the clinical concepts defined by the ADE to data definitions. Fourth, investigators need to adhere to appropriate epidemiological techniques for reporting rule test characteristics. Adherence to these guidelines will facilitate progress in this field as they will improve the generalizability and reproducibility of the published work.