This study assessed the impact on physician management of syncope patients after implementation of a CDSS based upon the 2007 ACEP Clinical Policy on Syncope into an EDIS.
The admission rate for syncope patients was significantly lower in the post-intervention period compared to the pre-intervention period. The head CT imaging rate for syncope patients was not significantly different during the pre- and post-intervention periods. Since there was no significant interaction between intervention and age for admission rate (p
0.258) or for head CT rate (p
0.420), we can therefore conclude that age in conjunction with pre-/post-intervention does not affect the admission rate or ordering head CT scan rate in our cohort.
Subset analysis of physicians seeing more than ten patients in both the pre- and post-intervention periods showed similar changes and did not suggest cluster-associated phenomenon. Observed changes in admission rates in adult syncope patients may be indicative of improved awareness, adoption, and adherence to ACEP practice guidelines.
Subset analysis of physician behavior when the CDSS was completed versus not completed and visible versus not visible revealed significant differences in admission and CT head rates when the CDSS was visible and significant differences in admission when the CDSS was completed. These findings suggest that although this intervention was a passive one, the CDSS’s presence may have had significant effect on physician practice behavior. An alternative conclusion might be that physicians who are likely to ignore CDSS may also be less likely to adhere to evidence-based clinical practice guidelines.
It is well-established in the literature that even when physicians are aware of evidence, they may not adhere to it [3
]. Lehrmann et al. [13
] found that increased knowledge of the ACEP Clinical Policy on Hypertension following distribution of the guidelines did not translate into changes in physician practice. Cabana et al. [3
] identified knowledge, attitudes, and behavior as barriers to physician adherence to clinical practice guidelines. Kirkpatrick’s hierarchy of levels of evaluation proposes that “complexity of behavioral change increases as evaluation of intervention ascends the hierarchy” [21
]. As evaluation ascends from (1) reactions to (2) learning to (3) behavior to (4) results, the impact of the intervention strengthens from (1) learner satisfaction to (2) knowledge to (3) transfer of learning to the workplace to (4) impact on society, respectively. Since acquired knowledge of ACEP Clinical Policy on Hypertension did not translate into changes in physician practice in Lehrmann’s trial, we approached the problem of adopting evidence-based guidelines in clinical practice at the next level in the hierarchy of interventions, namely, transfer of learning to the workplace via evaluation of behavior.
While developing the CDSS, we focused on following the provisions for improved clinical practice outlined by Kawamoto et al. [16
] Namely, the CDSS was included in the clinician workflow in our computer-based EDIS. We also developed a CDSS that used recommendations rather than assessments. Instead of explicitly assessing for compliance to ACEP Clinical Policy recommendations, we sought a measurable change in physician behavior. Although our population demographics were similar to previously studied populations, this admission rate is considerably higher than previously studied populations [10
]. We chose head CT imaging as our other outcome because in the absence of focal neurologic findings, head CT imaging is of low yield in determining the etiology of syncope [23
]. We suspected that clinicians were ordering more cranial imaging in syncope patients than necessary.
To our knowledge this is the first study to demonstrate a significant change in physician behavior after implementation of a CDSS based upon ACEP Clinical Policy. While the body of medical research and literature grows rapidly, practice guidelines provide a means to educate, summarize, and distill evidence-based medicine to the practicing physician. However, implementation and utilization of the ACEP Clinical Policies has historically been a challenge. Given the increased adoption of robust EDISs, these results will encourage further experimentation and implementation of CDSSs based upon evidence-based clinical practice guidelines into EDISs.
The next step for research and development of such CDSSs could include: (1) assessment of our CDSS closer to the point of decision-making or in a different practice environment, (2) integration of other ACEP Clinical Policies into similar CDSSs in an effort to create an EDIS with comprehensive decision support, or (3) focus on more complex outcome measures such as compliance with guidelines or patient outcome.
We have identified several limitations to our study. First, instead of explicitly assessing for compliance to ACEP Clinical Policy recommendations, we assessed for a measurable change in physician behavior. Our EDIS could not track specific use of the CDSS. Consequently, admissions or head CTs could have increased in one type of patient and decreased in another, leaving the global rate unchanged.
Second, the EDIS in our institution did not have the capability to incorporate decision support at the precise time and location of decision-making in physician work flow. Specifically, our EDIS provides decision support in the documentation template for History of Present Illness (HPI). Therefore, the CDSS was more passive than active, meaning that CDSS use did not actually result in the action of ordering a head CT or admitting a patient.
Although our chart extraction selected patients with a diagnosis of or including the term “syncope,” the physician documenting on such a patient had the option to choose the patient’s HPI template independently of the patient’s diagnosis. Therefore, a patient diagnosed with syncope might not have the Syncope HPI template. In such a scenario, the physician would not encounter the decision support tool. Furthermore, as is common in unpredictable environments like the ED, HPI documentation may occur before or after the decision to obtain imaging or admit the patient. Since this study was completed at a teaching hospital, the physician documenting the HPI was not always the physician deciding the patient’s diagnosis, need for imaging, or admission.
Next, to prevent delaying or disrupting physician work flow, we decided the HPI documentation could not be made to require use of the CDSS, so the CDSS could be bypassed without reading or completing it. In order to minimize interruption of work flow, we abbreviated the language of the recommendations from the ACEP Clinical Policy on Syncope to fit within the format of the drop-down menus seen in Fig. . We assume that the practice guideline was not originally authored with the intention of implementation in an abbreviated form. Thus, it is possible that rewording the recommendations creates a simplified or modified recommendation for clinical decision-making. In a separate performance improvement project, however, we had markedly improved documentation of aspirin and beta-blocker use in chest pain patients using a CDSS with a single reminder phrase just above the “ENTER” button of a given HPI template. We found the structure and placement of the chest pain CDSS provided a quick and accessible—yet not prohibitive—reminder.
It is important to note that the 2007 version of ACEP Clinical Policy on Syncope was revised to include new recommendations using the findings from the derivation and validation of the San Francisco Syncope Rule [20
]. Two studies have had problems validating the San Francisco syncope rule [26
A final limitation of this study was that the CDSS was trialed at the practice site of the CDSS’s creators. Garg et al. [15
] found that “studies in which authors also created the CDSS reported better performance compared with those in which the trialists were independent of the CDSS development process.” We attempted to minimize this “Hawthorne effect” bias by having a third party announce the implementation of the CDSS without mention of the fact that we would be gathering data on physician behavior associated with the CDSS. Additionally, within our department there was no specific notification or implementation of new ACEP Clinical Policies.