PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of bmcmidmBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Medical Informatics and Decision Making
 
BMC Med Inform Decis Mak. 2013; 13: 43.
Published online Apr 8, 2013. doi:  10.1186/1472-6947-13-43
PMCID: PMC3629995
“Reducing unnecessary testing in a CPOE system through implementation of a targeted CDS intervention”
Donald L Levick,corresponding author1 Glenn Stern,2 Chad D Meyerhoefer,3 Aaron Levick,4 and David Pucklavage5
1Office of Chief Medical Information Officer, Lehigh Valley Health Network, Cedar Crest and I-78, PO Box 689, Allentown, PA, 18105, USA
2Chasm Crossing Consulting, 1470 Limeport Pike, Coopersburg, PA, USA
3Rauch Business Center, 621 Taylor Street, Bethlehem, PA, 18015, USA
4Department of Physics, Cornell University, 109 Clark Hall, Ithaca, NY, 14853, USA
5Information Services, Lehigh Valley Health Network, Cedar Crest and I-78, PO Box 689, Allentown, PA, 18105, USA
corresponding authorCorresponding author.
Donald L Levick: donald.levick/at/lvh.com; Glenn Stern: gstern5000/at/gmail.com; Chad D Meyerhoefer: chm308/at/lehigh.edu; Aaron Levick: aml272/at/cornell.edu; David Pucklavage: david.pucklavage/at/lvhn.org
Received June 9, 2012; Accepted March 13, 2013.
Background
We describe and evaluate the development and use of a Clinical Decision Support (CDS) intervention; an alert, in response to an identified medical error of overuse of a diagnostic laboratory test in a Computerized Physician Order Entry (CPOE) system. CPOE with embedded CDS has been shown to improve quality of care and reduce medical errors. CPOE can also improve resource utilization through more appropriate use of laboratory tests and diagnostic studies. Observational studies are necessary in order to understand how these technologies can be successfully employed by healthcare providers.
Methods
The error was identified by the Test Utilization Committee (TUC) in September, 2008 when they noticed critical care patients were being tested daily, and sometimes twice daily, for B-Type Natriuretic Peptide (BNP). Repeat and/or serial BNP testing is inappropriate for guiding the management of heart failure and may be clinically misleading. The CDS intervention consists of an expert rule that searches the system for a BNP lab value on the patient. If there is a value and the value is within the current hospital stay, an advisory is displayed to the ordering clinician. In order to isolate the impact of this intervention on unnecessary BNP testing we applied multiple regression analysis to the sample of 41,306 patient admissions with at least one BNP test at LVHN between January, 2008 and September, 2011.
Results
Our regression results suggest the CDS intervention reduced BNP orders by 21% relative to the mean. The financial impact of the rule was also significant. Multiplying by the direct supply cost of $28.04 per test, the intervention saved approximately $92,000 per year.
Conclusions
The use of alerts has great positive potential to improve care, but should be used judiciously and in the appropriate environment. While these savings may not be generalizable to other interventions, the experience at LVHN suggests that appropriately designed and carefully implemented CDS interventions can have a substantial impact on the efficiency of care provision.
Keywords: Clinical decision support, Computerized physician order entry, Medical error, BNP testing, Alert fatigue, Sociotechnical, Unintended consequences, Appropriate testing
Computerized Physician Order Entry (CPOE) and embedded Clinical Decision Support (CDS) have been shown to improve quality of care and reduce medical errors [1-3]. CPOE and CDS can also improve resource utilization through more appropriate use of laboratory tests and diagnostic studies. While a few Random-Controlled Trials (RCTs) have addressed CDS and CPOE efficacy [2,4,5], there are inherent limitations on the external validity of such studies. As a result, observational studies are also necessary in order to understand how these technologies can be successfully employed by healthcare providers. However, documentation of provider-initiated interventions is often limited, and as a result, more studies of implementation at the point of care are needed to understand and measure CDS and CPOE effectiveness [6,7].
We describe the development and use of a CDS intervention in a CPOE system; an alert, in response to an administratively identified medical error of overuse of a diagnostic laboratory test. We discuss the complexity of effective CPOE implementation and demonstrate the benefits achieved by this quality improvement initiative through a reduction in inappropriate testing and lower medical care costs [2,7].
The medical error motivating this intervention was identified by the Test Utilization Committee (TUC) at the Lehigh Valley Health Network (LVHN), in Allentown, PA on September, 2008. The TUC noticed critical care patients were being tested daily, and sometimes twice daily, for B-Type Natriuretic Peptide (BNP). BNP is secreted by the heart in response to changes in pressure that occur in patients with congestive heart failure (CHF). It is used as a marker to gauge the change in severity of CHF over time. Repeat and/or serial BNP testing is inappropriate for guiding the management of heart failure and may in fact be clinically misleading [8]. Given that additional BNP tests during an acute exacerbation of CHF provide no relevant clinical information, it was initially unclear why so many tests were being ordered. This overuse did not likely result in adverse clinical events [9], but it did significantly increase treatment costs.
All CDS interventions in response to identified errors involve critical success factors such as a culture of acceptance of both technology and embedded ‘cognitive forcing strategies” (rules and alerts, standardized order sets, etc.) [10]. This report describes how one healthcare system designed a targeted CDS intervention to reduce unnecessary testing (and associated costs) in a CPOE system, and evaluates its effectiveness. This intervention was part of a larger initiative to create an “advanced” [11] CDS environment capacity at LVHN. Advanced CDS development moves beyond basic CPOE functions to respond to identified medical errors, evidence-based information changes, potential unintended consequences and continuous quality improvement (CQI) over time, benefiting from a supportive socio-technical infrastructure [11].
This study emerges from a growing body of literature that seeks to rigorously evaluate applications of Health IT in daily practice. A range of studies have focused on a sociotechnical approach in order to evaluate the multiple factors that contribute to effectiveness of these new tools [12-14]. Factors that have contributed the use and ability to evaluate decision support technologies include greater emphasis by providers on knowledge-management, and an increasing quantity of patient-specific data from genetic testing and electronic health records [15]. The enhanced ability of providers to analyze clinical practice patterns has also contributed to studies seeking to define what constitutes “appropriate testing” [16].
Among decision support tools, alerts and rules have received a great deal of attention in the literature due to the significant potential to improve care. CPOE systems have been shown to reduce medical errors and overall medical costs [1-3,17]. Several other studies and recent review articles [2,3] generally support the potential of CPOE and CDS to improve care, but note that most of the evidence related to effectiveness of these tools is from academic hospitals rather than community hospitals and or primary care facilities. Alerts and rules have also received a great deal of attention in the literature due to identified barriers to their acceptance and implementation [14,18-21], and unintended consequences [14,22-25]. For example, this CDS intervention, and the development of the CPOE system in general, was designed to minimize “alert fatigue” that may occur when clinicians are overwhelmed by the number of alerts [21].
CPOE development and implementation
Lehigh Valley Health Network (LVHN) is the largest health care provider in the Lehigh Valley region of Pennsylvania with: three not-for-profit hospitals; nine health centers in four counties; primary and specialty care practices throughout the region; pharmacy, imaging and lab services; and preferred provider services. In fiscal year 2012, LVHN had over 54,000 acute admissions and 173,000 emergency department visits. The primary service area of Lehigh, Northampton and Carbon counties has a combined population of more than 800,000, making it Pennsylvania’s third most populated region. Minorities comprise more than 54% of all Allentown residents (the region’s largest city), almost 43% of whom are Hispanic or Latino.
LVHN’s Information Technology (IT) Department was an early adopter of CPOE and CDS. The network has been named one of the 100 Most Wired and 25 Most Wireless hospitals by Hospitals and Health Networks magazine. IT applications at LVHN are implemented through a socio-technical infrastructure that facilitates input from providers and includes an evaluative component pre and post implementation. Examples of this infrastructure include the Test Utilization Council (TUC) that identified over-testing for BNP, and enterprise wide Clinical Decision Support Committee, safety committees, and technology assessment groups. These groups and committees maintain an ongoing sensitivity to work flow issues in the implementation of CDS, including just-in-time evidence-based decision support (clinical knowledge) [15]. The discussion below demonstrates an adaptation to overcome a socio-technical barrier to effective usage of this technology—intrusion into the physician work flow.
LVHN began implementation of CPOE in 2001; by 2006, all beds were converted and CPOE use was mandatory. Concurrent with the implementation of CPOE was the installation of a closed loop medication administration system including embedded clinical decision support, automated pharmacy robot dispensing, online medication administration documentation and bar code medication administration. At LVHN, the implementation of this closed loop medication administration system is estimated to have reduced potential harmful medication errors by 80%. a
The initial approach to CDS at LVHN was very deliberate. The implementation team was acutely aware of the impact that CPOE and CDS interventions had on physician workflow as well as their potential for generating alert fatigue. Consequently, interventions that would “add clicks” (such as alerts and rules) and standardized order sets were added judiciously during the early stages of the CPOE rollout—the “basic stage” [11]. As the physicians acclimated to the system, they became more receptive to an increasing number of CDS interventions. Moving physicians along this “continuum of intrusion” [26] was considered an important success factor by all stakeholders for this CDS intervention.
Identification of BNP over-testing as a medical error
In September 2008, the Senior Clinical Pathologist noticed critical care patients were being tested daily, and sometimes twice daily, for B-type natriuretic Peptide (BNP). The issue was brought to the TUC and prioritized as a high volume (and high cost) test requiring further investigation and intervention. Research revealed that repeat BNP testing is usually not indicated during a hospital stay [8]. The TUC recommended modifying LVHN’s existing CPOE system to employ an expert rule to alert the ordering clinician that a BNP had been performed during the current admission and that repeat testing would add no value to the clinical decision making process.
CDS alert development
The CDS alert was implemented through an expert rule that searches the system for a BNP lab value on the patient. If there is a value and the value is within the current hospital stay, an advisory is displayed to the ordering clinician. (See Figure 1 for a screenshot of the alert.) If there is no BNP lab value on the patient within the current episode of care, the rule evaluates pending orders for an active order for a BNP within the current episode. If an active order is found, a different advisory is displayed. If no lab value and no pending active order exist in the current episode of care, then no advisory is displayed. The alerts are advisory only and considered a “soft stop” because the clinician still has the opportunity to place the order after having been presented with relevant information. The CDS intervention for repeat BNP testing was implemented in June, 2009 and BNP ordering decreased by approximately 65% within six months of introduction of the alert. Comparing the three month period (January through March) before and after the intervention indicates that total BNP ordering on in-patient encounters decreased from 3,291 to 2,563, and the number of patients receiving two or more BNP tests during a single hospital stay decreased from 1,358 to 487 (p < .01). However, Figure 2 also reveals that there was a downward trend in BNP testing during the surrounding time period of the intervention. This trend could be the result of several factors, including behavioral change on the part of physicians or process changes caused by other aspects of the new medication administration system, as well as changes in the health and demographic characteristics of patients eligible for the BNP test.
Figure 1
Figure 1
Screenshot of alert providing recent BNP result information.
Figure 2
Figure 2
Number of monthly inpatient BNP tests, January 2008 – November, 2009.
Multivariate statistical analysis
In order to isolate the impact of the CPOE intervention on unnecessary BNP testing, we apply multiple regression analysis to the sample of 41,306 patient admissions with at least one BNP test at LVHN between January, 2008 and September, 2011. Although the average number of tests ordered was 1.8, the distribution of BNP orders is skewed to the right with a maximum value of 31 (see Table 1). Therefore, we model the number of BNP orders using a zero-truncated negative binomial (ZTNB) distribution to account for both the truncation of the dependent variable at 1, and the skewness of the distribution [27].
Table 1
Table 1
Descriptive statistics for analysis variables (N = 41,306)
We include a variety of controls in our models to account for changes over time in the characteristics of patients that may influence the propensity of physicians to order additional tests. These include indicator variables for patient age (age ≤ 40; 40 < age ≤ 60; 60 < age ≤ 80; age > 80), gender, race (white; black; Hispanic; other race), admission type (emergency; elective; urgent), and insurance type (private traditional plan; Medicare; Medicaid; self-pay; private managed care plan; Medicare managed care; Medicaid managed care). In addition, we include indicators for quartile of the inpatient cost risk distribution, where risk is based on patients’ Diagnostic Cost Group/Hierarchical Condition Category (DCG/HCC) risk scores. b
Our intervention indicator is equal to 1 for all patient admissions during or after June, 2009, the month that the CPOE alert was put and place, and is equal to 0 before that date. The mean, minimum, and maximum of the intervention indicator and all of the control variables are listed in Table 1.
We estimate three ZTNB regression models in order to test the sensitivity of our results to different specifications for capturing overall trends in BNP orders not due to the intervention during the sample period. These results are reported in Table 2, which contains the marginal effects of the control variables and the intervention indicator. The marginal effects are interpreted as the impact on the number of BNP orders corresponding to a one unit change in the given variable. The standard errors of the marginal effects in parenthesis are robust to heteroscedasticity.
Table 2
Table 2
Change in BNP orders associated with one unit change in selected variables from ZTNB regression
In the first specification in column 1 we include a single monthly time trend variable to capture aggregate trends in BNP orders net of the CDS intervention. The estimates from this specification suggest that the intervention reduced the number of BNP orders by 0.6 per inpatient admission, on average. However, when we include additional variables to more flexibility control for aggregate trends over time, this estimate is reduced by over 20%. In addition to a linear monthly time trend, the specification in column 2 also contains the square of the time trend, while the specification in column 3 includes separate indicator variables for each month of the sample. Among these models, we prefer the third and most flexible specification, which indicates that the intervention reduced the number of BNP tests by 0.4 per admission. This effect is precisely estimated, and represents a reduction in BNP orders of 21% relative to the mean.
The financial impact of the rule was also significant. In the three quarters prior to the implementation of the intervention there were 11,209 BNP tests ordered at LVHN. Based on our statistical model, we project that the intervention reduced BNP orders by 21%, or 2,454 tests over three quarters. Multiplying by the direct supply cost of $28.04 per test, this suggests the intervention saved approximately $92,000 per year. These savings are based on supply cost only, and do not account for the increased capacity for the laboratory to perform other tests. As a result, the full annual long run benefits of the intervention are greater than our estimate of supply cost savings, after the short term costs of implementing the intervention are recovered.
While there are few observational studies that evaluate the effectiveness of similar CDS interventions, Bates et al [3]. found using a randomized controlled trial that computerized reminders about redundant laboratory tests significantly reduced the final number of ordered tests. There is also a paucity of research on the impact of CDS and CPOE on costs. A comprehensive 2010 research review of the impact of CDS on test ordering and appropriate utilization in the UK was not able to identify any specific studies on reduced costs from interventions that reduced overuse of testing [4]. Nonetheless, the importance of studies that determine the impact of CDS and CPOE on cost is well recognized [2,4,7,11,28] A recent meta analysis of the relatively small number of studies currently available that do address the cost consequences of CPOE and CDS, found a significant amount of variability in the level of cost savings; ranging from $6,000 to $84,194 [29].
Much of the previous literature on CPOE addresses the unintended consequences that result from the implementation of new technologies, the most serious of which is e-iatrogenesis. Because there is no evidence that this over-testing error caused harm to patients, it is not an example of e-iatrogenesis. Furthermore, it is not clear whether over-testing for BNP was in fact an unintended consequence of CPOE. [22-25] In that case it may or may not be classified as a “new kind of error” [24] or an error made because of the existence of CPOE. LVHN front line staff concluded that the over-testing occurred because either the ordering physician did not understand the time-period of the reliability of the test, or the physician was unable to find previous orders/ results in the electronic record (or did not take the time to search the system). If the former hypothesis is accurate then the experience of LVHN demonstrates the benefit of CPOE in uncovering previously unrecognized instances of inefficient care, and the ability of complementary CDS technologies to increase the efficiency of care provision and lower costs. In contrast, if the latter hypothesis is correct, LVHN’s experience demonstrates how CDS can be used to correct an unintended consequence of CPOE .In both cases the CDS intervention led to a positive outcome, but the two underlying reasons for the over-use error have very different implications for the efficacy of CPOE. This highlights the need for future research studies that use a mix of qualitative and quantitative methods to fully characterize and evaluate the use of new technologies [11-13,15]. Irrespective of the source of the over-use error, LVHN’s method of identifying and correcting the error can be replicated by other health care networks with similar IT systems and socio-technical infrastructures.
In the specific case of CDS, the use of alerts has great potential to improve care, but should be used judiciously and in the appropriate environment [11,14,20,26,30]. The intervention implemented at LVHN to reduce unnecessary testing was effective because: 1) It alerted physicians that further BNP testing is potentially misleading; 2) It addressed an information failure associated with the EMR (previous episode of care results were not visible once results were posted) and; 3) The intervention was implemented and evaluated in the context of an advanced [11] CPOE system facilitating this cycle of implementation and evaluation. Although we cannot determine how each of these factors independently contributed to the success of the intervention, our statistical analysis indicates that they collectively resulted in the reduction of BNP testing by 21%; saving LVHN over $92,000 per year. While these savings may not be generalizable to other interventions, the experience at LVHN suggests that appropriately designed and carefully implemented CDS interventions can have a substantial impact on the efficiency of care provision.
Endnotes
aLehigh Valley Health Network, Internal Report, 2011
bDCG/HCC risk scores are derived from data on patient age, sex, and physician-reported diagnosis codes (ICD-9-CM). They have been validated as a proper measure of risk adjustment in the inpatient setting (Ash et al., 2003; Petersen, Pietz, Woodard & Byrne, 2005) and are also used by CMS to risk adjust Medicare payments to private insurers under Part C (Pope et al., 2004; Pope et al., 2000).
Competing interest
The author(s) declare that they have no competing interests.
Authors’ contributions
DL was involved in the development of the study, drafting and editing of the manuscript. DP participated in the development of the study and the initial draft of the document. AL performed the literature search and in the development of the manuscript. GS provided the theoretical frameworks and performed much of the editing of the manuscript. CM provided statistical analysis and contributed to the restructuring of the manuscript. All authors read and approved the final manuscript.
Pre-publication history
The pre-publication history for this paper can be accessed here:
Acknowledgements
Bala Carver, M.D., Department of Pathology and Laboratory Services, Lehigh Valley Health Network; for providing support from Laboratory Services and ideas for initial design. Geoffrey Correll, BS, MT (ASCP). Department of Pathology and Laboratory Services, Lehigh Valley Health Network; for providing the data regarding test utilization. Lizhong Peng and Jeffrey McFetridge, Lehigh University, for valuable research assistance.
  • Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. J Am Med Inform Assoc. 1998;13:1311–316. doi: 10.1001/jama.280.15.1311. [PubMed] [Cross Ref]
  • Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Loback D. Effect of Clinical Decision-Support Systems: A systematic Review. Ann Intern Med. 2012. [Epub ahead of print] [PubMed]
  • Bates DW, Kuperman GJ, Rittenburg E, Teich JM, Fiskio J, Ma’luf N, Onderdonk A, Wybenga D, Winkelman J, Brennan TA, Komaroff AL, Tanasijevic M. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999;13:144–150. doi: 10.1016/S0002-9343(98)00410-0. [PubMed] [Cross Ref]
  • Main C, Moxham T, Wyatt JC, Kay J, Anderson R, Stein K. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effsets and cost-effectiveness of systems. Helth Technol Assess. 2010;13(48):1–227. [PubMed]
  • Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, Jacques A, Dawes M, Abrahamowicz M, Perreault R, Winslade N, Poissant L, Pinsonneault A. A Randomized Trial of the Effectiveness of On-demand versus Computer-triggered Drug Decision Support in Primary Care. J Am Med Inform Assoc. 2008;13:430–438. doi: 10.1197/jamia.M2606. [PMC free article] [PubMed] [Cross Ref]
  • Velasco F. Clinical Decision Support and Meaningful Use. Adv Health Inform Exec. 2010;13:15–20.
  • Lehmannm H, Ohno-Machado L. Evaluation of informatics systems: beyond RCTs and beyond the hospital. Am Heart J. 2006;13(5):828–34. doi: 10.1016/j.ahj.2006.08.021. [PMC free article] [PubMed] [Cross Ref]
  • Wu AH. Serial testing of B-type natriuretic peptide and NTpro-BNP for monitoring therapy of heart failure: the role of biologic variation in the interpretation of results. Am Heart J. 2006;13(5):828–34. doi: 10.1016/j.ahj.2006.08.021. [PubMed] [Cross Ref]
  • Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L. Relationship between medication errors and adverse drug events. Gen Int Med. 1995;13:199–205. doi: 10.1007/BF02600255. [PubMed] [Cross Ref]
  • Croskerry P. Cognitive Forcing Strategies in Clinical Decisionmaking. Ann Emerg Med. 2003;13(1):110–120. doi: 10.1067/mem.2003.22. [PubMed] [Cross Ref]
  • Kuperman GJ, Bobb A, Payne TH, Avery AJ, Ghandhi TK, Burns G, Classen DC, Bates DW. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;13(1):29–40. [PMC free article] [PubMed]
  • Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E, Idema R. Multimethod evaluation of information and communication technologies in health in the context of wicket problems and sociotechnical theory. J Am Med Inform Assoc. 2007;13(6):746–755. doi: 10.1197/jamia.M2462. [PMC free article] [PubMed] [Cross Ref]
  • Georgiou A, Westbrook JI, Braithwaite J. An empirically-derived approach for investigating Health Information Technology: the Elementally entangled Organisational Communication (EEOC) framework. BMC Med Inform Decis Mak. 2012;13:68. doi: 10.1186/1472-6947-12-68. [PMC free article] [PubMed] [Cross Ref]
  • Harrison MI, Koppel R, Bar-Lev S. Unintended Consequences of Information Technologies in Health Care – An Interactive Sociotechnical Analysis. JAMIA. 2007;13:542–549. [PMC free article] [PubMed]
  • Wang XS, Nayda L, Dettinger R. Infrastructure for a clinical-decision-intelligence system. IBM Syst J. 2007;13(1):151–169.
  • van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. JAMA. 1998;13(6):550–558. doi: 10.1001/jama.280.6.550. [PubMed] [Cross Ref]
  • Galanter WL, Didomenico RJ, Polikaitis A. A trial of automated decision support alerts for contraindicicated medications using computerized physician order entry. J Am Med Inform Assoc. 2005;13(3):269–274. doi: 10.1197/jamia.M1727. [PMC free article] [PubMed] [Cross Ref]
  • Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ Decisions to Override Computerized Drug Alerts in Primary Care. Arch Intern Med. 2003;13(21):2625–2631. doi: 10.1001/archinte.163.21.2625. [PubMed] [Cross Ref]
  • Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and Override Rates of Order Checks in a Practitioner Order Entry System. Proc AMIA Annu Fall Symp. 2002. pp. 602–606. [PMC free article] [PubMed]
  • Wright A, Sittig DF, Ash JS, Bates DW, Feblowitz J, Fraser G, Maviglia SM, McMullen C, Nichol WP, Pang JE, Starmer J, Middleton B. Governance for clinical decision support: case studies and recommended practices from leading institutions. JAMIA. 2011;13:187–194. [PMC free article] [PubMed]
  • Osheroff JA, Teich JM, Levick D, Saldana L, Velasco FT, Sittig DR, Rogers KM, Jenders RA. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. 2. HIMSS Publishing; 2012.
  • Ash JS, Sittig DF, Dykstra R, Campbel E, Guappone K. The Unintended Consequences of Computerized Provider Order Entry: Finding from a Mixed Methods Exploration. Int J Med Inform. 2009;13(Suppl 1):S69–S76. [PMC free article] [PubMed]
  • Ash JS, Sittig DF, Poon EG, Guappone D, Campbell E, Dykstra RH. The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry. JAMIA. 2007;13(4):415–423. [PMC free article] [PubMed]
  • Campbell EM, Sittig DF, Ash JS, Guappone DP, Dykstra RH. Types of Unintended Consequences Related to Computerized Provider Order Entry. JAMIA. 2006;13:547–556. [PMC free article] [PubMed]
  • Koppel R, Metlayy JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors. JAMA. 2005;13(10):1197–1203. doi: 10.1001/jama.293.10.1197. [PubMed] [Cross Ref]
  • Byers J. CMIO Summit: Levick says DCS must be clear for success. Health Imaging. in press.
  • Cameron A, Trivedi P. Microeconometrics: Methods and Applications. New York: Cambridge University Press; 2005.
  • Qaseem A, Alguire P, Dallas P, Feinberg L, Fitzgerald F, Horwitch C, Humphrey L, LeBlond R, Moyer D, Wiese J, Weinberger S. Appropriate Use of Screening and Diagnostic Tests to Foster High-Value, Cost-Conscious Care. Ann Intern Med. 2012;13(2):147–149. doi: 10.7326/0003-4819-156-2-201201170-00011. [PubMed] [Cross Ref]
  • Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux R, Samsa G, Hasselblad V, Williams JW, Wing L, Musty M, Kendrick AS. Enabling Health Care Decisionmaking Through Clinical Decision Support and Knowledge Management. Evidence Report No. 203. (Prepared by the Duke Evidence-based Practice Center under Contract No. 290-2007-10066-I.) AHRQ Publication No. 12-E001-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2012.
  • Schedlebauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, Avery AJ. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? JAMIA. 2009;13:531–538. [PMC free article] [PubMed]
Articles from BMC Medical Informatics and Decision Making are provided here courtesy of
BioMed Central