|Home | About | Journals | Submit | Contact Us | Français|
Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research.
To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals.
Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE.
Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE.
We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites.
The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one’s perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.
Implementation of computerized provider order entry (CPOE), defined as a computer-based system that allows a physician or other ordering authority to directly enter medical orders, can be an effective mechanism for improving patient safety and practitioner performance. 1–14 There can be unintended consequences as well, however, 15–24 and it is likely that fear of adverse unintended consequences is impeding the diffusion of CPOE throughout hospitals in the U.S. 25–29
The unintended consequences of CPOE are especially intriguing because they can surprise implementers. Adverse unintended consequences unfortunately can become impediments to health care quality. The terms “unintended consequences” and “unanticipated consequences” are not synonymous. The “unintended” implies lack of purposeful action or causation, while the “unanticipated” means an inability to forecast what eventually occurred. Either kind of consequence can be adverse or beneficial. Unanticipated beneficial consequences are actually happy surprises. Unanticipated, unintended adverse consequences capture news headlines and are often what people imagine when they hear the term “unintended consequences.” An important goal of the current work is to identify the types of unintended consequences, so that they can be monitored and managed: once unintended consequences are predicted or detected, their management can knowingly involve tradeoffs. An example CPOE-related tradeoff is the degree to which the extra time physicians must exert to use CPOE (an undesirable consequence), is offset by the increased information physicians find available at the point of care via CPOE—making patient visits more effective. 30 The ability to maintain control over consequences may give hospital decision makers more confidence when making the determination to implement CPOE.
Recent descriptions in the literature of the unintended consequences of patient care information systems include a taxonomy of kinds of errors, 31 a list of kinds of errors related to CPOE, 18 a hierarchical model of unintended consequences of CPOE, 30 and a more general summary of the unintended consequences of patient care information systems. 19 In addition, the authors of the current study previously described in detail a typology of unintended adverse consequences of CPOE. 32 However, none of the previous studies provided a comprehensive, semi-quantitative description of the extent and importance of unintended consequences in hospitals across the nation. We therefore transformed our descriptions of types of unintended consequences into survey questions to discover how widespread and important the consequences are. We tested and, using short telephone interviews, administered the resulting instrument to representatives of 561 U.S. hospitals that have reported that they have installed CPOE systems, out of a total of 5,759 accredited hospitals. 33
The current study protocol for this three year-project, as proposed to the granting agency and approved by appropriate human subjects committees, included an extensive qualitative study to be followed by a national survey. In the initial qualitative study, we used semi-structured interviews and observations at five sites with experience in using CPOE to find out what types of unintended consequences exist. 32 Study sites for qualitative data gathering included: Wishard Memorial Hospital, a county hospital in Indianapolis which uses a system developed by the Regenstrief Institute; Brigham and Women’s Hospital in Boston, which developed its own system; Massachusetts General Hospital in Boston, which uses a modified version of that same system; The Faulkner Hospital in Jamaica Plain, MA, which uses a commercially available system; and Alamance Regional Medical Center in Burlington, NC, which also uses a commercial product. Institutional review boards at Oregon Health & Science University, Kaiser Permanente Northwest, and each of the individual study sites approved the qualitative study.
An experienced interviewer (JSA) conducted all of the taped interviews following an interview guide of questions. Thirty-two individuals, representing clinical, administrative, and technology roles, were formally interviewed. A multidisciplinary team of Ph.D.-level informatics researchers, physicians, a nurse, and a pharmacist conducted 390 hours of observation with a purposive sample of 142 individuals representing different clinical disciplines. The team performed 120 person-hours of observation at Wishard, 144 at the three Boston sites, and 126 at Alamance. For each visit, five to six researchers spent four days at each site in the period between August 2004 and April 2005. Observations took place in medicine, surgery, emergency, and intensive care areas in addition to the pharmacy, outpatient offices, and clinics.
Qualitative data analysis followed accepted principles for assuring trustworthy results. 34–38 All field notes and transcripts were entered into QSR N6, a qualitative analysis software program, and were individually coded by team members. The multidisciplinary team met 36 times to review transcripts and agree that what was seen or discussed was an unintended consequence and of what type.
Results of this preliminary study included identification of nine types of unintended consequences. Each of these categories can include desirable as well as undesirable consequences, depending on one’s point of view. However, our focus has been on those that are undesirable (adverse consequences), because institutions must manage them. The nine types are:
We developed an interview script, which included a short description of the purpose of the survey, five questions about the kind of CPOE system in place, and eight questions about possible unintended consequences. The five questions about the system were designed to collect descriptive information about the degree to which the system is infused into the institution. Infusion of technology is defined as the extent to which one uses an innovation in a complete and sophisticated manner, 40–41 and it occurs as information technology applications become more deeply embedded within the organization’s work systems. 42
Though we had previously discovered nine types of unintended adverse consequences, we decided not to ask interviewees about one of them, paper persistence, because as the role of paper has changed from a long-term storage medium to a temporary memory aid and disposable display device, its persistence should not now be considered adverse. The unintended consequences typology developed from the qualitative study was therefore transformed into eight survey questions, which were pilot tested (see ). The questions were designed to be as neutral as possible to avoid bias. We prefaced the questions by stating that the study concerned “what organizations have learned about CPOE.” We asked about surprises, “things that happened that you didn’t expect,” so the focus was on both unintended and unanticipated consequences; responses might then involve either desirable (beneficial) or undesirable (adverse) effects. The questions were asked first as yes/no queries to determine if the respondent thought the unintended consequence existed at that site (0 = No). If the answer was yes, the importance was determined on a 1 to 5-point Likert scale ranging from 1 for “not at all important” to 5 for “very important.” We received institutional review board approval from Oregon Health & Science University and Kaiser Permanente Northwest to conduct the survey.
We attempted to contact the entire population of U.S. hospitals using CPOE, instead of a sub-population from which we would make statistical inferences, because we wanted to gain insight into the nature of unintended consequences related to CPOE from organizations beyond those we visited. We selected acute care hospitals from the 2004 HIMSS Analytics™ Database that reported having CPOE in place (N= 448). Since that database did not include U.S. Veterans Affairs hospitals, which the authors believed to constitute important models of CPOE use, we also surveyed them (N= 113). We contacted hospital staff listed as appropriate contacts via electronic mail and then made phone appointments for the 10 to 20 minute survey. Follow-up included multiple phone calls until a knowledgeable informant was found. Each question asked the interviewee to respond on a 5-point Likert scale about the extent of the problem; researchers took notes on anything that was volunteered beyond that. Numerical data were analyzed with descriptive statistics and the text of comments was analyzed qualitatively, again with the aid of N6. Whenever a respondent spontaneously commented while answering a question, the interviewer wrote down the comment. At the end of the interview, we asked if the interviewee had anything else they would like to share, and these comments were also recorded.
Quantitative data were analyzed using SPSS software (V14.0, SPSS Inc. Chicago IL). We used descriptive statistics to understand the degree of CPOE infusion in the institutions. We explored the correlation between the length of time CPOE has been in place and the extent to which unintended consequences were present. Using bed size and ownership as established proxies for “other important organizational characteristics in studies of hospital organizational issues,” 43 as well as geographic location and teaching status, we compared respondents to non-respondents using logistic regression.
Qualitative data in the form of notes taken during the interviews were analyzed using QSR N6. This analysis simply entailed grouping comments with the questions to which they related, each representing a type of unintended consequence.
We were able to establish contact with 265 of the 561 hospitals and discovered that 89 of those listed in the HIMSS Analytics™ database did not actually have CPOE, so they were dropped from further consideration. We therefore conducted 176 full interviews. Thirty-four additional institutions had policies against responding to surveys, so they were added to the list of non-respondents. Using the Institute for Social and Economic Research (ISER) calculation for response rate, 44 which takes into consideration the ineligibility of some sites (e.g., listed as having CPOE when they actually do not), our response rate was 47%. To estimate how representative our responding sites were of all hospitals that have CPOE, we categorized the sites in four ways: bed size, management type, teaching hospital status, and geographic location 45 because these measures are considered important differentiators. 43 Logistic regression (forward, stepwise, likelihood ratio) comparing respondents and non-respondents showed no difference with respect to hospital bed-size and teaching hospital status. Management type differed between respondents and non-respondents, in that the VA hospitals were most likely to respond, followed by other governmental hospitals, private institutions and non-profits (compared to VA, governmental LR = 0.63, private not for-profit LR = 0.38 and private for-profit LR = 0.35; p < 0.001). There was also a difference in geographic location with a site in the Northeast region most likely to be a responder, followed by the West, South and the Midwest, in that order (compared to South, Northeast LR = 2.11, West LR= 1.65 and Midwest LR = 0.93; p = 0.007).
The length of time that CPOE had been in place ranged from 6 months to 25 years (median = five years). The percentage of orders entered electronically ranged from 1–100% (median = 91%). Greater than 96% of the sites used CPOE to enter pharmacy, laboratory, and imaging orders; 82% were able to access all aspects of the clinical information system with a single sign-on; 86% of the respondents had at least three types of clinical decision support (order sets, drug-drug interaction warnings, and pop-up alerts); and 90% had a CPOE committee in place. The overall infusion levels were therefore at the high end along all measures. 40
summarizes the responses to the eight questions about unintended consequences. For six of the questions, the responses indicated that the unintended consequences were important in those institutions. Workflow was rated as a 3, 4, or 5 by 88%, communication 84%, dependence on technology 83%, system demands 82%, emotions 80%, and more work or new work 72%. For two of the questions, many respondents answered that the unintended consequence had not been seen or was less important than others. Changes in the power structure was rated as 0 by 36% of respondents, and either 1 or 2 by 29%. New kinds of errors were rated 0 by 9% and either 1 or 2 by 44%. We considered that the number of years that CPOE had been in place might affect the occurrence of these consequences to either a lesser or greater extent. The response rating for these questions was evaluated for a correlation between it and duration of use in years and no correlation was found for any of the eight questions (Spearman’s rho statistic). 46
Our preliminary qualitative study identified major types of unintended adverse consequences of CPOE. This survey study goes beyond the prior study describing these types by advancing our knowledge about how widespread the problems are. The survey results verified the existence of the eight unintended adverse consequences types we asked about at most of the 176 hospitals with CPOE, and analysis of comments has offered insight into the perceived importance of the consequences. All types of consequences are indeed widespread although two of them, power shifts and new kinds of errors, were not considered as important as the others. While the current study was based on a categorization of adverse unanticipated consequences of CPOE from the prior study, we found that there are positive as well as negative unintended consequences. Often, the same consequence can be viewed different ways by different people depending on their perspectives.
Our mixed methods approach, using extensive fieldwork to gather data about types of unintended consequences at excellent sites, followed by a phone survey to verify the existence of these same types across the country and to gain further insight, yielded rich data.
The median length of time organizations have had CPOE in place is five years, which is relatively recent since systems have been available since the early 1970’s, 47 although only a few pioneering hospitals had it prior to ten years ago. Interestingly, there appears to be no correlation between the importance of each type of unintended consequence and the length of time that a system has been in place. Despite the recent timing of many implementations, the systems are highly infused, indicating that they are deeply embedded into the workflow, heavily used, and sophisticated.
One limitation of this study is the difficulty of asking for a simple score in response to a complex question. This difficulty was offset by allowing respondents to provide comments after each question—a distinct advantage of telephone surveys. The ability of the researcher to give examples and explain questions was helpful, but it is possible that some respondents may have misinterpreted some questions. There was no attempt to test the reliability and validity of the instrument. We asked to speak to someone in the organization who is most knowledgeable about CPOE, and we usually spoke to someone on the information technology staff with a clinical background. While these individuals seemed to offer balanced answers, they might have given answers biased in a positive way because they believe in CPOE’s benefits. This might help explain why the two categories of power shifts and new kinds of errors did not rank highly: often the power shifted to administration or information technology staff, so informants may not recognize their own gain in power. New kinds of errors may seem minor to these interviewees who seem confident that these errors are most often caught before harm is done. In addition, as representatives of these hospitals, informants may have been reluctant to discuss the sometimes-sensitive subject of errors in heath care. As previously noted, there was a bias toward eliciting adverse unintended consequences, based on the survey taxonomy being derived from a prior study of adverse consequences.
Responding sites tended to differ from non-respondents by management type, with VA hospitals the most likely to respond and for-profit centers, the least likely. Perhaps VA centers are proud of their systems and/or staff members are simply more willing to take the time to talk about them. Unfortunately, many organizations have policies against doing surveys, an increasingly common problem. 48 The reason for geographic differences in the likelihood of response is unclear. As stated, there was no difference in the level of response with respect to bed size or whether the site was a teaching institution.
Once unintended consequences are revealed, hospitals can work to either try to avoid them or to accept them as tradeoffs. For example, the negative impact of information technology on clinical workflow noted by others 49–52 is especially true of CPOE because it takes more of the physician’s time to enter orders than in the manual process, 53–54 but this might be accepted as a tradeoff for the added safety benefits of CPOE.
Our chief recommendation is that developers and implementers of CPOE systems consider each of the eight types of unintended adverse consequences carefully during their planning. The workflow issues are immense, and they affect nearly every hospital staff member. Communication changes, 55 and implementers must take steps to prevent or offset the altered communications created by electronic systems in this respect. As people become more and more dependent on such systems, 56 preparations for downtime must be more elaborate and serious. More work and new work are inevitable, as are increasing system demands, so ample resources must be made available on an ongoing basis. Finally, emotions run high for and against CPOE, 57 and efforts must be made to understand and manage them. Expectation management is therefore paramount. Informants indicated that power shifts and new kinds of errors were less important kinds of unintended consequences, but we must nevertheless be diligent about identifying them. 18,58–63 Unless we make a concerted effort to avoid, manage, and/or overcome unintended consequences, the implementation of clinical information systems may lead to detrimental results. 64–67
The eight kinds of unintended adverse consequences of CPOE we investigated are experienced by most CPOE sites, regardless of length of use of CPOE. CPOE is highly infused, or deeply embedded, at most of these sites, which is surprising considering that most implementations of CPOE are relatively recent. The telephone survey has given us insight into the extent and importance of unintended consequences. Management of possible unintended consequences is vital for maintaining control and avoiding problems. Implementation success depends on managing the unintended, as well as the intended, consequences, of CPOE.
This research was supported by grant LM06942 from the National Library of Medicine, National Institutes of Health, titled Overcoming the Unintended Consequences of Computerized Physician Order Entry Implementation. Emily Campbell and Dr. Guappone were also supported by National Library of Medicine training grant ASMMI0031. The authors extend special thanks to Caroline Wethern, Cody Curtis, and Ashley Jones for their assistance with data collection and organization.