|Home | About | Journals | Submit | Contact Us | Français|
To describe the foci, activities, methods, and results of a four-year research project identifying the unintended consequences of computerized provider order entry (CPOE).
Using a mixed methods approach, we identified and categorized into nine types 380 examples of the unintended consequences of CPOE gleaned from fieldwork data and a conference of experts. We then conducted a national survey in the U.S.A. to discover how hospitals with varying levels of infusion, a measure of CPOE sophistication, recognize and deal with unintended consequences. The research team, with assistance from experts, identified strategies for managing the nine types of unintended adverse consequences and developed and disseminated tools for CPOE implementers to help in addressing these consequences.
Hospitals reported that levels of infusion are quite high and that these types of unintended consequences are common. Strategies for avoiding or managing the unintended consequences are similar to best practices for CPOE success published in the literature.
Development of a taxonomy of types of unintended adverse consequences of CPOE using qualitative methods allowed us to craft a national survey and discover how widespread these consequences are. Using mixed methods, we were able to structure an approach for addressing the skillful management of unintended consequences as well.
When our study began in October of 2003, the unintended consequences of CPOE were a little-discussed area. Patterson et al. had identified and described what they called “side effects” of a different, but similar, kind of system, bar code medication administration (BCMA) . They used a relatively structured ethnographic approach to studying BCMA in Veterans Administration hospitals and identified side effects which they believed could lead to adverse drug events (ADEs). BCMA is supposed to prevent ADEs, but there are numerous unintended consequences they documented that could lead to mistakes. They offered suggestions about how to “eliminate these side effects before they contribute to adverse outcomes” [1, p. 540].
Problems related to clinical decision support had likewise been described in the literature. Although decision support is often cited as a reason for implementing CPOE, there has been controversy about the appropriate number of alerts and reminders, since too many tend to overwhelm and annoy users . One 1989 report described an experiment where, to reduce the time between alert posting and review by the clinician, a flashing light mechanism was placed on top of the computer and designed to flash to let a user know when an alert was present. The system was extremely effective in encouraging a rapid response to the alert, reducing the average acknowledgment time from 28 hours to .1 hour, but users insisted the experiment be halted because the lights were too annoying . This is a dramatic example of a negative unintended consequence of an otherwise effective system.
Medical error reduction is a prime reason for implementing CPOE, but users are also concerned that new kinds of errors are being made because of clinical systems. Many papers written about CPOE gave brief mention to this concern or cited anecdotes, but there were no published studies about mistakes that could be caused by CPOE. The Physician Order Entry Team (POET), a group of researchers based at Oregon Health & Science University in Portland, Oregon, U.S.A., was conducting a study of success factors for implementing computerized physician order entry (CPOE), defined as direct entry of orders into the computer by physicians or others with the same ordering privileges, when we began noticing unintended consequences (UCs) that might lead to errors. The clearest example is entry of an order for the wrong patient because of what we call a “juxtaposition error” when an item near the one actually desired is clicked by mistake.
Colleagues doing similar qualitative studies in Australia and The Netherlands were discovering these UCs as well, and a collaborative effort in 2002 produced a general description of kinds of adverse consequences caused by clinical information systems (CIS) . This was a rather startling revelation at a time when CPOE was being touted as the “leap” that hospitals should take in the interest of patient safety  and little attention was being paid to problems caused by CPOE. In three separate observational studies, these research teams had continually witnessed “wrong patient” juxtaposition errors. Similar errors occur when one clicks on a test or medication listed on the screen next to the one needed. The summary paper by Ash, Berg, and Coiera highlighted the phrase “unintended consequences of CPOE,” which has become widely accepted and used . This paper was influenced by several monographs that dramatically describe the unintended consequences of technology in general [6–9]. Since publication of the Ash, Berg, Coiera paper , numerous papers in both medical and the medical informatics journals have further described the unintended consequences of health information technology [10–18].
With funding from the U.S. National Library of Medicine, POET has been able to conduct an in-depth study over the past four years utilizing both qualitative and quantitative methods to discover more about these UCs of CPOE. Data were gathered via two expert panel conferences, fieldwork at a total of six sites (one outpatient and five primarily inpatient), and a national telephone survey of all CPOE sites in the U.S. The aims were to identify types of UCs and strategies for preventing, managing or overcoming them, and to provide tools to help implementers address them. The following presents a summary of the research foci, methods, and results, along with general conclusions about the overall project.
The main criterion for site selection for fieldwork was that organizations have a reputation for excellence in using clinical information systems. Excellent organizations learn from their mistakes [19–20] and therefore staff members in those organizations have analyzed the issues and are knowledgeable about strategies for overcoming obstacles. We were seeking sites with personnel who would be willing to 1) describe surprises they have experienced and managed, and 2) be observed during the order entry process. Sites represented a geographic distribution, different types (e.g. teaching and community hospitals) and ownership (e.g., public and private), varying durations of experience with CPOE, and both commercially and locally developed systems. Kaiser Permanente Northwest in Oregon, selected for excellence in outpatient CPOE, uses EpicCare (Epic Systems, Madison, WI), and is a health maintenance organization. Other sites included: Wishard Memorial Hospital, a county hospital in Indianapolis, IN using the locally developed Regenstrief system; The Brigham and Women’s Hospital in Boston, MA, which uses a locally developed system; Massachusetts General Hospital, Boston, MA, which uses a newer version of the Brigham system; Faulkner Hospital, a community hospital in Boston, MA, that uses the commercial MediTech system (Westwood, MA); and Alamance Regional Hospital in Burlington, NC, which uses the Eclipsys commercial system (Boca Raton, FL). We received human subjects approval from each site and from the researchers’ organizations. Within each site, informants for interviews were selected based on their knowledge of what had occurred during CPOE implementation and use and their representative roles (physician, nurse, pharmacist, implementer, champion, skeptic, etc.). We observed clinicians entering orders in all areas of the hospitals and clinics. Experts for the expert conferences were selected based on hands-on experience with CPOE implementation and included clinician implementers from a variety of hospital types and vendor organizations.
Figure 1 illustrates the progression of data gathering, analysis, and reporting of results that occurred between 2003 and 2007, starting with a transition period at Kaiser Permanente Northwest between the success factors study and the current UC study. We gathered data at Kaiser on an ongoing basis between the spring of 2003 and winter of 2004, developing refined semi-structured interview and observational techniques during the transition to studying UCs. During that period, we conducted 29 hours of observation in four clinics, shadowing 15 clinicians, and we interviewed 12 clinicians and staff members. In the spring of 2004 we held a conference of invited experts at the Menucha retreat center near Portland, OR to gather stories about UCs from the experts and to gain guidance about questions to ask and what to look for in the field. We then spent three to four days at each of our five inpatient study sites with four to six investigators on site at any one time. We conducted 390 hours of observation with 95 clinicians and did 32 interviews. Observations were documented with extensive field notes. Taped oral history interviews probed the history of surprises that occurred over time at each site. Realizing that it is hard to know when consequences are indeed unintended, we developed a short survey instrument to assess what clinicians being impacted by CPOE expect, and administered it to groups at three sites. Once we had identified the major types of UCs, we developed, piloted, and administered a telephone survey instrument designed to elicit information about the nature of UCs from all U.S. hospitals reporting that they have CPOE and to identify infusion, or levels of sophistication, of CPOE implementation efforts. A second conference of experts was held in May of 2006 to help interpret our results and to plan dissemination strategies, including publication of a set of tools that implementers could use for identifying, managing, and overcoming UCs in their hospitals.
We first reviewed data gathered at Kaiser  during the transitional period between projects to gain an overview of UCs and developed a general schema that included positive as well as negative UCs . Data collected during the expert conference and at the remaining five field sites yielded 750 pages of field notes and transcripts. The POET team reached consensus on each description of what seemed to be a UC, approximately 80 from the Menucha conference and 300 from fieldwork. As we analyzed them in detail, we realized that the initial UC schema we had developed  was too broad and superficial and that we would need to develop a more sophisticated taxonomy. Using a card sort method  and grounded theory approach , we iteratively developed a taxonomy of nine major types of adverse UCs into which all 380 instances fit, thus reaching saturation [25, p. 520]. Using an axial coding approach , we then explored each type in depth, discovering subcategories within each of the nine major categories. We developed survey questions based on what we had learned in the field and administered a telephone survey aimed at all CPOE sites in the U.S.A. We held another Menucha conference of experts to verify that these results made sense to the experts and to plan dissemination of tools to help implementers address each kind of UC. Finally, we developed each of the recommended tools with further guidance from the experts, using the Delphi technique and tree mapping.
The initial broad schema of consequences related to CPOE included categories of intended and unintended consequences, desirable and undesirable, direct and indirect, and “two sided” consequences that could be either desirable or undesirable depending on one’s point of view . This schema provided a valuable framework for fieldwork because it assured that we would not limit our foci to adverse consequences. While most informaticians are interested in undesirable consequences because they need careful management, it is also heartening to know that serendipitous, beneficial surprises occur as well.
Using the card sort method, we developed a “short list” of nine categories based on the larger list. The categories were validated as we analyzed the full complement of UCs. We have published a paper summarizing overall results , as well as specific papers about changes in the power structure resulting from CPOE , emotions related to CPOE ; the unintended consequences of CDS [30–31]; and overdependence on the technology . As we conducted our analysis, we discovered that a large number of consequences, over 20% of the total, emanated from issues with clinical decision support (CDS). Briefly, the nine categories, in descending order of frequency, are:
Physicians find that CPOE adds to their workload by forcing them to enter required information, respond to alerts, deal with multiple passwords, and expend extra time.
Many UCs result from mismatches between the CIS and workflow and include workflow process issues, workflow and policy/procedure issues, workflow and human computer interaction issues, workflow and clinical personnel issues, and workflow and situation awareness issues.
Because CPOE requires hardware technically advanced enough to support the clinical software, there is a continuous need for new hardware, more space in which to put this hardware, and more space on the screen to display information. In addition, maintenance of the knowledge base for decision support and training demands are ongoing.
It has long been hoped that CIS will reduce the amount of paper used to communicate and store information, but we found that this is not necessarily the case since it is useful as a temporary display interface.
The CIS changes communication patterns among care providers and departments, creating an “illusion of communication,” meaning that people think that just because the information went into the computer the right person will see it and act on it appropriately .
As outlined in the paper by Sittig et al. , these systems cause intense emotions in users. Unfortunately, many of these emotions are negative and often result in reduced efficacy of system use, at least in the beginning.
As noted by Koppel et al.  and Ash et al. , CPOE tends to generate new kinds of errors such as juxtaposition errors, in which clinicians click on the adjacent patient name or medication from a list and inadvertently enter the wrong order.
The presence of a system that enforces specific clinical practices through mandatory data entry fields changes the power structure of organizations. Often the power or autonomy of physicians is reduced in an effort to standardize , while the power of the nursing staff, information technology specialists, and administration is increased .
As hospitals become more dependent on these systems, system failures can wreak havoc when paper backup systems are not readily available .
We had already determined that the UC categories list was both useful and easy to understand and use, so we operationalized most of the categories by asking questions about UCs in a national survey. Those surveyed were to answer these questions as yes or no, and if yes, to rate the importance of this issue from 1 (not very important) to 5 (very important) (no = 0). The questions are shown in Table 1. We were also interested in discovering how sophisticated each CPOE implementation was, so we developed and piloted a set of questions about “infusion,” which is the depth or level of functionality and use of an innovation and it increases as an information system becomes more deeply embedded within an organization’s work processes [35–36]. Table 2 includes the infusion questions we asked each respondent during the interview survey. We surveyed the entire population of acute care hospitals listed in the 2004 HIMSS AnalyticsSM Database as having reported that they have CPOE in place (N = 448). Since that database did not include U.S. Veterans Affairs hospitals, which we feel are important models of CPOE use, we also surveyed VA hospitals (N = 113). We conducted interviews with staff at 176 hospitals, discovering that a large number listed as having CPOE did not in fact have functioning CPOE systems. We also found that a number of hospitals have policies against doing surveys.
In the 176 responding hospitals, we found the length of time that CPOE had been in place ranged from 6 months to 25 years (median = 5 years) while the percentage of orders entered electronically ranged from 1–100% (median = 90.5%). We found that greater than 96% of the sites used CPOE to enter pharmacy, laboratory and imaging orders and that 82% were able to access all aspects of their clinical information system with a single sign-on. Further, 86% of the respondents had access to order sets, drug-drug interaction warnings, and pop-up alerts even though nearly all hospitals surveyed were community hospitals with commercial systems. Finally, we found that 90% of the hospitals had a CPOE committee with a clinician representative in place .
The UC survey results verified the existence of these UCs, and analysis of comments offered insight into the nature of the consequences. All types of consequences are indeed widespread. Our informants did not consider two of them, power shifts and new kinds of errors, as important as the others, however. We verified that there are positive as well as negative unintended consequences, and often the same consequence can be viewed in different ways by different people, depending on their perspectives. We can only speculate about why power shifts and new kinds of errors were not considered as important as other types: those answering the questions were generally information technology professionals who may not realize that power is shifting in their direction. They may also believe that the new kinds of errors are not of a serious nature. A paper reporting more extensive results has been published .
To find out if the “unintended” or “unanticipated” consequences that we had identified were perhaps already known by others in the field and therefore actually anticipated during CPOE implementations, we designed and piloted an “anticipation survey.” It was designed to determine what end user clinicians were expecting to happen in organizations that were about to implement CPOE. The questions, which were asked in person, are shown in Table 3. The survey was administered as a short interview survey to 83 clinicians at three community hospitals at common gathering spots such as the cafeteria. Results from each hospital were fed back to the implementers at those individual sites. The research team conducted a comparative analysis across sites. Briefly, end users and others affected by CPOE were usually aware that CPOE was coming and that their workflow might be disrupted for a certain period of time, but they were optimistic that in the long run it would be of benefit. Other UCs were rarely mentioned. Results are summarized in a published paper .
The results of our fieldwork and the survey were disseminated to the experts prior to the Menucha conference held on May 4 and 5 of 2006. The experts affirmed our categories of UCs and recommended that we develop and disseminate a number of tools to assist implementers in avoiding or managing unintended consequences. These included a best practices guide for implementation to help avoid UCs, a checkup tool to help hospitals assess how well they are managing UCs, and a dashboard tool of recommended metrics that can help hospitals track their progress.
To develop recommendations about best practices, we looked at each UC type and asked how it could be prevented or mitigated. Since the resulting list of prevention measures resembled our list of success factors identified in our first project, we then performed an extensive exercise whereby we mapped each UC type and sub-type to each of the 80 success factors, aiming to enhance the success factors list. The result was a series of interactive “tree maps” published on our Web site, www.cpoe.org, which help a user look at a UC type and learn how to manage it. The success factors originally described withstood this test well: they were minimally edited, indicating that as a list of best practices, they were confirmed.
We designed and tested a number of survey instruments that form a series of checkup tools. The instruments illustrated in Table 1–Table 3 should prove helpful to hospitals over time. In addition, we have developed recommended metrics that form a dashboard tool, starting with a long list of ideal measures and honing them to a shorter, more realistic list, with the help of our experts [40–41]. The final list shown in Table 4 includes eight measures which are 1) important to measure on a continuing basis, 2) relatively easy to measure, and 3) are well defined. All of these tools are freely available on the Web site and in publications.
In this team’s earlier research on success factors for implementing CPOE, we cast a wide net because little was known at that time about factors leading to success or lack thereof. A rigorous yet open-ended grounded theory  approach was deemed most appropriate. As research questions became more focused, however, our qualitative methods became more structured. To investigate UCs of CPOE, we started with a broad schema of types and iteratively refined the schema by consulting experts, interviewing, and observing in the field . Once a taxonomy of types was defined, we were able to craft survey questions and administer a national telephone survey. Because such a large number of adverse UCs relate to CDS, our attention is becoming even more focused in that arena .
As the team and expert consultants developed tools for helping implementers avoid, manage, or overcome UCs, we realized that all of the success factors identified and published in the past  can actually serve as UC avoidance mechanisms.
While it is hard summarizing results of an intense four year study of UCs, we can draw some general conclusions about both methods and UCs. First, the selected methods served us well for this study. The more structured and rapid techniques such as the anticipation survey efficiently augmented other kinds of fieldwork. Second, development of a taxonomy of types and subtypes not only allowed us to craft survey questions, but was also useful in structuring an approach for addressing management of UCs. One key in prevention of UCs is to pay attention to success factors for implementing CPOE in the first place. Since so many UCs are related to CDS, it seems that a fruitful area for future research would be identification of success factors for implementation of CDS.
CPOE with CDS can reap benefits for hospitals, but use is not widespread in the U.S.A.
Fear of the unintended consequences of CPOE may be discouraging acceptance
Deeper understanding of the unintended consequences of CPOE
Knowledge about the levels of sophistication of CPOE implementations
Strategies for avoiding or dealing with the unintended consequences of CPOE
This paper is based on a presentation given at MedInfo 2007. This work was supported by grant LM06942 and training grant ASMM10031 from the U.S. National Library of Medicine, National Institutes of Health. The funding agency had no role in development of this paper and none of the authors has conflicts to declare. All authors contributed to data gathering, analysis, and writing.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
*Based on a paper presented at MedInfo 2007