Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Int J Med Inform. Author manuscript; available in PMC 2014 January 1.
Published in final edited form as:
PMCID: PMC3430828

Conducting an Efficient Proactive Risk Assessment Prior to CPOE Implementation in an Intensive Care Unit



To develop, conduct, and evaluate a proactive risk assessment (PRA) of the design and implementation of CPOE in an ICU.


We developed a PRA method based on issues identified from documented experience with conventional PRA methods and the constraints of an organization about to implement CPOE in an intensive care unit. The PRA method consists of three phases: planning (three months), team (one five-hour meeting), and evaluation (short- and long-term).


Sixteen unique relevant vulnerabilities were identified as a result of the PRA team’s efforts. Negative consequences resulting from the vulnerabilities included potential patient safety and quality of care issues, non-compliance with regulatory requirements, increases in cognitive burden on CPOE users, and/or worker inconvenience or distress. Actions taken to address the vulnerabilities included redesign of the technology, process (workflow) redesign, user training, and/or ongoing monitoring. Verbal and written evaluation by the team members indicated that the PRA method was useful and that participants were willing to participate in future PRAs. Long-term evaluation was accomplished by monitoring an ongoing “issues list” of CPOE problems identified by or reported to IT staff. Vulnerabilities identified by the team were either resolved prior to CPOE implementation (n = 7) or shortly thereafter (n = 9). No other issues were identified beside those identified by the team.


Generally positive results from the various evaluations including a long-term evaluation demonstrate the value of developing an efficient PRA method that meets organizational and contextual requirements and constraints.

Keywords: Risk assessment, CPOE, Electronic health records, Intensive care units, Human factors engineering

1. Introduction

There are known vulnerabilities associated with implementation of computerized provider order entry (CPOE) [14]. Many of these vulnerabilities occur by not considering the work system in which the technology is implemented and are related to the design of the technology and its impact on workflows and processes [5]. The human factors engineering discipline offers a range of approaches and methods for anticipating some of the vulnerabilities which can be addressed before technology implementation [69]. In this paper, we describe the development and use of a proactive risk assessment (PRA) method to identify and address known and potential vulnerabilities related to CPOE implementation in a particularly error-prone process, i.e. a transition in care within a hospital [1013].

PRA methods have been used to identify potential vulnerabilities associated with changes in processes [9] and technology implementation, including those related to the implementation of Smart infusion pump technology [14] and to identify strategies for dealing with vulnerabilities associated with IT-mediated medication management [15]. To our knowledge, there is only one published study on the use of PRA in the context of CPOE implementation [16]. Bonnabry and colleagues [16] used a specific PRA method – failure mode and effects analysis (FMEA) – to evaluate the medication prescription process before and after CPOE. While this application of FMEA identified a number of vulnerabilities in CPOE implementation, there was no information provided on the long-term effectiveness of the PRA.

The use of efficient PRA methods before health IT implementation will increase if we can demonstrate their long-term safety benefits. However, few PRA studies have attempted to assess the long-term impact of the method or its results. Our own research has described significant vulnerabilities identified through an FMEA method used prior to implementation of Smart infusion pump technology [14]. After implementation we assessed the various vulnerabilities related to the Smart infusion pump’s performance and determined which ones the FMEA team identified as well as which ones had not been anticipated. We demonstrated that the FMEA method identified many but not all vulnerabilities. A significant vulnerability related to the Smart infusion pump technology was identified only after implementation [17].

Another barrier to the use of PRA methods before health IT implementation is the significant investment of resources and time necessary. Conventional PRA methods such as FMEA have demonstrated shortcomings due to the excessive resources required [18] and expectations placed on participants [19]. By following the method prescriptively, FMEA teams generally spend a significant amount of time attempting to fully understand the problem and current process(es), to generate extensive issue and problem lists, and to identify possible solutions. Likewise, organizational constraints (e.g., staff availability, implementation timelines) are not always taken into account when initiating PRAs. When implementing health information technology (IT) such as CPOE in a complex healthcare environment, it is even more critical to ensure that the PRA method is conducted in an efficient manner. Research on PRA has produced a set of guidelines to use when preparing for the team portion of a PRA [9, 20, 21].

In this study we developed, implemented, and evaluated a PRA method that identifies vulnerabilities in CPOE design and implementation.

2. Methods

The PRA was part of a larger study evaluating the impact of a computerized provider order entry (CPOE) implementation in intensive care units (ICUs) ( The PRA was conducted jointly by human factors engineering (HFE) researchers at the University of Wisconsin-Madison and staff from the various departments that would be affected by or were involved in the CPOE implementation at Geisinger Medical Center (GMC), in Danville, PA.

2.1. Organizational setting

GMC, a flagship quaternary hospital, has 403 beds and four intensive care units (ICUs): a 24-bed semi-closed adult ICU (medical/surgical shock and trauma), an 18-bed open cardiac ICU (cardiac medical/surgical, transplants, adult ICU overflow), an 11-bed semi-closed pediatric ICU (all pediatric medical/surgical patients excluding neonates), and a 38-bed closed neonatal ICU (critically & seriously ill newborns). Electronic health record (EHR) technology was first implemented in 1996 in the ambulatory setting.

In conjunction with EHR implementation, workflow redesign efforts have aimed at improving the quality and efficiency of the care provided [22]. Through the PRA we describe here, hospital leaders and IT management further demonstrated support for identifying critical issues associated with CPOE by obtaining input from multiple individuals representing various clinical stakeholders. Hospital leaders were especially interested in assessing a process that: 1) was complex in the paper world and would be equally challenging to transform to an electronic format, 2) required numerous transitions in care, and 3) was currently error-prone.

Because there were limitations on staff availability and a tight development and implementation timeline, management believed the amount of group meeting time required by many conventional PRA methods [9, 18] would be problematic. The organization was committed to conducting a PRA, however, and was willing to pay staff overtime to participate. Including benefits, this cost the organization approximately $2500. The participating stakeholders demonstrated their commitment to the PRA by the fact that each of them worked an 8-hour shift immediately prior to convening for the PRA team meeting. All necessary Institutional Review Board approval was obtained before proceeding with the PRA.

Given these constraints, we determined that the PRA would be divided into three phases (Figure 1): a planning phase requiring considerable up-front effort by a limited number of content and methodological experts; a team phase that would capture the unique experience and expertise of the stakeholders related to the process selected; and an evaluation phase that would include collecting feedback related to the PRA team phase, the success of the team in identifying vulnerabilities, and the long-term success of the PRA. Each of the phases required varying degrees of input from a number of different individuals. The team phase of the PRA occurred eight months before the actual CPOE implementation.

Figure 1
PRA timeline1

2.2. PRA planning phase (Figure 1)

In the planning phase, a team comprised of two HFE researchers who later led the PRA team (SVD and ASH) and the organization’s IT director (JAA) met on a biweekly basis. The meetings included relevant IT staff/analysts as their direct input was needed. The research team’s past experience with FMEA led to the development of ground rules for selecting a process to review [20]. These ground rules were reviewed every time the planning team convened and require that:

  1. the process is clearly defined,
  2. its scope is limited,
  3. the process is relevant to ICUs,
  4. the process poses significant risk with the implementation of CPOE,
  5. workflows related to the process are complete and verified,
  6. process owners are willing and available to participate in the team phase,
  7. proposed changes and recommendations will be addressed,
  8. top management supports the activity, and
  9. content experts will provide necessary input prior to the meeting.

The ground rules related to the process selection criteria (first four listed above) were reinforced at each meeting. After five 30-minute meetings, a process was selected: the transition of patients from the OR to the ICU; this is a well-documented high risk process in hospitals [1012]. The practice at the organization for handling ICU patients after surgery was, and would continue to be after CPOE implementation, to bypass the post-anesthesia care unit and transfer these patients directly to the ICU. This process posed significant risk and inherent vulnerabilities associated with potentially missing, unconfirmed, or duplicate orders and other hand-off communication associated with ICU patients’ orders.

After selecting the process, flowcharts depicting the workflows of the current non-IT-related tasks (Figure 2) as well as the proposed tasks (Figure 3) were developed by IT analysts. Current workflows were confirmed, whereas the workflow the analysts envisioned after CPOE implementation would be updated as necessary during and after the PRA team phase. Both workflows were later presented to the full PRA team. The planners also agreed that having these readily visible to the team would help maintain focus and efficiency during the team phase.

Figure 2
Existing workflow - Transfer from OR directly to ICU (pre-CPOE)
Figure 3
Proposed workflow - Transfer from OR directly to ICU (with CPOE)

To help the team further understand both the proposed workflow and the CPOE application that would support it, CPOE simulations were also created prior to the team phase. The simulations were designed by planning team members, an IT department physician liaison, and an IT nurse informatician. Two simulations depicted the patients that would be affected by the process being assessed: immediate post-operative pediatric and adult ICU patients. These realistic test patient records (the simulations) contained diagnoses, orders, and laboratory values. The simulations would be demonstrated during the team phase to afford team members the opportunity to observe the computer interface as they considered its impact on the proposed workflow. This would be the first time any future users of the system would see this aspect of the CPOE interface.

2.3. PRA team phase (Figure 1)

One of the most important considerations for a successful PRA is the involvement of key and relevant stakeholders. Therefore, the planning team asked IT analysts working on this facet of the CPOE design to nominate participants based on their role in the process. Thus the final team was comprised of 18 people from: 1) Nursing staff, including 1 floor nurse, 2 nurse educators, and 2 unit desk clerks (UDCs), 2) Medicine – 1 critical care physician representing the pediatric ICU and neonatal ICU, 2 critical care physicians from the adult ICUs, and a trauma surgeon from the adult ICU who also represented the operating room (OR), and 3) Information Technology (IT), including 1 analyst, 1 physician liaison, 3 nurse informaticians, and 3 management staff. The chief health information officer (CHIO - JMW) represented top management and was an ex-officio member. The two HFE researchers served as co-facilitators and an HFE graduate student (BLP) served as note-taker.

The meeting began promptly with an overview of the PRA’s objective provided by the CHIO. Subsequently, a nurse informatician reviewed the workflows, including both the current and proposed versions of the process. (Figures 2 and and3)3) This served as a means of clearly defining the process and as a visual reference for participants throughout the remainder of the meeting. It also reinforced the limited scope of the task and made it significantly easier to readily identify unrelated issues suggested by the participants. Any unrelated issues were recorded on a “parking lot” and later forwarded to relevant IT analysts.

A nurse informatician then gave a demonstration of the CPOE interface by projecting it on a screen and presenting each of the simulations. This provided a clear understanding of how the interface would influence or complement the proposed workflow. At the same time it helped participants identify potential vulnerabilities related to the interface design and workflow.

Participants then brainstormed and identified potential vulnerabilities associated with the proposed workflow and interface. All of these were recorded on flip charts posted on two sides of the perimeter of the meeting room regardless of whether or not they were previously noted during this phase. No vulnerabilities had been identified prior to the team phase. After the meeting was approximately half over, the group concurred that they reached saturation [23] and that all of the potential vulnerabilities they could think of had been identified.

The group then began identifying similar and/or redundant vulnerabilities captured during the brainstorming session and grouped them. Once they were grouped, the participants rated the vulnerabilities according to a taxonomy commonly used within the organization. This taxonomy rated the vulnerabilities according to their respective impact on: efficiency, medical-legal issues, patient safety, quality of care, and regulatory compliance. Efficiency and regulatory compliance were rated as relevant (“yes”) or not (“no”) to the respective vulnerability, whereas the three other categories were rated as having “high”, “medium”, or “low” negative impact associated with the vulnerability. Both the grouping and rating steps utilized a group consensus method. In most instances participants quickly concurred and the group proceeded efficiently.

After completing the grouping and rating, participants reviewed their efforts and the results of the PRA – a list of vulnerabilities and their probable impact on organizational and patient outcomes. Approximately half of the participants provided verbal feedback on the team effort during a short group-based debriefing. No further vulnerabilities were identified and the group’s five-hour effort was deemed complete.

2.4. PRA evaluation phase (Figure 1)

The PRA evaluation consisted of an immediate evaluation that occurred upon completion of the PRA team phase and a long-term evaluation that occurred after CPOE implementation. Immediately after the PRA team phase, everyone remaining in the room completed a short written evaluation. Those who had previously left were sent evaluations electronically or through interdepartmental mail and asked to promptly complete and return them. Eleven out of the 18 surveys distributed were completed and returned.

An additional evaluation of the team phase occurred the following morning during a wrap-up and planning meeting that included the HFE researchers, physician liaison, CHIO, IT manager, and IT analysts who attended the PRA team meeting. In preparation for the wrap-up meeting, members of the planning team had condensed and summarized the outcome of the PRA team meeting held the previous day. After reviewing the team’s work, those participating in the wrap-up session developed plans on how to address the vulnerabilities.

In addition to the immediate evaluation of the PRA team phase, participants in the wrap-up session agreed they would review all reported CPOE vulnerabilities and focus on those associated with the OR-ICU transfer process (as addressed during the PRA) after CPOE implementation. This long-term ongoing review, a unique feature of our study, continues as of this writing to measure the effectiveness of the PRA.

2.5. Traditional PRA versus our modified PRA

In summary, our PRA method attempted to compensate for many problems associated with the manner in which traditional PRA methods such as FMEA have been executed. Like many other PRA methods, our PRA method involved top leadership, required interdisciplinary involvement of key stakeholders, and addressed a well-defined process with a limited scope. Table I summarizes the differences between our PRA method and traditional PRA methods.

Table I
Comparison of traditional and modified PRA methods

3. Results

3.1. Vulnerabilities and solutions identified by PRA team

A total of 59 vulnerabilities were initially identified by the PRA team: 40 were within the scope of the PRA, 16 were out-of-scope but still relevant to other aspects of the IT implementation, and 3 were deemed irrelevant to implementation. By combining duplicate and similar vulnerabilities (from the 40 pertinent ones), 16 relevant vulnerabilities resulted and were prioritized. (Table II) The vulnerabilities demonstrate a range of potentially negative consequences, including 1) patient safety and quality of care issues, 2) non-compliance with regulatory requirements, 3) increases in cognitive burden on CPOE users, and 4) worker inconvenience or distress. Nearly half of the vulnerabilities (7/16, 44%) solely have potential patient safety and quality of care consequences; five (31%) have a combination of patient safety/quality of care and worker distress/inconvenience consequences; two vulnerabilities (12%) solely impact worker distress/inconvenience; and potential patient quality and safety issues combined with either increased cognitive burden or non-compliance with regulatory requirements account for one consequence each (6% each).

Table II
Relevant vulnerabilities identified in PRA and steps leading to their resolution

The organizational and timeline constraints associated with the CPOE implementation required that IT staff further review the prioritized vulnerabilities and determine when and what type of action should be taken. (Table II) Those known to pose the greatest risk were investigated first. Once these vulnerabilities were confirmed with usability evaluations on simulations, various actions or combinations of them ensued. A technology redesign required changes to the user interface or system software. Process redesigns required changes to the anticipated workflow. Initial and/or ongoing user training occurred by using computer-based training modules and through classroom-based super user training. Monitoring occurred for instances in which the vulnerability could not be confirmed or the solution was deemed potentially insufficient. In this study monitoring refers to “watchful waiting”, a term adopted from the medical literature that implies that a known vulnerability exists and focused monitoring for consequences of the vulnerability needs to occur.

In slightly over half of the instances (13/24), only one type of action was taken: technology redesign occurred in 3 instances, training addressed vulnerabilities 3 times, a process redesign occurred once and monitoring alone occurred 6 times. Training was combined with: technology redesign four times, process redesign once and both technology and process redesign twice. Redesigns of both the technology and process occurred twice. In two other instances, no change occurred; the feature was not installed and action was deferred at the request of nursing staff.

Vulnerability 1 – “CPOE feature (if installed) allows for automatic discontinuation of all orders” – referred to a “button” feature of the EHR that, if selected, would discontinue all orders. No prompt, alert or warning would appear that informed the user of the consequences of selecting the “button”. Continued patient care would be compromised and standard processes such as medication reconciliation would be significantly more challenging to accurately perform. With these consequences in mind it was obvious that this “button” feature should not be installed.

Vulnerability 4 – “hard to distinguish STAT vs now orders2” – was one of the vulnerabilities addressed in a number of ways: through initial and ongoing training and by modifying the user interface. Training occurred in three mandatory formats. One training mechanism utilized at GMC is “physician champion lectures”. In this case, the champion highlighted the risks associated with confusion over “STAT” versus “now” orders and clarified the definitions and timing of the two order types. During the lecture, the champion also reinforced the organization’s policy that “all STAT orders must be verbally communicated”. Second, the issue, as well as the organization’s policy, was presented during classroom and computer-based training. Finally, immediate post-implementation “at the elbow” support staff reinforced the distinctions and appropriate use of these order types. These same support staff also distributed one-page tip sheets that clarified order types. Aside from training, the computer interface was simplified and clearly noted the order types by sequencing orders and prominently displaying the order timing (STAT/now).

Vulnerability 6 – “Need to record intra-operative medications on paper” – was especially relevant to the constraints associated with the CPOE implementation because the anesthesia module of the EHR was not being implemented at the same time. The decision to scan the anesthesia medication records assured that all information would be available electronically.

Three vulnerabilities (14, 15 and 16) refer to unit desk clerks’ and nurses’ “worry state” related to order awareness (from “release” of the order by the system to the receiving service, to order completion). These vulnerabilities were addressed through numerous means. During classroom training, instructors paid significant attention to the ordering process and demonstrated how the redesigned (post-PRA) CPOE system provided real-time status reporting. Similarly, workflows were further analyzed and redesigned to incorporate the need for staff to pay attention to the enhanced status reporting function. Finally, IT support staff monitored for problems related to order awareness after implementation.

3.2. Immediate evaluation by PRA team members

At the conclusion of the PRA team meeting, we asked participants to share their feedback verbally and to complete a paper survey. Verbal feedback was positive. The participants pointed out that they had never previously provided feedback on interface design and its associated workflow in a group that included individuals from outside their professional domain. Prior to this PRA, staff solely provided input on interface design in peer groups.

Feedback was also provided through the short surveys the participants completed at the end of the team phase. (Appendix A) Eleven of the eighteen PRA team participants (61%) completed a questionnaire rating the usefulness of the PRA (1 = completely useless, 5 = extremely useful) with a mean of 4.18, median of 4, and standard deviation of 0.98. (This result was similar to that reported in another work by our research team where the average response to this question was 4.2 [15].) The five comments made were: “Utility appears to vary by unit.” “Without the introduction it was unclear to me what the meeting was attempting to accomplish.” “Positive: having new eyes looking at the workflows find new ideas.” “Negative: new eyes that are inexperienced with [software] are distracted.” “For the first PRA, I thought that the feedback was good.”

Overall, participants indicated they were willing to “participate in a PRA again” (1 = not at all willing, 5 = extremely willing) with a mean of 4.27, median of 4, and standard deviation of 0.79. The only comment provided was, “…time permitting”. This result is also similar to that in Faye et al [15] who reported an average response of 4.0 to this question.

Suggestions for improving future PRAs included two comments: “…[software] should be utilized by clinicians in a test environment before proceeding with hospital-wide implementation.” “I thought this was a very useful [method] with different perspectives generating good discussion of potential issues.” Three comments indicated that participants had insufficient preparation for the PRA: “Provide preview to participating members in advance; may help those in session to be better prepared.” “Review process and functionality with end users prior to PRA.” “Everyone should attend the introduction. Those who came late should be taken aside to explain the point of the meeting.”

3.3. Immediate evaluation by IT staff and top management

The IT staff, planning team, top management representative (CHIO), and researchers convened the morning after the PRA team phase to review the PRA method and the outcome of the team’s effort. The group agreed that assembling a heterogeneous group of stakeholders was worthwhile and, in this case, resulted in significant input to the IT staff, possibly more than would have been achieved if homogeneous groups of stakeholders would have been given the task. Peer group feedback had been the conventional means of capturing stakeholder input prior to this PRA. The group noted that what made the contributions even more significant was the “deference to expertise” [24] by team members. In this case, input by unit desk clerks, nurses, and physicians was valued equally, with slightly higher attention paid by stakeholders to the vulnerabilities the unit desk clerks identified.

3.4. Long-term evaluation of the PRA

Issues related to the vulnerabilities were closely monitored in conjunction with ongoing “issues lists” maintained by Geisinger IT staff. Much like the combined prospective and retrospective risk analyses discussed by Kessels-Habraken, et al [25], this “issues list” contains problems and vulnerabilities identified by users and IT staff related to the design and use of any aspect of GMC’s EHR prior to and after implementation. We used the issues list to monitor whether any of the vulnerabilities identified in the PRA occurred during the eight months prior to go-live (pre-implementation) and whether these or any new vulnerabilities related to the OR-ICU transition process were identified after CPOE implementation.

Action taken on 7 of the 15 vulnerabilities (Table II, #1–7) prior to implementation was deemed successful since no negative event or consequence has been reported or occurred. Action was also taken on 8 other vulnerabilities prior to implementation (Table II, # 8–15). Through focused monitoring by IT staff and/or user reporting after implementation, it became apparent that each of the 8 vulnerabilities required further action. The subsequent action taken appears to have effectively addressed the vulnerabilities since no negative event or reporting related to any of these 8 has occurred. One other vulnerability (#16), having no identified negative consequence on patient care, yet invoking a “worry state” for stakeholders, was addressed post-implementation and resolved. This issues list continues to be maintained and monitored for all EHR applications. To date no issues related to the OR-ICU transition process have occurred or been reported that were not identified during the PRA.

4. Discussion

The PRA findings described here, in conjunction with other research [14, 14, 16, 19, 22], demonstrate the value of conducting risk assessments on planned health IT implementations. In contrast to challenges associated with FMEA as discussed by others [18, 19], this method follows guidelines as proposed elsewhere [9, 20, 21] that are designed to improve the efficiency of the PRA. In this case we divided the method into three phases to more appropriately utilize resources and personnel.

In this study, top management of GMC recognized organizational and implementation timeline constraints that guided the PRA method we developed. By lessening the “up front” burden on stakeholders and accomplishing a significant amount of descriptive work during the planning phase (issue identification and planning: 3 people, 7 ½ hours total; simulation: 3 people, 5 hours total), we then utilized the stakeholders’ expertise and efficiently captured their input during the team phase of this PRA (18 people, 3–5 hours in attendance each). During a single team session, stakeholders identified 16 potential vulnerabilities in the CPOE technology design. IT staff and top management then addressed the prioritized vulnerabilities and rectified them before (44% of vulnerabilities) or shortly after (56% of vulnerabilities) CPOE implementation based on both organizational and timeline constraints and the calculated risks/benefits for each vulnerability.

The short-term evaluation of the team phase provided generally positive immediate feedback to those who oversaw the PRA. Unlike Bonnabry et al [16] and Faye et al [15], this PRA “closed the loop” by providing feedback on the long-term effectiveness of the PRA. Long-term ongoing evaluation continues to provide feedback on the effectiveness of this PRA method.

Throughout the PRA, IT management and staff provided extensive input and direction while HFE researchers developed and facilitated the method. We believe this contributed to the efficiency with which the PRA was organized, conducted and evaluated in both the short- and long-term. The PRA confirmed that continued stakeholder involvement in the development of user-friendly health IT that supports critical workflows (e.g., order management during patient transitions of care) is extremely valuable [26, 27]. Modifying an established PRA method (e.g., FMEA), however, requires that those leading the effort must have knowledge and an understanding of PRA requirements (e.g., clearly defining a process, analyzing workflows, identifying and rating vulnerabilities, determining potential solutions) and know how to functionally incorporate the method within an organization’s and the stakeholders’ time and resource constraints. Likewise, content expertise related to the process scrutinized (OR to ICU transfer) increased the planning team’s credibility in the eyes of the stakeholders. Finally, following the “ground rules” previously identified by the research team helped guide the various phases of the PRA.

This PRA provided a structured means of identifying vulnerabilities in CPOE design and the anticipated workflows associated with CPOE. The vulnerabilities identified were prioritized, addressed, and resolved. Heightened awareness of the identified vulnerabilities promoted continuous focused monitoring of the OR-ICU transfer and its impact on CPOE. Resolution of issues identified and/or confirmed generally occurred promptly. Long-term evaluation provides a means of continued monitoring for the issues identified and a means of identifying issues the participant did not foresee. In our study, no unanticipated issues arose but, if any had occurred, the IT team and those involved in the PRA long-term would have used this as a source of feedback to improve future PRAs.

One limitation of this study relates to the challenge associated with identifying and then clearly defining a process that meets the ground rules we followed [20]. This effort consumed five 30-minute meetings between three people. Although this commitment of time should not be overlooked, this was the first attempt to utilize this PRA method at GMC. We also recognize that conducting the team phase during one long meeting may be a limitation. More vulnerabilities may have been identified if additional meetings were held and team members had more time to reflect on both the proposed workflow and interface. However, we faced a number of constraints including time and staff availability that curtailed our ability to convene more than one meeting. An additional limitation of this study is that the participants at the various phases had limited knowledge of PRA. We dealt with this by sharing PRA knowledge while also facilitating the various phases. In this manner participants gained knowledgeable of PRA concurrently. Likewise none of the participants in the team phase previously viewed the interface or considered the proposed workflow. Organizers agreed that no team phase participants would have an advantage over the others regarding familiarity with the proposed system. Finally, although we were able to demonstrate benefit of this PRA method at GMC, we do not know if some other method may have proven equally or more effective.

Because of the resources necessary for PRA (e.g., time, expertise, commitment), it may not be possible to apply this method to implementation process of significantly larger scope. Attempting to conduct PRA on an organization-wide full-scale implementation of health IT should require identification of high-risk processes and a means of prioritizing them. Specific high-risk processes could then undergo a PRA, following a method such as we discuss here.

5. Conclusion

In this paper we describe the use of a proactive risk assessment prior to implementation of health IT, specifically CPOE. Feedback from participants indicated they found value in the method. The corrective actions taken before and immediately after implementation of CPOE confirmed the potential existence of the vulnerabilities identified. Prioritizing the vulnerabilities allowed those with the potentially most negative consequences to be corrected prior to or promptly after implementation. The value and success of the PRA as part of EHR implementation received significant attention at GMC and as a result, the method we discuss here, without overtime and its associated costs, is now incorporated in every health IT project plan.

Further research in this area should continue to address the need to better execute PRAs by understanding and assessing their long-term impact. Post-implementation monitoring for consequences of identified vulnerabilities that may have been inadequately addressed, as well as vulnerabilities not identified in the PRA provides a mechanism for continuous improvement of the PRA method, the specific process the PRA focuses on (in our case, CPOE associated with the OR-ICU transfer process) and the health IT overall.

This work demonstrates the value of conducting PRA in addition to performing workflow analyses and usability evaluations in anticipation of health IT implementation. Workflow analysis is generally incorporated in PRAs and we, like others [28] found that simulation of the technology promotes better participant understanding of the process being reviewed. PRAs can be efficiently conducted by recognizing the expertise required at the various phases discussed in this paper. PRAs can also provide significant information for the IT design team and others involved in an organization’s health IT implementation.

Summary points

What is known about this topic:

  • Negative consequences commonly result with health IT implementation.
  • PRA methods provide useful means of identifying known or potential problems.
  • Many PRA methods require extensive commitment by team members.
  • The long-term effectiveness of PRA methods has not been examined for health IT implementation.

What this study added to our knowledge:

  • Effective PRA methods can be developed and conducted by recognizing and then incorporating methodological and organizational constraints.
  • PRA identifies vulnerabilities before CPOE implementation which can be addressed through technology redesign, training, process redesign, and monitoring.
  • The long-term effectiveness of the PRA method was demonstrated as no new vulnerabilities were identified post-CPOE implementation.


  • An exploratory study examining perceived attitudes and readiness of older women to adopt a mobile phone-based intervention
  • A telephone survey was developed from concepts identified from various user acceptance theories and models.
  • Partial least square and boot-strapping procedures were used to estimate the significance of the path coefficients.
  • Validated some ageing-specific and female-posited variables to suggest as constructs in future studies about older women.

Supplementary Material


This research was made possible by funding from the Agency for Healthcare Research and Quality (AHRQ). Grant Number: R01 HS15274. Principal Investigator: Pascale Carayon, Ph.D., and was supported by the Clinical and Translational Science Award (CTSA) program, previously through the National Center for Research Resources (NCRR) grant 1UL1RR025011, and now by the National Center for Advancing Translational Sciences (NCATS) grant 9U54TR000021. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Appendix A – PRA Evaluation Survey

1. Please indicate your primary role at Geisinger Health System:

     * MD clinician          * IT staff
     * RN clinician          * other (specify) ______________________

2. What portion(s) of the PRA were you present for? (please check all that apply)

     * Introduction
     * Walk-through and demonstration of [system]/workflow
     * Vulnerability identification
     * Prioritization (this includes “clean up” of vulnerability table)
     * Root cause identification (use of fault tree analysis tool)
     * Solution development

In light of the portion(s) of the PRA in which you participated, please answer the following

3. How useful was the PRA?

          Completely                         Extremely
           useless                             useful
              1        2        3        4        5


4. How willing are you to participate in a PRA again?

          Not at all                         Extremely
           willing                            willing
             1        2        3        4        5


5. In hindsight, what would you have done to make this PRA better?


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1Activities in highlighted boxes are performed by more than one group, as denoted by the overlap of the box.

2* STAT: abbreviated form of Latin term (“statim”) meaning “immediately”. In medicine, and at GMC, this refers to medication orders being immediately processed, filled and administered; and to a specimen being immediately collected, processed and then reported. “Now” orders refer to an order for a specimen to be immediately drawn but results are reported as part of scheduled results reporting.

Authors’ contributions

Dr. Hundt, Ms. Adams and Mr. Douglas were responsible for designing and conducting the PRA. Dr. Carayon was principal investigator for the overall study and provided input in the design and evaluation of the PRA method. Ms. Adams, Ms. Musser and Mr. Schmid monitored the “issues list” for vulnerabilities post-CPOE implementation. Dr. Walker was project sponsor for the study and coined the term “watchful waiting” as applied to PRA. Dr. Wetterneck provided input in the evaluation of the PRA method. Ms. Paris participated in the PRA. Dr. Hundt was involved in all aspects of the PRA and took primary responsibility for writing the manuscript which was reviewed and edited by all of the authors.

Conflict of interest

The authors have no conflict of interest to report.


1. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. Journal of the American Informatics Association. 2004;11(2):104–112. [PMC free article] [PubMed]
2. Ash JS, Sittig DF, Dykstra RH, Guappone KP, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. International Journal of Medical Informatics. 2007;76(Supplement):S21–S27. [PubMed]
3. Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. Journal of the American Medical Association. 2005;293(10):1197–1203. [PubMed]
4. Horsky J, Kuperman GJ, Patel V. Comprehensive analysis of a medication dosing error related to CPOE. Journal of the American Medical Informatics Association. 2005;12(4):377–382. [PMC free article] [PubMed]
5. Carayon P, Schoofs-Hundt A, Karsh B-T, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: The SEIPS model. Quality and Safety Health Care. 2006;15:50–58. [PMC free article] [PubMed]
6. Israelski EW, Muto WH. Human factors risk management in medical products. In: Carayon P, editor. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. Mahwah, NJ: Lawrence Erlbaum Associates; 2007. pp. 615–647.
7. Carayon P, editor. Handbook of Human Factors in Health Care and Patient Safety. Mahwah, New Jersey: Lawrence Erlbaum Associates; 2007.
8. Lyons M. Towards a framework to select techniques for error prediction: Supporting novice users in the healthcare sector. Applied Ergonomics. 2009;40(3):379–395. [PubMed]
9. Carayon P, Faye H, Hundt AS, Karsh B-T, Wetterneck TB. Patient safety and proactive risk assessment. In: Yuehwern Y, editor. Handbook of Healthcare Delivery Systems. Taylor & Francis; 2010.
10. Nagpal K, Vats A, Ahmed K, Smith AB, Sevdalis N, Jonannsson H, et al. A systematic quantitative assessment of risks associated with poor communication in surgical care. Archives of Surgery. 2010;145(6):582–588. [PubMed]
11. Pearse RM, Dana EC, Lanigan CJ, Pook JA. Organisational failures in urgent and emergency surgery - A potential peri-operative risk factor. Anaesthesia. 2001;56(7):684–689. [PubMed]
12. Friesdorf W, Gross-Alltag F, Konichezky S, Schwilk B, Fattroth A, Fett P. Lessons learned while building an integrated ICU workstation. International Journal of Clinical Monitoring and Computing. 1994;11(2):89–97. [PubMed]
13. Abraham J, Reddy M. Challenges to inter-departmental coordination of patient transfers: A workflow perspective. International Journal of Medical Informatics. 2010;79:112–122. [PubMed]
14. Wetterneck TB, Skibinski KA, Roberts TL, Kleppin SM, Schroeder ME, Enloe M, et al. Using failure mode and effects analysis to plan implementation of smart i.v. pump technology. Am. J. Health. Syst. Pharm. 2006;63(16):1528–1538. [PubMed]
15. Faye H, Rivera-Rodriguez AJ, Karsh BT, Hundt AS, CBaker C, Carayon P. Involving intensive care unit nurses in a proactive risk assessment of the medication management process. The Joint Commission Journal on Quality and Patient Safety. 2010;36(8):376–384. [PMC free article] [PubMed]
16. Bonnabry P, Despont-Gros C, Grauser D, Casez P, Despond M, Pugin D, et al. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. Journal of the American Medical Informatics Association. 2008;15(4):453–460. [PMC free article] [PubMed]
17. Schroeder ME, Wolman RL, Wetterneck TB, Carayon P. Tubing misload allows free flow event with Smart intravenous infusion pump. Anesthesiology. 2006;105(2):434–435. [PubMed]
18. Habraken MMP, Van der schaaf TW. Healthcare Systems Ergonomics and Patient Safety 2008. Vol. 2008. Strasbourg, France: 2008. Jun 25–27, Requirements for prospective risk analysis in health care. 2008.
19. Wetterneck TB, Skibinski K, Schroeder M, Roberts TL, Carayon P, editors. Challenges with the performance of failure mode and effects analysis in healthcare organizations: An IV medication administration HFMEATM. Annual Conference of the Human Factors and Ergonomics Society; 2004; September 20–24, 2004; New Orleans, LA: The Human Factors and Ergonomics Society; 2004.
20. Wetterneck TB, Hundt AS, Carayon P. FMEA team performance in health care: A qualitative analysis of team member perceptions. Journal of Patient Safety. 2009;5(2):102–108. [PubMed]
21. Ashley L, Armitage G, Neary M, Hollingsworth G. A practical guide to failure mode and effects analysis in health care: Making the most of the team and its meetings. The Joint Commission Journal on Quality and Patient Safety. 2010;36(8):351–358. [PubMed]
22. Walker JM, Bieber EJ, Richards F, editors. Implementing an Electronic Health Record System. London: Springer-Verlag; 2005.
23. Sandelowski M. Sample size in qualitative research. Research in Nursing & Health. 1995;18(2):179–182. [PubMed]
24. Weick KE, Sutcliffe KM. Managing the Unexpected: Assuring High Performance in an Age of Complexity. San Francisco, CA: Jossey-Bass; 2001.
25. Kessels-Habraken M, Van der Schaaf T, De Jonge J, Rutte C. Defining near misses: Towards a sharpened definition based on empirical data about error handling processes. Social Science & Medicine. 2010;70(9):1301–1308. [PubMed]
26. Karsh B-T. Beyond usability: Designing effective technology implementation systems to promote patient safety. Quality and Safety in Health Care. 2004;13(5):388–394. [PMC free article] [PubMed]
27. Stead WW, Lin HS, editors. Computational Technology for Effective Health Care. Washington DC: The National Academies Press; 2009.
28. Davis S, Riley W, Gurses AP, Miller K, Hansen H. Failure modes and effects analysis based on in situ simulations: A methodology to improve understanding of risks and failures. In: Henriksen K, Battles JB, Keyes MA, editors. Advances in Patient Safety: New Directions and Alternative Approaches. Rockville, MD: Agency for Healthcare Research and Quality; 2008. [PubMed]