|Home | About | Journals | Submit | Contact Us | Français|
Failure mode and effects analysis (FMEA) is a widely used tool for prospectively evaluating safety and reliability. We report our experiences in applying FMEA in the setting of radiation oncology.
We performed an FMEA analysis for our external beam radiation therapy service, which consisted of the following tasks: (1) create a visual map of the process, (2) identify possible failure modes; assign risk probability numbers (RPN) to each failure mode based on tabulated scores for the severity, frequency of occurrence, and detectability, each on a scale of 1 to 10; and (3) identify improvements that are both feasible and effective. The RPN scores can span a range of 1 to 1000, with higher scores indicating the relative importance of a given failure mode.
Our process map consisted of 269 different nodes. We identified 127 possible failure modes with RPN scores ranging from 2 to 160. Fifteen of the top-ranked failure modes were considered for process improvements, representing RPN scores of 75 and more. These specific improvement suggestions were incorporated into our practice with a review and implementation by each department team responsible for the process.
The FMEA technique provides a systematic method for finding vulnerabilities in a process before they result in an error. The FMEA framework can naturally incorporate further quantification and monitoring. A general-use system for incident and near miss reporting would be useful in this regard.
A safe and effective radiotherapy treatment necessarily consists of treating the correct tissue in the correct patient with the correct dose. This is to be accomplished in one of the most complex settings in healthcare—one that is steadily growing more complex. Reported error rates have ranged from 0.06% to 4.66%, depending on how errors are quantified (1–5). Although most of these errors are minor and treatment can be adjusted to allow effective therapy, there is a strong incentive to reduce these rates even further, given the potentially catastrophic nature of an error.
The discipline of radiation oncology has many well-established methods in place to prevent errors or to mitigate their effects. Examples include independent checks of delivered dose via diodes on the patient’s skin (6), weekly review of patient films and charts, and standard quality assurance measures designed to uncover more systematic errors (7). What is less well established in the field of radiation oncology is the view of systems-wide safety and the associated methods for systematic risk analysis and improvement.
Systematic risk analysis is a powerful tool that can help to identify vulnerabilities in a specific clinic and to suggest where resources might be concentrated for the most significant impact. Such methodologies are now in regular use in other healthcare disciplines: for example, to prevent chemotherapy errors (8), to improve performance in trauma (9), or to improve the design and use of equipment (10). The Joint Commission on Accreditation of Healthcare Organizations has also called for regular prospective safety analyses (11).
Here we report our experience with one particular safety improvement methodology, failure mode and effects analysis (FMEA). The FMEA technique is a well-established tool for safety analysis and improvement (12). It is widely used in manufacturing, where many vendors use it to fulfill the requirements for prospective risk analysis by the Food and Drug Administration (21 CFR 820) and the Internal Organization for Standardization 9000 standards. The FMEA technique is also beginning to be used in other disciplines in healthcare (8–10). The goal of this article is to present an overview of FMEA as specifically applied to a radiation oncology clinic. A similar report by American Association of Physicists in Medicine Task Group 100 has appeared, which describes a basic outline of FMEA and two example applications in radiation oncology (13).
The initial analysis presented here was performed over a 3-month period, starting in January 2007. The steering committee consisted of 11 members, representing a cross-section of department staff including administrators, nurses, clinical research coordinators, radiation therapists, physicists, information technologists, resident physicians, and attending physicians. The committee was further assisted by staff from the Johns Hopkins Center for Innovation in Quality Patient Care.
A first step in FMEA (or, indeed, in any systematic risk analysis), is to generate an overview of the entire process, a document that we refer to as a process map, see for example the “process tree” in Huq et al. (13). To generate this, it is helpful to consider the workflow from the patient’s point of view and to record each event that happens to that patient or that patient’s medical record. In most health-care process mapping, the focus is only on those parts of the process that directly interact with the patient. Our process map could be divided into four subprocesses: patient consult, simulation, treatment planning, and patient treatment. Three of these four directly involve the patient. Treatment planning, although it does not involve direct interaction with the patient, does rely on image data that acts as a kind of “surrogate” for the actual patient. Because treatment planning is a critical part of the radiation therapy process, we track the medical record through that subprocess. Our maps were reviewed and revised by both the 11-member committee and the teams responsible for each part of the process.
Using the process map as a guideline, we undertook the whole FMEA. This consisted of first identifying the possible failure modes in the process. We interviewed each group in the department that was responsible for part of the process and asked the group members to identify possible failure modes, that is, vulnerabilities in the process map.
The goal of FMEA is to rank these failure modes in order of importance. The FMEA ranking is a semiquantitative approach in which each failure mode is assigned a score consisting of three attributes: (1) severity if the failure occurs, S; (2) probability of occurrence of this particular failure, O; and (3) the probability that this particular failure would go undetected, D. For the last category, D, a higher score means that the failure is less likely to be detected. A 10-point scale was used for scoring each of these categories, with 10 being the most severe, most frequent, or least detectable. Huq et al. use a similar 10-point scoring system (13). To determine scores for frequency of occurrence, O, and probability of not detecting, D, we polled the individuals closest to that particular potential failure mode and took an average of their scores. Severity scores, S, were assigned by consensus with the entire committee present. Table 1 outlines the basis for assigning these scores and the numerical values that we used. Using these scores, a risk probability number (RPN) was calculated for each failure mode (12). The failure modes are ranked numerically by RPN, with higher scores being more important to mitigate first.
One important step in performing FMEA is to establish the scoring scale. This is particularly important when an FMEA is performed, as here, based on qualitative scoring. All parties submitting the scores must agree on what constitutes a 10, for example, in severity, frequency, or detectability. We used the values shown in Table 1 as a guideline. It is to be noted that the overall normalization of the scale is unimportant because the final goal is simply to rank risk scores for failure modes relative to each other. What is more important is that all parties understand and adhere to the same, uniform scoring scale.
The final aspect of the analysis is to develop process improvements that can lower the overall risk score for each failure mode. This is accomplished by either decreasing the occurrence frequency (ideally making it impossible for that particular error to occur), by increasing its detectability and thereby lowering the detectability score, or by decreasing its severity if it does occur (e.g., stopping treatment after one fraction if diode readings indicate an error). Accepted and proven steps to mistake-proofing are as follows (14–19): (1) eliminate the task or part, thereof, (2) replace the task or part with a more reliable one, (3) engineer the task or part to make the error impossible, (4) make work easier to perform (5) make variations and deviations more obvious, and (6) minimize the effects of errors.
We considered failure modes in order of decreasing RPN. The full committee was convened to develop potential process improvements for each of these failure modes. Suggestions were also sought from the groups directly involved in each potential failure mode.
Once these potential process improvements were established and recorded, the task was to sort them and determine which should be implemented. To do this, the full committee assigned two scores to each solution, one for feasibility and one for effectiveness. This was done on a 10-point scale, with 10 being most feasible or most effective. Solutions that had both a score for feasibility and effectiveness that was scored as greater than 5 were considered for immediate implementation. Solutions with a high effectiveness score but a low feasibility score were considered for implementation on a longer time scale, with the idea that, although effective and worthwhile, these solutions would take more time and/or effort to implement.
Presentations were then set up with individual groups that would be responsible for implementing the suggested changes. The suggested change was presented to the group for discussion and possible implementation. In some cases, the changes were rejected because the group thought they were impractical.
Our process map consisted of 269 separate nodes, each indicating an action to be taken or information to be manipulated in some manner (Fig. 1). The map falls naturally into four categories: patient consult, simulation, treatment planning, and treatment. An example of one section of the map is shown in Fig. 2. Note that this section is part of a much larger map and represents approximately 10% of the total number of nodes. Nevertheless, even with such a large process map we emphasize that the detail is relatively coarse. One square, for example, indicates “export DRR to R&V system”; another square in another part of the map indicates “port films are obtained.” Each of these processes itself, of course, contains many steps. To maintain a map of manageable scale, however, we elected to not consider the processes at a finer resolution.
We identified 127 total possible failure modes and assigned a risk score to each one as outlined above. The RPN scores ranged from 2 to 160. Typical of FMEA (12), our highest score is much lower than the maximum possible score of 1,000 (RPN = S × O × D = 10 × 10 × 10). This is a reflection of the fact that very severe events rarely occur and their detectability is considered high given the checks already in place. Table 2 shows the failure modes that had the top five RPN scores in our analysis. In our treatment, the 15 top-ranked failure modes were considered for interventional solutions. This represents RPN scores as low as 75. These top-ranked nodes are highlighted in black in Fig. 1, where it can be appreciated that failure modes are not restricted to only one area of the process map.
We highlight a few of the failure modes listed in Table 2 as examples of the analysis and the method for arriving at solutions. Each failure mode is associated with a cause. In the first failure mode listed in Table 2, the patient would be treated at an incorrect location because a digitally reconstructed radiograph (DRR) was generated for the wrong isocenter location. Based on this DRR image, the patient would be shifted and treated at the wrong isocenter. It is easy to envision how this might happen if a patient’s treatment plan contained numerous points of interest to which treatment beams could be assigned. As with almost all of the failure modes identified, already there are checks in place to prevent the failure from occurring. These include second checks of the DRR by a physicist, a physician, and a radiation therapist. Marks or tattoos on the patient’s skin should be consistent with the DRR plus any shifts from the marked location. Because of these checks, the detectability score for this failure mode is fairly low (4 = “easy to detect”; Table 1). However, the overall FMEA risk probability number is high because the severity of this event, if it did happen, is considered quite high (8, corresponding to a potentially large dose deviation), and this failure mode has been observed.
The proposed solutions for this particular failure mode included the following: (1) color code isocenters so that everyone always expects the isocenter point to appear on the DRR with a particular color; (2) implement an explicit checklist for radiation therapists that includes a check of the isocenter coordinates in the treatment plan; (3) implement a regular continuing education program for radiation therapists that reviews the issue of reading and checking treatment plans; and (4) eliminate the step of having the physics postdoctoral residents import the DRR into the record and verify (R&V) system (Fig. 1). This last solution is interesting because it is an example of simplifying the process. The physics residents were originally assigned to DRR import to help with an R&V interface that was extremely cumbersome. With improved R&V software, however, it became possible to eliminate this step and have the DRR exported directly into the R&V system from the treatment planning software by the treatment planner. This eliminated one person in the chain and thereby decreased the chances of miscommunication or error that could lead to this failure mode.
None of the proposed solutions made the DRR failure mode impossible, but they were thought to make it more detectable and less frequent. These solutions listed above all had both high feasibility and effectiveness scores. There were, however, proposed solutions that were ranked low in either feasibility, effectiveness, or both and were not considered further. An example was the suggested possibility of bringing a patient back into the computed tomography simulator just before that patient’s first treatment for marking and verifying the final isocenter point. This would reduce the reliance on DRRs. This solution, however, was deemed impractical with current clinical operations.
Another example of a failure mode from Table 2 is one in which a patient would be treated with the wrong beams because the wrong patient record was pulled up in the R&V system at the time of treatment. This lead us to the following solutions: (1) verify the patient’s name before treatment; (2) standardize beam names to include information on patient’s anatomy as a further obvious indicator; (3) display the patient’s picture in the treatment room, not just in the accelerator control room as was done at the time; and (4) implement software that uses a bar-code reader and automatically pulls up the patients record when the bar code (attached to the patient’s ID card) is scanned.
In all, 28 solutions were proposed to address the 15 potential failure modes with the top RPN scores. Of these changes, 12 were to be performed primarily by physics/dosimetry, 10 by therapists, one by information technologists, two by nursing staff, and three by physicians, although there was significant overlap with multiple groups sharing responsibility for various solutions. Implementation is ongoing.
A systematic consideration of factors related to reliability and safety can reduce error rates. This is germane to the generally recognized need to improve safety in healthcare (20). Anesthesiology is an often-cited success story in this regard. By actively using safety measures, mortality rates have been reduced by more than a factor of 20 in the last 25 years (21). A handful of studies have measured the error rates of delivering radiotherapy treatment fields (1–5). There is a wide variability in the details of reporting, but these studies typically quote failure rates of roughly 0.5% of all treated fields (1). Although this number might appear to be small, it must be considered in context. It has been noted that even a 0.1% rate of process failures would result in two unsafe landings per day at O’Hare airport and 880,000 credit cards produced each year in the US with the incorrect information on their magnetic strips (12). The measurements of treatment field delivery errors are also surely an underestimate of total error rates, given all the possible failures that can arise in the process of delivering an external beam radiotherapy treatment. Here we explored one particular prospective tool for improving patient safety in healthcare, namely, FMEA. The FMEA technique is widely used in manufacturing (12) and is beginning to be used in various branches of medicine as well (8–10).The AAPM Task Group 100 is also considering this tool for quality assurance improvement and recently issued a report describing it (13).
Several learning experiences were worthy of note. First, although many improvement projects using the FMEA tool do not explicitly consider a visual process map (12), we found this to be extremely useful in identifying problem areas and putting them in a context that all parties involved in the process could appreciate. This is also a recommendation of Huq et al. (13). Process maps have appeared before in the context of radiation therapy; a simple example can be found in the report of AAPM Task Group 59 on high dose-rate brachytherapy (22) and in the preliminary report of Task Group 100 (13). Our process map was quite large, consisting of 269 separate nodes. With this level of complexity, a map can be very helpful. The map also serves as an active document that can be updated and maintained as processes change and new technologies are introduced over time.
Another important lesson relates to the FMEA scoring system. There are other simpler methods for ranking and selecting failure modes (12), but the RPN method is somewhat more objective. It is, however, critical to establish an explicit scoring scale on which all parties agree. It is also necessary to determine how many failure modes will be considered for intervention. Our cutoff at an RPN score of 75 was somewhat arbitrary and was motivated largely by the need for a manageable number of solutions. Standard texts suggest that RPN scores greater than 100 should be flagged when performing an FMEA (12); but the absolute values of RPN scores depend on the scaling used for severity, occurrence, and detectability, and thus no definite statement can be made about cutoff values.
The FMEA analysis presented here required approximately 5 months for completion, meeting once weekly. Of this, approximately 2 months were spent generating a process map, 1 month was spent scoring and performing FMEA analysis, and the remainder was used to develop solutions and to settle on implementation. Use of FMEA analysis is therefore a substantial investment. The overall process of gathering individuals from across the department to discuss the radiation therapy process, however, is a valuable educational tool and, in our experience, frequently helped to highlight the difference between how steps were supposed to happen vs. how they actually happened.
The FMEA technique offers several advantages as a safety analysis tool. Most importantly, it allows one to identify vulnerabilities before failures actually occur. Second, it allows one to explicitly consider the severity and detectability of a failure mode in addition to its occurrence frequency. A number of studies have appeared that carefully measure the occurrence frequency of various delivery errors (1–5). These studies, however, have been less explicit in considering the severity or detectability associated with these errors. The FMEA tool provides a mechanism for incorporating this information.
Another advantage of FMEA is that it naturally encourages one to think beyond the confines of a particular part of the process. An examination of R&V-related delivery errors, for example, considers only one small part of the overall process of designing and delivering external beam radiotherapy. The FMEA provides a framework for considering this larger process as a whole. In addition, the FMEA results also serve as a “living document” that can be easily updated and modified as process or error rates change (12).
Our treatment of FMEA has several limitations. One is that individuals and groups were sometimes slightly biased by recent incidents or events when suggesting failure modes. In general, this may be good, because known failure modes are then addressed and considered. Focusing too much on these, however, may cause other important failure modes to be overlooked. We believe that our review process minimized this problem.
A second limitation is that the solutions to particular failure modes tended, on the whole, to add tasks or information to the process. This occurred in an effort to make potential failures more visible. Every attempt was made to not add onerous tasks. This approach must be used carefully, as the safety literature has shown that more process steps can lead to more errors. We note that the tendency to add tasks is not inherent to FMEA itself but, rather, is an issue with how the group uses the results of the FMEA to modify the process. Other quality improvement techniques, used in conjunction with the FMEA approach describe here, may be very helpful in finding the most effective process design for a given department (23).
A third limitation of our study is the semiquantitative nature of the scoring system (Table 1). The Huq et al. report from AAPM Task Group 100 (13), suggest values for the scoring system as follows: O (1 = 1 per 104, 10 = 5%), S (1 = “no appreciable danger,” 10 = “catastrophic”), D (1 = <0.01%, 10 = 20%). The difficulty is that exact quantitative values are often not available. We therefore relied on a human assessment of severity, occurrence, and detectability, which may be unreliable. This is not a fundamental limitation of FMEA, however. More quantitative measures are certainly possible. This is illustrated by previous studies of error rates (1–5), which demonstrate that it is possible to acquire high-quality data in the radiation oncology environment.
This last point underscores the need for a system of safety-related data collection for the radiation oncology field. As complex and contentious as such a system might be, it could serve the field well. Reports on errors and near misses have been vital to the improved safety in the airline industry, starting with the Aviation Safety Reporting System (ASRS) administered by the National Aeronautics and Space Administration (NASA) since the 1970s and later modified and adopted by the airlines themselves as the Aviation Safety Airways Program (ASAP). An excellent review of these nonmedical reporting systems can be found in Barach et al. (24). The Institute of Medicine 2000 report also advocates the use of mandatory incident reporting and voluntary near miss reporting as a means of improving safety (20). Such reporting systems are beginning to be used in medicine. An example is the Veteran Administration’s Patient Safety Reporting System (PSRS), launched in 2002 and modeled on the NASA ASRS system (25). Other disciplines and settings are also piloting such measures: for example, in the intensive care unit (26), perioperative nursing (27), transfusion services (28), and nursing homes (29). Individual hospital-based patient safety reporting systems are also available in various forms. A well-designed reporting system that is specific for radiation oncology could prove an important tool.
Data on near misses and other process deviations (as opposed to actual failure incidents) can provide valuable information in radiation oncology. To take a specific example, consider a case in which a physics check of a patient’s chart is not performed. The patient’s treatment may not be adversely affected, but such an omission does reflect the quality of the process and can provide information for improvement. To take another example, consider a case in which a major setup error is noticed on the review of image data. In this case, a mistreatment is prevented from happening. This represents a near-miss. Typically data on such near-misses and process deviations are not gathered in any systematic manner. A multi-institutional network that systematizes this, perhaps with vendor support, would be a step in this direction. The FMEA tool could provide a system for explicitly incorporating such data by way of the detectability score.
The authors thank the following individuals for their active participation in this failure mode and effects analysis process: Ruth Bell, R.T.T., Daryl Carson, B.S., Kelly Cavallio, J.D., Timothy Chan, M.D., Andrea Cox, R.N., Marsha Goldberg, R.T.T., Mary Ellen Holland-Callender, R.T.T., Richard McCarthy, B.S., and members of the Johns Hopkins Center for Innovation in Quality Patient Care.
Conflict of interest: none