|Home | About | Journals | Submit | Contact Us | Français|
Clinical protocols to standardize care may improve patient outcomes but worsen trainee education. Our objective was to determine the availability of clinical protocols in teaching medical ICUs.
We administered an electronic questionnaire regarding protocol availability in five specific clinical areas. All directors of adult medical ICUs in US teaching hospitals were eligible to participate.
The response rate was 70%. 86% of ICU directors reported availability of protocols for ventilation liberation, 73% for sedation, 62% for sepsis resuscitation, 60% for lung-protective ventilation, and 48% for life support withdrawal. Ventilation liberation protocols are most often started and driven by respiratory therapists (40% and 90%); sedation started by residents (41%) and driven by nurses (95%); sepsis resuscitation started and driven by residents (49% and 46%); lung-protective ventilation started by attending physicians (39%) and driven by respiratory therapists (67%); and life support withdrawal started by attending physicians (93%) and driven by nurses (47%).
There is wide variation in clinical protocol availability among teaching hospitals. Further study of the effect of protocols on education is needed.
Clinical protocols are formal pathways with specific inclusion and exclusion criteria that provide standardized algorithms for caring for patients with specific conditions. Protocols are commonly used to help implement evidence-based therapies and reduce unnecessary practice variation in the intensive care unit (ICU). Some evidence suggests that protocol use is associated with improved patient-centered outcomes, especially in the areas of sedative administration and weaning from mechanical ventilation.1–3 As a result, many international professional societies promote the use of clinical protocols in their campaigns to improve the quality of critical care.4 However, the overall benefits of routine protocol use in the ICU are still widely debated.5 Proponents argue that protocols help to standardize care and improve patient outcomes by facilitating evidence-based practice.3, 6 7 Opponents argue that protocols discourage clinical reasoning resulting in what is sometimes called “cookbook medicine”8, 9 and note that protocols are not consistently associated with improved patient outcomes in all studies.7
Additionally, some individuals express concern about the impact of protocol use on the training of new physicians. Practice-based learning is a core tenet of medical education. By creating a disconnect between trainees and the patient, protocols may rob trainees of important clinical experiences and thus impair acquisition of new skills.10 Protocols may also prevent trainees from seeking assistance from senior physicians about complex management issues, eliminating an opportunity for supervised education. Although one study showed no difference in trainee performance on standardizing questions about respiratory care whether or not they trained in an institution with respiratory protocols11, the overall effect of protocols on medical education is as yet unknown.
A first step to understanding the potential impact of clinical protocols on ICU education is to understand the prevalence of protocol availability in teaching ICUs and the ways in which these protocols are used in daily practice. The purpose of this study was to determine the availability of clinical protocols in medical ICUs of United States (US) teaching hospitals and the hospital and ICU characteristics associated with their use. To do so, we conducted a nationwide survey of medical directors of teaching medical ICUs regarding their use of clinical protocols.
We performed an internet-based survey of directors of adult medical ICUs in United States teaching hospitals. All 129 primary training sites for accredited US pulmonary and critical care fellowship programs in 2008 were eligible to participate. We specifically selected the ICUs of fellowship training programs because these ICUs are most likely to have a strong educational mission and highly prioritize evidenced-based practice and training. We obtained a list of all training programs from the web site of the Accreditation Council of Graduate Medical Education.12 We then identified the primary teaching ICU of each program by searching the websites of each program. We assembled a list of email addresses of the medical directors of each ICU through web searches, contact with the fellowship program, and direct contact with the ICUs. In addition, we obtained data about hospital and ICU characteristics from the 2005 American Hospital Association (AHA) Annual Survey.
To develop the survey we first conducted semi-structured interviews of four medical directors of teaching ICUs. The purpose of the interviews was to develop a standard definition of clinical protocols distinct from clinical pathways or decision support tools, to identify the key content items of the survey, and to develop a preliminary understanding of which protocols they consider important in their own ICUs. We created an initial draft of the survey based on these interviews. For further content and construct validity we assembled two separate focus groups of attending and trainee physicians, one with physicians trained in critical care medicine (8 participants) and one with physicians of various clinical backgrounds (6 participants). Focus groups provided feedback about the content, form, and structure of the questionnaire. We then piloted the questionnaire electronically to ten Canadian ICU clinicians experienced in survey research, protocol implementation and ICU management. This population was chosen because they would not otherwise be included in the main study population but their medical practices and training programs are very similar to that of the United States. These respondents answered the questions as would a study participant but also had an opportunity for question-specific and general feedback. The survey was revised and finalized based on their feedback.
The final survey was 36 items and could be completed in 5–7 minutes (see Appendix). Based on the interviews and focus groups, we defined a protocol as “written, standardized orders that allow a nurse or respiratory therapist to evaluate the patient and change a therapy accordingly” and assessed availability in five specific areas: liberation from mechanical ventilation, sedation during mechanical ventilation, early-goal directed therapy for severe sepsis, lung-protective ventilation for acute lung injury, and withdrawal of life support. We selected the first four of these because we believe that there is a sound evidence base to support best practices that may be associated with improved clinical outcomes in the ICU1, 2, 7, 13, 14 and the last one because protocol-driven withdrawal of care may improve clinician satisfaction with the dying process.15 The survey assessed four domains regarding protocol use: the availability of the protocol for routine clinical use, the length of time the protocol had been in place, the clinician most responsible for initiating the protocol, and the clinician most responsible for driving the protocol. We specifically did not ask how frequently the protocol was used in clinical practice, as we did not think that physicians’ assessment of their own behavior would be reliable.16, 17 For each protocol we offered hypothetical clinical scenarios to aid in respondent interpretation. We also collected basic ICU characteristics.
The survey was administered between July, 2008, and September, 2008 using a standardized internet survey application (SurveyMonkey, Portland, Oregon). All eligible participants were sent a link to the survey by electronic mail. We e-mailed reminders to non-respondents at intervals of two weeks for a total of eight weeks. At the end of eight weeks, we contacted non-respondents by phone to request that they complete the survey. No incentives were offered.
Hospital and ICU characteristics between respondents and non-respondents were compared using t-tests and chi-square tests, as appropriate. Self-reported ICU characteristics of respondents, availability of clinical protocols and protocol characteristics were analyzed using standard summary statistics. We used chi-square tests to study the association of the individual protocols with the following individual hospital and ICU characteristics, modeled as binary variables: teaching intensity (resident-to-bed ratio ≤0.6 vs. >0.618, 19), number of hospital beds (≤500 vs. >500), number of ICU beds (≤18 vs. >18), and duration of ICU directorship (≤5 years vs. >5 years). The results are presented as a ratio of clinical protocol use, with a 95% confidence interval. To understand if having one individual protocol was correlated with having other individual protocols, we estimated a kappa statistic for each pair of protocols. We interpreted the kappa as follows: 0.01–0.20, slight concordance; 0.21–0.40, fair concordance; 0.41–0.60, moderate concordance; 0.61–0.80, substantial concordance; and >0.8, almost perfect concordance.20
Responses of “I’m not sure” were recoded as “no” for yes/no items and were dropped for multiple choice items. The questionnaire was designed to require an answer to continue to prevent missing responses.
For all statistical analyses, α=0.05 was considered the threshold for significance. All statistical analyses were performed using Stata 10.0 (StataCorp; College Station, Texas). This project was reviewed and approved by the University of Pennsylvania Institutional Review Board.
A total of 90 ICU directors responded to the survey (final response rate: 70%). . Hospitals represented by respondents (n=90) were similar to those of non-respondents (n=39). Most hospitals in both groups had greater than 250 beds (97% of respondents and 95% of nonrespondents),, were either private, non-profit or government hospitals (95% and 98%), and were medical-school affiliated (96% and 100%). Similar proportions were high intensity teaching hospitals, with resident:bed ratios of >0.6 in 65% and 57% of respondents and non-respondents, respectively. Table 1 summarizes the self-reported characteristics of the ICUs of respondents. ICUs were of varying size and typically admitted only medical patients. Attending physician staffing policies required transfer of care to an intensivist upon admission in most ICUs (82 of 90, 91%). About one quarter of ICUs were also staffed by non-physician practitioners (i.e., physician assistants or advanced practice nurses).
Almost all (88 of 90, 98%) had at least one clinical protocol (Figure 1). More than half (66 ICUs, 73%) had three or more. Figure 2 summarizes the availability of protocols by clinical area. The most common was liberation from mechanical ventilation (77 ICUs, 86%). 66 ICUs (73%) had protocols for sedation, 54 (87%) of which reportedly include a strategy for daily interruption of sedative medications. Of the 56 ICUs (62%) with protocols for early goal-directed therapy for severe sepsis, 45 (80%) shared the protocol with the emergency department. 54 ICUs (60%) reported having a clinical protocol for lung-protective ventilation for patients with acute lung injury. Finally, 43 ICUs (48%) reported availability of protocols for withdrawal of life support, 22 (51%) of which include specific instructions for the order in which therapies should be withdrawn. Table 2 summarizes additional characteristics of these protocols, including the duration that the protocol has been in place, which clinician type decides to start a patient on the protocol, and which clinician type drives the protocol (i.e., who assesses the patient and makes decisions according to the protocol guidelines). The time since implementation of protocols differed. Programs with protocols for early goal-directed therapy for severe sepsis were most commonly implemented within the past three years (39 of 56, 69%); whereas mechanical ventilation protocols were largely implemented more than three years ago (49 of 77, 65% of liberation protocols; 42 of 66, 68% of sedation protocols; and 33 of 54, 71% of lung-protective ventilation protocols). For the most common protocol, liberation from mechanical ventilation, a majority (52 protocols, 67%) are either started by respiratory therapists or are automatic and do not require physician direction. Trainees (residents and fellows) commonly start and drive protocols for early goal-directed therapy (44 of 56, 77% and 38 of 56, 67%, respectively); however, they are rarely the primary decision-makers with other clinical protocols.
Table 3 summarizes the concordance between pairs of individual protocols, that is, when both protocols were either present or both absent. There was fair concordance (kappa 0.21–0.40) between lung protective ventilation protocols and each of liberation from mechanical ventilation and early goal-directed therapy protocols, suggesting that programs with one protocol were slightly more likely to have the other.
To understand the factors that are associated with the stated availability of individual clinical protocols, we studied the relationship of individual hospital and ICU characteristics with protocol availability. The higher teaching intensity hospitals (resident-to-bed ratio >0.6) were 1.31 times as likely to have a clinical protocol for liberation for mechanical ventilation (95% confidence interval 1.03 to 1.66). No other hospital or ICU characteristics were statistically significantly associated with availability of the individual clinical protocols in univariate analyses.
In a national survey of academic medical ICU directors, we found wide variation in the stated number, types and clinical area of protocols available for the care of critically ill adults. Furthermore, in those protocols that are available, variation exists in their content and structure, even for protocols supported by well-developed medical evidence. Specifically, the providers that start and drive the protocols vary widely, both within and among the different content areas.
Our results have important implications for clinical care for critically ill patients. Despite the large body of literature suggesting that protocols for liberation from mechanical ventilation and sedation of mechanically ventilated patients are associated with improved patient care1–3, many teaching programs do not yet make them available for daily practice. The barriers to clinical protocol adoption may be similar to those to adopting evidence-based guidelines in general and can be categorized into barriers of knowledge, attitudes, or behavior.21, 22 In this case we expect that knowledge barriers are rarely a factor, as it is unlikely that ICU directors in teaching hospitals are unaware of the clinical evidence underpinning protocol use in the ICU. In contrast, barriers of attitude and behavior are more likely contributors to the variation. The attitudinal barriers may include skepticism about specific results of research studies or more generally about the usefulness of clinical protocols as a tool to incorporate those results into practice. Indeed, that clinical protocols are broadly effective is passionately debated. ICU directors may also be concerned about the impact of protocols on education, and feel that the potential clinical benefits do not outweigh the potential negative effects on trainee education. There is limited information on this important question, but what is available suggests that protocols do not affect trainee knowledge.11 Behavioral barriers include structural barriers to protocol adoption in academic hospitals. Protocols are time consuming to develop and frequently require multiple levels of administrative review and oversight. There are also financial costs associated with training clinicians to implement and measure the efficacy of new protocols. In the absence of robust data about the clinical benefit of protocols in all practice settings, institutions may deem that such efforts are not worthwhile. However, these barriers to guideline adoption may not explain all the apparent variation in clinical protocol availability. Were this the case, we would expect higher concordance among individual protocols. An ICU that overcomes these barriers in one clinical area should be more likely to overcome them in other clinical areas. That there was at best fair concordance between individual protocols suggests that other forces are in play. More study of clinical protocols is needed to better understand how their development and implementation differs from adoption of clinical guidelines.
In addition to general variation in protocol availability between hospitals, we observed differences in availability between protocols. Surprisingly, protocols were less common in the clinical situations in which the standardized practice has the greatest impact on mortality, such as early goal-directed therapy for severe sepsis13 and lung-protective ventilation for acute lung injury.14 This pattern may reflect that clinicians are unwilling to create protocols for care in settings of diagnostic uncertainty—as sepsis and acute lung injury are both clinical syndromes that are difficult to reliably identify.23 Conversely, liberation from mechanical ventilation and sedation management of mechanically ventilated patients manage elements of care for a patient category without diagnostic uncertainty. We found that these more common protocols are also more likely to be started and driven primarily by non-physicians. Perhaps this finding reflects a greater comfort with the use of clinical protocols to appropriately turn over management responsibilities to non-physicians when less clinical uncertainty exists, or physicians’ unwillingness to develop protocols to standardize their own behaviors compared to the behaviors of other providers.
Our results also have important implications for medical education. Protocols in teaching ICUs may deprive trainees of important educational experiences. Indeed, respondents indicated that trainees are infrequently the primary drivers of clinical protocols. This may result in multiple degrees of separation between trainees and clinical decision-making. Conversely, in teaching ICUs without clinical protocols, trainees may not learn the evidence behind protocol use in the ICU. Furthermore, they are less likely to learn the skills necessary to develop, implement, and test the efficacy of protocol-driven care, although even with clinical protocols, trainees may have limited exposure to these processes, as they are often time-intensive, behind-the-scenes, and incompletely performed. Still unanswered is whether the presence of evidence-based protocols facilitates education about the best management practices by providing a consistent structure or impedes education by removing opportunities to engage in critical thinking. Further exploration of this question is imperative as protocol use and training practices continue to evolve.
The only association we found between hospital or ICU characteristics and protocol availability was between high teaching intensity and having a protocol for liberation from mechanical ventilation. Due to the small number of training programs, the study was underpowered to detect all but the largest differences in protocol availability between different hospital types. As a result, it is difficult to draw conclusions about the associations between hospital characteristics and protocol availability. Further studies of broader populations of ICUs could elucidate these relationships further.
Our study has several limitations. First, as with all survey research, some questions were subject to some interpretation by respondents. We attempted to provide clarification whenever possible, by describing clinical scenarios, for example, but we recognize this as a potential limitation. Second, we restricted our study population to adult medical ICUs of training programs because we aimed to study how clinical protocols may be useful in such settings and how they may affect trainee education. Although there was variation in ICU size, region, and teaching intensity, our findings may not extend to non-academic or non-medical ICUs. Third, although there was variation in the presence of specific protocols, very few ICUs had none or only one protocol, limiting our ability to study characteristics of programs that prefer not to use any protocols. Although it is possible that this represents a response bias (i.e., that ICUs without protocols did not complete the questionnaire), we found no significant differences in hospital characteristics among respondents and non-respondents. Fourth, the survey can only evaluate stated practice and cannot evaluate adherence to or use of the protocols. Our data likely overestimate the actual availability and use of protocols.24 The mere existence of a protocol may not influence practice or education in any way if the protocol is not actually employed. In fact, in one study of adherence to a lung-protective ventilation protocol, less than 40% of appropriate patients received low tidal volume ventilation within two days of diagnosis of acute lung injury.25 However, we did not believe that any information on the subject of adherence or use would be reliable in questionnaire format.16, 17 In addition, we only obtained limited details on the content of the protocols, in order to keep the questionnaire short so as to obtain a high response rate. Further study of the content of clinical protocols will be an important future direction to better understand their potential impact. Finally, although our study describes whether clinicians have access to protocols in teaching ICUs, it does not directly address the impact of protocols on the quality of medical education. This issue is another key area for future research.
There is variation in the availability, content, and structure of clinical protocols in teaching medical ICUs, even in content areas with well-accepted evidence where standardized care may be appropriate. The reasons for these variations are unclear, and the best use of clinical protocols is still controversial. Further research is needed to better understand the impact of protocols on patient outcomes and medical education and to understand how we may use clinical protocols to balance the advantages of standardized care with those of individualized decision-making.
Financial support: NIH T32 HL007891-10
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.