We developed an instrument to measure pediatric residents’ self-efficacy in CRM skills and found validity evidence for the scores from our instrument through content validity, internal structure, and relationship to other variables including known group comparison and comparison with performance of CRM skills. We will review our findings in light of other work on CRM skills, self-efficacy and performance, consider possible reasons why only two of our self-efficacy factors significantly correlated with performance, and reflect on where our work fits within the research on self-assessment.
We drew heavily from previous work in the field of crisis resource management and personal experience of the principal investigators and other experts when developing our instrument. In spite of this attempt to ensure content validity, two of our items, “follow Pediatric Advanced Life Support algorithm” and “consider a variety of explanations for the symptoms”, were eliminated during factor analysis likely because they refer more to medical knowledge than CRM skills. The reasons for the poor performance of the other two eliminated items, “stay calm yourself” and “elicit suggestions from other team members”, are less clear however in a subsequent study by our group, residents were found to be particularly unable to self-assess their level of calmness and their degree of interaction with the team while leading simulated resuscitations (Plant et al. 2010
). Our residents may lack insight into their ability and performance of these two specific skills. Factor analysis in this study indicated the remaining 20 items fell with good internal consistency reliability into four distinct areas of CRM skills: situation awareness, team management, environment management, and decision making. In spite of the fact that we referenced both the ANTS (Fletcher et al. 2003
) and Ottawa GRS (Kim et al. 2006
) instruments when developing ours, two of our factors (team management and environment management) are not shared with either of these instruments. This discrepancy is likely because of the complexity of the construct of CRM skills and possibly because our methods and target audiences differ from those of Fletcher et al. and Kim et al. Interestingly, in our study, the inter-item correlations for observers’ scores on the ANTS and Ottawa GRS instruments were so high that we created composite observer scores for each. It is unclear whether our high degree of inter-item correlations was a function of the instruments or the “halo effect” (Nunnally 1978
We found significant positive correlations between two factors from our self-efficacy instrument and performance as measured by the composite observer scores on the ANTS and Ottawa GRS instruments. Since published guidelines suggest that correlations with an absolute value between 0.3 and 0.5 are considered moderate (Cohen et al. 2003
), our findings give some support the assertion that self-efficacy is related to performance of the associated skills, at least in the context of resuscitation (Bandura 1977
; Maibach et al. 1996
). Maibach et al. (1996
, p. 95) have discussed the theoretical importance of self-efficacy in resuscitation training, stating, “it is likely to influence the development of and real-time access to other cognitive, affective, psychomotor, and social aspects of resuscitation proficiency”. Studies have provided conflicting evidence regarding this hypothesis. In a study of internal medicine residents’ ability to follow ACLS algorithms during simulated resuscitations, there was no correlation between self-efficacy and performance (Wayne et al. 2006
). In another study, a large majority of pediatric residents expressed confidence in technical skills such as endotracheal intubation, whereas only a minority performed those skills properly (Nadel et al. 2000a
). A more recent study found positive correlations between self-efficacy in technical skills and the initiation, but not the successful performance, of those behaviors during simulated resuscitations. This same study showed a moderate correlation (r = 0.48) between self-efficacy in general resuscitation skills and observers’ assessment of their global performance (Turner et al. 2009
). While their study provides the first positive evidence for a link between self-efficacy and performance during resuscitations, the study did not allow for the demonstration and measurement of CRM skills, since the simulated resuscitations were not performed in a team context. Our study is unique in that we examined the relationship between self-efficacy and performance of CRM skills during interdisciplinary simulated resuscitations.
We found correlation with performance for self-efficacy in situation awareness and environment management, but not for team management and decision making. A possible explanation for this finding is that level of experience may affect ability to predict skill level. “Personal performance mastery experiences” are thought to be among the most direct and powerful factors affecting self-efficacy (Bandura 1997
). It is assumed that successful practice of skills increases self-efficacy whereas unsuccessful practice decreases self-efficacy. A lack of experience with skills may, therefore, limit an individual’s ability to accurately assess self-efficacy, a concept called the “dual burden” by Kruger and Dunning (1999
). In addition, there is some evidence that exposure to benchmarking examples and feedback improves an individual’s ability to self-assess accurately (Lane and Gottlieb 2004
; Martin et al. 1998
). Pediatric residents have considerable experience with patient assessment and resource acquisition, activities corresponding to situation awareness and environment management. While engaged in these patient care activities, they are exposed to benchmarks as they observe their peers and attending physicians demonstrate the related skills, receive feedback regarding their own performance and are likely to reflect on their performance in light of this feedback. Accordingly, due to their experiences requiring situation awareness and environment management skills, residents in this study may have been better able to assess their abilities. In contrast, pediatric residents have less experience, especially early in their training, with independent decision-making and team management, the other two factors on our self-efficacy instrument and, as a result, may be unable to accurately assess these abilities.
Our results support that there is a role for assessment of self-efficacy in crisis resource management training. In light of our findings and the fact that self-efficacy is a form of self-assessment, the conclusion that “physicians are inaccurate self-assessors” may be premature and an oversimplification. Many of the studies on which this conclusion is based have been criticized for their suboptimal quality (Colthart et al. 2008
; Davis et al. 2006
). In this study, we specifically addressed the issue of measurement and used psychometrically robust approaches to measuring both self-efficacy and performance. We were able to find some correlation between self-efficacy and performance suggesting that the ability to self-assess may be task and context specific.
Our study has several limitations. First, since this study was incorporated into a resident resuscitation curriculum and a variety of scenarios were used to maximize the educational experience, the scenarios were not identical for all residents. The complexity and therefore difficulty of scenarios may have varied, however, the residents were not evaluated on their medical knowledge or technical skills and should have been able to demonstrate the same level of CRM skills in each scenario. Second, we were unable to find a validated rating instrument for CRM skill performance that was developed specifically for use in pediatrics. Since CRM skills are generic behavioral skills, generalizable across fields of medicine and we were able to achieve good inter-rater reliability with instruments developed for adult practitioners, the ANTS and Ottawa GRS seem to have been appropriate for our setting. Finally, our study through its quantitative nature was not designed to elucidate the underlying reasons for our pattern of findings. Self-assessment is a qualitative process and qualitative inquiry into an individual’s approach to self-assessment may shed further light on the factors that determine its accuracy (Colliver et al. 2005
; Colthart et al. 2008
; Ward et al. 2002