Procedure training, certification, and outcome measurements for certification have long been an area of uncertainty in IM training. Methods for measuring procedural skills in Internal Medicine residency training have not been rigorously studied, nor has the relationship between procedural skills and other resident performance measurements been clearly defined. In this study, we found low complication rates for ABIM-required procedures, similar to previous reports. We also found that certified residents were significantly less likely to report a future complication in that procedure (both lower number of complications and lower complication rates) during residency. This finding is reassuring—it appears that competence-based certification is a valid method for attaining independence in core ABIM procedures. There was also no protection from having a complication in 1 procedure by being certified as competent in another, suggesting internal reliability of our procedural competence certifying method.
The lack of relationship between how quickly residents became certified and complication rates suggests that resident self-assessment of preparedness correlates with proficiency in a given procedure (manifest by future complications in a procedure). This is also supported by the finding that prompt or delayed certification status in any procedure did not correlate with future complications—our data suggest that resident confidence in performing procedures unsupervised, when given clear and specific performance criteria, is associated with improved procedural outcomes (fewer complications).
There was no association between other commonly used resident measurements and complications, including ABIM-MEF ratings of procedural competence or scores on the ITE-2, ITE-3, or ABIM certifying examination (ABIM-CE). This important finding provides data to support the ABIM and ACGME’s movement to remove procedural skill ratings from the new ABIM-MEF that incorporates the core competencies. This may be because of the complex relationship between factors that influence complications (i.e., supervision, experience) and issues that influence resident ratings (i.e., rater skills, amount of observation). ITE and ABIM-CE scores reflect primarily cognitive competencies. Whereas it may be possible to assess whether or not an individual has the cognitive understanding of the indications and potential complications of procedures with a written test, it appears that actual complications relate more to performance of the procedure rather than cognitive understanding of the procedure. Further, on some occasions, attending physicians rated a resident’s procedural skills without direct observation of their skills (i.e., no procedures for the month on cosigned log sheet, yet they filled in a procedural skills score on the ABIM-MEF).
The lack of association between ITE and ABIM-CE scores with having 1 or more complications in an ABIM-required procedure supports the need for separate tools for assessing procedural competence. Likewise, the significant association between certification status and complications (number and rate) underscores the need for direct observation to properly monitor resident procedure performance before certification.
Our reported complication rate is consistent with nationally reported rates for thoracentesis (25/1,000; reference3
) and immediate complications from central line insertion (15/1,000; reference4
). Of note, central line insertion included internal jugular, subclavian, and femoral vein insertion; the latter may have contributed to our lower complication rate.
Our study was limited by several factors. First, one of our underlying assumptions was a relationship between competency and having 1 or more procedural complications. Not only does this assumption have face validity, it makes sense. In this study, we did find a relationship between competence certification and lack of complications, suggesting that this assumption is partially valid. It is not surprising that the relationship we found was only modest, as complications can happen even in the most expert of hands. An ideal study looking at the relationship between competence and procedural complications would have included a rigid, gold standard assessment of each resident’s competence. However, our study was “real-world.” The measures we report, attending ratings of competency, resident procedural logs, ABIM-MEFs and ITEs, are commonly used in US residency programs. Second, our data was collected from a single institution in a single specialty and may not be generalizible to other residency programs and/or specialties. The duration of our study period, the completeness of data, and our prior published description of tools used in this study strengthen our findings. Third, we included no “longer term” procedural outcome measurements. Fourth, our findings were limited by the low number of major complications, and complications were self-reported. However, the resident’s supervising attending physician for the month is required to proofread and sign the procedure log. Attending physicians corrected procedures and reported complications on over 5% of procedure logs. Also, our complication rate was consistent with national published rates, and random chart review confirmed the accuracy of complications on procedural log sheets. Additionally, most of the procedures studied generally have a low complication rate, and a lack of competence in a procedure may not be reflected by having 1 or more complications. Complications are so rare and underreported that they may be a poor marker for lack of competence unless there are repeated complications of the same type of procedure. For example, a previous study of resident skill in performing flexible sigmoidoscopy revealed that limited experience can be associated with poor performance, although there were no complications.5
Fifth, because of the low rate of complications, we were underpowered to show the correlations between complication rates and ABIM MEF ratings, ABIM board scores or in-service training examinations to be statistically significant. For all of these, the correlations we found were quite low, with Pearson’s rho for each at less than 0.05. Even if we had the power to show such weak associations to be statistically significant, the clinical significance would be questionable. Sixth, we did not explore the potential impact of simulation to teach procedural skills. This was explored in a recent study;6
the impact for IM procedure training is unknown. Seventh, our study was limited by range restriction of our resident measurements. Whereas it is possible that associations may have been found if range restriction was not present, the extremely weak correlation we found suggests that these rating methods are poor predictors of resident procedural complication rates. There is literature to support poor correlation between resident confidence and competency. Our system of competency certification has shown the opposite, which may, in part, be because of resident and faculty understanding of the components of competency validation—this is outlined to both residents and faculty in departmental meetings throughout the year. We also acknowledge that 1 competency certification observation has limited reliability. We supplement this observation with procedure note review and discussion with the resident regarding alternate routes, indications, potential complications. Our finding of a lower complication rate after certification is reassuring; providing some support regarding the reliability and validity of our system. Furthermore, we have previously demonstrated that our residents received their competency certification for each required procedure within 1 SD of the corresponding ABIM recommendation.2
Finally, we randomly reviewed a sample of the charts of minor complications; a complete review would allow for description of minor complications.
Our data support the need for a resident procedural competence certification system based on direct observation. Our data also provide preliminary support for moving from a system of procedural competence assessment, used at some residency training programs, that is solely based on attaining a given number of attempts in a procedure to a system based on resident perception of procedural proficiency with clear and specific observed performance criteria. In our system, residents, whether obtaining certification early or late, had significantly lower rates of complication after certification than before.
For future studies of procedures, outcomes could include whether the procedure was successfully performed, whether a complication occurred, and whether patient management or outcomes were effected by the procedure. To date these data have not been comprehensively reported for any Internal Medicine (IM) residency program. Future research could try to tease out what is the “best” method for appraising procedural competence. Our study suggests that 1 possible model could be competency validation.
Our data support the ABIM’s movement to remove resident procedural competence from the monthly ABIM-MEF ratings. The lack of correlation between ABIM-MEF ratings of resident procedural competence and resident procedural complications suggests the lack of validity of this measure for rating procedural skill. This may reflect lack of sufficient direct observation in any given month by supervising physicians or the possibility that the independent system of rating and certifying procedural competence did not cross over to ABIM-MEF scores.
Our findings suggest that commonly used cognitive measurements in Internal Medicine are not a proxy for poor procedural performance, as measured by complications, and that procedural proficiency needs to be assessed by direct observation.