This study of a national sample of practicing physicians showed a positive association between the use of electronic medical knowledge resources and maintenance of certification examination scores, although the magnitude of this association was small. Additional associations were found between IM–MOCE score and prior certification experience, demographic factors, and practice characteristics, as discussed below.
The strongest predictor of IM–MOCE score examined in this study was the physician’s initial certification score, which by itself accounted for 46% of the variability in IM–MOCE score. This observation is consistent with prior studies demonstrating positive correlations between standardized test scores, including correlations between the Medical College Admissions Test (MCAT) and Steps 1, 2 and 3 of the United States Licensing Examination USMLE,34
between USMLE scores and the Internal Medicine In-Training Examination (ITE),35
and between ITE scores and the ABIM initial certification examination.36,37
In addition to the initial certification examination score, several notable associations were seen. First, this study showed a decrease in examination scores with increasing age. This finding is consistent with prior research showing declines in knowledge and clinical performance over time.9,21,22
Physicians face a tremendous challenge in keeping current with rapidly evolving medical advances over careers that may span several decades. Ongoing research into best practices for CME may help address this issue.22,24
Second, working primarily in a residency clinic was associated with a 15-point higher IM–MOCE score, which was equivalent in association to approximately 217 days of electronic resource use. Although further study is needed, this novel finding may suggest that interacting with residents in outpatient clinics may provide valuable intellectual stimulation for the supervising physician and assist physicians in keeping their knowledge and clinical practice current. Studies of the behaviors of clinical supervisors in residency clinics show that they frequently use internet-based resources to answer questions raised during the interaction with learners38
and role model positive behaviors for asking and seeking answers for clinical questions.39
These behaviors may contribute to more effective maintenance of current medical knowledge. In addition, faculty likely participate in a number of other educational activities in the program such as morning report, grand rounds, teaching rounds and curriculum development.
Third, physicians in multi-specialty practices had higher IM–MOCE scores compared to those in single-specialty practices, as did those with primarily inpatient hospital practices as opposed to solo private practices. It is unknown whether these differences are attributable to better access to traditional CME activities or to greater opportunities for more proximal interactions with colleagues with diverse expertise. Perhaps being a member of a multifaceted “learning community”, as exists in most multi-specialty practices, hospitals, and residency clinics, offers an advantage in terms of maintaining one’s knowledge base.40
There are several limitations to this study. First, this is a cross sectional study. Causal relationships between electronic resource use and examination scores cannot be assumed from these data. Second, we were able to examine use of just two electronic resources. Although PIER
are two very large resources with over 400,000 users combined,31,32
this study cannot account for use of other electronic resources or for general internet searching. Third, only physicians with individual licenses to these electronic resources were included. It is noteworthy that resource users scored higher on the certification examination than non-users, independent of the amount of resource use and after including other explanatory variables in the multivariate models. A possible explanation for this finding is that physicians who access information resources may be more likely than others to engage in the broader process of evidence-based practice which involves active learning at the point of care, and may result in greater knowledge acquisition and retention.29
However, this study measured only resource access and examination scores; further research is needed to examine relationships between other aspects of evidence-based practice and medical knowledge. Fourth, resource usage was directly measured through usage logs identified by individual accounts; therefore the analyses do not consider potential access by participants to these resources through institutional licenses. Fifth, electronic resources may differ in the extent to which they “push” or “pull” evidence to clinicians, meaning some resources provide general information to clinicians through online and print media while other resources require physicians to actively acquire specific information when needed.41
We were unable to explore such differences in this study and this remains an important area for future research. Finally, although participants in this study had higher IM–MOCE scores on average, it is reassuring that the score patterns across available demographic factors were very similar for participants and the remainder of the IM–MOCE examinees. However, the observed associations between physician demographics, practice characteristics, and IM–MOCE scores should be confirmed for other physicians participating in MOC programs.
This study also has several strengths. It examined a large, national sample of physicians from a variety of practice settings recertifying in internal medicine over a 3-year period. Additionally, a substantial body of validity evidence supports the primary outcome, the IM–MOCE score, which is a well-established and credible measure of medical knowledge. Finally, the primary independent variable, daily electronic medical knowledge resource use, was measured directly and objectively.
In summary, the results of this study indicate that regular, frequent use of electronic medical knowledge resources may be associated with modestly enhanced performance on the Internal Medicine Maintenance of Certification Examination, while relatively larger score increases are associated with involvement in multi-specialty practice and active participation in a residency education clinic. The factors underlying these associations are worthy of further study.