|Home | About | Journals | Submit | Contact Us | Français|
Maintenance of certification examination performance is associated with quality of care. We aimed to examine relationships between electronic medical knowledge resource use, practice characteristics and examination scores among physicians recertifying in internal medicine.
We conducted a cross-sectional study of 3,958 United States physicians who took the Internal Medicine Maintenance of Certification Examination (IM–MOCE) between January 1, 2006 and December 31, 2008, and who held individual licenses to one or both of two large electronic knowledge resource programs. We examined associations between physicians’ IM–MOCE scores and their days of electronic resource use, practice type (private practice, residency teaching clinic, inpatient, nursing home), practice model (single or multi-specialty), sex, age, and medical school location.
In the 365 days prior to the IM–MOCE, physicians used electronic resources on a mean (SD, range) of 20.3 (36.5, 0–265) days. In multivariate analyses, the number of days of resource use was independently associated with increased IM–MOCE scores (0.07-point increase per day of use, p=0.02). Increased age was associated with decreased IM–MOCE scores (1.8-point decrease per year of age, p<0.001). Relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM–MOCE scores by 29.1 and 20.0 points, respectively (both p<0.001).
Frequent use of electronic resources was associated with modestly enhanced IM–MOCE performance. Physicians involved in residency education clinics and hospital inpatient practices had higher IM–MOCE scores than physicians working in private practice settings.
The purpose of certification of physicians is to ensure the safety of patients and the public by providing evidence of individual physician competency.1 In 1990, the American Board of Medical Specialties required all certification boards to issue time-limited certificates, thereby encouraging physicians to demonstrate maintenance of competency by renewing their certification at regular intervals throughout their careers. Although there is debate among physicians about the role and relevance of maintenance of certification (MOC),2 many practicing physicians believe that MOC is necessary,3 and some advocate that it is an important element of professionalism and public accountability.4 Patients value certification and many indicate that they would change physicians if they discovered that their doctor had failed to maintain certification.5
Maintenance of certification includes successfully passing a secure medical knowledge examination1; for internists this examination is the American Board of Internal Medicine Maintenance of Certification Examination (IM–MOCE).8 Substantial evidence for the validity of certification examinations has been demonstrated.5–7 Certification examination scores correlate with training factors (type of medical school and residency program)9,10 and assessments of competency by program directors,11 and recertification scores discriminate between individuals with and without subspecialty content expertise.12 A growing body of evidence demonstrates relationships between certification status and quality of care including outcomes following acute MI,13–15colorectal surgery,16 diabetes care,17 mammography screening,17 and other preventive services.18
Unfortunately, unrelenting time pressures in clinical practice have made it increasingly difficult for physicians to maintain habits of lifelong learning,19,20 resulting in decrements in medical knowledge over time.9,21,22 Traditional continuing medical education (CME) courses vary in effectiveness23–25 and often do not provide the “just in time” learning needed at the point of care.26 Critics of the MOC process question whether correlations between certification examination scores and quality of care measures represent true cause and effect or whether examination scores may be merely a surrogate marker of competency.27 They suggest that the ability to access relevant information resources to support decision-making is more important than fund of knowledge.27
However, research into clinical reasoning and cognition demonstrates that a robust fund of knowledge in working memory is necessary to recognize gaps in knowledge and when to use information resources for the benefit of patients.28,29 Since physicians need both a strong foundation of knowledge and the skills to effectively use information resources, it follows that regular and active use of electronic information resources could lead to gains in foundational knowledge. Electronic knowledge resources are capable of providing physicians quick access to evidence-based information, but the effectiveness of these resources in enhancing physicians’ knowledge is unknown. One study demonstrated a positive association between use of an electronic resource by residents and performance on a standardized examination30; however, this relationship has not been examined on a national scale.
The primary aim of this study was to examine relationships between electronic knowledge resource use and MOC examination scores among practicing internal medicine physicians in the United States. This study was designed to address current gaps in the literature by using a direct and objective measure of electronic resource use, a standardized, well-validated measure of medical knowledge (the IM–MOCE), and a large, national sample of physicians. A secondary aim of this study was to identify demographic and practice factors associated with IM–MOCE scores.
We conducted a cross-sectional study of 3,958 physicians in the United States who took the IM–MOCE for their first time between January 1, 2006 and December 31, 2008 and who held individual licenses to one or both of the electronic medical knowledge resources examined in the study. This study was approved by the Mayo Clinic Institutional Review Board.
This study examined usage of two leading electronic medical resources: UpToDate® and the American College of Physicians Physicians’ Information and Education Resource (PIER). UpToDate® and PIER are web-based resources that provide peer-reviewed, evidence-based information on a wide range of clinical topics, and are designed to answer clinical questions at the point-of-care.31,32 To maintain confidentiality of resource-specific data according to data access agreements with each resource, data were evaluated in aggregate and randomly labeled only as resource A and resource B.
The primary dependent variable was the physician’s IM–MOCE equated score on their first exam attempt. Equating is a measurement technique that enables scores from exams across administrations or forms to be used interchangeably so they can be directly compared. The score scale ranged from 200 to 800, with a mean of 500 and standard deviation of 100. To pass, physicians needed to correctly answer approximately 65% of the exam questions; this cutoff was set using an absolute standard through the well-established modified-Angoff method.33 This cut-score corresponds to a percentile of 18–22% for those taking the exam for the first time, depending on which form of the exam was taken. Considerable content and criterion validity evidence for IM–MOCE scores has been demonstrated.5
We also examined factors known or hypothesized to affect examination performance, including physicians’ age, gender, medical school location (United States or International), practice model (single or multi-specialty) and practice type (including private practice, academic medical center, hospital inpatient-based, residency clinic, and nursing home).
Electronic resource usage data were linked with IM–MOCE score data for each physician using a matching algorithm combining physicians’ first and last names, birth dates, and postal addresses. We required a 100% match of characters for birth date and zip code to determine a positive match between resource usage data and examination scores for each physician. A de-identified data set was created for use in all analyses.
Standard univariate statistics were used to characterize the sample. Bivariate and multivariate analyses were conducted using generalized linear models, with IM–MOCE score as a continuous outcome. Independent variables (or sets of variables if linked, such as for licenses to electronic resources A, B, or both) were entered into the multivariate model if the univariate p<0.20 and retained in a backward stepwise model selection procedure if p<0.05. Two-way interaction terms and nonlinearity were assessed for all variables. Models were also evaluated for undue effects of outlier data. Statistical analyses were conducted using SAS Version 9.1 (SAS Institute Inc, Cary, NC).
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). The co-authors affiliated with the ABIM (EH, RL, CJ) had a partial role in the design and conduct of the study, collection of the data, review and approval of the manuscript. No funding was received from UpToDate® or PIER resource companies.
Of the 15,148 physicians who took the IM–MOCE between January 1, 2006 and December 31, 2008, 3,958 physicians were identified holding individual licenses to one or both electronic medical knowledge resources. The demographic characteristics of the 3,958 physicians with resource licenses were generally similar to the examinees without resources licenses, although those with licenses were somewhat younger (Table 1). The majority of physicians taking the IM–MOCE were graduates of United States medical schools, and most were between 40 and 49 years of age. Among physicians with resource licenses, approximately half (49.2%) reported working in a single-specialty practice. Nearly one-fifth (18.6%) were affiliated with an inpatient hospital practice, 15.6% were in an academic practice, and 6.6% worked in a residency clinic. Physicians with resource licenses were more likely to pass the IM–MOCE on the first attempt (88.9% vs. 80.8%, p<0.001) and had a higher mean IM–MOCE score (555.8 vs. 537.5, p<0.001).
The mean (SD) equated score on the IM–MOCE among all 3,958 physicians with licenses to electronic resources was 534.9 (92.8). A total of 3,553 (89.8%) physicians had a license to only one of the two resources in this study, and 405 (10.2%) had a license to both resources. In the 365 days prior to the IM–MOCE, physicians with licenses to either or both resources used at least one resource on a mean (SD, range) of 20.3 (36.5, 0–265) days. Physicians with a license to only one resource used the resource on a mean of 18.3 (34.5, 0–264) days, while physicians with licenses to both resources used one or both of the resources on a mean of 38.4 (47.1, 0–265) days in the year prior to the recertification exam. Those with licenses to both resources were more likely to be United States medical graduates (68.6% vs. 58.7%, p<0.001) and were more likely to report working primarily in a hospital inpatient practice (24.4% vs. 17.9%, p<0.001). In addition, those with licenses to both resources had slightly higher mean initial certification examination scores (491.8 vs. 481.8, p=0.04). A total of 1,193 (30.1%) physicians with licenses to electronic resources did not use the resources at all during the year. The mean IM–MOCE score for this group was slightly but not statistically significantly lower than that of the group with any use of the resources (531.7 vs. 536.2, p=0.16).
Bivariate analyses are shown in Table 2. Total days of use of electronic resources was associated with a modest but statistically significant increase in mean IM–MOCE score of 0.14 points per day of use (p<0.001). Physicians with licenses to both resources had higher mean IM–MOCE scores by 25.7 and 18.0 points compared to those with licenses to resource A and resource B, respectively (p<0.001 and p<0.001, respectively). The strongest explanatory associations as indicated by the R2 measure of explained variance occurred for age and prior internal medicine certification experience. IM–MOCE score decreased by 5.9 points for each year of age (p<0.001). IM–MOCE score increased by 0.67 points for each point on the initial certification examination (p<0.001), and decreased markedly as the number of attempts required to pass the initial certification examination increased (p<0.001). Additional bivariate associations with IM–MOCE score were observed for medical school location and practice characteristics (Table 2).
The multivariate model adjusting for demographic and practice characteristics accounted for 51.2% of the variability in the data. The number of days of use of electronic resources remained independently associated with an increase in IM–MOCE score, with an increase in IM–MOCE score of 0.07 points per day of use (p=0.02) (Table 3). Thus, every one day of resource use within the 365 days prior to taking the IM–MOCE was associated with an examination score increase of 0.07 points (i.e., every 100 days of resource use was associated with an IM–MOCE score increase of 7 points). Days of resource use accounted for less than 1% of the variability in IM–MOCE score explained by the multivariate model.
Initial score and fewer attempts to pass also remained strongly associated with higher IM–MOCE scores in the multivariate model. The remaining multivariate associations were also consistent with the bivariate results, although the effect estimates were generally attenuated. For example, each year of increased age was associated with a 1.8-point lower IM–MOCE score in the multivariate model (p<0.001). Also, relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM–MOCE scores by 29.1 and 20.0 points, respectively (both p<0.001).
This study of a national sample of practicing physicians showed a positive association between the use of electronic medical knowledge resources and maintenance of certification examination scores, although the magnitude of this association was small. Additional associations were found between IM–MOCE score and prior certification experience, demographic factors, and practice characteristics, as discussed below.
The strongest predictor of IM–MOCE score examined in this study was the physician’s initial certification score, which by itself accounted for 46% of the variability in IM–MOCE score. This observation is consistent with prior studies demonstrating positive correlations between standardized test scores, including correlations between the Medical College Admissions Test (MCAT) and Steps 1, 2 and 3 of the United States Licensing Examination USMLE,34 between USMLE scores and the Internal Medicine In-Training Examination (ITE),35 and between ITE scores and the ABIM initial certification examination.36,37
In addition to the initial certification examination score, several notable associations were seen. First, this study showed a decrease in examination scores with increasing age. This finding is consistent with prior research showing declines in knowledge and clinical performance over time.9,21,22 Physicians face a tremendous challenge in keeping current with rapidly evolving medical advances over careers that may span several decades. Ongoing research into best practices for CME may help address this issue.22,24
Second, working primarily in a residency clinic was associated with a 15-point higher IM–MOCE score, which was equivalent in association to approximately 217 days of electronic resource use. Although further study is needed, this novel finding may suggest that interacting with residents in outpatient clinics may provide valuable intellectual stimulation for the supervising physician and assist physicians in keeping their knowledge and clinical practice current. Studies of the behaviors of clinical supervisors in residency clinics show that they frequently use internet-based resources to answer questions raised during the interaction with learners38 and role model positive behaviors for asking and seeking answers for clinical questions.39 These behaviors may contribute to more effective maintenance of current medical knowledge. In addition, faculty likely participate in a number of other educational activities in the program such as morning report, grand rounds, teaching rounds and curriculum development.
Third, physicians in multi-specialty practices had higher IM–MOCE scores compared to those in single-specialty practices, as did those with primarily inpatient hospital practices as opposed to solo private practices. It is unknown whether these differences are attributable to better access to traditional CME activities or to greater opportunities for more proximal interactions with colleagues with diverse expertise. Perhaps being a member of a multifaceted “learning community”, as exists in most multi-specialty practices, hospitals, and residency clinics, offers an advantage in terms of maintaining one’s knowledge base.40
There are several limitations to this study. First, this is a cross sectional study. Causal relationships between electronic resource use and examination scores cannot be assumed from these data. Second, we were able to examine use of just two electronic resources. Although PIER and UpToDate® are two very large resources with over 400,000 users combined,31,32 this study cannot account for use of other electronic resources or for general internet searching. Third, only physicians with individual licenses to these electronic resources were included. It is noteworthy that resource users scored higher on the certification examination than non-users, independent of the amount of resource use and after including other explanatory variables in the multivariate models. A possible explanation for this finding is that physicians who access information resources may be more likely than others to engage in the broader process of evidence-based practice which involves active learning at the point of care, and may result in greater knowledge acquisition and retention.29 However, this study measured only resource access and examination scores; further research is needed to examine relationships between other aspects of evidence-based practice and medical knowledge. Fourth, resource usage was directly measured through usage logs identified by individual accounts; therefore the analyses do not consider potential access by participants to these resources through institutional licenses. Fifth, electronic resources may differ in the extent to which they “push” or “pull” evidence to clinicians, meaning some resources provide general information to clinicians through online and print media while other resources require physicians to actively acquire specific information when needed.41 We were unable to explore such differences in this study and this remains an important area for future research. Finally, although participants in this study had higher IM–MOCE scores on average, it is reassuring that the score patterns across available demographic factors were very similar for participants and the remainder of the IM–MOCE examinees. However, the observed associations between physician demographics, practice characteristics, and IM–MOCE scores should be confirmed for other physicians participating in MOC programs.
This study also has several strengths. It examined a large, national sample of physicians from a variety of practice settings recertifying in internal medicine over a 3-year period. Additionally, a substantial body of validity evidence supports the primary outcome, the IM–MOCE score, which is a well-established and credible measure of medical knowledge. Finally, the primary independent variable, daily electronic medical knowledge resource use, was measured directly and objectively.
In summary, the results of this study indicate that regular, frequent use of electronic medical knowledge resources may be associated with modestly enhanced performance on the Internal Medicine Maintenance of Certification Examination, while relatively larger score increases are associated with involvement in multi-specialty practice and active participation in a residency education clinic. The factors underlying these associations are worthy of further study.
The authors wish to thank Peter A. Bonis, MD of UpToDate for his assistance in providing UpToDate® usage data and Gary Glenn for his assistance in providing PIER usage data.
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). This organization had a partial role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Specifically, Dr. Holmboe, Dr. Lipner, and Ms. Jacobs, each of whom are affiliated with the ABIM, reviewed the study protocol, provided ABIM administrative data, and reviewed and approved the final manuscript. This study was also supported in part by the Mayo Clinic Internal Medicine Residency Office of Educational Innovations as part of the ACGME Educational Innovations Project. No funding was received from UpToDate® or PIER resource companies. The electronic medical knowledge resource companies examined in this study, UpToDate® and PIER, had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
This study was presented at the national meeting of the Society of General Internal Medicine in May, 2011.
The authors declare they do not have a conflict of interest. Dr. Holmboe, Dr. Lipner, and Ms. Jacobs, each of whom are affiliated with the ABIM, reviewed the study protocol, provided ABIM administrative data, and reviewed and approved the final manuscript.
This study received funding from the American Board of Internal Medicine Foundation (ABIMF). Please see the Acknowledgments for the role of the funding source and conflicts of interest.