|Home | About | Journals | Submit | Contact Us | Français|
To describe the development and implementation process and assess the effect on self-reported clinical practice changes of a multidisciplinary, collaborative, interactive continuing medical education (CME)/continuing education (CE) program on chronic obstructive pulmonary disease (COPD).
Multidisciplinary subject matter experts and education specialists used a systematic instructional design approach and collaborated with the American College of Chest Physicians and American Academy of Nurse Practitioners to develop, deliver, and reproduce a 1-day interactive COPD CME/CE program for 351 primary care clinicians in 20 US cities from September 23, 2009, through November 13, 2010.
We recorded responses to demographic, self-confidence, and knowledge/comprehension questions by using an audience response system. Before the program, 173 of 320 participants (54.1%) had never used the Global Initiative for Chronic Obstructive Lung Disease recommendations for COPD. After the program, clinician self-confidence improved in all areas measured. In addition, participant knowledge and comprehension significantly improved (mean score, 77.1%-94.7%; P<.001). We implemented the commitment-to-change strategy in courses 6 through 20. A total of 271 of 313 participants (86.6%) completed 971 commitment-to-change statements, and 132 of 271 (48.7%) completed the follow-up survey. Of the follow-up survey respondents, 92 of 132 (69.7%) reported completely implementing at least one clinical practice change, and only 8 of 132 (6.1%) reported inability to make any clinical practice change after the program.
A carefully designed, interactive, flexible, dynamic, and reproducible COPD CME/CE program tailored to clinicians' needs that involves diverse instructional strategies and media can have short-term and long-term improvements in clinician self-confidence, knowledge/comprehension, and clinical practice.
Limitations of current continuing medical education (CME)/continuing education (CE) programs are well documented.1-3 Most CME/CE activities in which only attendance and satisfaction are evaluated are unable to document improved clinician performance. Several factors may explain this inability; most programs include content experts who lack expertise in instructional design principles, have limited resources, and often do not have guidance from educational experts. Research results support the fact that activities planned using specific adult learning principles can demonstrate improved clinician confidence and performance.4 The Institute of Medicine highlights 8 areas within the current continuing health care professional educational model. The primary concerns are research gaps, regulation, financing, and team-based educational activities.3 The medical community has now been called by this recent Institute of Medicine report to design CME/CE offerings tailored to clinician learners' educational needs by using diverse instructional models, strategies, and media.1
Using a collaborative effort, we created a CME/CE program that targets primary care clinicians, including physicians (medical doctors [MDs]/doctors of osteopathy [DOs]), advanced practice nurses (APNs), and physician assistants (PAs), aiming to improve their knowledge/comprehension, self-confidence, and, ultimately, clinical practice. A multidisciplinary team of subject matter experts (SMEs), including pulmonologists (S.G.A., E.J.D., N.A.H.), a family physician (B.P.Y.), an APN (J.W.), and educational specialists (J.P., E.D.), designed and implemented this program based on the Institute of Medicine recommendations.3 Our first priorities were establishing group trust and mutually agreed on goals. We ensured the program included key chronic obstructive pulmonary disease (COPD) content relevant to the interdisciplinary target audience. Our objectives in this article are to describe the development and implementation process and assess the effect on self-reported clinical practice changes of a multidisciplinary, collaborative, interactive CME/CE program on COPD. We report the process of developing and measuring the effectiveness of this interactive CME/CE program on self-confidence, knowledge/comprehension, and changes in clinical practice. For the first time, we introduce the 6 learning categories developed by one of the authors (E.D.) and depict how these categories can be applied to develop an effective CME/CE program. We describe our experience of translating concepts into practical application by applying a systematic approach based on sound educational principles. We also describe how to facilitate, collaborate, plan, design, and deliver a high-quality CME/CE program on COPD best practices in multiple US cities.
We used Analysis, Design, Development, Implementation, and Evaluation (ADDIE), a systematic approach to instructional development, to design the program. Step 1 (analysis phase) was to perform a needs assessment and identify clinical and educational practice gaps. Because physician self-assessment is unreliable,2,5 we used literature reviews, interviews, survey data, and expert opinion to assess COPD care gaps.6 The needs assessment highlighted practice gaps of what currently is happening in practice compared to what should be happening. One major practice gap is the late diagnosis of COPD, which results in suboptimal management.7-9 Primary care clinicians provide most COPD care.7
The American Academy of Nurse Practitioners (AANP) conducted a survey in 2008 to assess interest in educational topics, and COPD ranked 15th of more than 240 topics. Respondents identified the following topics: management of acute COPD exacerbations, spirometry, and pharmacotherapy. Results from another survey administered in US primary care physicians found that 41% to 48% were unaware of COPD guidelines.8 Barriers to optimal COPD recognition and care included patient-related and practice-related factors.8 Patients often defer medical attention for early COPD, and many primary care physicians rely on overt respiratory symptoms to recognize COPD.8,10 Few physicians surveyed reported high confidence in detecting (30%-36%) or treating (27%-34%) COPD.8 On the basis of these needs and practice gaps, our group of multidisciplinary SMEs (S.G.A., E.J.D., N.A.H., J.W., B.P.Y.) determined that the program had to (1) be based on the Global Initiative for Chronic Obstructive Lung Disease (GOLD) recommendations, (2) focus on recognizing clinical COPD risk factors, (3) provide practical information and tools for translating guideline recommendations into clinical practice, and (4) target primary care clinicians.
On the basis of the analysis phase, we designed the course entitled “COPD: What Really Works? A Best Practices Workshop for Primary Care” with these overall learning goals: (1) recognize clinical factors that identify patients at risk for COPD, (2) translate best practice recommendations for COPD into clinical practice, and (3) apply best practice recommendations for COPD in clinical scenarios.
We developed robust formative assessment tools consisting of 7 self-assessment confidence questions and 10 pretest and 10 posttest questions. We linked each pretest and posttest question to instructional objectives, and all SMEs and an independent reviewer with expertise in item writing and COPD peer reviewed each question. We tested and modified the questions only during the first 5 courses. To reduce testing bias, we developed the pretest and posttest questions to evaluate the same concepts but used different content. The faculty did not reveal the answers until after the posttest was completed. Five pretest and posttest questions measured Bloom's taxonomy level 1 (knowledge), and 5 questions measured levels 2 and 3 (comprehension/application).11 During courses 6 through 20, we implemented a commitment-to-change form and encouraged participants to write as many as 4 concrete, measurable goals that they were willing to implement in their practices as a result of program participation (Bloom's taxonomy level 3—application).12 Willing participants returned their completed commitment-to-change form, which we mailed approximately 3 months after the program, in the self-addressed, stamped envelope we provided. Rather than a name, these forms included a unique number for each participant, which was linked to an administrative database to identify who had not returned a survey. We sent up to 2 follow-up e-mail reminders 5 and 6 months after the course with an online survey link for nonresponders. We did not require participation to receive CME/CE credit and did not offer financial incentives.
On the basis of our target audience and learning objectives, we created a collaborative approach to develop and implement this program. We selected educational methods and media based on the educational model of 6 learning categories developed by one of the authors (E.D.) (Table 1). These learning categories are based on Bloom's taxonomy of learning, which describes different adult learning types.11 In this article, we describe and define these learning categories for the first time in the literature and explain their application to an educational program. Reinforcing existing or obtaining new knowledge for clinician learners typically is provided through lecture (learning category 1). As learning category levels increase, learners receive knowledge in more complex and integrated formats, allowing information application in sophisticated ways, such as case-based or simulated environments. Ultimately, the educational program goals are to allow participants to process information on the basis of various experiences.
In a CME/CE environment, information may not appear chronologically. The education model deploys a learning continuum in which knowledge generates appropriate actions and allows participants to judge whether particular actions will yield intended results. By including multiple learning strategies, the model facilitates learners' realization of their full intellectual potential. The education model involves formative assessment tools to allow learners to practice applying information before incorporating it into clinical practice. Miller et al13 described modular-type educational activities based on learner capabilities and goals. Many educators believe that such personalized approaches in educational delivery can meet future demands of high-end learners. Using these 6 learning categories as a foundation, we incorporated multiple learning strategies into our program. A multispecialty team of 5 faculty (COPD experts and respiratory therapists) collaborated to teach the program.
The SMEs worked with educational staff to develop the course content. As highlighted in evidence-based education guidelines and other resources, multiple educational activities, including clinician engagement and implementing various instructional strategies, are substantially more effective than passive single activities.1,2,4,14-16 Therefore, we developed the course based on these principles.
The American College of Chest Physicians (ACCP) and AANP partnered to implement and evaluate this COPD CME/CE program for clinicians in 20 US cities from September 23, 2009, through November 13, 2010 (Table 2). We chose many of these cities on the basis of the high incidence of smoking and COPD. In addition, we chose other cities on the basis of limited opportunities for CME/CE activities. The participants stayed in a group of as many as 36 participants for the morning sessions and divided into small groups of 4 to 12 clinicians in the afternoon (Table 1). We limited the participants to 36 per course to optimize the faculty-to-student ratio for the small-group sessions. The New England Institutional Review Board approved the study. All faculty members attended a 3-hour face-to-face training session. We offered as many as 6.75 hours of AMA PRA Category 1 credits, AANP CE credits, and the American Academy of Family Physicians prescribed credits.
We started the day with an introduction, icebreaker, and pretest. We then showed a 17-minute, faculty-facilitated video, which used the predisposing-enabling-reinforcing instructional framework developed by Green and Kreuter.17 The video highlighted skillful clinician questioning in a routine clinician-patient interaction (Table 1). This method includes creating and reinforcing a teachable moment (predisposing) by allowing learners to compare themselves to an ideal standard.17 We based the 3 morning lectures on the evidence-based GOLD recommendations and used case presentations and the audience response system to encourage learner participation (Table 1).7 Because clinicians learn best by doing,2 participants spent the afternoon in 3 small-group workshops (12 participants maximum per workshop), including simulated role-play exercises, hands-on spirometry, and case-based spirometry billing/coding and interpretation. We developed educational scenarios (on identifying and managing initial COPD diagnosis, medication nonadherence, comorbidities, end-of-life, and others) to allow participants to practice interview techniques to assess COPD symptoms and apply GOLD recommendations. The third portion of the instructional framework included providing reinforcing instructional materials to assist in competence recall; therefore, we provided a precourse suggested reading list, syllabus, pocket guidelines, tobacco-dependence treatment toolkit, and key articles. Finally, on the basis of concepts contributing to clinician behavior change,18 we addressed barriers to implementing changes in clinical practice.
As part of the pilot program, SMEs attended each session and reviewed participants' course evaluations. After the program, the SMEs and staff met to determine areas for improvement. On the basis of the observations and evaluations, we made adjustments to the program during the first 5 courses.
Creating a 3-hour faculty-training program was crucial to design an interactive and flexible but easily reproducible program. The training focused on time management, adult-learning principles, and group facilitation skills. We provided faculty members with detailed lesson plans and links to previously recorded programs to ensure reproducibility.
We described demographic characteristics and participant characteristics, with frequencies and percentages for discrete variables and mean and SD for continuous variables. We used a McNemar test to compare matched pairs of pretest and posttest questions for clinician self-confidence and knowledge. We also examined the score change from pretest to posttest for all 10 questions and subscores of knowledge (5 questions) and comprehension/application (5 questions) by using a paired t test. We examined the proportion of GOLD criteria among the practice types (ie, APN, PA, and MD/DO) by using a Pearson χ2 test. We evaluated the measure of agreement (improved or worsened) of pretest to posttest for each question among the practice types, number of practice years, and age group by using a κ statistic. P<.05 was considered significant for all analyses. We performed all analyses with SAS statistical software, version 9.1 (SAS Institute Inc, Cary, NC).
Three hundred fifty-one participants attended one of the interactive CME/CE programs. We recorded responses to demographic, self-confidence, and knowledge/comprehension questions by using an audience response system. Of the respondents to the audience response system, 237 of 322 (73.6%) were women and 180 of 351 (51.3%) were APNs (Table 2). Before attending the program, 173 of 320 participants (54.1%) had never used the GOLD recommendations. Clinician self-confidence improved after the course in all areas measured (Figure 1). In addition, clinician knowledge/comprehension significantly improved (mean ± SD pretest percentage correct, 77.1%±16.4; 95% confidence interval [CI], 76.2%-78.9%; and mean ± SD posttest percentage correct, 94.7%±8.7; 95% CI, 94.2%-95.2%; P<.001), with an absolute percentage change of 17.6%±13.2%. Of the 5 knowledge (recall) questions, the mean ± SD improvement in pretest vs posttest scores was 14%±5.0% (95% CI, 9.6%-18.4%), from 83.1% to 97.1% (P=.001). The mean improvement in the 5 comprehension/application questions was 22.7%±17.5% (95% CI, 7.4%-38.0%), from 68.8% to 91.5% (P<.001).
We noted participant differences in demographic characteristics, self-confidence, and knowledge/comprehension according to discipline. In general, APNs were in practice fewer years than were MDs/DOs or PAs (P<.001). A total of 97 of 180 NPs/APNs (53.9%), 37 of 55 PAs (67.3%), and 40 of 74 MDs/DOs (54.0%) were unaware of the GOLD recommendations before the course (P=.002). The NPs and PAs generally reported significantly less comfort with managing COPD and interpreting/coding spirometry than did MDs/DOs (P<.001). These trends continued after the course for interpreting spirometry (P<.001), managing acute exacerbations (P=.002), and billing/coding (P<.001). However, the groups no longer differed in comfort with GOLD recommendations (P=.90) or managing COPD (P=.19). Practitioners also did not differ in 9 of 10 knowledge/comprehension questions. Significantly more MDs/DOs correctly answered a spirometry flow-volume loop interpretation/application question than did APNs and PAs on the pretest (P=.048), but no differences were found after the course (P=.20). Despite substantial differences in confidence and knowledge before the program, many gaps between disciplines closed after this educational intervention.
Three hundred thirteen clinicians participated in the last 15 courses, in which we implemented the commitment-to-change statement. A total of 271 of 313 participants (86.6%) participated in the commitment-to-change process and completed 971 citations (mean ± SD, 3.58±0.72 commitments per person). Participants cited commitments related to spirometry (ordering/interpreting tests and/or acquiring a spirometer), identification of new COPD patients (incorporating screening questionnaires into practice), implementing guidelines, smoking cessation, and others (Figure 2). A total of 132 of 271 participants (48.7%) responded to the 3- to 6-month follow-up survey and reported the status of completing 444 commitments (46% of original commitments; mean, 3.36 per respondent).
Notably, 92 of 132 respondents (69.7%) completely implemented at least one commitment. Only 8 of 132 (6.1%) were unable to implement any changes, 7 of whom were no longer employed or in practice. Overall, clinicians reported at least partially implementing most commitments (Figure 2). No differences were found in practice type (MD/DO vs APN vs PA), number of practice years, or age among those who were or were not able to at least partially implement their commitments. On the basis of lessons learned from this interactive program, 125 of 132 respondents (94.7%) self-reported making substantial practice changes.
Participants were asked about barriers encountered when implementing these changes. Respondents reported 223 barriers while trying to implement 444 commitments. The most common barriers included time limitations (42 of 223 [18.8%]), limited resources (27 of 223 [12.1%]), lack of fiscal support (25 of 223 [11.2%]), difficulty remembering to incorporate into daily routine (20 of 223 [9.0%]), organizational issues (20 of 223 [9.0%]), and inadequate knowledge. The respondents identified substantial barriers, but most clinicians still reported practice changes despite these barriers.
We describe our experience designing and implementing a successful interactive, multimedia, multidisciplinary, collaborative CME/CE COPD program for primary care clinicians in 20 cities across the United States. We based the foundation of this unique program on adult learning and instructional design principles and the ACCP 6 learning categories. Our data suggest that this program is associated with clinical practice changes and significant improvements in clinician self-confidence and knowledge/comprehension. Similar to our findings during the initial needs assessment in the analysis phase, we found that 173 of 320 clinician attendees (54.1%) had never used the GOLD recommendations and admitted to substantial discomfort with managing COPD and spirometry interpretation and billing. A total of 125 of 132 respondents (94.7%) to the follow-up commitment-to-change survey reported substantial clinical practice changes after the program.
Our results indicate that this COPD CME/CE program appears to have helped close the clinical practice gaps between what was happening before the program and the positive practice changes after the program. Adults seek information that they can apply pragmatically to their immediate circumstances. The more relevant and practical the information, the more likely they are to change behavior. Chronic obstructive pulmonary disease is important to these clinicians as evidenced by high response rates and interest in staying engaged with the learning and practice improvement process during and after the course, despite the lack of financial incentives.
Our study is unique in that we evaluated self-confidence, knowledge/comprehension, and commitments to change from a large, multidisciplinary group of clinicians. Other investigators have implemented the commitment-to-change strategy to evaluate CME/CE programs.19-29 Most of these studies included a small number of participants (N=26-84),19-21,23-25,27 and only 3 included nonphysician clinicians.24,25,27 A total of 952 of 971 (98.0%) of the commitments written by clinicians attending our CME/CE program were directly related to learning objectives, and all learning objectives had at least one commitment statement. In contrast, Dolcourt and Zuckerman25 reported 32% of commitment-to-change statements were unmatched to their objectives (classified as unanticipated learning), and there were no commitment statements for 34% of their objectives.
Our study has 2 main limitations. The first is that we did not design the study for independent verification of the degree of adherence to self-reported commitments. Adams et al29 performed a systematic review, describing limitations inherent to data reported by clinicians and not corroborated by objective data. They reported that 8 of 10 selected studies found considerable bias in self-reported practice guideline adherence when compared with objective measures.29 However, investigators in 2 studies not included in the review, but specifically involving the commitment-to-change process, evaluated the association of self-reported commitments with actual behavioral changes.21,22 These investigators found that physicians who committed to change prescribing practices were truly likely to change their prescribing habits.21,22 These study results indicate that self-reported changes can be a proxy for actual change.21,22 A second limitation of our study includes the potential for reporting bias of the nonresponders.24 Nonresponders may be uncomfortable reporting their potential inability to implement changes. Because of the limitations of our study design, it is not possible to analyze the magnitude of this potential bias accurately. However, in the worst-case scenario, even if all 139 nonresponders did not implement a single change, then still nearly half (132 of 271 [48.7%]) of all participants reported successfully implementing substantial practice changes.
Our data stress the need for future programs to implement similar strategies in designing and delivering CME/CE programs and for measuring long-term outcomes. Although the commitment-to-change assessment is not perfect and may be subject to reporting bias, it may shed light on actual clinical practice changes. Our strategy, however, does not specifically measure health outcomes in COPD patients because we did not directly assess new clinical diagnosis rates or specific management changes. In future programs, investigators should evaluate these outcomes.
Using diverse instructional models, teaching techniques, and media, we created an interactive, dynamic, collaborative, multidisciplinary, and reproducible CME/CE program for primary care clinicians throughout the United States. This program addresses practice gaps in COPD recognition and management and is associated with substantial self-reported changes in clinical practice. We developed this unique educational program by (1) linking instructional methods to outcome strategy, (2) creating teachable moments, (3) using formative assessment throughout the process, and (4) fostering true collaboration among the various disciplines. In addition to making considerable self-reported changes in clinical practice, clinicians who attended this program significantly improved their self-confidence and knowledge/comprehension of COPD. This program may be used as a template for creating CME/CE programs on other medical topics.
Potential Competing Interests: Dr Adams discloses the following: investigator/grant research: National Institute of Health, Veterans Affairs Cooperative Studies Program, Bayer Pharmaceuticals Corp, Boehringer Ingelheim Pharmaceuticals Inc, Centocor Inc, GlaxoSmithKline, Novartis Pharmaceuticals AG, Pfizer Inc, and Schering-Plough Corp; honoraria for speaking at CE programs (unrestricted grants for CE): AstraZeneca Pharmaceuticals LP, Bayer Pharmaceuticals Corp, Boehringer Ingelheim Pharmaceuticals Inc, GlaxoSmithKline, Novartis Pharmaceuticals AG, Pfizer Inc, and Schering-Plough Corp. Ms Pitts and Mr Dellert are both employees of the ACCP. Ms Wynn is employed by the AANP. Dr Yawn discloses the following: research support from Aerocrine, Boehringer Ingelheim, Forrest, GlaxoSmithKline, and Novartis. Dr Hanania discloses the following: investigator/research support from Astra Zeneca, Boehringer Ingelheim, GlaxoSmithKline, MedImmune, Novartis, Pfizer, and Sunovion; speaker bureau of Boehringer Ingelheim, GlaxoSmithKline, and Pfizer; advisory board/consultancy for Dey Inc, GlaxoSmithKline, Novartis, Pearl, and Pfizer. The programs described were conducted and sponsored by the ACCP and the AANP, which received unrestricted grants from AstraZeneca Pharmaceuticals LP, Wilmington, DE, Boehringer Ingelheim Pharmaceuticals Inc, Ridgefield, CT, and GlaxoSmithKline Pharmaceuticals Ltd, Philadelphia, PA. The funders of the unrestricted grants were not involved in the development or implementation of these programs, in the interpretation of the data, or in the preparation, review, or any part of the manuscript.
Author Interview Video