Search tips
Search criteria 


Logo of jgimedspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
J Gen Intern Med. 2011 February; 26(2): 221–225.
Published online 2010 November 4. doi:  10.1007/s11606-010-1547-y
PMCID: PMC3019318

Teaching Internal Medicine Residents to Sustain Their Improvement Through the Quality Assessment and Improvement Curriculum

Julie Oyler, MD,corresponding author1 Lisa Vinci, MD,1 Julie K. Johnson, MSPH PhD,2 and Vineet M. Arora, MD, MAPP1



Although sustainability is a key component in the evaluation of continuous quality improvement (CQI) projects, medicine resident CQI projects are often evaluated by immediate improvements in targeted areas without addressing sustainability.


To assess the sustainability of resident CQI projects in an ambulatory university-based clinic.


During their ambulatory rotation, all second year internal medicine residents use the American Board of Internal Medicine’s Clinical Preventive Services (CPS) Practice Improvement Modules (PIM) to complete chart reviews, patient surveys, and a system survey. The residents then develop a group CQI project and collect early post data. Third year residents return to evaluate their original CQI project during an ambulatory rotation two to six months later and complete four plan-do-study-act (PDSA) cycles on each CQI project.


From July 2006 to June 2009, 64 (100%) medicine residents completed the CQI curriculum. Residents completed six group projects and examined their success using early (2 to 6 weeks) and late (2 to 6 months) post-intervention data. Three of the projects demonstrated sustainable improvement in the resident continuity clinic.


When residents are taught principles of sustainability and spread and asked to complete multiple PDSA cycles, they are able to identify common themes that may contribute to success of QI projects over time.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-010-1547-y) contains supplementary material, which is available to authorized users.

KEY WORDS: resident education, quality improvement, sustainability, practice-based learning and improvement, system-based practice


In 2009, the Accreditation Council for Graduate Medical Education (ACGME) introduced new continuity clinic requirements for all internal medicine residents. These requirements included: 1) performance data review for each resident’s continuity clinic panel, and 2) involvement in a continuous quality improvement (CQI) process1. Although practice performance evaluation has been a required element of practice-based learning and improvement (PBLI), many residency programs have had to develop unique curricula to fulfill these requirements24. While there are many challenges to teaching CQI412, the American Board of Internal Medicine’s (ABIM) Practice Improvement Modules (PIM) have been used as a tool to teach physicians important CQI principles6,1315. The previously published results of the University of Chicago Quality Assessment and Improvement Curriculum (QAIC) showed that residents were able to make significant changes in their continuity clinic using the Clinical Preventive Services (CPS) PIM14,16.

Sustainability is a key component of the evaluation of CQI projects. However, many CQI projects are evaluated by immediate improvements in targeted areas and do not address long-term effects of improvement efforts. Sustainability is especially relevant in residency training due to the transient nature of resident education. QI projects which are embedded into institutional memory have greater potential for long term sustained improvements after the champions of that project graduate. Sustainability of CQI has been assessed in manufacturing17 but little is published in the medical literature1821. McDonald and colleagues developed the Quality Improvement Proposal Assessment Tool (QIPAT-7) to assess residents’ CQI proposals, including evaluation of sustainability22. However, despite involvement in CQI processes, residents are often not expected to evaluate the sustainability of these processes.


The aim of this educational innovation is to describe a novel sustainability curriculum for internal medicine residency CQI projects, to assess the sustainability of resident CQI projects in an ambulatory university-based clinic, and to determine qualitative factors that contributed to the sustainability of these projects.


We previously described the first year of QAIC, in which post graduate year 2 (PGY2) residents use the ABIM’s CPS PIM to complete chart reviews, gather patient surveys, and complete a system survey. They use the data gathered by the PIM to reflect on the quality of care they provide in their continuity clinic and develop a plan for improvement. Residents work together in groups of 9-12 during their second PGY2 ambulatory month to develop an aim statement, map the targeted process, interview stakeholders, and perform a small test of change14.

The second year of the curriculum teaches PGY3 residents about principles of sustainability and spread (Table 1). PGY3 residents are assigned to complete their ambulatory rotation during one month in the summer/fall and one month in the winter/spring. Each month, there are 9-12 residents who had previously worked on a group project during their PGY2 year. For example, in January 2007 10 residents completed the first PDSA cycle of the BMI project, in March 2007 10 residents completed the first PDSA cycle of the Smoking Cessation Project, and in May 2007 10 residents completed the first PDSA cycle of the Medication Refill project. In August 2007, 3-5 residents from each of these projects (BMI, Smoking, and Refills) came together to participate in the Sustainability Block. During the Sustainability Block, the residents work in small groups to complete further PDSA cycles (Fig. 1). This structure enables representatives from each project to be present over the course of three sustainability blocks so that residents can complete PDSA cycles 2 thru 4, while also giving feedback on fellow residents’ QI projects.

Table 1
Quality Assessment and Improvement Curriculum Outline
Figure 1
Plan do study act cycles for the tobacco cessation project.

During Week 1 of the Sustainability Block, residents use class time to review their previous group project, assess the strengths and weaknesses of that project, and determine how their aim statement needs to be changed. The residents are given a lecture on sustainability during Week 2 and develop a plan to measure long term impact of their project. Faculty use Britain’s National Health Service sustainability model to discuss barriers and facilitators to projects, including project factors, staff factors, and organizational factors23.

During this session, faculty recorded the successes and weaknesses brainstormed by the residents for each project. After reviewing their long term follow-up data during Week 3, the residents develop a sustainability plan for their project to address the resident turnover in the continuity clinic. Faculty supervise this sustainability plan and residents present the plan to peers to get feedback. During Week 4, the residents receive a lecture on spread of QI projects which includes social networking theory24, diffusion of innovation25, and adopter categorization26. They are then asked to consider spread of their QI project and develop a short one page handoff document for their PGY3 peers from their QI project group to use as they reassess the project again in two months (Appendix 1). After all PGY3 residents have completed the Sustainability and Spread Block, a total of four plan-do-study-act (PDSA) cycles (Appendix 2) have been completed on each QI project (Fig. 1). The PGY3 residents return 6 months later to receive four ninety minute lectures about pay for performance and review essential QI concepts. These lectures are available in the online appendices (Appendix 3). During this time, they finalize poster presentations of their projects and complete the ABIM PIM.


From July 2006 to June 2009, 64 (100%) internal medicine residents participated in the sustainability portion of QAIC. All residents completed written consent forms and the study was approved by the University of Chicago Institutional Review Board. Residents completed 6 group projects which they evaluated with both early (2 to 6 weeks) and late (2 to 6 months) post-intervention data. As previously described, residents brainstormed qualitative themes that contributed to the sustainability of each project. Sustained projects were identified as projects which were able to make a significant improvement after early post intervention data collection and were able to maintain these improvements during late post intervention follow-up. Stata 10 was used to perform statistical analysis. Two sample tests of proportions were used to assess changes observed in each project from baseline. Statistical significance was defined as p < 0.05. Three of the six groups showed a sustainable improvement in the resident continuity clinic. Each group developed resident and patient education components and targeted specific clinic interventions.

Projects with Sustained Improvements

Three groups were able to show sustained improvement over time (Fig. 2a). The medication bottle project added a reminder to the clinic hotline for patients to bring their medication bottle to clinic each visit and educated resident physicians about Joint Commission requirements for patients to receive an updated medication list at every visit. There was an initial improvement in the measure “doctor giving patient medications lists” from 56% to 76% 2 weeks post (p = 0.043), which was sustained at 74% 9 months post (p = 0.032). The number of patients who brought in their medication bottles initially did not change (17% to 18% 2 weeks post, p = 0.861), but then improved significantly to 40% 9 months post (p = 0.002).

Figure 2
Results from Internal Medicine Resident Quality Improvement Projects (a) Sustained Improvements and (b) Non-Sustained Improvements. Blue Bar: Pre-Intervention (baseline). Red Bar: Early Post-Intervention (2-6 weeks). Grey Bar: Late Post Intervention ...

The smoking cessation project worked on documentation of smoking status and then initiated a referral process to a smoking cessation clinic. This group was able to improve resident documentation of smoking status from 41% to 67% 4 weeks post (p < 0.001) and continued to increase to 82% 6 mosnth post (p < 0.001). The aspirin project added current aspirin guidelines to the residents’ clinic screening form and posted informational flyers for patients about the importance of aspirin use in exam rooms. The project resulted in an initial modest improvement in aspirin documentation for patients with any indication for aspirin use from 55% to 67% 4 weeks post (p = 0.070) that increased to 75% 4 months post (p = 0.002).

Projects with Non-sustained Improvements

Three of the groups were not able to sustain their projects over time (Fig. 2b). The body mass index (BMI) project incorporated height and weight into clinic triage and BMI was calculated and documented by residents using newly posted BMI charts. Although BMI documentation improved initially from 4% to 79% at 2 weeks post (p < 0.001), documentation of BMI had fallen to 41% at 6 months post (but still increased over baseline, p < 0.001). The medication refill center project attempted to improve the accuracy of clinic note medications lists that were used by the clinic refill center to refill medications. Inaccuracies in these medication lists led to residents getting paged by pharmacies, the refill center, and the patients about medication refills. The residents educated peers, updated business cards with the correct clinic refill center hotline, and integrated correct medication dictation instructions into intern clinic orientation. They then assessed the accuracy of dictated medication lists, counting medications without dose, route or directions as inaccurate. While they initially made an improvement in the accuracy of the medication lists from 75% to 91% 4 weeks post (p < 0.001), the accuracy dropped back down to 75% by 4 months post (p = 0.808). Finally, the social service resource project assessed resident confidence in providing social resources in clinic and then added accessible wall folders with updated information on transportation, food, mental health, and homemaker services. Although not significant, residents made an initial modest improvement in residents’ ability to provide social support at least once per clinic (10% pre to 25% post, p = 0.075). However, the group was not able to collect late post data and could not ascertain whether these initial improvements were sustained.

Common Themes for Quality Improvement Projects

The following were the common themes that residents noticed while evaluating their projects: 1) QI initiatives that included only educational interventions tended to wane over time, 2) QI initiatives which were integrated into the clinic system were more sustainable, 3) QI initiatives can integrate quickly into “clinic memory” because of resident turnover, 4) Residents were generally flexible and open to change and 5) QI initiatives which decreased burden in clinic were more likely to be sustained. For instance, developing a referral source for tobacco cessation or having patient bring in medication bottles simplified residents’ clinic visits, while requiring residents to calculate BMI on each patient was seen as an additional burden. Some interventions took a longer time to show significant change. For instance in the medication bottle project, changing the phone message which encouraged patient to bring their medication bottles to clinic did not make a difference until the late post follow-up 6 months later.


Sustainability is often overlooked when teaching and evaluating quality improvement projects. QI curricula teach residents about the principles of PDSA cycles, but rarely require residents to complete more than a few PDSA cycles on a given project. By assessing QI projects multiple times over a 2-year period, residents are able to evaluate barriers and facilitators to sustainability and spread.

There are a myriad of challenges to sustainability in QI projects and these challenges provide an opportunity to teach residents that ongoing efforts are needed to hold the gains from their improvement projects. Group projects may have varying levels of involvement between residents. Faculty addressed this issue in our curriculum by having resident groups fill out a PDSA form during each cycle listing each resident’s responsibilities (Appendix 2). By reviewing the PDSA form each week, residents and faculty were able to report on individual contributions to the larger group project. Although attending clinics were in the same physical location and have a common structure, we observed that the residents’ QI project initiatives were often hard to spread to attending clinics and met resistance to changing the “old ways” of doing things. Structural changes to the clinic, while positive for clinic workflow, often had unintended consequences that affected the sustainability of QI projects. For example, adding cubicles to resident workrooms and redecorating patient rooms led to loss of BMI charts, aspirin posters, and social services resource wall files. One resident team had difficulty completing long term follow-up, due to the timing of their project. There was another instance when the supporting guidelines changed in the middle of a project. The aspirin project sought to increase the number of diabetes patients who were prescribed aspirin. However, conflicting evidence about the use of aspirin to prevent heart disease in diabetic patients surfaced during their intervention.

Sustainability is a key learning objective when teaching internal medicine residents about quality improvement. It is important for residency programs to train residents to address the sustainability of a project during the initial design of a project as well as during subsequent PDSA cycles. Faculty teaching CQI should incorporate sustainability concepts early in QI curricula and plan for residents to evaluate sustainability of CQI projects. Lectures and worksheets for the University of Chicago QAIC are available to guide this process (Appendix 1-3)27. When residents are taught principles of sustainability and spread and asked to complete multiple PDSA cycles they can learn to identify common themes that may contribute to success of their QI projects over time.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Online Appendix 1(24K, doc)

(DOC 23 kb)

Online Appendix 2(37K, doc)

(DOC 37 kb)

Online Appendix 3(1.0M, pdf)

All lectures.Handout format (PDF 1046 kb)

Online Appendix 3(310K, pdf)

QAIC Toolkit (PDF 309 kb)


We wish to thank the University of Chicago Internal Medicine Classes of 2007 and 2008 for their excellent quality improvement projects and participation in this research. We thank Ms. Meryl Prochaska for her assistance in editing and manuscript preparation. This manuscript has been presented at the national SGIM meeting in Miami, Florida in May 2009, the SGIM Midwest Regional meeting in September 2009, and at the AAMC quality improvement meeting in Chicago, IL, June 2010. This curriculum also won the SGIM Innovations in Clinical Practice Award and the APDIM Award for Innovation in Medical Education at the Spring 2008 APDIM Conference. This work was funded by the University of Chicago Graduate Medical Education Committee and the Department of Medicine.

Conflicts of Interest Dr. Vinci, Dr. Johnson, and Dr. Oyler have no conflicts of interest to declare. Dr. Arora reports receiving funding from the American Board of Internal Medicine Foundation, the Accreditation Council for Graduate Medical Education, and the Agency for Healthcare Research and Quality.


1. ACGME Program Requirements for Residency Education in Internal Medicine. Available at: Accessed September 29, 2010
2. Leach DC. A model for GME: shifting from process to outcomes. a progress report from the Accreditation Council for Graduate Medical Education. Med Educ. 2004;38(1):12–14. doi: 10.1111/j.1365-2923.2004.01732.x. [PubMed] [Cross Ref]
3. Mosser G, Frisch KK, et al. Addressing the challenges in teaching quality improvement. Am J Med. 2009;122(5):487–491. doi: 10.1016/j.amjmed.2009.01.013. [PubMed] [Cross Ref]
4. Morrison L, Headrick L. Teaching residents about practice-based learning and improvement. JT Comm J Qual Patient Saf. 2008;34(8):453–459. [PubMed]
5. Voss JD, May NB, et al. Changing conversations: teaching safety and quality in residency training. Acad Med. 2008;83:1080–1087. doi: 10.1097/ACM.0b013e31818927f8. [PubMed] [Cross Ref]
6. Holmboe E, Meehan T, et al. Promoting physicians’ self assessment and quality improvement: The ABIM diabetes practice improvement module. J Contin Educ Health Prof. 2006;26(2):109–119. doi: 10.1002/chp.59. [PubMed] [Cross Ref]
7. Staker L. Teaching performance improvement : An opportunity for continuing medical education. J Contin Educ Health Prof. 2003;23(S1):S34–S52. doi: 10.1002/chp.1340230408. [PubMed] [Cross Ref]
8. Holmboe ES, Prince L, Green M. Teaching and improving quality of care in a primary care internal medicine residency clinic. Acad Med. 2005;80(6):571–77. doi: 10.1097/00001888-200506000-00012. [PubMed] [Cross Ref]
9. Orginc G, Headrick LA, et al. Teaching and assessing resident competence in practiced based learning and improvement. J Gen Intern Med. 2004;15(5 pt2):496–500. [PMC free article] [PubMed]
10. Varkey P, Reller MK, et al. An experiemental interdisciplinary quality improvement education initiative. Am J Med Qual. 2006;21(5):317–322. doi: 10.1177/1062860606291136. [PubMed] [Cross Ref]
11. Djuricich, AM, Ciccarelli M, et al. A continuous quality improvement curriculum for residents. Acad Med. 2004;79(10 suppl):S65-S67. [PubMed]
12. Boonyasai RT, Windish DM, et al. Effectiveness of teaching quality improvement to clinician: a systematic review. J Am Med Assoc. 2007;298(9):1023–1037. doi: 10.1001/jama.298.9.1023. [PubMed] [Cross Ref]
13. Bernabeo ED, Conforti LN, Holmboe ES. The impact of a preventative cardiology quality improvement intervention on residents and clinic: a qualitative exploration. Am J Med Qual. 2009;24:99–107. doi: 10.1177/1062860608330826. [PubMed] [Cross Ref]
14. Oyler JL, Vinci L, Arora V, Johnson J. Teaching internal medicine residents quality improvement techniques using the ABIM’s practice improvement modules. J Gen Intern Med. 2008;23(7):927–30. doi: 10.1007/s11606-008-0549-5. [PMC free article] [PubMed] [Cross Ref]
15. Shunk R, et al. Using ABIM practice improvement modules to teach internal medicine residents practice improvement. J Grad Med Educ. submitted. [PMC free article] [PubMed]
16. Vinci L, Oyler J, Johnson J, Arora V. Effect of a quality improvement curriculum on resident knowledge and skills in improvement”. Qual Health Care. Published Online First: 31 May 2010. doi:10.1136/qshc.2009.033829. [PubMed]
17. Prajogo DI, Sohal AS. The sustainability and evolution of quality improvement programmes – an Australian Case Study. Total Qual Manag. 2004;15(2):205–220. doi: 10.1080/1478336032000149036. [Cross Ref]
18. Curtis JR, et al. Intensive care unit quality improvement: A “how-to” guide for the interdisciplinary team. Crit Care Med. 2006;34(1):211–218. doi: 10.1097/01.CCM.0000190617.76104.AC. [PubMed] [Cross Ref]
19. Hogg W, Lemelin J, et al. Improving prevention in primary care. Can Fam Physician. 2008;54:712–20. [PMC free article] [PubMed]
20. Edwards JC, et al. Sustainability of partnership projects: a conceptual framework and checklist. JT Comm J Qual Patient Saf. 2007;33(12):37–47. [PubMed]
21. Wallin L, et al. Sustainability in changing clinical practice promotes evidence-based nursing care. J Adv Nurs. 2003;41(5):509–518. doi: 10.1046/j.1365-2648.2003.02574.x. [PubMed] [Cross Ref]
22. Leenstra JL, et al. Valiadation of a Method for Assessing Resident Physicians’ Quality Improvement Proposals. J Gen Intern Med. 2007;22:1330–1334. doi: 10.1007/s11606-007-0260-y. [PMC free article] [PubMed] [Cross Ref]
24. Freeman LC. Centrality in Social Networks. 1978;1:215-239.
25. Rogers EM. Diffusion of Innovations. New York: Free Press; 1983.
26. Rogers, EM.Diffusion of Innovations, Glencoe: Free Press, 1962.27.
27. Accessed August 23, 2010.

Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine