|Home | About | Journals | Submit | Contact Us | Français|
Chart review represents a critical cornerstone for practice-based learning and improvement in our internal medicine residency program.
To document residents’ performance monitoring and improvement skills in their continuity clinics, their satisfaction with practice-based learning and improvement, and their ability to self-reflect on their performance.
Retrospective longitudinal design with repeated measures.
Eighty Internal Medicine residents abstracted data for 3 consecutive years from the medical records of their 4,390 patients in the University of Wisconsin-Madison (UW) Hospital and Clinics and William S. Middleton Veterans Administration (VA) outpatient clinics.
Logistic modeling was used to determine the effect of postgraduate year, resident sex, graduation cohort, and clinic setting on residents’ “compliance rate” on 17 nationally recognized health screening and chronic disease management parameters from 2003 to 2007.
Residents’ adherence to national preventive and chronic disease standards increased significantly from intern to subsequent years for administering immunizations, screening for diabetes, cholesterol, cancer, and behavioral risks, and for management of diabetes. Of the residents, 92% found the chart review exercise beneficial, with 63% reporting gains in understanding about their medical practices, 26% reflecting on specific gaps in their practices, and 8% taking critical action to improve their patient outcomes.
This paper provides support for the feasibility and practicality of this limited-cost method of chart review. It also directs our residency program’s attention in the continuity clinic to a key area important to internal medicine training programs by highlighting the potential benefit of enhancing residents’ self-reflection skills.
The Accreditation Council of Graduate Medical Education (ACGME) identified practice-based learning and improvement (PBLI) as a key competency in 1999.1 Shortly thereafter, in 2000, our internal medicine residency program began incorporating annual chart review, guided by a follow-up meeting between the resident and project director, as a critical exercise for ensuring that residents developed both an attitude and an aptitude for self-reflection and a lifelong commitment to self-improvement. For the past 5 years, residents in our training program, under the direct supervision of the Associate Chief of Staff for Medicine and Ambulatory Care (CH), routinely conducted annual chart review. They studied the relevant practice guidelines, reviewed the medical records of continuity care patients in their outpatient clinics, documented their adherence to national preventive and chronic disease care recommendations, and reflected on their practices.
In 2003, when the American Board of Internal Medicine (ABIM) developed its practice improvement modules (PIM), which are now required for maintenance of certification, it acknowledged that quality improvement was inherently linked to professionalism and self-regulation.2 External audits might lead to improvement in the quality of patient care,3 but such improvements were not found consistently,4,5 tended to be small to moderate,6 and were most evident when residents’ performance was particularly poor or when educational interventions were particularly intense.6 Moreover, residents often challenged the accuracy of external reports,7,8 and physicians’ belief in the credibility of their own data was essential for quality improvement.2 By establishing self-assessment and reflection on practice as the foundation for the PIMs, the ABIM acknowledged the potential for bias, but affirmed our belief that residents’ honest reflections on their own medical practices provided an element of learning that could not be gleaned through standard didactic medical education or passive review of externally extracted data.
Research reported within the past year raised serious concerns about the quality of residency clinic care practices.9,10 It also provided practical suggestions for improving patient outcomes by strengthening clinic operations,11,12 adapting instruction to residents’ prior knowledge,13 increasing physician-patient continuity,14 and re-designing the ambulatory block structure.15 New methods for teaching and assessing PBLI support clinical understanding,16–19 and disease registries help programs achieve benchmarks for critical care processes.20,21 However, creating meaningful improvement in medical practice requires individual commitment to change.22
In our 5-year retrospective study, we assessed residents’ improvement over time in their performance monitoring skills and outcomes, their skills at self-reflection, and their satisfaction with our PBLI practices. We also considered contextual effects of resident sex and clinic, and the feasibility and value of PBLI to the residency program as we assessed the impact of these practices on residents’ commitment to improved patient care outcomes.
The sample included 80 residents from five consecutive graduating cohorts (2003 – 2007) who performed chart review during all 3 years of residency in the University of Wisconsin-Madison Internal Medicine residency program. There were 37 residents (60% male) in the primary care continuity clinic of the Veterans Administration Hospital (VA) and 43 residents (40% male) in one of four University of Wisconsin (UW) health-care clinics. Clinics served a mixed income, urban/rural patient population. The same weekly outpatient medical curriculum was implemented at all sites under the direct supervision of the faculty member who had responsibility for each resident’s clinic. Assigned faculty were on site during all continuity clinics, personally seeing interns’ patients, and providing indirect staffing for patients of PGY-2 and PGY-3 residents. Residents who did not complete all 3 years of chart review (n=36) were not included in this study.
Residents were instructed to select, in good faith, 20 charts from their panels of approximately 60 adult patients whom they had seen in their continuity clinics within the past year, with the intention that nearly all of their patients would be reviewed within the 3-year context of the program. To confirm the absence of bias in chart selection, we examined medical record numbers across 3 years for a randomly selected 10% of residents (n=8) and documented that 17% of their 453 patient charts had undergone re-review. Prior to conducting chart review, small groups of residents were guided through a master sheet summarizing the evidence-based and/or expert opinion references for each parameter and the standardized data abstraction tool. Criteria were documented as “met” if test results were evident in the patient’s chart or if written documentation indicated that the test had been performed, scheduled, or offered; as “not met” if documentation was missing or incomplete; or as “not applicable” if screening or management was not indicated due to age, sex, or medical history. Chart review was performed during 2 h of unscheduled clinic time. The Institutional Review Board categorized this project as exempt, and patient identifiers were anonymized as data collection was completed.
Data abstracted from chart review included patient age and sex as well as residents’ compliance with each of 11 preventive medicine and 6 chronic disease management measures reflecting standards of care as accepted by the US Preventive Services Task Force.23 Measures included (1) immunizations (tetanus booster, pneumococcal 23-valent, inactivated influenza), (2) general health screening (diabetes, cholesterol, hypertension), (3) cancer screening (colorectal, breast, cervical), (4) behavioral risk factor screening (tobacco, alcohol), and (5) chronic disease management for type 2 diabetes mellitus (HgbA1c, blood pressure, LDL, microalbumin, foot care, eye exam). These measures have been endorsed by the National Committee for Quality Assurance (NCQA)24 as quality indicators in ambulatory care.
Following chart review, residents completed a brief questionnaire indicating whether they had benefited from the chart review exercise (yes or no) and why. Content analyses were conducted on residents’ qualitative responses using a modified version of four categories developed by Mezirow (1991):25 (1) Lack of engagement captured responses in which residents suggested an absence of learning or otherwise complained about the chart review experience; (2) Understanding captured a range of positive statements about the importance of the process or general appreciation of new learning regarding residents’ strengths or weaknesses; (3) Reflection included positive statements in which residents appeared to interpret and give meaning to their performance on a particular quality indicator; (4) Critical Reflection included statements in which residents clearly addressed a particular problem in their chart review and either planned action to change or specified a commitment to change the situation. Inter-rater reliability between one of the authors (MAR) and a trained graduate assistant indicated excellent reproducibility (kappa=0.84).
For chart review, each patient’s outcome was coded as 1 if the criterion was met (termed “compliance”), coded as 0 if the criterion was not met and should have been met, and coded as “not applicable” if screening or managing that measure was not indicated given that patient’s risk factors. We assumed that if a measure was coded as met, it was indicated for that patient.
For each resident, postgraduate year, and health-care measure, compliance was modeled against program year, clinic, and known demographic variables of the resident and patient. Each resident had a maximum of 1,020 observations (17 measures × 3 postgraduate years × 20 patients). Logistic modeling was used to determine the effect of postgraduate year in the residency program (PGY-1, PGY-2, PGY-3), resident sex (male, female), graduation cohort (2003, 2004, 2005, 2006, 2007), clinic setting (VA, UW), patient sex (male, female), and patient age. Generalized Estimating Equations were used to adjust for correlation between repeated observations for a resident and postgraduate year in the program.26 Chart review data were analyzed using the Statistical Analysis Software (SAS) package.27 Resident sex, clinic setting, postgraduate year, patient sex, and patient age were treated as categorical variables; graduation cohort was treated as a continuous variable. Compliance rates were analyzed using a logistic regression model that adjusted for repeated measures and year in the program. Descriptive statistics were used to analyze the survey data.
Eighty residents abstracted data from the medical records of 4,390 patients (mean=18 patients per resident for each of 3 years) on 17 core measures.
Comparisons of patient sex and age characteristics within resident sex and clinic setting indicated that at the UW clinic, female residents were significantly more likely than male residents to have female patients (68% vs. 30%, χ2=287.8, p<0.0001); this was not true at the VA clinic, where both male and female residents had 29% female patients (χ2=0.03, p=0.87). Patients at the VA clinic were older than patients at the UW clinics (χ2=578.0, p<0.0001), with about two-thirds of patients at or above age 55 at the VA and two-thirds below age 55 at the UW.
Residents’ compliance rates for monitoring core health-care measures within postgraduate years are presented in Table 1. A multivariate model using results from all 17 health-care measures for the 80 residents indicated a significant effect of postgraduate year. Residents exhibited a significantly higher compliance rate during their second [OR=1.72, CI=(1.39, 2.13), p<0.0001] and third [OR=1.98, CI=(1.60, 2.44), p<0.0001] years than during their first year of residency. There was no significant difference in residents’ compliance rates between the 2nd and 3rd years. Following the first postgraduate year, significant improvement was evident in residents’ rates of immunization (tetanus and pneumococcal vaccines), general health screening (diabetes and cholesterol), and cancer screening (colorectal and cervical). For management of patients with diabetes, residents demonstrated significant improvement for screening for LDL, testing for microalbumin, and conducting foot and eye exams. Residents’ compliance rates for addressing patients’ use of tobacco and alcohol during clinic visits also increased significantly. No significant improvements were found for administering influenza vaccines, screening and monitoring for hypertension, conducting mammograms, or checking for HgbA1c in patients with diabetes. Residents’ success at screening and successfully monitoring blood pressure are reflected in Table 1 as the percentages of all patients in the residents’ clinic who either had normal blood pressure during screening or had a treatment plan to address their elevated blood pressure (≥140/90, ≥130/80 for diabetics).
Residents at the UW clinics exhibited significantly lower compliance rates than residents at the VA clinics [OR=0.58, CI=(0.44, 0.76), p<0.0001], and there was a small but significant increase in compliance rates by graduation cohort (11% increase in compliance rate, compared to the previous cohort year) [OR=1.11, CI=(1.04, 1.19), p<0.002]. Compliance rates for residents did not differ significantly by resident sex, patient sex, or patient age. A comparable logistic model, developed for all 116 residents, indicated that residents who completed all 3 years of chart review (n=80) exhibited significantly higher rates of compliance than residents who did not complete 3 years of chart review (n=36) [OR=1.35, CI=(1.10, 1.67), p<0.005].
The 80 residents completed a total of 160 annual surveys during their 3 years of residency, with 28 residents completing all three surveys, 49 residents completing at least one but not all three surveys, and 3 residents completing no surveys. Among completed surveys, the overwhelming majority of residents (92%) indicated that the chart review exercise was beneficial, and positive perceptions were consistently high across postgraduate year (PGY-1=90.0%; PGY-2=96.5%; PGY-3=90.5%).
In response to the follow-up question as to why they thought chart review was beneficial, only 60% of the resident surveys included open-text commentary. Responses fell into four categories.25 The majority of residents (63%) indicated general understanding about the importance of self-assessment or identification of general perceived strengths or deficits, e.g., “It helped me identify areas in which I need to improve.” A total of 26% of the residents identified specific insights or associations among quality indicators, e.g., “I missed general maintenance for complicated patients.” Some residents (8%) emphasized specific actions taken or commitments to change in the future, e.g., “I changed my computer template after the last chart review to incorporate preventative medicine reminders into each visit.”
Chart review represents a well-established technique for measuring and continuously improving the quality of health care.28 Academic medical centers have provided benchmarking data by postgraduate year,29–31 and evaluation of personal clinical performance against benchmarks has become an accepted intervention in professional development outside of academic settings.32 Results of this study add to the literature by demonstrating that residents who participated in annual reviews of their own patients’ charts exhibited increased adherence to nationally recognized preventive and chronic disease standards. Residents showed significant improvement over postgraduate year in their rates of administering immunizations (tetanus, pneumococcal), screening for diabetes, cholesterol, and cancer (colorectal, cervical), and screening for behavioral risk factors, as well as in their management of diabetes (LDL, microalbumin, foot and eye care). These findings are consistent with the notion that physicians naturally seek to close gaps in their performance once they become aware of them, especially if those gaps are self-discovered.33
Although our focus has always been on residents’ individual practice-based learning and improvement, differences in compliance rates clearly exist within our residency program and in comparisons between our residency and other residency programs. For example, although all of our residents were exposed to the same curriculum and model for faculty supervision, compliance rates were significantly higher at the VA than at the UW. This might be explained by the fact that practice guidelines at the VA were fully integrated into an established electronic medical record system, and residents had primary responsibility for their patient panels, whereas electronic medical records were unavailable at the UW, and residents had to schedule their patients through their attendings, thereby limiting access to patient information and the ability to monitor quality improvements. It is more difficult to explain how specific clinic practices (e.g., reminder systems, protocols for immunizations) might have led to differential improvements in some screening measures (e.g., cervical cancer) and not others (e.g., breast cancer) or to specify reasons for incremental improvements over time given the retrospective nature of our data. Moreover, the fact that we, like Kern (1990),29 found improved outcomes across residency years where others did not,31 or that our compliance rates appear similar to baseline rates in some studies,22 but higher than those reported in other studies15 suggests potential differences in practice sites that may be too numerous to speculate (e.g., patient populations, frequencies of re-visits to clinic, core curriculum).
Residents consistently reported benefiting from the process of reviewing their patients’ charts. Those who completed the survey typically cited gains in understanding about their medical practices, with some, although not a majority of residents, articulating specific progress toward commitments to change. These findings suggest that the residency program may wish to strengthen PBLI skills in continuity clinics,34 while reinforcing residents’ skills at self-reflection through focused mentorship,35 since residents who fail to engage in reflection or critical reflection25 on this exercise may be less capable in general of addressing their gaps and improving their medical practices.
This paper provided support for the feasibility and practicality of our limited cost method of chart review. In contrast to a more expensive model of quality assessment, we estimate that our method could be implemented in a similar-sized program at an annual cost of about $3,000, based on 15 h per year of faculty time (at $100 per hour), supported by 20 h for a program assistant to monitor compliance (at $15 per hour), and 40 h by a data analyst to compile, analyze, and report program data (at $30 per hour), plus the opportunity costs associated with resident time to complete the chart review.
Inherent limitations in the methodological features of our study must also be recognized. We assumed that the value of spending time looking at one’s own work relative to accepted standards of care outweighed the potential for resident bias in chart audit, but in the absence of a control group and double data extraction, we can neither claim that this PBLI activity changed resident behavior nor can we rule out the possibility that improvement over time reflected additional years of training. Similarly, we integrated practice-based learning into the ambulatory care setting assuming that this type of learning would lead to improved patient care practices, but because we did not actually measure improvements in the quality of patient care, we can not confirm this association. Finally, we laid a foundation for lifelong learning by protecting the chart review process from outside evaluation, thereby encouraging residents to take responsibility for their own foibles. However, absence of outcome-based assessments or other evidence to show that our former residents continued to apply their knowledge to self-improvement in their later medical practice hampered our ability to claim success at establishing lifelong self-regulation.
Conclusion Our method of chart review focused on inculcating a spirit of independent reflective thinking and providing residents with an opportunity to improve their practices based on what they learned. Our data collected over 5 years may have been retrospective, but they resulted in the accrual of a large number of individual measurements, and our findings seem promising enough to recommend more rigorous testing. Indeed, if this practical, meaningful, limited cost PBLI method could be shown to lead to change in resident behavior within controlled studies, it seems likely that many residency programs would choose to implement it. It is our hope that our enhanced counseling practices will likewise move residents along a continuum toward increasing critical self-reflection and commitment to individual and system change, further accelerating residents’ progress toward improved patient outcomes.
Completion of this paper was made possible by a grant from the Medical Education and Research Committee (MERC) of the University of Wisconsin-Madison School of Medicine and Public Health as well as by the support of the Education Innovation Project of the Residency Review Committee for Internal Medicine, of which we are a participating residency. This research was also supported by the University of Wisconsin Institute for Clinical and Translational Research, funded through an NIH Clinical and Translational Science Award (CTSA), grant no. 1 UL1 RR025011. Earlier results of this paper were presented at the Association of Program Directors in Internal Medicine meeting (2004). We wish to acknowledge the programmatic efforts of Suzy Griffiths, Toni Prisk, Vonnie Schoenleber, Jessalyn Richter, and Erik Stava at various stages in the process of this work.
Conflict of Interest None disclosed.