PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Am J Med. Author manuscript; available in PMC 2014 April 1.
Published in final edited form as:
PMCID: PMC3666348
NIHMSID: NIHMS469359

Effects of Clinical Pathways for Common Outpatient Infections on Antibiotic Prescribing

Timothy C. Jenkins, MD,1,2,5,6 Amy Irwin, DNP, RN,1 Letoynia Coombs, EdD,7 Lauren DeAlleaume, MD,3,7 Stephen E. Ross, MD,5 Jeanne Rozwadowski, MD, MPH,3,5 Brian Webster, MD,8 L. Miriam Dickinson, PhD,7 Allison L. Sabel, MD, PhD, MPH,4,9 Thomas D. MacKenzie, MD, MSPH,3,4,5 David R. West, PhD,5,7 and Connie S. Price, MD1,2,5,6

Abstract

Background

Antibiotic overuse in the primary care setting is common. Our objective was to evaluate the effect of a clinical pathway-based intervention on antibiotic use.

Methods

Eight primary care clinics were randomized to receive clinical pathways for upper respiratory infection, acute bronchitis, acute rhinosinusitis, pharyngitis, acute otitis media, urinary tract infection, skin infections, and pneumonia and patient education materials (study group) versus no intervention (control group). Generalized linear mixed effects models were used to assess trends in antibiotic prescriptions for non-pneumonia acute respiratory infections and broad-spectrum antibiotic use for all eight conditions during a 2-year baseline and 1-year intervention period.

Results

In the study group, antibiotic prescriptions for non-pneumonia acute respiratory infections decreased from 42.7% of cases at baseline to 37.9% during the intervention period (11.2% relative reduction) (p <.0001) and from 39.8% to 38.7%, respectively, in the control group (2.8% relative reduction) (p=0.25). Overall use of broad-spectrum antibiotics in the study group decreased from 26.4% to 22.6% of cases, respectively, (14.4% relative reduction) (p <.0001) and from 20.0% to 19.4%, respectively, in the control group (3.0% relative reduction) (p=0.35). There were significant differences in the trends of prescriptions for acute respiratory infections (p<.0001) and broad-spectrum antibiotic use (p=0.001) between the study and control groups during the intervention period, with greater declines in the study group.

Conclusions

This intervention was associated with declining antibiotic prescriptions for non-pneumonia acute respiratory infections and use of broad-spectrum antibiotics over the first year. Evaluation of the impact over a longer study period is warranted.

Keywords: Clinical pathways, guidelines, antimicrobial stewardship, antibiotic prescribing, primary care, acute respiratory infection

Introduction

Despite national efforts to promote appropriate antibiotic prescribing in the United States, unnecessary antibiotic use in the primary care setting remains common.1 Antibiotic overuse fosters the spread of antimicrobial-resistant organisms.2,3 In an era of progressive antimicrobial resistance and a diminishing pipeline of new antibiotic development,4 ensuring optimal antibiotic prescribing for common infections is of fundamental importance to conserve our current arsenal of antibiotics for as long as possible.5 Although there has been an increasing focus on antibiotic stewardship in the hospital setting,6 outpatient prescribing accounts for the majority of antibiotic consumption and is an important factor in the emergence of resistance in both the community and hospitals;7,8 therefore, preventing unnecessary antibiotic use in the primary care setting remains paramount to overall stewardship efforts.

Excessive antibiotic use is often manifest by prescriptions for conditions where antibiotics are known to provide little or no benefit, such as non-pneumonia acute respiratory infections,9 and use of antibiotics with a broader spectrum of activity than necessary.10 Prior studies to address these prescribing errors have led to modest reductions in antibiotic use.1,9,1114 A recent evaluation of various interventions concluded that active clinician education targeting multiple conditions is most likely to impact community antibiotic use.15 Clinical pathways, or algorithms, are an effective means to change antibiotic prescribing behavior.1620 We hypothesized that implementation of clinical pathways for a group of infections commonly seen in the primary care setting would decrease prescribing for non-pneumonia acute respiratory infections – conditions where antibiotics are typically not indicated – and decrease overall use of broad-spectrum antibiotics.

Methods

Study setting and population

The study was performed in a diverse group of Family Medicine and Internal Medicine outpatient clinics from the Distributed Ambulatory Research in Therapeutics Network (DARTNet) and Denver Health. DARTNet is a federated network linking patient-level clinical and pharmacy data from 86 organizations made up of 450 practices, 3000 clinicians, and over 4.5 million patients.21 Electronic health records from DARTNet clinics are integrated into a single system provided by Clinical Integration Networks of America, Inc. (CINA), allowing central electronic data abstraction. Denver Health is avertically-integrated public safety net institution22 with eight community health centers and 13 school-based clinics. Patient-level clinical and pharmacy data from Denver Health are electronically accessible via a data warehouse. Clinics were recruited for this study to include adult and pediatric patients, urban, suburban, and rural locations, academic and private providers, and varying geographic locations.

Study design

We developed clinical pathways for eight common adult and pediatric outpatient infections: non-specific upper respiratory infection, acute bronchitis, acute rhinosinusitis, pharyngitis, acute otitis media, urinary tract infection, skin and soft tissue infection (cellulitis or cutaneous abscess), and community-acquired pneumonia (Appendix). Each pathway was a one-page decision support algorithm designed to assist providers in determining whether an antibiotic should be prescribed, the optimal antibiotic choice when indicated, and the shortest appropriate duration of therapy. In addition to the clinical pathways, the intervention consisted of patient education materials developed as part of a prior community antibiotic stewardship campaign.23

Eight participating clinics were randomized to receive either the clinical pathways and patient education materials (study group) or no intervention (control group). To avoid overrepresentation of clinics with similar characteristics or clinics from one institution in either arm, the eight clinics were initially stratified into groups of two based on the clinic population and institution (e.g., two Family Medicine clinics from Denver Health). Within each of these groups, one clinic was randomized to the intervention arm (study group) and the other to the control via a random number generator. Study clinics received binders containing hard copies of the clinical pathways for examination rooms and provider work areas as well as web-based access to the documents. One study clinic incorporated the clinical pathway recommendations into encounter templates in its electronic medical record. A provider at each of the four study clinics was selected as a peer champion to advocate for use of the clinical pathways during the intervention period. The study was approved by the Colorado Multiple Institutional Review Board.

The primary study outcomes were two related measures of antibiotic prescribing: 1) change over time in antibiotic prescriptions for non-pneumonia acute respiratory infections (hereby referred to as acute respiratory infections), conditions where antibiotics are typically not indicated,9 and 2) change over time in broad-spectrum antibiotic prescriptions for all clinical pathway conditions, including those for which antibiotic therapy is indicated (urinary tract infection, skin and soft tissue infection, and pneumonia). Acute respiratory infections were defined as upper respiratory infection, acute bronchitis, pharyngitis, acute rhinosinusitis, or acute otitis media. Broad-spectrum antibiotics were defined as second- and third-generation cephalosporins, fluoroquinolones, macrolides, β-lactam/β-lactamase inhibitor combinations, and aminoglycosides.10 We performed a quasi-experimental study to assess changes in antibiotic prescribing at the participating clinics during a 2-year period immediately prior to the intervention (baseline period: May 1, 2008 through April 30, 2010) and during a 1-year period after implementation of the pathways (intervention period: May 1, 2010 through April 30, 2011). International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes were used to identify clinic visits for the clinical pathway conditions (Table 1). Instances where two of these conditions were diagnosed at the same visit were excluded. Visits for clinical pathway conditions occurring greater than 90 days apart were analyzed as separate events. Antibiotic prescriptions were captured from electronic medical records at DARTNet clinics and from patient-level pharmacy fill data at Denver Health clinics.

Table 1
International Classification of Diseases, 9th Revision, Clinical Modification

To evaluate the safety of the intervention, we collected data on adverse events within 30 days of the index visit. Late antibiotic prescriptions and late follow-up visits were defined as those occurring 8 to 30 days after the index visit, respectively. Since all adverse event data could not be obtained through the CINA database, this analysis was limited to Denver Health sites (2 study clinics, 2 control clinics).

Data Analysis

The primary analytic technique was generalized linear mixed effects models to extend the traditional logistic model to accommodate repeated observations within clinics over time. This was applied to the two primary outcomes described above to model the probability of an antibiotic being prescribed over time in the study group and control group. A piecewise approach was used to model pre- and post-intervention time periods resulting in a mixed effects piecewise logistic regression model.24 Each model included an intercept, a variable indicating group membership (study vs. control), a time trend (slope) for the baseline period, a time trend for the intervention period, interactions of these time trends with group (study vs. control), and 11 seasonal indicator variables for the months January through November. Separate models were also developed for individual clinics. Aggregated proportions of antibiotic prescriptions and adverse events were compared between the baseline and intervention periods using the Pearson chi-square test. Comparisons of antibiotic prescriptions for individual conditions were not performed given the likelihood of confounding due to multiple comparisons. We used SAS Version 9.3 (SAS Institute, Cary, NC) for data analysis.

Results

Characteristics of the eight participating clinics are described in Table 2. The study group included clinics with more providers (46 vs. 34) and more patients served (52,766 vs. 48,881) than the control group. Most index visits for the clinical pathway conditions were due to acute respiratory infections (68.0% – 76.4%), and the proportions were similar among the baseline and intervention periods (Table 3).

Table 2
Characteristics of participating clinics by group.
Table 3
Index visits by group and time period.

The proportion of acute respiratory infections where an antibiotic was prescribed decreased in the study group from 42.7% of cases during the baseline period to 37.9% during the intervention period (Χ2 (1) = 50.8, p <.0001), representing a relative reduction of 11.2% (Table 4). Most of this effect was due to fewer prescriptions for upper respiratory infections (21.6% vs. 15.6%) and acute bronchitis (60.5% vs. 54.9%). In the control group, the overall change in antibiotic prescriptions for acute respiratory infections was not statistically significant (39.8% vs. 38.7%, relative reduction of 2.8%, Χ2(1) = 1.3, p = 0.25).

Table 4
Antibiotic prescriptions for non-pneumonia acute respiratory infections*

The results of the mixed effects piecewise logistic regression model of antibiotic prescriptions for acute respiratory infections are displayed in Figure 1. During the baseline period, there was a lower probability of antibiotic use in the control group than in the study group; however, there was not a significant difference in the trend of antibiotic use between the groups (F(1, 35968) = 0.5, p = 0.49) (Figure 1A). During the intervention period, there was a significant time trend (F(1, 35968) = 66.9, p<.0001) and a significant difference in trend between the study and control groups F(1, 35968) = 23.1, p<.0001) with a greater decline in antibiotic use in the study group (Figure 1A). There was also a significant seasonal effect with higher prescribing in the winter months and lower prescribing in the summer months. When the models were stratified by individual clinics, antibiotic use declined during the intervention period for 3 of the 4 study clinics (Figure 1B).

Figure 1
Mixed effects piecewise logistic regression models predicting antibiotic prescriptions for acute respiratory infections over time for study and control groups (Panel A) and for individual clinics (Panel B). Models for two control clinics are similar and ...

The proportion of all clinical pathway conditions where a broad-spectrum antibiotic was prescribed decreased from 26.4% to 22.6% (Χ2 (1) = 0.9, p<.0001) in the study group (Table 5), representing a relative reduction of 14.4%. This included reductions in broad-spectrum antibiotic use for upper respiratory infection, acute bronchitis, pharyngitis, acute rhinosinusitis, skin and soft tissue infections, and community-acquired pneumonia. Broad-spectrum antibiotic prescriptions for urinary tract infections increased. In the control group, broad-spectrum antibiotics were prescribed in 20.0% of cases during the baseline period and 19.4% during the intervention period, representing a relative reduction of 3.0% (Χ2 (1) = 50.8, p=0.35).

Table 5
Broad-spectrum antibiotic prescriptions for all clinical pathway conditions

In the mixed effects piecewise logistic regression analysis of broad-spectrum antibiotic use, there was not a significant difference in the trend of prescriptions between the study and control groups during the baseline period (F(1, 48367) = 1.1, p = 0.29) (Figure 2A). During the intervention period, there was a significant time trend (F(1, 48367) = 41.5, p<.0001) and a significant difference in the prescribing trend between the study and control groups, with a greater decline in broad-spectrum antibiotic use in the study group (F(1, 48367) = 10.7, p=0.001). Use of broad-spectrum antibiotics declined during the intervention period for 3 of the 4 individual study clinics (Figure 2B).

Figure 2
Mixed effects piecewise logistic regression models predicting broad-spectrum antibiotic prescriptions for all eight clinical pathway conditions over time for study and control groups (Panel A) and for individual clinics (Panel B).

Potential adverse events for the subset of clinics where these data were available are presented in Table 6. In the study clinics, there was a trend toward fewer late antibiotic prescriptions (4.9% vs. 3.9%, p=0.06) during the intervention compared with the baseline period. In the control clinics, there were more frequent late follow-up visits (3.3% vs. 4.2%, p=0.02) and a trend toward more frequent late antibiotic prescriptions (6.1% vs. 7.1%, p=0.06) during the intervention period.

Table 6
Adverse events within 30 days of index visit*

Discussion

Unnecessary antibiotic prescriptions and use of overly broad-spectrum antibiotics remain common in the primary care setting. During the first year of this clinical pathway-based intervention, clinics randomized to the intervention prescribed antibiotics 11% less frequently for non-pneumonia acute respiratory infections and used broad-spectrum antibiotics 14% less frequently for all eight clinical pathway conditions compared with the baseline period. Logistic regression models revealed significant differences in the trends of antibiotic use for acute respiratory infections and overall broad-spectrum antibiotic use between the study group and control group during the intervention period, with greater declines in the study group.

Clinical pathways for infections in the primary care setting have been studied previously. Samore and colleagues demonstrated a 10% relative reduction in antibiotic prescriptions for acute respiratory infections through dissemination of paper- and handheld device-based algorithms to primary care clinicians along with a community educational campaign.17 However, this result was achieved only during the second year of the intervention; during the first year, antibiotic prescriptions did not decrease. More recently, Weiss and colleagues described a large-scale clinical pathway-based intervention in Quebec that led to a 4.2% reduction in outpatient antibiotic prescriptions in the first year.19 Our intervention is notable in that we observed 11% and 14% relative reductions in prescriptions for acute respiratory infections and total broad-spectrum antibiotic use, respectively, during the first year. However, similar to these previous studies, antibiotics use remained common despite the intervention, particularly for acute bronchitis and rhinosinusitis where an antibiotic was prescribed in approximately half and two-thirds of cases, respectively.

In the studies by Samore and Weiss and colleagues, population-based estimates of antibiotic prescribing were made through combining retail pharmacy and census data. Since antibiotics may be prescribed for many different indications, a number of factors could affect this measure of antibiotic utilization. Our study is unique in that we were able to capture patient-level data regarding visits for clinical pathway conditions and antibiotic prescriptions associated with those visits. This is likely a more accurate means to assess changes in prescribing practices for specific conditions.

This intervention was designed to be practical and widely generalizable with three simple components: 1) clinical pathways, 2) patient education materials, and 3) peer champion support. The availability of our clinical pathways in the appendix and widely accessible patient education materials should enable implementation at any primary care practice. We encourage clinicians to advocate for use of these resources in their own offices to scale-up this low-resource and evidence-based antimicrobial stewardship intervention.

Given our methods of data capture, we were not able to assess the appropriateness of antibiotic prescriptions nor the duration of therapy prescribed. Although antibiotics do not provide substantial clinical benefit in the majority of acute respiratory infections, they are indeed indicated in a subset of cases (e.g., group A streptococcal pharyngitis).9 Similarly, broad-spectrum antibiotics have a role in certain clinical settings (e.g., complicated urinary tract infection). Our clinical pathways were developed to help clinicians identify specific clinical scenarios where antibiotics – narrow- or broad-spectrum – are warranted and the shortest appropriate duration of therapy. Therefore, this intervention may have had unmeasured effects on both the appropriateness and duration of antibiotic therapy.

Despite less overall antibiotic exposure in the study group during the intervention, adverse events such as emergency department visits and late antibiotic prescriptions were not increased in the subset of clinics where these data were available. Although the study was not powered to detect small differences in adverse events, the trends in the safety data between the baseline and intervention periods are consistent, supporting the relative safety of decreasing antibiotic exposure for the conditions studied.

This study has several limitations that warrant further discussion. First, our data revealed lower baseline antibiotic use for the study group than the control group. However, it is important to point out that this study was not designed to compare prescribing between groups given the small numbers of clinics leading to imbalances in group characteristics. Moreover, methods of data capture among DARTNet and Denver Health sites differed. As Denver Health prescriptions were obtained through pharmacy fill data, the prescribing rates for these clinics are underestimates. Data collection methods were consistent over time at each site; therefore, changes in antibiotic prescribing within each clinic (or group of clinics) before and after the intervention were the outcomes of interest. Second, use of electronic data to identify visits for clinical pathway conditions, antibiotic prescriptions, and adverse events leads to inevitable misclassification. The large number of patients and the pre-intervention post-intervention study design lessen the potential impact on results. Third, data on adverse events such as emergency department visits and hospitalizations were not available for DARTNet clinics. Fourth, this study design is subject to the Hawthorne effect, whereby providers may have changed prescribing practices simply because they were being observed in a study. Fifth, since we combined clinical pathways, patient education materials, and peer champion advocacy in our intervention, the relative contribution of each aspect of the intervention cannot be known. Multi-faceted interventions to change provider behavior have been shown to be more effective than single interventions;25,26 therefore, our intervention strategy likely had a larger impact than would have been observed with passive dissemination of the clinical pathways alone. Last, the 12-month intervention period may not have been of sufficient duration to demonstrate the true effect of this intervention over time.

Relative strengths of the study are its inclusion and cluster-randomization of diverse outpatient practices, large number of visits for the conditions under study, generalizability of the clinical pathways to locations where antimicrobial resistance rates and formularies vary, and practical nature of the intervention.

In summary, a widely generalizable clinical pathway-based intervention to improve antibiotic use for common outpatient infections modestly decreased antibiotic prescriptions for acute respiratory infections and overall use of broad-spectrum antibiotics during the first year. Further study is needed to evaluate the impact and sustainability of this intervention over a longer time period. Antibiotic use for acute bronchitis and rhinosinusitis was remarkably common despite the intervention and should be a focus of future research.

Supplementary Material

supplemental material

Acknowledgments

We wish to thank John Ogle (Denver Health), Ralph Gonzales (University of California San Francisco), Adam Hersh (University of Utah), and Thea Carruth (HealthTeamWorks) for their input on the clinical pathways and Bryan Knepper and Carolyn Valdez (Denver Health), Elias Brandt, Wilson Pace, and Diane Fairclough (University of Colorado Denver) for assistance with data abstraction and statistical analyses.

Funding: Funding was provided by the federal Agency for Healthcare Research and Quality (AHRQ) under Task Order Contract Number: HHSA290200710008, Task Order No. 7 (AHRQ Task Order Officer: Michael Parchman, MD, Task Order Leader: Connie Price, MD). Dr. Jenkins was also supported by the National Institute of Allergy and Infectious Diseases (1K23AI099082-01A1). The authors of this article are responsible for its content. No statement may be construed as the official position of the Agency for Healthcare Research and Quality of the United States Department of Health and Human Services.

Footnotes

Conflicts of interest: All authors, no conflicts.

All authors contributed to the study design, had access to the data, and participated in authorship of the manuscript.

References

1. Office-related antibiotic prescribing for persons aged </= 14 years--United States, 1993–1994 to 2007–2008. MMWR Morb Mortal Wkly Rep. 2011 Sep 2;60(34):1153–1156. [PubMed]
2. Goossens H, Ferech M, Vander Stichele R, Elseviers M. Outpatient antibiotic use in Europe and association with resistance: a cross-national database study. Lancet. 2005 Feb 12–18;365(9459):579–587. [PubMed]
3. Lipsitch M, Samore MH. Antimicrobial use and antimicrobial resistance: a population perspective. Emerging Infectious Diseases. 2002 Apr;8(4):347–354. [PMC free article] [PubMed]
4. Boucher HW, Talbot GH, Bradley JS, et al. Bad bugs, no drugs: no ESKAPE! An update from the Infectious Diseases Society of America. Clin Infect Dis. 2009 Jan 1;48(1):1–12. [PubMed]
5. Rice LB. The Maxwell Finland Lecture: for the duration-rational antibiotic administration in an era of antimicrobial resistance and clostridium difficile. Clin Infect Dis. 2008 Feb 15;46(4):491–496. [PubMed]
6. Dellit TH, Owens RC, McGowan JE, Jr, et al. Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America guidelines for developing an institutional program to enhance antimicrobial stewardship. Clin Infect Dis. 2007 Jan 15;44(2):159–177. [PubMed]
7. Gallini A, Degris E, Desplas M, et al. Influence of fluoroquinolone consumption in inpatients and outpatients on ciprofloxacin-resistant Escherichia coli in a university hospital. The Journal of Antimicrobial Chemotherapy. 2010 Dec;65(12):2650–2657. [PubMed]
8. Hicks LA, Chien YW, Taylor TH, Jr, Haber M, Klugman KP. Outpatient antibiotic prescribing and nonsusceptible Streptococcus pneumoniae in the United States, 1996–2003. Clinical Infectious Diseases: an official publication of the Infectious Diseases Society of America. 2011 Oct;53(7):631–639. [PubMed]
9. Gonzales R, Malone DC, Maselli JH, Sande MA. Excessive antibiotic use for acute respiratory infections in the United States. Clin Infect Dis. 2001 Sep 15;33(6):757–762. [PubMed]
10. Steinman MA, Landefeld CS, Gonzales R. Predictors of broad-spectrum antibiotic prescribing for acute respiratory tract infections in adult primary care. JAMA. 2003 Feb 12;289(6):719–725. [PubMed]
11. McCaig LF, Besser RE, Hughes JM. Trends in antimicrobial prescribing rates for children and adolescents. JAMA. 2002 Jun 19;287(23):3096–3102. [PubMed]
12. Roumie CL, Halasa NB, Edwards KM, Zhu Y, Dittus RS, Griffin MR. Differences in antibiotic prescribing among physicians, residents, and nonphysician clinicians. The American Journal of Medicine. 2005 Jun;118(6):641–648. [PubMed]
13. Hersh AL, Shapiro DJ, Pavia AT, Shah SS. Antibiotic prescribing in ambulatory pediatrics in the United States. Pediatrics. 2011 Dec;128(6):1053–1061. [PubMed]
14. Arnold SR, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Database Syst Rev. 2005;(4):CD003539. [PubMed]
15. Ranji SR, Steinman MA, Shojania KG, Gonzales R. Interventions to reduce unnecessary antibiotic prescribing: a systematic review and quantitative analysis. Med Care. 2008 Aug;46(8):847–862. [PubMed]
16. Jenkins TC, Knepper BC, Sabel AL, et al. Decreased antibiotic utilization after implementation of a guideline for inpatient cellulitis and cutaneous abscess. Arch Intern Med. 2011 Jun 27;171(12):1072–1079. [PubMed]
17. Samore MH, Bateman K, Alder SC, et al. Clinical decision support and appropriateness of antimicrobial prescribing: a randomized trial. JAMA. 2005 Nov 9;294(18):2305–2314. [PubMed]
18. Marrie TJ, Lau CY, Wheeler SL, Wong CJ, Vandervoort MK, Feagan BG. A controlled trial of a critical pathway for treatment of community-acquired pneumonia. CAPITAL Study Investigators. Community-Acquired Pneumonia Intervention Trial Assessing Levofloxacin. JAMA. 2000 Feb 9;283(6):749–755. [PubMed]
19. Weiss K, Blais R, Fortin A, Lantin S, Gaudet M. Impact of a multipronged education strategy on antibiotic prescribing in Quebec, Canada. Clin Infect Dis. 2011 Sep;53(5):433–439. [PubMed]
20. Dellit TH, Chan JD, Skerrett SJ, Nathens AB. Development of a guideline for the management of ventilator-associated pneumonia based on local microbiologic findings and impact of the guideline on antimicrobial use practices. Infect Control Hosp Epidemiol. 2008 Jun;29(6):525–533. [PubMed]
21. Pace WD, Cifuentes M, Valuck RJ, Staton EW, Brandt EC, West DR. An electronic practice-based network for observational comparative effectiveness research. Ann Intern Med. 2009 Sep 1;151(5):338–340. [PubMed]
22. Gabow P, Eisert S, Wright R. Denver Health: a model for the integration of a public hospital and community health centers. Ann Intern Med. 2003 Jan 21;138(2):143–149. [PubMed]
23. Gonzales R, Corbett KK, Wong S, et al. “Get smart Colorado”: impact of a mass media campaign to improve community antibiotic use. Med Care. 2008 Jun;46(6):597–605. [PubMed]
24. Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2. Thousand Oaks, CA: Sage Publications, Inc; 2002.
25. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ. 1998 Aug 15;317(7156):465–468. [PMC free article] [PubMed]
26. Grimshaw JM, Shirran L, Thomas R, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001 Aug;39(8 Suppl 2):II2–45. [PubMed]