The Ottawa Knee Rule: Examining Use in an Academic Emergency
Department

doi:10.5811/westjem.2012.2.6892

West J Emerg Med. 2012 September; 13(4): 366–372.

doi: 10.5811/westjem.2012.2.6892

PMCID: PMC3523897

The Ottawa Knee Rule: Examining Use in an Academic Emergency Department

Bryan G. Beutel, BS,^* Samir K. Trehan, AB,^* Robert M. Shalvoy, MD,^† and Michael J. Mello, MD, MPH^*^‡^§

^*Injury Prevention Center at Rhode Island Hospital, Providence, Rhode Island

^†Department of Orthopedics, The Warren Alpert Medical School of Brown University, Providence, Rhode Island

^‡Department of Emergency Medicine, The Warren Alpert Medical School of Brown University, Providence, Rhode Island

^§Department of Community Health, The Warren Alpert Medical School of Brown University, Providence, Rhode Island

Address for Correspondence: Michael J. Mello, MD, MPH, Injury Prevention Center at Rhode Island Hospital, 55 Claverick Street, 2^nd Floor, Providence, RI, USA 02903. Email: mjmello/at/lifespan.org.

Supervising Section Editor: Ali S. Raja, MD

Reprints available through open access at http://escholarship.org/uc/uciem_westjem

Received September 11, 2011; Revised December 8, 2011; Accepted February 6, 2012.

This is an Open Access article distributed under the terms of the Creative Commons Non-Commercial Attribution License, which permits its use in any digital medium, provided the original work is properly cited and not altered. For details, please refer to http://creativecommons.org/licenses/by-nc-nd/3.0/. Authors grant Western Journal of Emergency Medicine a nonexclusive license to publish the manuscript.

Abstract

Introduction:

The Ottawa Knee Rule is a validated clinical decision rule for determining whether knee radiographs should be obtained in the setting of acute knee trauma. The objectives of this study were to assess physician knowledge of, barriers to implementation of, and compliance with the Ottawa Knee Rule in academic emergency departments (EDs), and evaluate whether patient characteristics predict guideline noncompliance.

Methods:

A 10 question online survey was distributed to all attending ED physicians working at three affiliated academic EDs to assess knowledge, attitudes and self-reported practice behaviors related to the Ottawa Knee Rule. We also performed a retrospective ED record review of patients 13 years of age and older who presented with acute knee trauma to the 3 study EDs during the 2009 calendar year, and we analyzed ED records for 19 variables.

Results:

ED physicians (n = 47) correctly answered 73.2% of questions assessing knowledge of the Ottawa Knee Rule. The most commonly cited barriers to implementation were “patient expectations” and system issues, such as “orthopedics referral requirement.” We retrospectively reviewed 838 records, with 260 eligible for study inclusion. The rate of Ottawa Knee Rule compliance was retrospectively determined to be 63.1%. We observed a statistically significant correlation between Ottawa Knee Rule compliance and patient age, but not gender, insurance status, or provider type, among others.

Conclusion:

Compliance with the Ottawa Knee Rule among academic ED healthcare providers is poor, which was predicted by patient age and not other physician or patient variables. Improving compliance will require comprehensive educational and systemic interventions.

INTRODUCTION

Knee pain is a common presenting complaint in the emergency department (ED) and accounts for approximately 1.3 million ED visits annually in the United States (US).¹ Despite the prevalence of acute knee trauma, fractures are only observed in approximately 6% of patients.² Knee radiographs to detect fractures, however, remain one of the most commonly ordered studies and account for $1 billion in healthcare spending annually.³^,⁴

The Ottawa Knee Rule was published in 1995 by Stiell et al⁵ as a tool for determining whether knee radiographs should be obtained to detect a fracture in the setting of acute knee trauma. The rule states that knee radiographs are indicated in these patients if at least one of the following criteria is satisfied: the patient is at least 55 years old, has an inability to bear weight immediately after trauma and in the ED for four steps (regardless of limp), has isolated patellar tenderness, fibular head tenderness, or an inability to flex the knee to 90°. ¹^,⁵^,⁶ Stiell et al1 validated the rule in 2 prospective studies, which showed the criteria to be 100% sensitive for identifying clinically-significant fractures. Additionally, Bachmann et al⁷ published a meta-analysis concluding that the Ottawa Knee Rule was 99% sensitive and 49% specific for knee fracture. While the criteria were initially validated in adults, prospective studies and meta-analyses also demonstrated that they are 99–100% sensitive and 43–46% specific in children over 5 years of age.⁸^,⁹

When used appropriately the Ottawa Knee Rule has been shown to reduce unnecessary imaging. In an implementation trial conducted by Stiell et al⁶, educational interventions on the Ottawa Knee Rule targeting ED physicians decreased the proportion of knee radiographs by 26.4%, ED visit costs by $34 per patient, and time spent in the ED by 33.1 minutes.¹⁰ Another prospective trial demonstrated a 35% decrease in knee radiographs.¹¹

Despite evidence supporting use of the Ottawa Knee Rule, its adoption has been limited. A multi-national survey of ED physicians 10 years ago investigated this issue.¹² According to the results, self-reported knowledge of the Ottawa Knee Rule was highest in English-speaking countries, yet use of the rule among those with knowledge of the criteria was lowest in the U.S. While the majority of respondents agreed that clinical decision rules are intended to reduce healthcare costs and improve quality of care, physicians in the U.S. were most likely to agree that such rules are oversimplified, increase the likelihood of being sued, and challenge physician authority. These studies suggest that a multitude of factors contribute to limited use of the Ottawa Knee Rule and consequent radiograph overuse. Knee radiographs remain one of the most commonly performed imaging studies and, despite low fracture rates, are obtained in 60–80% of acute knee trauma cases.³^,¹³^,¹⁴ While the low adoption rate of the Ottawa Knee Rule is known, there are currently no published studies assessing physician knowledge of the rule, or patient or physician predictors of compliance with it.

In the present study, 15 years after publication of this clinical decision rule, we aimed to (1) evaluate knowledge, attitudes and self-reported practice behaviors regarding the Ottawa Knee Rule among attending academic U.S. ED physicians; (2) determine Ottawa Knee Rule adherence among ED providers; and (3) examine if patient level characteristics predict guideline noncompliance. We hypothesized that Ottawa Knee Rule adherence is poor. Additionally, we hypothesized that systems-based barriers inherent to the ED setting and patient factors prevent appropriate application of the rule.

METHODS

The Institutional Review Boards of the participating hospitals and the university approved all aspects of the study protocol.

ED Physician Survey

To assess ED physician knowledge, and barriers to implementation, of the Ottawa Knee Rule, the authors designed a 10-question online survey. The questions were developed by extrapolating scenarios from the rule criteria, and by phrasing inquiries about basic demographic and other information in plain language agreed upon by all authors. The survey was administered to all 76 ED attending physicians (all board certified in emergency medicine and/or pediatric emergency medicine) who were faculty at one of the nation’s largest academic emergency medicine departments in June 2010. The three study institutions’ EDs were staffed by members of the same university-based physician group and included an academic trauma center, a community teaching hospital and an academic pediatric specialty hospital with a combined annual volume of approximately 200,000 ED visits.

Of the questions, 2 related to the demographics of the respondents, 5 evaluated knowledge of the Ottawa Knee Rule through case vignettes and guideline questions, one probed for self-reported adherence to the rule, and 2 inquired about potential barriers to implementation (see Appendix for full survey questions). One of the barriers-to-implementation questions asked respondents to identify the top three reasons (out of nine choices) for ordering a knee radiograph in the absence of the Ottawa Knee Rule criteria being met; many of these answer choices were adapted from Graham et al’s¹² physician survey. We disseminated the survey link via email. All data were collected with identification of the respondent, and a $5 gift card was offered for survey completion. The survey responses, however, were not linked to the respondent’s name.

We evaluated knowledge of the Ottawa Knee Rule by comparing the proportion of participants who answered the 5 vignettes and guideline questions correctly with the total number of participants. Similarly, we calculated simple percentages for the responses to the 2 survey questions relating to the most commonly reported barriers.

Medical Record Review

To corroborate self-reported data on adherence to the Ottawa Knee Rule, as well as collect patient level characteristics that may influence guideline compliance, we conducted a retrospective ED medical record review. The review notes were not directly linked to the physicians who responded to the survey.

Our study population consisted of patients 13 years of age and older who presented with acute knee trauma to the three study institutions’ EDs between January 1, 2009 and December 31, 2009. To include the entire teenage population and broaden the potential data set we chose 13 as a lower age limit Patients were initially identified by querying the ED billing database for 16 International Classification of Diseases 9 (ICD-9) codes related to acute knee and lower extremity trauma (716.1, 717, 718.86, 822, 823, 823.1, 823.8, 823.9, 827, 836, 844, 891, 916, 924, 928, 959.7). We then applied the same exclusion criteria used in the Ottawa Knee Rule’s validation studies (except for age less than 18 years), which excluded patients who were referred from outside hospitals with knee radiographs, sustained knee trauma more than seven days previously, returned for reassessment of the same injury, had isolated skin injury, had multiple trauma, were pregnant, were paraplegic or had an altered level of consciousness.¹^,⁵ Two data abstracters then examined medical records of eligible patients for 19 principal variables gathered from ED nursing records, ED physician records and radiology reports. We entered data into Microsoft Excel; any missing data were noted. The authors managed any discrepancies between nursing and physician records by deferring to physician documentation for assessment of principal variables.

We organized the principal variables by categories such as “patient characteristics” (e.g. age, gender, insurance status, previous knee injury), “injury features” (e.g. mechanism, setting, the 5 Ottawa Knee Rule criteria, diagnosis), “radiograph ordering” (e.g. imaging, type of provider ordering films), and “other” (e.g. time spent in the ED, returned to the ED within 2 weeks for reassessment). From these, we evaluated Ottawa Knee Rule compliance by comparing the proportion of patients who had knee radiographs obtained with the proportion of patients in whom knee radiographs were indicated according to the rule. We assessed potential association of Ottawa Knee Rule adherence with patient variables, adjusted for multiple comparisons, using the Fisher’s Exact test, which was calculated in Statistical Analysis System 8.2 (SAS, SAS Institute Inc., Cary, NC). Calculations involving time spent in the ED excluded patients who were admitted to the hospital or taken to the operating room, and were made using the Wilcoxon Rank Sum Test. We determined inter-rater reliability between the 2 reviewers by calculating the Cohen’s kappa value based upon whether the Ottawa Knee Rule criteria were met from a random subset of 33 medical records.

RESULTS

ED Physician Survey

Forty-seven out of 76 ED physicians responded to our survey (61.8% response rate). On average, respondents worked 22.3 (standard deviation (SD) 7.8, 95% confidence interval (CI) 20.1–24.5) hours per week in the ED and had 8.7 (SD 8.3, CI 6.4–11.1) years of experience.

Physicians scored an average of 73.2% (CI 66.6–79.8) on questions assessing Ottawa Knee Rule knowledge, with 2.1% of respondents correctly answering all of the questions, 78.7% answering all but 1 of the questions correctly, and 0% answering all of the questions incorrectly. Only 36.2% of responding physicians, however, chose to withhold diagnostic imaging in a case vignette that did not satisfy the Ottawa Knee Rule criteria (an ambulatory adolescent with a severe limp who exhibited full range of motion of the knee with no focal tenderness).

Generally, self-reported adherence to the Ottawa Knee Rule was poor. More than one-third (36.2%) of physicians reported never using the guideline, while only 23.4% of physicians used the rule “always” or “most of the time.” The most commonly cited barriers to rule implementation were “patient expectations” and “patient satisfaction” (Figure). Other barriers frequently identified included an orthopedics referral requirement and lack of confidence in physical exam findings. Finally, ED physicians reported that the majority (53.2%) of radiographs for knee injury patients were ordered by non-attending physician providers (e.g. residents, triage nurses and physician assistants).

Figure.

Self-reported barriers to implementation of the Ottawa Knee Rule.

Medical Record Review

The ED billing database query for ICD-9 codes, restricted by target ages and dates, identified 838 patient visits. Upon review of these records, 437 did not have knee trauma, 129 met exclusion criteria, and 12 did not record sufficient information to determine if the Ottawa Knee Rule criteria were met. Consequently, 260 records were eligible for study inclusion. The inter-rater reliability was high (Cohen’s kappa = 0.939).The demographics, community setting, and etiology of acute knee trauma in our study population are summarized in Table 1. Of the 260 patients, 198 (76.2%) had a knee radiograph and 1 had a magnetic resonance image (Table 2). Forty-one patients had clinically-significant fractures (39 involving the patella). Only 17 patients returned to the ED within 2 weeks for reevaluation of the same injury. Of these patients, 16 had radiographs at the initial visit and none had a revised diagnosis based on reevaluation. The mean time spent in the ED was 4.1 hours (median 3.4; interquartile range 2.7 hours). Upon excluding the 12 patients who were admitted or taken to the operating room, the mean time was 3.9 hours (median 3.3; interquartile range 2.3 hours). Patients who had a radiograph spent more time in the ED (mean 4.1, median 3.5, interquartile range 2.5 hours) than those who did not (mean 3.2, median 2.8, interquartile range 1.6 hours) (p < 0.05).Ottawa Knee Rule criteria fulfillment is summarized in Table 3. While some data were missing from the medical records, all cases had sufficient information to assess rule fulfillment. Taken together, 65.4% of patients fulfilled at least 1 of the Ottawa Knee Rule criteria and therefore warranted a knee radiograph. Additionally, the rule was 100% sensitive and 40.7% specific for fracture in the 260 cases. We assessed Ottawa Knee Rule compliance by comparing criteria fulfillment with radiograph obtainment (Table 4). Out of 260 cases, 164 had knee radiographs either appropriately obtained or appropriately not obtained, and were therefore considered compliant with the Ottawa Knee Rule. Conversely, 96 cases had knee radiographs either inappropriately obtained or inappropriately not obtained, and were considered noncompliant. Patients who had a radiograph inappropriately obtained spent an average of 3.7 hours (median 3.5; interquartile range 1.7 hours) in the ED. Overall, the rate of Ottawa Knee Rule compliance was 63.1%.

Table 1.

Acute knee trauma patient demographics and injury description.

Table 2.

Acute knee trauma diagnosis.

Table 3.

Ottawa Knee Rule criteria fulfillment.

Table 4.

Ottawa Knee Rule compliance as determined by ED medical record review.

The association between Ottawa Knee Rule compliance and various patient and provider factors is summarized in Table 5. A statistically significant association (p = 0.01) was only observed with patient age. Compliance was higher in patients aged 13–18 years old (76.5%) and ≥55 years old (68.7%), as compared to 19–54 year olds (54.0%). No statistically significant (p<0.05) differences were noted with provider type, patient gender, patient insurance status, previous ipsilateral knee injury, sports injury or mechanism of injury.

Table 5.

Association of Ottawa Knee Rule compliance with patient and provider factors.

DISCUSSION

To our knowledge, this is the first retrospective record review to analyze Ottawa Knee Rule compliance and potential associations between compliance with provider and patient factors.

The survey response rate was acceptable, and consistent with other surveys targeting physicians.¹⁵ The survey demonstrated that ED physician knowledge of the Ottawa Knee Rule was good, but self-reported adherence was poor. Interestingly, of the 5 vignettes, the scenario with the lowest correct response rate (36.2%) was the only case in which imaging was not indicated according to the Ottawa Knee Rule. That is, despite overall acceptable knowledge of the Ottawa Knee Rule criteria and its application, physicians were still hesitant to withhold imaging. This was also found in our record review, where one-third of radiographs were ordered for patients not meeting any criteria. Furthermore, physicians noted that the primary barriers to Ottawa Knee Rule implementation were related to patient and systems barriers rather than the criteria themselves. The results of our recent survey coupled with the early findings by Graham et al¹² suggest that noncompliance with the Ottawa Knee Rule is currently likely more attributable to systemic concerns, such as orthopedic consultation demands and malpractice implications (as indicated by the “Legal concern” column in Figure), than lack of knowledge. Addressing these systemic concerns is important to maximize adherence to the Ottawa Knee Rule.

Poor compliance with the Ottawa Knee Rule, as reported by ED physicians in our survey, was confirmed by retrospective review. This demonstrated a compliance rate of 63.1%. Overall, 76.2% of patients received knee radiographs, consistent with the previously published result of 74% in Stiell et al’s³ retrospective study. Our study population, however, had a fracture rate of 15.8%, which is higher than previously published figures of 6–7%.²^,³ This discrepancy may be due to the fact that 2 of our study hospitals are Level I trauma centers.

Additionally, physician respondents to our survey reported that the majority of radiographs were ordered by non-attending physician providers. Review of medical records confirmed this – only 34.4% of radiographs were ordered by attending physicians, while 24.2% were ordered by residents, 18.8% by nurses, and 22.6% by physician assistants. Recognizing that other healthcare providers influence radiograph ordering, Matteucci et al¹⁶ performed a prospective study in which both physicians and triage nurses were educated on the Ottawa Knee Rule. This training led to 37% and 21% relative reductions in radiograph ordering among physicians and triage nurses, respectively, although triage nurses still ordered 3.6 times more radiographs than physicians. Our hypothesis that rule compliance would be higher when physicians ordered radiographs proved incorrect, as there was no association with provider type. Future educational efforts should target all ED healthcare providers, as well as consulting and follow-up services such as orthopedics, given the significant proportion of radiographs ordered by non-attending physicians.

We further aimed to determine which patient level variables correlated with Ottawa Knee Rule compliance. Patient age was the only factor to have a statistically significant correlation. Compliance was significantly higher in younger (≤ 18 years old) and older (≥ 55 years old) patients, as compared to patients aged 19–54 years. The higher compliance rate in the older group is consistent with the fact that all patients in this age range warrant a radiograph (per rule criteria). Additionally, the higher compliance rate in the younger group may be attributed to provider hesitation to order imaging in pediatric patients given concerns of radiation exposure.

We also examined whether Ottawa Knee Rule compliance correlated with reduced ED wait times and radiograph ordering. After excluding patients taken to the operating room or admitted to the hospital from the ED, patients who had a knee radiograph spent 53 minutes longer in the ED (mean of 4.1 hours) compared to those who received no radiograph (mean of 3.2 hours). Since this difference was potentially confounded by injury severity, we also performed the analysis after excluding fracture diagnoses. Even after removing these patients, those who had a knee radiograph spent 47 minutes longer in the ED. These figures are comparable to 2 previous studies by Stiell et al,¹^,⁶ which showed that patients receiving knee radiographs spend 33–39 minutes longer in the ED.

Finally, the Ottawa Knee Rule was 100% sensitive and 40.7% specific for fracture in our retrospective study. Our study was not implicitly designed to determine these calculations, as a fracture diagnosis may have been missed in cases where radiographs were not obtained. Of the minority of patients who did return to the ED within 2 weeks with similar complaints, however, none had a missed fracture. These sensitivity and specificity values are consistent with results published in the literature.⁷^,⁹

As noted in multiple reviews, a multi-faceted approach is often the most effective technique in enhancing adherence to clinical guidelines.¹⁷^,¹⁸ As such, several interventions could be employed to improve Ottawa Knee Rule compliance. For example, reminders could be introduced by incorporating a diagram outlining the Ottawa Knee Rule criteria on ED history and physical examination templates. Alternatively, prompts can be integrated into electronic ordering systems asking the provider whether or not the patient has satisfied rule criteria when ordering a knee radiograph. Similar computerized decision support systems have yielded significant benefits on provider performance outcomes.¹⁹ The results of our study also highlight the importance of focusing these educational and system level efforts not only on attending physicians but all ED providers.

LIMITATIONS

The physician survey and medical record review share certain common limitations. Our ability to generalize our findings is limited since the survey only queried ED attending physicians (it did not query ED nurses, residents, or physician assistants), and the medical records were from three hospitals affiliated with the same academic institution. Additionally, this study investigated an academic emergency medicine population that worked an average of 22.3 clinical hours per week, and consequently may not represent general community emergency physician practices. Specific limitations exist for the online survey. Our response rate of 61.8%, while acceptable for physician surveys, is not ideal.²⁰ Furthermore, while the questions assessing Ottawa Knee Rule knowledge were carefully designed by the authors, they were not validated or reviewed by a committee to establish face validity. In addition, the clinical vignettes involved patients aged 14–16 years old. Note that although the Ottawa Knee Rule has been validated in the pediatric population, the original study was conducted in patients aged 18 years or older. Additionally, using a more open-ended question when exploring barriers to implementation may have offered more diverse responses, although this would not have necessarily made for a stronger design. Moreover, it is possible that respondents gave socially-desirable answers to the survey that may not reflect their actual practice. This, however, is less likely considering that the survey was administered without being linked to the respondent’s name. Several factors also limit our record review. Missing data from records limited our analysis. Furthermore, even in cases where all data was present, it is impossible to determine whether compliance was secondary to coincidence or conscious determination by healthcare providers to employ the decision tool. Nonetheless, this would have the effect of overestimating the adherence rate, suggesting that actual compliance is lower than our results indicate.

CONCLUSION

Compliance with the Ottawa Knee Rule among academic ED healthcare providers is poor. Patient concerns and system issues, rather than issues intrinsic to the rule itself, continue to serve as barriers to proper implementation of this validated decision tool. Addressing these concerns is essential to maximizing guideline adherence and mitigating unnecessary imaging. Improving compliance will require a comprehensive approach involving both education (of attending and non-attending providers alike) and system interventions, such as have been used with other clinical decision rules.

Appendix

Click here to view.^{(566K, pdf)}

Acknowledgments

The first two authors contributed equally to this study. Additionally, the authors would like to thank Anthony Nunes for his assistance with the statistical analysis.

Footnotes

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1. Stiell I, Greenberg G, Wells G, et al. Prospective validation of a decision rule for the use of radiography in acute knee injuries. JAMA. 1996:611–15. [PubMed]

2. Tandeter H, Shvartzman P. Acute knee injuries: use of decision rules for selective radiograph ordering. Am Fam Physician. 1999;60:2599–608. [PubMed]

3. Stiell I, Wells G, McDowell I, et al. Use of radiography in acute knee injuries: need for clinical decision rules. Acad Emerg Med. 1995:966–73. [PubMed]

4. Verma A, Su A, Golin AM, et al. A screening method for knee trauma. Acad Radiol. 2001;8:392–7. [PubMed]

5. Stiell I, Greenberg G, Wells G. Derivation of a decision rule for the use of radiography in acute knee injuries. Ann Emerg Med. 1995:405–13. [PubMed]

6. Stiell I, Wells G, Hoag R. Implementation of the Ottawa knee rule for the use of radiography in acute knee injuries. JAMA. 1997:2075–9. [PubMed]

7. Bachmann L, Haberzeth S, Steurer J, et al. The accuracy of the Ottawa knee rule to rule out knee fractures: a systematic review. Ann Intern Med. 2004:121–7. [PubMed]

8. Bulloch B, Neto G, Plint A, et al. Validation of the Ottawa Knee Rules in children: A multicenter study. Ann Emerg Med. 2003:48–58. [PubMed]

9. Vijayasankar D, Boyle A, Atkinson P. Can the Ottawa knee rule be applied to children? A systematic review and meta-analysis of observational studies. Emergency Medicine Journal. 2009;26:250–3. [PubMed]

10. Nichol G, Stiell I, Wells G, et al. An economic analysis of the Ottawa Knee Rule. Ann Emerg Med. 1999:438–47. [PubMed]

11. Jenny J, Boeri C, El Amrani H. Should plain X-rays be routinely performed after blunt knee trauma? A prospective analysis. The Journal of Trauma. 2005:1179–82. [PubMed]

12. Graham I, Stiell I, Laupaois A. Awareness and use of the Ottawa knee and ankle Rules in 5 counties: can publication alone be enough to change practice? Ann Emerg Med. 2001:259–66. [PubMed]

13. McConnochie K, Roghmann K, Pasternack J, et al. Prediction rules for selective radiographic assessment of extremity injuries in children and adolescents. Pediatrics. 1990:45–57. [PubMed]

14. Jackson JL, O’Malley PG, Kroenke K. Evaluation of acute knee pain in primary care. Ann Intern Med. 2003;139:575–88. [PubMed]

15. Balter KA, Balter O, Fondell E, et al. Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology. 2005;16:577–9. [PubMed]

16. Matteucci M, Roos J. Ottawa Knee Rule: a comparison of physician and triage-nurse utilization of a decision rule for knee injury radiography. J Emerg Med. 2003:147–50. [PubMed]

17. Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8:1–72. [PubMed]

18. Davis DA, Thomson MA, Oxman AD, et al. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995;274:700–5. [PubMed]

19. Main C, Moxham T, Wyatt JC, et al. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems. Health Technol Assess. 2010;14:1–227. [PubMed]

20. Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol. 1997;50:1129–36. [PubMed]

Articles from Western Journal of Emergency Medicine are provided here courtesy of
the University of California, Irvine