|Home | About | Journals | Submit | Contact Us | Français|
The primary care evaluation of chest pain represents a significant diagnostic challenge.
To determine if electronic alerts to physicians can improve the quality and safety of chest pain evaluations.
Randomized, controlled trial conducted between November 2008 and January 2010 among 292 primary care clinicians caring for 7,083 adult patients with chest pain and no history of cardiovascular disease.
Clinicians received alerts within the electronic health record during office visits for chest pain. One alert recommended performance of an electrocardiogram and administration of aspirin for high risk patients (Framingham Risk Score (FRS)≥10%), and a second alert recommended against performance of cardiac stress testing for low risk patients (FRS<10%).
The primary outcomes included performance of an electrocardiogram and administration of aspirin therapy for high risk patients; and avoidance of cardiac stress testing for low risk patients.
The majority (81%) of patients with chest pain were classified as low risk. High risk patients were more likely than low risk patients to be evaluated in the emergency department (11% versus 5%, p<0.01) and to be hospitalized (7% versus 3%, p<0.01). Acute myocardial infarction occurred among 26 (0.4%) patients, more commonly among high risk compared to low risk patients (1.1% versus 0.2%, p<0.01). Among high risk patients, there was no difference between the intervention and control groups in rates of performing electrocardiograms (51% versus 48%, p=0.33) or administering aspirin (20% versus 18%, p=0.43). Among low risk patients, there was no difference between intervention and control groups in rates of cardiac stress testing (10% versus 9%, p=0.40).
Primary care management of chest pain is suboptimal for both high and low risk patients. Electronic alerts do not increase risk-appropriate care for these patients.
The evaluation of acute chest pain represents a substantial burden on the US health care system. The challenge is to reliably identify life-threatening causes such as acute myocardial infarction, while avoiding unnecessary evaluations for those patients with less worrisome etiologies.1,2 This challenge is highlighted in the primary care office, where one-quarter of all chest pain evaluations take place,3 and where important gaps in the quality and safety of chest pain evaluations exist.4,5
There is an extensive literature to guide the evaluation of chest pain in the emergency department;1,2 however, the optimal primary care approach may be fundamentally different. Primary care offices have limited capability to perform serial cardiac enzymes or immediate exercise stress testing as is done in the emergency department,6 and the patients are relatively low risk for cardiovascular disease.7 The relative infrequency of serious diagnoses, combined with time pressures and fear of litigation may lead to a combination of performing many unnecessary cardiac stress tests for low risk patients,8 and a high potential to miss the diagnosis of the uncommon acute myocardial infarction. Indeed, the majority of primary care patients with chest pain do not have an electrocardiogram performed or receive aspirin therapy,4,5 despite their key role in the evaluation and management of this condition.9–11 Ultimately, missed diagnosis of myocardial infarction in primary care is common.5
The dual nature of chest pain evaluations challenges primary care clinicians to limit excess testing, while also avoiding missed diagnoses of acute myocardial infarction. Electronic health records are promoted as an important solution to improving safety and efficiency.12 Providing clinicians with real-time information to promote risk-appropriate clinical decisions may increase the accuracy and efficiency of chest pain care,13,14 and the Framingham Risk Score represents a reliable tool to risk stratify patients with chest pain in primary care.4,5 We conducted a randomized, controlled trial to assess the impact of delivering real-time cardiac risk information within an electronic health record to primary care clinicians evaluating patients complaining of chest pain.
This 15-month trial was conducted from November 2008 through January 2010 at Harvard Vanguard Medical Associates, an integrated multispecialty group practice in eastern Massachusetts. These practices utilize a common electronic health record (Epic Systems). We have documented underutilization of electrocardiograms and aspirin therapy, as well as missed diagnoses of acute myocardial infarction among patients presenting with chest pain in this group practice.5
We randomized 212 physicians, nurse practitioners, and physician assistants practicing across 15 health centers (Fig. 1). We consecutively enrolled patients ≥30 years old on the first occasion of presenting with chest pain to these clinicians during the study period. We excluded any patients with a prior history of cardiovascular disease, emergency department evaluation for chest pain within the prior 30 days, or those patients presenting for an annual physical examination.
Medical assistants underwent training for 12 months prior to the initiation of the randomized trial. We conducted a 1-hour training session with all medical assistants to review the definition of chest pain. Medical assistants entered a chest pain-specific code into the electronic health record chief complaint field prior to the clinician evaluating the patient. Medical assistants received detailed performance reports every two weeks documenting their accuracy with identifying chest pain based on manual chart reviews. During visits for which chest pain was the primary complaint documented by clinicians, 70% of encounters coded by the clinicians were captured by the medical assistants.
The Brigham and Women’s Hospital Human Studies Committee approved the study protocol. A waiver of informed consent was approved for clinicians and patients.
We developed a set of electronic alerts based on automated calculation of the patient’s Framingham Risk Score at the time of the office visit.15 We stratified patients according to their Framingham Risk Score, with ‘high risk’ patients defined as those with a score≥10% and ‘low risk’ patients defined as those with a score<10%. This differentiation within the Framingham Risk Score has good predictive ability in identifying those primary care patients with chest pain at high risk for acute myocardial infarction, with a sensitivity of 85% and specificity of 75%.5 The required variables were extracted from the electronic record, including patient age, sex, total and HDL cholesterol, smoking status, systolic blood pressure, and presence of antihypertensive therapy. Presence of diabetes was treated as a coronary heart disease risk equivalent and these patients were assigned a risk score of 20%. Our previous work has demonstrated minimal missing data for these variables.5 In the case of missing data, we imputed values into the Framingham risk calculator that did not raise the overall risk score.
We developed two electronic alerts that triggered based on the presence of a coded chief complaint of chest pain (Fig. 2). During office visits for ‘high risk’ patients, clinicians received an alert recommending the performance of an electrocardiogram and the administration of aspirin; and these alerts facilitated “one-click” ordering of these recommendations. We targeted electrocardiogram performance based on its diagnostic importance and minimal associated patient risk. We targeted aspirin therapy based on its proven effectiveness in the early management of acute myocardial infarction along with its recommendation for routine preventive use among even asymptomatic adults with a Framingham Risk Score greater than 10%.16
During office visits for ‘low risk’ patients complaining of chest pain, clinicians ordering cardiac stress tests received an alert recommending against performance of this test. This recommendation is consistent with published guidelines regarding the evaluation of chest pain and the questionable utility of cardiac stress testing for patients with a pre-test probability of heart disease of less than 10%.14 The alerts did not trigger if a qualifying diagnosis of cardiovascular disease was on the electronic problem list.
The alerts were present in both a passive and active form within each patient’s electronic chart. The active alert displayed when clinicians accessed the electronic ordering module of the patient chart, and required acknowledgement from clinicians. Clinicians could view the passive alert at any point during an encounter within the electronic visit summary screen. Immediately prior to the intervention, we educated clinicians regarding the use of the Framingham Risk Score and the alerts via a one-hour presentation at each center.
The intervention was randomized at the individual clinician level. Within each health center, we paired clinicians based on training background (physician versus non-physician) and number of patients with chest pain evaluated in the prior 6 months, and then randomly assigned one clinician in each pair to receive electronic reminders. Intervention group clinicians received both high risk and low risk reminders, and control group clinicians received no reminders.
All data were collected from manual reviews of the electronic record. The primary study outcomes for high risk patients included performance of an electrocardiogram and administration of aspirin therapy on the day of the office evaluation for patients with no documented allergy. The primary study outcome for low risk patients was performance of an outpatient cardiac stress test within 2 months of the office evaluation. For patients evaluated in the emergency department within 30 days of the index office visit, we collected information on diagnoses, treatments, and outcomes from the discharge summaries.
A diagnosis of coronary artery disease was based on the presence of a positive cardiac stress test, performance of coronary angioplasty or coronary artery bypass graft surgery, or a diagnosis of acute myocardial infarction. We ascertained diagnoses of acute myocardial infarction within 30 days of the index outpatient office visit based on the universal definition of myocardial infarction requiring symptoms of ischemia along with an elevated serum creatine kinase-MB fraction or serum troponin level.17 We defined missed diagnosis of acute myocardial infarction as those occurrences where the patient was triaged home from the primary care office and was subsequently diagnosed with acute myocardial infarction within 30 days.
We surveyed all 292 clinicians at the conclusion of the 15-month study period. Clinicians reported how often they felt the Framingham Risk Score represented a valid tool when evaluating patients complaining of chest pain on a 5-point scale ranging from ‘always’ to ‘never’. Clinicians in the intervention group rated the effectiveness of the electronic alerts at improving care on a 3-point scale of ‘very effective’, ‘somewhat effective’, and ‘not effective’. Intervention clinicians rated whether they felt the threshold of 10% for the Framingham Risk Score to identify high versus low risk patients was ‘too high’, ‘too low’, or ‘about right’. The survey was implemented via an initial paper mailing, followed by a reminder email to non-responders, and a final paper mailing at 4 weeks, achieving a 76% response rate.
Balance between patient characteristics in the two randomized arms was checked using a t-test for patient age, Fisher exact tests for binary variables, and chi-square tests for categorical variables. Comparisons of care patterns and outcomes between high and low risk patients were carried out using Fisher’s exact tests. We analyzed the impact of the intervention by fitting clustered logistic regression models with performance of each of our three outcomes as the dependent variable and clinician intervention status as the independent variable. These models used random effects to account for the clustering of patients within clinicians. The study had 90% power to detect an approximately 10% increase in treatment rates for high risk patients, and a 5% decrease in stress testing for low risk patients. All analyses were performed using SAS version 9.1.
Patient characteristics in the intervention and control group were similar (Table 1). The mean number of eligible patients among both intervention and control clinicians was 24 (range 1 to 89).
The majority (81%) of patients with chest pain were classified as low risk. The clinical evaluation was generally more aggressive among high risk patients compared to low risk patients (Table 2), including rates of performing electrocardiograms (50% versus 43%, p<0.001) and cardiac stress tests (17% versus 10%, p<0.001).
Non-emergent diagnoses including musculoskeletal pain and gastroesophageal reflux disease were considered by clinicians more often among low risk compared to high risk patients (Table 2). Potential coronary ischemia was considered more commonly among high risk patients compared to low risk patients (25% versus 14%, p<0.001).
Only 6.4% of patients were evaluated in the emergency department following the primary care visit, and 3.5% were ultimately hospitalized, with high risk patients more likely than low risk patients to be evaluated in the emergency department (11% versus 5%, p<0.01) and to be hospitalized (7% versus 3%, p<0.01). Among patients evaluated in the emergency department, 55% were hospitalized, 39% underwent cardiac stress testing and 10% underwent cardiac catheterization.
A diagnosis of coronary artery disease was established among 42 (0.6%) patients, more commonly among high risk compared to low risk patients (1.1% versus 0.5%, p<0.01). Acute myocardial infarction occurred among 26 (0.37%) patients, also more commonly among high risk compared to low risk patients (1.06% versus 0.21%, p<0.01). Among 26 diagnoses of acute myocardial infarction, 10 (38.5%) represented missed diagnoses in the primary care setting. There were no deaths among either high or low risk patients.
Among high risk patients, there was no difference between the intervention and control groups in rates of performing electrocardiograms (51% versus 48%, p=0.33) or administering aspirin (20% versus 18%, p=0.43). Among low risk patients, there was no difference between intervention and control groups in rates of cardiac stress testing (10% versus 9%, p=0.40, Table 3).
Among 212 responding clinicians, nearly three-quarters (72%) of clinicians reported most commonly using electronic order entry while the patient was still in the office with them, with 28% using it after the patient leaves the office. The majority of clinicians felt the Framingham Risk Score represented a valid tool either “often” (40%) or “sometimes” (47%) when evaluating patients complaining of chest pain; with 5% feeling it was “always” valid, and only 8% feeling it was “rarely” or “never” valid.
Among clinicians in the intervention group, a majority felt that the electronic alerts for high risk patients were “very” (9%) or “somewhat” (49%) effective at improving their management of chest pain. Similarly, 52% of clinicians felt that the alerts for low risk patients were very or somewhat effective at improving their management of chest pain. A majority of clinicians (81%) felt the cut-off of 10% for the Framingham Risk Score to identify high risk patients was “about right”, with 7% feeling it was too low and 12% feeling it was too high.
We conducted a post hoc analysis among the subgroup of intervention clinicians that reported the alerts to be very or somewhat effective at improving management of chest pain, and still found no effect of the intervention for both high and low risk patients.
In a large cohort of primary care patients presenting with chest pain, we demonstrated important gaps in quality and safety, with high risk patients not receiving recommended care and low risk patients undergoing many low-yield tests. Our electronic alerts provided evidence-based recommendations based on real-time calculation of the Framingham Risk Score, however did not significantly impact clinical practice patterns.
Electronic health records are promoted as an important patient safety tool,12,18 as well as having great potential to increase the efficiency of health care and reduce unnecessary testing.19,20 As the nation considers a substantial investment to support the broad implementation of electronic health records,21 the failure of our intervention highlights the need for deeper insight into how to use them to change physician behavior. While there are data to support the use of decision support to improve quality and medication safety,22,23 our data add to prior evidence suggesting that these benefits are not universally realized.24–26
The design of electronic decision support systems play a large role in their effectiveness.27 Many chest pain algorithms rely on symptom description and are difficult to implement using real-time decision support.28–30 The Framingham Risk Score represents a promising decision support tool for patients with chest pain.5,31,32 In our study, a risk score of at least 10% was associated with a higher occurrence of acute myocardial infarction within 30 days of presentation, highlighting the relevance of this risk assessment in the primary care evaluation of chest pain. The electronic health record can automate the calculation of this risk score and provide real time recommendations directly integrated into the workflow without requiring additional information input on the part of clinicians.
Our surveys indicate that the electronic alerts were well received by clinicians. Rather than problems of workflow integration or usability, it may be that the clinical benefits of electronic decision support as a stand-alone intervention do not extend to more complex clinical scenarios. Clinicians bring significant clinical intuition and experience to these encounters, and this experience may have superseded the information provided by the risk scores. Electronic alerts may be only one piece of a multi-targeted effort required to improve the management of complex scenarios such as chest pain.
An important distinction is whether our program failed to change care patterns due to limitations of the technology-based intervention, or due to lack of clinician trust in the clinical recommendations being offered. Our data indicate the latter issue did not play a large role as the majority of clinicians endorsed the validity of the Framingham Risk Score as a tool when evaluating patients complaining of chest pain.
To our knowledge, this is the largest prospective analysis of chest pain management in primary care. Errors in diagnosis represent a leading ambulatory patient safety concern,18,33,34 and our data showed that while the occurrence of acute myocardial infarction was infrequent, misdiagnosis was common. Over one-third of acute myocardial infarctions were misdiagnosed, compared to less than 5% in the emergency department setting.35 This is not surprising, given that only one-half of high risk patients had an electrocardiogram performed, despite its key role in evaluating patients with chest pain.9–11 The challenges to improving patient safety in the outpatient setting are substantial,34 particularly as primary care physicians may not view errors in diagnosis as an important patient safety concern.36
The majority of the patients in our study were low risk, and approximately 10% of these patients underwent cardiac stress testing. Current guidelines recommend against the use of such testing for low risk patients based on the poor positive predictive value.14,37 The common use of this low yield test represents a key area for improving efficiency, though recommending against such testing for patients with chest pain may be particularly challenging given physician concerns regarding patient safety and malpractice. Prior studies of decision support have focused on reducing unnecessary testing represented by avoiding redundant testing, though greater value may be achieved by avoiding tests that are not needed at the outset.
Our study should be interpreted in the context of some limitations. We relied on medical assistants to identify patients with chest pain. This decision was based on the need to identify patients prior to the evaluation by the clinician to deliver real-time risk information. We conducted extensive training of all medical assistants including performance feedback, and validated their identification of patients using medical record review. We chose the Framingham Risk Score as a tool to risk stratify primary care patients with chest pain. Many other risk prediction instruments exist to risk stratify primary care patients with chest pain.28,30 The Framingham Risk Score provides a substantial advantage over other instruments by producing a valid risk estimate without the need for additional input such as detailed symptom description, and without requiring testing not typically available in primary care such as serial cardiac enzymes or immediate cardiac stress testing.
In conclusion, this study of primary care management of acute chest pain demonstrated important quality and safety concerns. A well-designed electronic decision support system was acceptable to clinicians, but did not impact clinical practice patterns, with errors in diagnosis and overuse of stress testing persisting. Future work is needed to understand how advanced electronic health records can be used to improve the quality and safety of health care delivery.
We would like to thank the clinicians and patients of Harvard Vanguard Medical Associates for participating in this study. This project was supported by grant number R18HS017075 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The funding agency played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Dr. Sequist had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Dr. Sequist has served as a consultant on the Aetna External Committee on Racial and Ethnic Equality.
This study was funded by the Agency for Healthcare Research and Quality (R18HS017075). The study protocol was registered at www.ClinicalTrials.gov. (ID number NCT00674375).