PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of cmajCMAJ Information for AuthorsCMAJ Home Page
 
CMAJ. 2010 July 13; 182(10): 1039–1044.
PMCID: PMC2900327

Diagnostic accuracy of the TIMI risk score in patients with chest pain in the emergency department: a meta-analysis

Abstract

Background

The Thrombolysis in Myocardial Infarction (TIMI) risk score uses clinical data to predict the short-term risk of acute myocardial infarction, coronary revascularization or death from any cause. It was originally developed for use in patients with unstable angina or non–ST-elevation myocardial infarction. We sought to expand the clinical application of the TIMI risk score by assessing its prognostic accuracy in patients in the emergency department with potential acute coronary syndromes.

Methods

We searched five electronic databases, hand-searched reference lists of included studies and contacted content experts to identify articles for review. We included prospective cohort studies that validated the TIMI risk score in emergency department patients. We performed a meta-regression to determine whether a linear relation exists between TIMI risk score and the cumulative incidence of cardiac events.

Results

We included 10 prospective cohort studies (with a total of 17 265 patients) in our systematic review. Data were available for meta-analysis in 8 of the 10 studies. Of patients with a score of zero, 1.8% had a cardiac event within 30 days (sensitivity 97.2%, 95% CI 96.4–97.8; specificity 25.0%, 95% CI 24.3–25.7; positive likelihood ratio 1.30, 95% CI 1.28–1.31; negative likelihood ratio 0.11, 95% CI 0.09–0.15). Meta-regression analysis revealed a strong linear relation between TIMI risk score (p < 0.001) and the cumulative incidence of cardiac events.

Interpretation

Although the TIMI risk score is an effective risk stratification tool for patients in the emergency department with potential acute coronary syndromes, it should not be used as the sole means of determining patient disposition.

Chest pain is a common presenting complaint in the emergency department that requires efficient risk stratification, timely initiation of treatment in high-risk patients and safe determination of patient disposition. Several studies have been published that stratify the risk of patients in the emergency department with chest pain.15 However, only the Thrombolysis in Myocardial Infarction (TIMI) risk score, which was initially developed for use in patients with unstable angina or non–ST-segment elevation myocardial infarction or both,6 has been broadly validated in several independent emergency department populations with chest pain and thus constitutes the highest level of evidence available.

The TIMI risk score assigns each of seven predictors a value of one point, allowing stratification of patients into one of eight prognostic categories (Box 1).6 The clinical end points are acute myocardial infarction, coronary revascularization and death from any cause.

Box 1. Predictor variables included in the TIMI risk score*

  • Age of more than 65 years
  • Three or more risk factors for atherosclerosis
  • Known coronary artery disease
  • Two or more episodes of anginal chest pain in the preceding 24 hours
  • Acetylsalicylic acid use in the seven days before hospitalization
  • ST-segment deviation of 0.05 mV or more
  • Elevated cardiac markers

A robust estimate of the performance of the TIMI risk score obtained from a systematic review may prove useful to both clinicians and researchers. Clinicians would have a reliable quantitative estimate of a patient’s short-term risk of a cardiac event. This could be used as an adjunct to clinical acumen and as a tool to communicate risk to patients in a shared decision-making model of care.7 Researchers would also have an estimate of the prognostic accuracy of the TIMI risk score derived from different practice settings and patient populations that represent a wide variety of ethnic backgrounds. This estimate may serve as a useful baseline for comparison as emerging clinical prediction rules and imaging modalities continue to refine our approach to diagnosis and risk stratification in patients in the emergency department with potential acute coronary syndromes.

We conducted a comprehensive systematic review and meta-analysis to assess the methodological quality and prognostic performance of studies that had prospectively validated the TIMI risk score in patients in the emergency department.

Methods

This systematic review and meta-analysis adheres to the reporting guidelines of Meta-analysis of Observational Studies in Epidemiology (MOOSE)8 and the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) statement as applicable to meta-analyses of observational studies.9

Eligibility criteria

We included studies that prospectively validated the TIMI risk score in patients in the emergency department with chest pain. We excluded retrospective studies, case reports, studies not based on original research and studies not involving patients in the emergency department (e.g., hospital ward, intensive care unit).

Search strategy

An expert librarian (P.J.E.) designed a comprehensive search strategy with input from the clinical lead (E.P.H.). Our electronic search included five databases (MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, Web of Science and Scopus; Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). We applied no language restrictions to the search strategy. We also hand-searched conference proceedings from 2007 to 2009 from the Society for Academic Emergency Medicine, the Canadian Association of Emergency Physicians and the American College of Emergency Physicians. We reviewed the reference lists of eligible articles and consulted with content experts (J.E.H., I.G.S.) to identify additional published reports.

Study selection

Two investigators (E.P.H., D.A.) independently screened the titles and abstracts of all identified records (phase I). If either reviewer believed that the study might be eligible, we obtained the full report. The same two investigators then independently assessed the eligibility of each full report (phase II). We used the Cohen κ to measure the chance-corrected agreement between reviewers for each phase of study selection. Any disagreements were resolved by consensus.

Quality assessment

Two reviewers (D.A., S.C.) independently assessed the methodological quality of each eligible article. Because there is no single quality assessment tool that has been designed to assess the methodological quality of studies of risk scores, we selected criteria from assessment tools1012 that, based on a consensus among investigators, most closely indicated study quality. We selected the following seven criteria: patients selected in an unbiased fashion (consecutive or random sample); study sample representative of a wide spectrum of the severity of disease; predictor variables assessed without knowledge of the outcome; outcome assessed without knowledge of the predictor variables; outcomes accurately defined (particularly, acute myocardial infarction); explicit interpretation of the risk score by clinicians in practice without knowledge of the outcome; and adequate follow-up (arbitrarily defined as a lost-to-follow-up rate of < 10%). We assessed the degree of interobserver agreement for quality assessment using the Cohen κ. Any disagreements were resolved by consensus.

Data extraction and outcomes

Using a standardized data extraction sheet, two authors (D.A., S.C.) independently extracted data from each included report. Because most reports did not differentiate between patients with TIMI risk scores from 5 to 7, we classified patients with TIMI risk scores ≥ 5 into a single risk stratum. If the data were not available in the original report or were unclear, we contacted the corresponding author for clarification. Clinical outcomes were acute myocardial infarction, coronary revascularization or death from any cause. Disagreements were resolved by consensus.

Data synthesis and analysis

We present all continuous data as either means with standard deviations (SD) or medians with interquartile ranges as reported in the primary study. Categorical data are presented as per cent frequency of occurrence. We performed meta-regression to determine whether a linear relation exists between TIMI risk score (independent variable) and the logit event rate (dependent variable) using Comprehensive Meta-Analysis version 2 (Biostat Inc., Englewood, NJ). We assessed the prognostic performance of the TIMI risk score at potential decision thresholds using Meta-DiSc software (Unit of Clinical Biostatistics and the Ramon y Cajal Hospital).

We used the I2 statistic to quantify statistical heterogeneity between studies. We hypothesized that statistical heterogeneity identified by the I2 statistic could be explained by differences in the prevalence of cardiac events between studies. To test this hypothesis, we arbitrarily defined low prevalence as less than 10% and conducted an a priori subgroup analysis of studies with low (< 10%) versus high (≥ 10%) prevalence using a test of statistical interaction as defined by Altman.13 Estimates from each subgroup are analyzed on a log scale. The test is defined as the ratio of the difference in the log estimates to the standard error of this difference. The associated p value is derived from the normal distribution.

Results

Study selection

Figure 1 shows the study selection process. The initial search strategy identified 383 records. After removing 68 duplicate records, 315 records remained for screening in phase I. We identified 27 potentially eligible records for further review based on review of the titles and abstracts (κ = 0.87, 95% CI 0.76–0.97). Review of the full articles in phase II identified 10 that met our inclusion criteria (κ = 0.78, 95% CI 0.54–1.00). All 10 studies were included in the qualitative analysis. We were able to contact authors for missing data and verification of data extraction for 8 of the 10 included studies. We were unable to obtain TIMI risk score data in two studies,14,15 leaving eight studies for the quantitative synthesis.

Figure 1
Flow diagram of the study selection process. Note: ED = emergency department.

Characteristics of the included studies

The 10 studies included in the review enrolled 17 265 emergency department patients with chest pain from 5 countries (Table 1).1623 Three studies were conducted in the United States, two in the United Kingdom, two in Spain, two in Italy and one in Canada. The mean age ranged from 53.6 to 71.3 years. The proportion of males to females enrolled in each study was approximately equal.

Table 1
Characteristics of the studies* included in the systematic review of the use of the Thrombolysis in Myocardial Infarction (TIMI) risk score to predict the short-term risk of acute myocardial infarction, coronary revascularization or death

Quality assessment

Assessment of study quality for all 10 studies had an inter-observer agreement of 92.8% (κ = 0.85, 95% CI 0.71–0.98). Although the predictor variables were assessed without knowledge of the outcome in all studies, only one study16 explicitly reported blinding outcome assessors to knowledge of the predictor variables (Appendix 2, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). Two studies did not report how acute myocardial infarction was defined.14,15 Physicians explicitly interpreted the TIMI risk score in practice without knowledge of the outcome in three studies.1719 All studies that reported the fidelity of patient follow-up had lost-to-follow-up rates of less than 10%.

Heterogeneity

Cardiac troponin or creatine kinase–muscle and brain (CK-MB) or both were used as cardiac biomarkers in all eight studies included in the quantitative analysis. However, only three studies16,20,21 reported the specific assay used (including the manufacturer, platform, 99th percentile reference limit and coefficient of variation) as recommended by standardized reporting guidelines.24 This makes it difficult to assess potential heterogeneity between studies because of the different cardiac biomarker thresholds used to define acute myocardial infarction. However, all studies validated the TIMI risk score in emergency department patients and assessed the same predictor variables and the same composite outcome. Thirty-day outcome data were available for all studies included in the quantitative synthesis except one17 in which outcomes were assessed at the time of discharge from hospital or emergency department. The prevalence of cardiac events varied from 3.7% to 25.5% between studies.

Classification characteristics and diagnostic performance

The number of patients and the number with cardiac events in each risk stratum of the TIMI risk score for the eight studies included in the quantitative synthesis are shown in Appendix 3 (available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). The prevalence of cardiac events in patients with a TIMI risk score of zero was 1.8%, and the prevalence in patients with a score of one was 4.0%. Although the total number of patients in each risk stratum progressively decreased with each successive increase in TIMI risk score, the proportion with adverse outcomes progressively increased up to a score of 5 or greater.

The results of the meta-regression of the TIMI risk score on the logit event rate are shown in Figure 2. The regression coefficient was significant (p < 0.001), indicating a strong linear relation between TIMI risk score and cumulative incidence of cardiac events.

Figure 2
Meta regression of TIMI risk scores on the logit event rate. Each circle represents data from an individual study, and the size of the circle represents the size of the cohort. There is a strong linear relation (p < 0.001) between the TIMI risk ...

Table 2 shows the classification performance of the TIMI risk score at potential decision thresholds. At a cut-off of > 0, the sensitivity was 97.2% (95% CI 96.4–97.8; I2 = 99.9%) and the specificity was 25.0% (95% CI 24.3–25.7; I2 = 100.0%). The negative likelihood ratio was 0.11 (95% CI 0.09–0.15; I2 = 83.3%). At a cut-off of > 1, the sensitivity decreased to 90.6% (95% CI 89.3–91.8; I2 = 97.9%) and the specificity increased to 50.9% (95% CI 50.0–51.7; I2 = 99.5%).

Table 2
Classification characteristics of the Thrombolysis in Myocardial Infarction (TIMI) risk score at potential decision thresholds

We also conducted a sensitivity analysis excluding the study by Pellicia and colleagues,17 because this study reported in-hospital rather than 30-day outcomes. There were no substantial differences in the classification performance of the TIMI risk score when this study was excluded (data not shown).

Subgroup analyses

The test for interaction comparing studies with a low (< 10%) prevalence of cardiac events at a threshold of TIMI > 0 was significant (diagnostic odds ratio 5.58, 95% CI 4.21–7.39 v. (34.21, 95% CI 17.01–68.78; p < 0.001). The test for interaction at a threshold of TIMI > 1 was also significant (diagnostic odds ratio 5.12, 95% CI 4.27–6.15 v. 18.55, 95% CI 13.88–24.78; p < 0.001).

Interpretation

The results of our meta-regression analysis indicated a strong linear relation between TIMI risk score and the short-term incidence of cardiac events. The incidence of cardiac events in the lowest risk stratum (TIMI score of zero) was 1.8%; the sensitivity was 97.2% and specificity was 25.0% at this decision threshold. All potential decision thresholds were insufficiently sensitive or specific to recommend use of the TIMI risk score as the sole means of determining patient disposition. Although the analysis was associated with significant heterogeneity, interaction testing indicated that the heterogeneity could be explained by differences in the prevalence of cardiac events between studies.

Strengths and limitations

The strengths of this study include a search strategy that involved five electronic databases, searching the bibliographies of the included articles and contact with content experts and authors of the included studies. This minimized, but did not exclude, the potential for publication bias. We also used sound methods in conducting the review, including assessment of interrater reliability for study selection and quality assessment and dual data extraction. Our findings also represent the performance of the TIMI risk score in unselected “real life” patients. These patients are generally at increased risk of adverse events compared with patients in randomized controlled trials. Thus, the estimates of risk provided in this review may be more relevant to general cardiology practices.

The study is limited by the relatively small number of studies included in the review and the statistical heterogeneity between studies, which limits the quality of the evidence and our ability to assess publication bias. Although the Global Registry of Acute Coronary Events (GRACE) risk score has been shown to have greater discriminative ability than the TIMI risk score for mortality in-hospital and at six months in patients with confirmed acute coronary syndromes,25 few studies have assessed the performance of the GRACE score in emergency department patients with potential acute coronary syndromes. One study reported a sensitivity of 100% and a specificity of 13%, with data needed to calculate a complete GRACE score missing for 24% of patients.22 The focused nature of the research question, however, enabled careful assessment and meta-analysis of studies with limited clinical heterogeneity.

Our review is also limited by the lack of reporting of the characteristics of the cardiac biomarker assays and the thresholds used to define acute myocardial infarction. All of the studies included in the meta-analysis, however, used either CK-MB or cardiac troponin as recommend by the Joint European Society of Cardiology, American College of Cardiology, American Heart Association and World Health Federation Task Force for the Redefinition of Myocardial Infarction.26

Reporting bias may also have affected the results of this review. Despite repeated attempts to contact the primary authors to obtain data about TIMI risk scores, we were unable to obtain data for two studies.14,15 The large number of remaining patients included in the review and the strong linear relation between TIMI risk score and the cumulative incidence of cardiac events in the eight studies included in the quantitative synthesis suggests that the lack of data from these two studies may not substantially reduce the scientific value of this research.

External validity, clinical implications and future research

This review included 10 validation studies conducted in 5 countries, encompassing 17 265 patients. The broad sample of patients from which the statistical estimates were derived suggests a high degree of external validity of the findings. The 1.8% miss rate suggests that this degree of accuracy may be achieved using clinical, electrocardiography and laboratory data without need for urgent stress testing or complex rule-out protocols, which could result in important cost savings in practice settings without observation units. However, a miss rate of 1.8% may not be acceptable in all practice settings.

Conclusion

Our analysis supports the consistent performance of the TIMI risk score in the risk stratification of patients in the emergency department with potential acute coronary syndromes. For every 1000 patients in the lowest risk category (TIMI score of zero), 20 will experience a cardiac event within 30 days of the visit to the emergency department. Although this rate may render the TIMI risk score an unacceptable single reason to dismiss patients from the emergency department, this rate highlights the potential utility of this score as an adjunct to clinical acumen, as a source of information for patient decision aids and as a benchmark for researchers developing new prognostic and disposition tools.

Supplementary Material

[Online Appendices]

Acknowledgement

Erik Hess is the recipient of a Fellow-to-Faculty Transition award from the American Heart Association, the Emergency Medicine Foundation and the Society for Academic Emergency Medicine.

Footnotes

Previously published at www.cmaj.ca

Competing interests: None declared.

Contributors: Erik Hess conceived study, analyzed the data and wrote the manuscript. Dipti Agarwal and Subhash Chandra contributed substantially to data acquisition. Mohammed Murad contributed to the design of the study and conducted the meta-regression analysis. Patricia Erwin designed and conducted the search. Judd Hollander and Ian Stiell contributed to the interpretation of the data. Victor Montori contributed to the design of the study and supervised the meta-analysis. All authors revised the manuscript for important intellectual content and approved the final version submitted for publication.

*To calculate the TIMI risk score, sum the number of positive predictor variables (0–7).

Funding: This study was jointly funded by a grant from the American Heart Association, the Society for Academic Emergency Medicine and the Emergency Medicine Foundation.

This article has been peer reviewed.

REFERENCES

1. Baxt WG, Shofer FS, Sites FD, et al. A neural computational aid to the diagnosis of acute myocardial infarction. Ann Emerg Med. 2002;39:366–73. [PubMed]
2. Goldman L, Cook EF, Brand DA, et al. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med. 1988;318:797–803. [PubMed]
3. Goldman L, Cook EF, Johnson PA, et al. Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. N Engl J Med. 1996;334:1498–504. [PubMed]
4. Pozen MW, D’Agostino RB, Selker HP, et al. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med. 1984;310:1273–8. [PubMed]
5. Selker HP, Beshansky JR, Griffith JL, et al. Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia. A multi-center, controlled clinical trial. Ann Intern Med. 1998;129:845–55. [PubMed]
6. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST elevation MI: A method for prognostication and therapeutic decision making. JAMA. 2000;284:835–42. [PubMed]
7. Nannenga MR, Montori VM, Weymiller AJ, et al. A treatment decision aid may increase patient trust in the diabetes specialist. The Statin Choice randomized trial. Health Expect. 2009;12:38–44. [PubMed]
8. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–12. [PubMed]
9. Moher D, Liberati A, Tetzlaff J, et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. BMJ. 2009;339:b2535. [PubMed]
10. Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25. [PMC free article] [PubMed]
11. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277:488–94. [PubMed]
12. Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analyses. Ottawa (ON): Ottawa Health Research Institute; 2009. [(accessed 2010 May 21)]. Available: www.ohri.ca/programs/clinical_epidemiology/oxford.htm.
13. Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ. 2003;326:219. [PMC free article] [PubMed]
14. Conti A, Pieralli F, Sammicheli L, et al. Updated management of non-ST-segment elevation acute coronary syndromes: selection of patients for low-cost care: an analysis of outcome and cost effectiveness. Med Sci Monit. 2005;11:CR100–8. [PubMed]
15. Garcia Almagro FJ, Gimeno JR, Villegas M, et al. Use of a coronary risk score (the TIMI Risk Score) in a non-selected patient population assessed for chest pain at an emergency department [article in Spanish] Rev Esp Cardiol. 2005;58:775–81. [PubMed]
16. Hess EP, Perry JJ, Calder LA, et al. Prospective validation of a modified thrombolysis in myocardial infarction risk score in emergency department patients with chest pain and possible acute coronary syndrome. Acad Emerg Med. 2010;17:368–75. [PubMed]
17. Pelliccia F, Salvini P, Cartoni D, et al. Frequency and clinical correlates of changes in Thrombolysis In Myocardial Infarction risk score during observation period at emergency department in “low-risk” patients with acute chest pain. Am J Cardiol. 2006;97:781–4. [PubMed]
18. Campbell CF, Chang AM, Sease KL, et al. Combining thrombolysis in myocardial infarction risk score and clear-cut alternative diagnosis for chest pain risk stratification. Am J Emerg Med. 2009;27:37–42. [PubMed]
19. Pollack CV, Jr, Sites FD, Shofer FS, et al. Application of the TIMI risk score for unstable angina and non-ST elevation acute coronary syndrome to an unselected emergency department chest pain population. Acad Emerg Med. 2006;13:13–8. [PubMed]
20. Body R, Carley S, McDowell G, et al. Can a modified thrombolysis in myocardial infarction risk score outperform the original for risk stratifying emergency department patients with chest pain? Emerg Med J. 2009;26:95–9. [PubMed]
21. Sanchis J, Bodi V, Nunez J, et al. New risk score for patients with acute chest pain, non-ST-segment deviation, and normal troponin concentrations: a comparison with the TIMI risk score. J Am Coll Cardiol. 2005;46:443–9. [PubMed]
22. Lyon R, Morris AC, Caesar D, et al. Chest pain presenting to the emergency department — to stratify risk with GRACE or TIMI? Resuscitation. 2007;74:90–3. [PubMed]
23. Tong KL, Kaul S, Wang XQ, et al. Myocardial contrast echocardiography versus thrombolysis in myocardial infarction score in patients presenting to the emergency department with chest pain and a nondiagnostic electrocardiogram. J Am Coll Cardiol. 2005;46:920–7. [PubMed]
24. Hollander JE, Blomkalns AL, Brogan GX, et al. Standardized reporting guidelines for studies evaluating risk stratification of emergency department patients with potential acute coronary syndromes. Ann Emerg Med. 2004;44:589–98. [PubMed]
25. Aragam KG, Tamhane UU, Kline-Rogers E, et al. Does simplicity compromise accuracy in ACS risk prediction? A retrospective analysis of the TIMI and GRACE risk scores. PLoS One. 2009;4:e7947. [PMC free article] [PubMed]
26. Thygesen K, Alpert JS, White HD, et al. Universal definition of myocardial infarction. Circulation. 2007;116:2634–53. [PubMed]

Articles from CMAJ : Canadian Medical Association Journal are provided here courtesy of Canadian Medical Association