|Home | About | Journals | Submit | Contact Us | Français|
There are currently more than 20 risk-scoring systems that attempt to predict peri-operative mortality following coronary artery bypass surgery (CABG). All these scoring systems use objective criteria to assess operative risk. Angiographic data are currently not included in any of these systems. This pilot study assessed the value of coronary angiography in predicting peri-operative mortality following CABG.
Fourteen patients who died following first-time isolated CABG surgery were identified. These were matched with 14 patients of similar age, sex, left ventricle function and European System for Cardiac Operative Risk Evaluation (EuroSCORE). A panel of 25 clinicians were given details of the patients' age, sex, diabetic status, family history, smoking history, hypertensive status, lipid status, pre-operative symptoms, left ventricle ejection fraction and weight and shown the coronary angiograms of the patient. They were asked to predict the outcome following CABG for each patient.
Receiver operator characteristic curves were constructed and the area under the curves calculated and analysed using a commercially available statistical package (PRISM). The area under the curve for the group was 0.6820 for the group. Consultant clinicians achieved an area of 0.6789 versus their trainees 0.6844 (P = NS). The cardiologists achieved an area of 0.7063 versus the cardiothoracic surgeons 0.6491 (P = NS).
Despite the EuroSCORE predicting equal risk for the two groups of patients, it would appear that clinicians are able to identify individual higher risk patients by assessing pre-operatively the quality of the patient' coronary vasculature. Although the clinicians were able to predict individual patient mortality better than the EuroSCORE, the area under the curve indicates that it is not a robust method and clinicians, with all the clinical information to hand, are only moderately good at predicting the outcome following coronary artery bypass surgery.
Interest in mortality following coronary artery bypass surgery is at its highest level ever in the UK. The Healthcare Commission (a UK Government body) has published, on the internet, the operative mortality following cardiac surgery not only of each institution performing cardiac surgery but also each individual cardiothoracic surgeon. The index operation is isolated, first-time, coronary artery bypass surgery, but mortality following isolated aortic valve replacement and all cardiac surgery are in the public domain.1 In order to reduce the bias produced by the differing pre-operative case mix, each surgeon' operative mortality is represented alongside his or her mean EuroSCORE.2,3
Parsonnet et al.4 produced the first risk stratification system for acquired cardiac surgery in 1989, the Parsonnet score.
The inadequacies of risk scores in predicting operative mortality, however, is represented by the fact that there are at least 20 different risk scores currently used world-wide. Authors have tried to identify the relative merits and validity of each system5,6 but no one system has demonstrated clear superiority over another.
One consistent feature of the scoring systems is that they all use objective risk factors as data fields (Table 1). This has the benefit of allowing patient profiles to be compared across different units. However, important subjective information about the quality of the distal coronary vasculature, which is considered when planning the strategy of managing a patient with ischaemic heart disease, is excluded from all scoring systems. The exclusion of this information could account for why no current risk scoring system can accurately predict outcome following coronary artery bypass surgery.
Prospectively collected data within our unit were analysed. Fourteen patients undergoing first-time coronary artery bypass surgery (CABG) who died in hospital were identified and their pre-operative angiograms located. The 14 patients who survived first-time CABG were matched for age, sex, left ventricle function and EuroSCORE score (Table 2). Datasheets were prepared showing the parameters that would be available to the clinician whilst assessing the patient at the bedside. These were the patients age, gender, weight, symptom status (New York Heart Association [NYHA] and Canadian Classification System [Cleveland score]), diabetic status, family history of ischaemic heart disease and hypertensive (systolic > 140 mmHg) and lipid status (serum cholesterol > 6.0 mmol).
The panel was then shown the angiograms in random order. From the information they had to hand alone, they were asked to predict whether the patient lived or died following the operation.
The sensitivity and specificity of each clinician' responses were calculated and a separate receiver operator characteristic (ROC) curve constructed for each group of clinicians as well as a ROC curve for the whole panel. We sub-analysed by specialty and by grade (consultant or trainee). A Student' t-test was used to look for statistical significance between groups. Results are expressed as area under the curve and, when relevant, ± SD.
The risk profile of the two groups was similar (Table 3).
ROC analysis (Table 4) showed that the area under the curve (ROC area) for the whole panel was 0.6820 ± 0.0890, consultant cardiologists 0.6692 ± 0.0894, consultant cardiothoracic surgeons 0.6595 ± 0.0417, cardiology residents 0.7168 ± 0.0125, and cardiothoracic residents 0.6305 ± 0.0916.
There was no significant difference between the specialities of cardiology and cardiothoracic surgery or between the consultants and their trainees (Table 5).
It has long been proposed that clinicians are better at predicting clinical outcomes than those predicted by risk scores. This has not been conclusively demonstrated but clinicians were able to predict which patients required less than 48 h on an intensive care unit (ICU) in 87% of cases.7 On a day-to-day basis, clinicians assess patients and make judgements about the suitability of a patient for cardiac surgery.
Operative mortality is widely used as an indicator of the quality of surgery. However, comparison of different surgeons, institutions or different time periods is misleading if the mortality is not adjusted by the patient' characteristics that can adversely affect survival during and early after surgery. Consequently, during the last decade, several models8–12 have been developed in an attempt to compare outcomes of open heart surgery retrospectively (based on a reliable stratification of case mix) and to identify high-risk patients prospectively (as a basis for a meaningful, informed, pre-operative consent).
Current risk stratification systems approximate the probability of mortality between 0% and 100% for each patient, based on individual risk factors. However, the outcome for each patient is death or survival. Therefore, it is not a useful measure of the efficacy of the risk-scoring systems when directly comparing the predicted and observed outcome for the individual patient.13
In a study by Pintor et al.,14 418 consecutive patients undergoing cardiac surgery were risk scored by four different models. The Parsonnet score, Cleveland score, French score and EuroSCORE were used and found to be accurate in predicting the observed seven deaths (1.7%) within 30 days of the operation. The patients who survived surgery had low mean scores of Parsonnet score (1.1 ± 1.5), Cleveland score (1.3 ± 1.7), French score (1.7 ± 2.2) and EuroSCORE (2.2 ± 1.9). However, the patients who died had only slightly higher mean scores of Parsonnet score (2.1 ± 2.6), Cleveland score (4.2 ± 2.5), French score (4.1 ± 1.6) and EuroSCORE (4.6 ± 1.0). The scores were significantly different between the two groups of patients, but for the individual patients, the score was entirely unsuccessful in predicting their demise. This lack of prediction is not a fault of any single model, on the variables included in that model or indeed the statistical analysis involved. It is an intrinsic fault of any attempt to predict a low-rate event.15 The positive predictive value of a test will be low, according to Bayes' theorem, when the disease prevalence is low, even if the discriminatory test has a high sensitivity and specificity. Most models consider a high-risk patient to have a probability of death of > 10%. Consequently, the current models will never be able to predict the death of an individual patient. The models are, of course, better at predicting survival than mortality and their use in comparing surgical outcomes between surgeons or institutions is now common practice.
Whilst no risk-stratification system used to date includes the angiographic data relating to quality of coronary vasculature, some risk-stratification systems use left main stem stenosis > 70% and number of vessels diseased as data fields.16 The Quality Measurement and Management Initiative Coronary Revascularisation Project (QMMI)17 undertook a study to develop a clinical prediction rule for major adverse outcomes in CABG. Left main stem coronary artery stenosis > 70% (P = 0.004) correlated with adverse outcome in univariate analysis, but triple vessel disease was not significantly so (P = 0.02). Nevertheless, no quantitative or qualitative analysis of the coronary anatomy was included in the validation rule. In their analysis of variables needed for risk adjustment of short-term mortality following CABG, Jones et al.18 found left main stem stenosis and number of coronary vessels with > 70% stenosis to be predictive of mortality following CABG.
These risk-stratification systems vary in their predictive characteristics, but many are comparable to each other in risk stratifying groups of patients undergoing coronary artery bypass surgery. Although these systems have been used extensively in the last decades, difficulty still remains in predicting operative mortality for an individual patient. Nevertheless, risk scores currently form a basis for informed consent, patient counselling and comparing practices between different surgeons and units.
Having noted the inability of any scoring system to predict an individual patient mortality described above, the EuroSCORE nevertheless has been shown to predict reasonably accurately the mortality of a cohort of patients undergoing CABG in low- to medium-risk patients.19,20 The EuroSCORE has been constructed from a multinational European adult cardiac surgery database and has gained wide acceptance within Europe.2,3 This risk-stratification system has some superiority in discriminatory power compared to other systems,21 and has been shown to be superior to the Society of Thoracic Surgeons risk algorithm.22 Recently, it was demonstrated that EuroSCORE could predict intensive care unit stay and costs of open-heart surgery.23
However, the model is less accurate in predicting operative outcomes in high-risk patients. In an attempt to make scores more accurate, more data fields can be included for analysis, but this makes the score less easy to use. The American Heart Association (AHA) or The American College of Cardiology (ACC) score was found to predict risk of operative mortality (and stroke and mediastinitis) at least as well as the EuroSCORE, but is considerably easier to use than the EuroSCORE and Parsonnet score due to the smaller number of variables used.24 The increasing pre-operative complexity of the patients undergoing CABG is represented by an increase in the proportion of patients with a EuroSCORE > 10 and rising mean EuroSCORE over the last 4 years (Figs 1 and and2).2). As we operate on more ‘high-risk’ patients, the relevance of these scoring systems is diminished.
The importance of the patient' coronary anatomy is addressed in the New Zealand priority criteria project.25 This project looked to identify patients at high risk of coronary events prior to their surgical procedure, and subsequently reduce rates of myocardial infarction and death whilst on a waiting list for CABG. The investigators identified the degree of coronary artery obstruction (percentage diameter occluded) and the number of coronary vessels diseased criteria for inclusion.
The ROC curve is the best statistical tool for describing performance.26 The area under the ROC curve is considered a more appropriate statistical measure of the ability of a model to predict what it intends to. It is a plot of sensitivity versus 1 – specificity, and the area under the curve is a useful summary measure of the diagnostic accuracy of the tool. An area of 1 suggests a perfect predictor, and a value of 0.5 is a test of no value. Values between 0.7 and 0.9 are useful for some purposes, and higher values represent a high accuracy.
In our study, the clinicians were not asked for an operative risk assessment, but to predict whether the patients survived or died. All sub-groups of clinicians achieved ROC areas that were significantly > 0.5 indicating that the clinicians were better in this group of patients of predicting outcome than the EuroSCORE. Whilst only the cardiology residents actually achieved a ROC area of > 0.7, there was no significant difference between any of the groups indicating that the clinicians were reasonably good at this prediction. Higher ROC values would only be achieved with a crystal ball, which sadly none of us have access to.
Clinicians are able to identify ‘individual’ higher risk patients by assessing the quality of the patients' coronary vasculature. However, this is not a robust method. Even with all the clinical information to hand, clinicians are ‘only moderately’ good at predicting the outcome following CABG.
This work was presented, in part, at the 56th International Congress of the European Society of Cardiovascular Surgery, Venice, May 2007.