|Home | About | Journals | Submit | Contact Us | Français|
Background. Risk-adjusted mortality rates are used to compare quality of care of different hospitals. We evaluated the EuroSCORE (European System for Cardiac Operative Risk Evaluation) in patients undergoing isolated coronary artery bypass grafting (CABG).
Patients and method. Data of all CABG patients from January 2004 until December 2008 were analysed. Receiver-operating characteristics (ROC) curves for the additive and logistic EuroSCOREs and the areas under the ROC curve were calculated. Predicted probability of hospital mortality was calculated using logistic regression analyses and compared with the EuroSCORE. Cumulative sum (CUSUM) analyses were performed for the EuroSCORE and the actual hospital mortality.
Results. 5249 patients underwent CABG of which 89 (1.7%) died. The mean additive EuroSCORE was 3.5±2.5 (0-17) (median 3.0) and the mean logistic EuroSCORE was 4.0±5.5 (0-73) (median 2.4). The area under the ROC curve was 0.80±0.02 (95% confidence interval (CI) 0.76 to 0.84) for the additive and 0.81±0.02 (0.77 to 0.85) for the logistic EuroSCORE. The predicted probability (hazard ratio) was different from the additive and logistic EuroSCOREs. The hospital mortality was half of the EuroSCOREs, resulting in positive variable life-adjusted display curves.
Conclusions. Both the additive and logistic EuroSCOREs are overestimating the in-hospital mortality risk in low-risk CABG patients. The logistic EuroSCORE is more accurate in high-risk patients compared with the additive EuroSCORE. Until a more accurate risk scoring system is available, we suggest being careful when comparing the quality of care of different centres based on risk-adjusted mortality rates. (Neth Heart J 2010;18:355–9.)
In order to decide which therapeutic option (surgery, percutaneous intervention or medical therapy) is best for patients with coronary artery disease, a cost-benefit evaluation has to be performed for each therapeutic option. On the benefit side is the relief of symptoms and possible better long-term survival. On the cost side there is operative and postoperative morbidity and mortality. Several demographic data and preoperative, operative and postoperative variables (risk factors) may have impact on the results of the different therapies. Only when the impact of risk factors on morbidity and early and late mortality is known can a balanced decision about the best therapeutic option be made and a true informed consent from patients be obtained.
Nowadays, results of medical care given by different institutions and doctors are increasingly scrutinised, and the production of hospital league tables is now widespread. Such league tables usually fail to account for case mix, and great caution should be exercised in their interpretation. The development of tools to allow results from different hospitals and surgeons to be compared in a meaningful way is obviously important. Operative mortality is an indicator of the quality of cardiac surgery. Comparing different institutions or surgeons on the basis of crude mortality figures may be misleading as mortality is affected by various preoperative characteristics of the patients.1 The hospital standardised mortality ratio (HSMR) as first described by Jarman et al.2,3 uses administrative data that are readily available in every hospital such as year, age, sex, ICD-9 code, Charlson index for comorbidity, social-economic status of the patient, urgency, and length of stay in the hospital. The HSMR to measure quality of healthcare has been debated in the Dutch literature as it has many flaws and is considered inaccurate.4-13 The ideal risk stratification model should be easy to implement, objective, an accurate predictor of observed mortality, and in widespread use. Various models have been developed for use in cardiac surgery, and the first to become popular was the Parsonnet risk stratification system,14 which was developed in the USA in the 1980s. Thereafter, many other risk scoring models were developed such as the Cleveland Clinic score,15 French score,16 Pons score17 and the Ontario Province score.18 Nowadays the most used risk stratification models are the EuroSCORE and the Society of Thoracic Surgeons (STS) score.
The European System for Cardiac Operative Risk Evaluation (EuroSCORE) was first described in 1999.19,20 Nearly 20,000 consecutive patients from 128 hospitals in eight European countries were studied. Information was collected on 97 risk factors in all patients. The outcome (survival or death) was related to the preoperative risk factors. In the newly developed Dutch national database the data from every Dutch cardiothoracic centre are collected including the EuroSCORE and the hospital mortality. The variables that are used to calculate the EuroSCORE are shown in table 1.
Two risk calculators are available: in the simple additive EuroSCORE the weights (score in table 1) are added to give an approximate percent predicted mortality. This simple additive EuroSCORE model is now well established and has been validated in many cardiac surgery patient populations across the world. It is very valuable in quality control in cardiac surgery and gives a useful estimate of risk in individual patients. However, particularly in very high-risk patients, the simple additive model may sometimes underestimate the risk when certain combinations of risk factors co-exist. The logistic version of EuroSCORE produces more accurate risk prediction for a particular high-risk patient using the ‘Beta’ from table 1 for its calculation. Its main disadvantage is that the risk has to be calculated in quite a complex way.
When the STS risk scoring model was compared with the EuroSCORE in patients undergoing coronary artery bypass graft surgery (CABG), Ad et al.21 found both models to be good predictors of operative mortality with a slight advantage for the STS model. However, Nilsson et al.22 found that the additive EuroSCORE had a better discriminatory power compared with the STS risk model.
The goal of this study was to evaluate the EuroSCORE as a risk stratification model for patients undergoing (CABG) surgery using the data we collected in our computerised database.
From January 2004 we collected the EuroSCORE of all cardiac surgery patients in the Catharina Hospital, Eindhoven, the Netherlands. We analysed the data of all patients undergoing isolated CABG from January 2004 till December 2008. We measured the discriminatory power of the EuroSCORE for operative mortality. For this we calculated the receiver operating characteristics (ROC) curve and the area under the ROC curve. We calculated the cumulative summation of the EuroSCORE of all consecutive patients (CUSUM) and the cumulative hospital mortality as defined by mortality during the same hospital stay as the operation. From this we calculated the variable life-adjusted display (VLAD) curve as described by Lovegrove et al. in 1997.23 In constructing the VLAD curves, the EuroSCORE minus the mortality per patient are cumulatively summarised. Logistic regression analyses were used to calculate the predicted probability per EuroSCORE value, which reflects the actual hospital mortality.
During the five-year study period, 5249 patients underwent CABG of which 89 (1.7%) died. The mean additive EuroSCORE was 3.5±2.5 (0 to 17) and the mean logistic EuroSCORE was 4.0±5.5 (0 to 73). The median additive EuroSCORE was 3.0 and the median logistic EuroSCORE was 2.4. The numbers of patients per additive EuroSCORE are shown in figure 1.
The ROC curves of both the additive and the logistic EuroSCORE are shown in figure 2. The area under the ROC curve was 0.80±0.02 (95% confidence interval (CI) 0.76 to 0.84) for the additive EuroSCORE and 0.81±0.02 (0.77 to 0.85) for the logistic EuroSCORE. The predicted probability (actual mortality) was different from the additive and the logistic EuroSCORE (figure 3). The cumulative additive EuroSCORE, logistic EuroSCORE and the hospital mortality are depicted in figure 4. The cumulative mortality is half of the predicted mortality according to the EuroSCORE. The VLAD curve was positive indicating that lives were spared (figure 5).
In our patients, the EuroSCORE has a high discriminatory power shown by an area under the ROC curve of 0.80 for the additive EuroSCORE and of 0.81 for the logistic EuroSCORE indicating that most patients who died had a high EuroSCORE. The area under the ROC curve is considered to be an indicator for the discriminatory power of the model. The discriminatory power is thought to be excellent if the area under the ROC curve is >0.80, very good if >0.75, and good if >0.70.23 Others found the similar results22 in a group of 4497 patients undergoing CABG. When they made a comparison with the STS score they found an area under the ROC curve of 0.85 (95% CI 0.80 to 0.88) for the EuroSCORE compared with 0.71 (95% CI 0.66 to 0.77) for the STS risk scoring system. They concluded that the EuroSCORE showed a larger discriminatory power compared with the STS risk scoring system. Geisler et al.24 reported an area under the ROC curve for the EuroSCORE of 0.78 in a group of 505 patients undergoing all kinds of heart surgery. When we plot the predicted probability calculated with logistic regression analyses in our CABG patients against the additive and logistic EuroSCORE (figure 3) it is clear that the logistic EuroSCORE is a better predictor of hospital mortality, especially in patients with a higher score. For lower logistic EuroSCOREs there was an overestimation of the risk. The additive EuroSCORE underestimates the risk in high-risk patients. For a perfect risk scoring system the picture should show a diagonal line. One can conclude that the EuroSCORE model, though having a very good discriminatory power, is not accurate in predicting hospital mortality in CABG patients.
Recently, data on all cardiac surgery patients including the EuroSCORE, the EuroSCORE variables and the hospital mortality were collected in a Dutch national database. The CABG patients form an important and large group in this database. In order to compare the results of different centres using risk-adjusted mortality rates, we need reliable and accurate risk stratification models. If the calculated risk score is not accurate, comparison of risk-adjusted mortality may lead to the wrong conclusions. If, for example, the risk model underestimates the risk in high-risk patients and overestimates the risk in low-risk patients, as is the case in our CABG patients, the results of the comparison will always be in favour of the centres that are performing operations with the lowest risk scores. The VLAD curve will be more positive in centres with more low-risk patients. Improvement of the VLAD curve can be easily accomplished by denying operations to high-risk patients. Especially when the results of the VLAD curves are made public this issue will become more important. Re-evaluation of the variables used in the EuroSCORE and considering new ones in order to improve the accuracy of the scoring system is warranted. Only when a more accurate scoring system for CABG (and for all other cardiac operations) has been developed, as suggested by Nissinen et al.,26 can publication of VLAD curves per centre be done more accurately without the risk that a hospital that is offering operations to the highest risk patients will be unfairly accused of ‘bad quality’.
Both the additive EuroSCORE and the logistic EuroSCORE are overestimating hospital mortality risk in low-risk patients undergoing CABG. The logistic EuroSCORE is more accurate in high-risk patients compared with the additive EuroSCORE. Until a more accurate risk scoring system is available, we must be careful with comparing quality of care of different cardiac surgery centres based on risk-adjusted mortality rates.