Response by David Henry
Address: University of Newcastle, Faculty of Health, Level 5, Clinical Sciences Building, Newcastle Mater Hospital, Waratah, NSW, Australia, 2298.
Professor Måns Rosén raises concerns about our 2007 Cochrane review of anti-fibrinolytic drugs [2
]. The topic is an important one – many patients receive these drugs as an adjunct to reduce blood loss during a variety of surgical procedures. Some of his concerns go to the heart of assumptions that are made when conducting meta-analyses of randomized controlled clinical trials.
First, we should point out that we have updated some of the meta-analyses of trials contained in the Cochrane review and have recently published the results. This analysis is confined to the use of anti-fibrinolytic trials in cardiac surgery. The full text is available at [20
]. In the updated report we found an increased risk of death in subjects treated with aprotinin compared with tranexamic acid or aminocaproic acid. Our conclusions were "The risk of death tended to be consistently higher with use of aprotinin than with use of lysine analogues. Aprotinin had no clear advantages to offset these harms...The conclusions of our updated review conflict with those of our published Cochrane review.... The addition of data from the large BART study increased the relative risk of death with the use of aprotinin compared with the use of either tranexamic acid or epsilon aminocaproic acid."
So, the updated review is consistent with the observational studies of aprotinin (recently presented in a systematic review [21
]) in finding an increased risk of death with aprotinin compared with the lysine anti-fibrinolytic agents. But significantly, the updated meta-analysis found no increase in the risk of death with aprotinin compared with placebo/no treatment when used in cardiac surgery (summary RR for death 0.93; 95% CI 0.69, 1.25). This summary analysis comprised results from forty-nine trials of aprotinin, which included 7439 participants and reported on 182 deaths. This finding troubles Professor Rosén as it is at odds with the results of large observational studies and is based on small randomized trials that were not designed to show a change in the incidence of death or indeed cardiovascular events. It should be noted here that neither was the BART study.
The five main criticisms made by Rosén are: that few of the trials had mortality as a stated outcome; none of the trials had statistical power to detect differences in mortality; very few trials described the follow-up methods and follow-up time was not specified in most of them; follow-up may have been too short to quantify late deaths. One large study was not a randomized trial. While we don't dispute the accuracy of these observations about the individual studies we disagree with the inferences that Professor Rosén makes regarding the systematic review.
If at least one large trial had specified thrombosis (a theoretical adverse effect of these drugs) and death as outcomes, and had adequate power and sufficient follow-up, there would have been little need for the meta-analysis. One of the purposes of systematic reviews is to examine events that were not primary outcomes of the individual trials. By denying this Rosén is arguing against a central purpose of systematic reviews. In terms of patient follow-up, this varied between the aprotinin trials, but should not have varied between the treatment and control arms of the individual trials. In other words, it is unlikely that this was a source of bias either toward or away from the null. While some non-fatal events could lead to late mortality, we found no increase in the risk of non-fatal thrombosis. In addition, the BART trial3, cited by Rosén, found a separation of survival curves early in the post-operative phase and the curves are roughly parallel from Day 10 onwards. So, it is unlikely that undetected late mortality accounts for the lack of risk that we found in the meta-analysis of the placebo/inactive controlled aprotinin trials.
However, we do share one of Rosén's worries, which have also been expressed by Ray [22
]. This concerns the completeness of reporting of uncommon events in small clinical trials. We put considerable effort into identifying trials that appeared to report mortality, but we have no way of assessing how rigorously this was done. One concern about trials of drugs is the tradition to report 'adverse reactions' – events that are reported as 'possibly' or 'probably' caused by the drug. The use of causality assessment, traditional in the assessment of voluntary adverse reaction reports, could lead to under-reporting of events. This process has no place in the reporting of the results of clinical trials.
Regarding the alleged inclusion of one non-randomized study in the meta-analysis, the methods section in this study states: "Patients were randomly assigned to either an aprotinin treatment group (group A) or to a control group without aprotinin (group C)". We accepted this information in good faith, but scored the methodological quality of the study as low. We did not routinely contact study authors to confirm the details of randomization. Like most meta-analysts we accepted and scored the written description of the methods. We subsequently contacted the senior author, Professor Wulf Dietrich of the University of Munich, who kindly reviewed his files (the study was reported in 1992 [11
]) and has advised us that in his opinion the study does not meet contemporary standards for being considered 'randomized', that there was a possibility of selection bias, but this would have led to sicker patients receiving aprotinin. It should be noted that exclusion of this study does not change the overall estimate of mortality with aprotinin compared with control: Pooled RR = 1.02 (95%CI 0.71 to 1.47).
So did our Cochrane review miss an adverse effect of aprotinin? That is possible. Are there significant problems with meta-analyses of infrequent outcomes measured in small clinical trials? Yes there are. But in our view the major criticisms voiced by Professor Rosén concerning the specification of outcomes, statistical power of individual studies and variable follow up of trial participants are not the key issues. Under-reporting of infrequent events is pivotal and if non-differential (the most likely scenario) will lead to a bias to the null. This could account for the fact that we found no increase in mortality in the aprotinin trials. We have acknowledged this in the updated review.3 Despite considerable methodological improvements, meta-analysis remains an imperfect science, being an observational not an experimental discipline, which relies heavily on the diligence of trial investigators and authors of reports.
Systematic reviews must be rigorously performed, but Professor Rosén has not made a comprehensive assessment of the quality of our work. Tools exist to enable appraisal of systematic reviews (for instance the recently validated AMSTAR instrument) [23
]. Systematic reviews have value in summarizing literature, providing overall estimates of effect and in assisting in the planning of clinical trials. In the latter regard it is significant that two of the authors of the Cochrane review (Fergusson and Laupacis [1
]) were involved in the planning conduct and monitoring of the BART trial. This trial owed a lot to the results of the many published meta-analyses of this literature.