shows the flow of reviews through the study. After exclusions from the 52 citations initially identified (double hits, non-systematic or incompletely reported reviews,19
reviews that lacked quantitative effect estimates, and an ongoing Cochrane review (due for publication in late 2006),20
10 articles were included ().12,21-29
All were originally submitted between May 2003 and July 2004, published between August 2003 and March 2005, and with no overlap in authorship (49 investigators). gives details of the randomised controlled trials that were pooled for each included systematic review. Reviews carried out later could identify and pool randomised controlled trials reported after the publication of the earliest reviews. In three cases the journal in which articles appeared was a general medicine journal, in three cases a cardiology journal, and in four cases a nephrology journal. The median impact factor was 3.9 (range 18.3 to 0). All the studies used the same primary end point of contrast associated nephropathy (defined as > 44.2 μmol/l or > 25% increase in serum creatinine concentration from baseline to 48 hours).
Flow of reviews through study
Included systematic reviews comparing acetylcysteine with controls for the prevention of contrast associated nephropathy
Pooled randomised controlled trials (RCTs) for each of included systematic reviews*
Funding was from for profit sources in five cases, not for profit sources in seven cases, and specific for the review in one case ().
Search strategies varied from extensive (five databases in one case) to relying on only Medline. Summary estimates used for pooling were odds ratios (three reviews), relative risks (six), and risk differences (one), by means of random effects (eight) or fixed effects (one) methods (). Quantitative results were reported using both P values and confidence intervals in all cases, the practice supported by most reviewers.1,2
Exploratory metaregression was carried out in light of statistical heterogeneity in six studies. Although most metaregressions included covariates likely to accommodate variability in underlying risk, none formally tested as covariate the role of control group event rates, a practice advocated by some authors and criticised by others.16,30
Methodological characteristics of included studies
Five reviews recommended routine, or almost routine, use of acetylcysteine in the prevention of contrast nephropathy. The other five were more cautious and, although not dismissing the potential benefit and limited toxicity and costs of acetylcysteine, called for further well designed randomised controlled trials on the subject (). These discrepancies occurred despite similar pooled effect estimates.
Authors' recommendations on current role of acetylcysteine in prevention of contrast associated nephropathy
Compliance with the QUOROM statement
Agreement before consensus between reviewers was high for compliance with the QUOROM items (174/180 (91%), κ = 0. 89, 95% confidence interval 0.80 to 0.97;
= 0.89), and 100% (10 reviews) for authors' recommendations. The median compliance with the QUOROM checklist was 16 (range 11 to 17; ).
Compliance of included studies with quality of reporting of meta-analyses (QUOROM) checklist and Oxman and Guyatt index*
In only three cases could the report be identified from the title as a meta-analysis or systematic review of randomised controlled trials. A structured abstract was available in seven reviews, and objectives and conclusions were explicitly stated in 10 and nine, respectively. In the abstract sections, data sources, review methods, and results complied with the QUOROM statement in only six cases.
All the studies fulfilled the QUOROM requirements for the introduction and discussion sections, as they reported search strategies, selection criteria, and methods for quantitative data synthesis employed for the systematic review. Most of the studies complied with the recommendations for reporting of the assessment of the methodological quality of the primary studies (seven reviews), data abstraction (eight), and study characteristics (nine).
Despite thorough reporting of methods by most studies, no review quantified the agreement between reviewers on selection and appraisal of the methodological quality of the primary studies. A diagram showing the flow of the trial was provided in only three cases. Details on study characteristics and quantitative data synthesis in the results section were, however, provided in enough detail in all the reviews.
Quantitative analyses and predictors of QUOROM score
A significant association was found between number of published pages and overall QUOROM score (Pearson R = 0.73, P = 0.016, Spearman ρ = 0.64, P = 0.046), suggesting that the longer the manuscript, the greater the likelihood of complying with the QUOROM statement (). Although this finding might have relevant implications for editorial policies, it should be interpreted with caution given the small sample size and the fact that two studies largely drive the hypothesis tests.
Scatter plot showing relation between manuscript length and quality of reporting of meta-analyses (QUOROM) score. Thorough and complete reports achieved higher scores
Language restrictions were associated with fewer trials analysed (7, range 5-11 v 15, range 7-21, P = 0.040) and fewer patients analysed (805, range 643-1207 patients v 1539, 805-2420; P = 0.033) (tables and ).
Studies from reviewers reporting previous not for profit funding were more likely to score higher on the Oxman and Guyatt index (6, range 3-7 v 2, range 1-4; P = 0.037), to search more databases for original articles (4, 3-5 v 1.3, 1-2; P = 0.014), and to be published in a journal with a greater impact factor (4.9, 3.1-18.3 v 1.3, 1.2-1.5; P = 0.020). Conversely, funding was not significantly associated with authors' recommendations supporting the routine use of acetylcysteine (43% (three of seven) for studies reporting previous not for profit funding compared with 66% (two of three) for the others; P = 0.42) or QUOROM score (16, range 11-17 for studies with previous not for profit funding compared with 14, range 12-16 for the others; P = 0.38).
Journal readership on the basis of type of journal was significantly associated with quality of reporting, with a significant trend towards increasing QUOROM scores from reviews published in cardiology journals compared with those published in general medicine and nephrology journals. Moreover, QUOROM scores and Oxman and Guyatt scores were not associated (Pearson R = -0.06, P = 0.86, Spearman ρ = -0.11, P = 0.76). In most cases this was due to lack of assessment of bias in selecting and abstracting studies (six reviews) or lack of appraisal of internal validity of individual randomised controlled trials (five).
Limits of the included studies
Clinical and statistical heterogeneity are a major source of discordance in meta-analyses. A thorough appraisal of this issue is paramount in any research synthesis. Moreover, some authorities consider heterogeneity (P < 0.05 to 0.10 with hypothesis tests or I2
> 50%) a potential hurdle to the completion of a formal meta-analysis.1,2
Heterogeneity was appraised in most of the included meta-analyses (nine reviews), by means of Cochran Q, Der Simonian-Laird, Breslow-Day, or χ2
tests. In all cases evidence for statistical heterogeneity was present (P values ranging from 0.05 to < 0.001), and in all studies except one23
reviewers computed pooled effect size, on the basis of random effects methods (seven reviews), on fixed effects (one), and on both fixed and random effects (one).
A major source of bias in meta-analyses is small study size, which predisposes to a higher probability of publishing, quoting, and disseminating the results of randomised controlled trials with significant or positive results. This leads to pooled effect estimates being skewed to more positive findings, as small but significant randomised controlled trials are not counterbalanced in the statistical pooling procedure by similarly small but negative or non-significant studies, as shown graphically by funnel plots.1,2
Bias due to small study size was acknowledged and explicitly tested in most (eight) reviews, using Begg, Egger, or Rosenthal tests. Such tests provided significant results, suggesting the potential presence of such bias in five cases, prompting reporting and discussion of funnel plots in four of the five, plus another review that had none the less reported a non-significant test (). Despite checking for small study bias among the same pool of seven studies, Birck et al12
and Isenbarger et al21
reached disparate conclusions, respectively in favour and against the likely presence of such bias, a finding that can be partly explained by the use of risk ratios by Birck et al and odds ratios by Isenbarger et al. Moreover, despite such likelihood (P < 0.10) for small study bias at specific tests, all meta-analyses went forward with the pooled estimates thus providing potentially biased results. Duong et al carried out two separate analyses, the first limited to randomised controlled trials published as complete reports and the second also including randomised controlled trials available only as abstracts.29
They thus showed that bias due to small study size was likely when selecting studies published only as complete reports (P = 0.02), but that it was no longer evident when including studies reported only as abstracts (P = 0.22).29