A total of 377 complete reviews were published in CDSR Issue 2, 1998. Thirty one Cochrane Collaborative Review Groups contributed to these reviews, with the greatest representation (33%) from the Cochrane Pregnancy and Childbirth Group.
Figure outlines the status in CDSR Issue 2, 2002 of all the original reviews from CDSR Issue 2, 1998. During this time period, 14 reviews were withdrawn and one was split, leaving 362 reviews that were present in both issues of CDSR. Of these 362 reviews, approximately one third (38%) had been updated with new included studies, one third (32%) had re-run searches but included no new studies, and 30% had not been updated at all. The median number of studies per review increased from 5 (range 0 to 72) in 1998 to 6 (range 0 to 108) in 2002. Of the 254 updated reviews with and without new studies included, only 23 (9%) had a changed conclusion.
The status of all the reviews from CDSR Issue 2, 1998 in CDSR Issue 2, 2002.
Of the 137 reviews updated with new studies, 18 reviews were excluded from our analysis because a summary statistic for the primary outcome was not available in one or both versions of the review, for example, if the review authors decided that the results of the included studies were not suitable for meta-analysis. Nine of these reviews had a change of conclusion and nine were unchanged. For the reviews with an unchanged conclusion, seven of the nine did not have a summary analysis available in either versions of the review, and two changed their research question resulting in a change in primary outcome. In the nine reviews with a changed conclusion, three did not have a summary analysis available in either version of the review and in four reviews either the research question changed or the outcomes examined changed resulting in a change in conclusion. In the remaining two reviews with a changed conclusion, a meta-analysis was not possible in the 1998 version and was then subsequently possible in the 2002 version.
This left 119 reviews in which new studies had been added to the updated review and for which data were available from the meta-analysis of the primary outcome from both 1998 and 2002. Further analysis was conducted only on these reviews to determine the effects of updating.
Ratio of confidence intervals
Relative precision of the review pairs has been determined as a ratio of the width of the CI in 2002 to the width of the CI in 1998. For 85 of the 119 reviews (71%), the width of the CI around the primary outcome changed by less than 20% as a result of adding new studies (Figure ). Of these, the width of the CI increased, remained the same, or decreased in five, 36, and 44 of the updated reviews respectively. For the five reviews with widened CIs, two had the same number of studies but had re-extracted the data from their original studies and in the other three reviews the updated meta-analysis led to only a small change in the precision (ratio range 1.04–1.11). The mean ratio of the width of the CI in 2002 to the width of the CI in 1998 was 0.81 (95% CI; 0.75, 0.86). For reviews with unchanged conclusions (n = 105) and changed conclusions (n = 14), this ratio was 0.85 (0.81, 0.89) and 0.56 (0.36, 0.81), respectively.
Ratios of confidence intervals of the summary statistic of the primary outcome of Cochrane reviews from Issue 2, 2002 to Issue 2, 1998.
An increase in the precision of the CI in 2002 compared to 1998 (that is, a decrease in the width of the CI in 2002), increased the odds of a change in conclusion of the systematic review. In particular, for each percentage increase in the precision of the CI in 2002, the odds of a change in conclusion were 3.3% (95% CI; 1.0%, 5.6%) higher than the previous odds. Therefore, for a 19.1% increase in precision, as was observed on average between 1998 and 2002, the odds of a change in conclusion were 85.7% (95% CI; 21.1%, 184.6%) higher than those of no change in precision.
Determination of significance change
Of the 119 reviews, 11 summary statistics of the primary outcome changed statistical significance between 1998 and 2002. Five changed from significant (p < 0.05) to non-significant (p ≥ 0.05), while the remaining six changed from non-significant to significant.
For the 11 reviews where there was a change in statistical significance of the summary statistic for the primary outcome, four reviews changed conclusion (36.4%). Of the 108 reviews where there was no change in the statistical significance, 10 reviews changed conclusion (9.3%). The difference in these percentages was 27.1% (95% CI; 0.7%, 60.3%).
Ten of the 14 reviews that changed conclusion did not have a change in significance of their summary statistic for the primary outcome. For eight of these reviews, the change in conclusion was related to a change for an outcome other than the primary outcome. Of the remaining two reviews, the change in conclusion in one was related to a sub-group analysis, and for the other review, the 1998 analysis was significant but based on only one small trial, and a further trial confirmed this in the 2002 review leading the review authors to be firmer in their conclusions.