Our study shows that SRs are now being produced in large numbers, with about 2,500 new publications indexed annually on Medline, of which about one-fifth are Cochrane reviews. This value is an underestimate of the total number of new non-Cochrane SRs, as we examined only a single database and our searching might not have identified all SRs, and we restricted our study to English-language publications.
Our results indicate that SRs predominantly address questions about the effectiveness of interventions, and about half of them report combining their results statistically. Recent SRs include more studies and participants than previously reported [4
], and while Cochrane reviews are smaller than other therapeutic reviews in our sample, their size corresponds to more recent data on Cochrane reviews [16
]. Similarly, and like randomized trials [17
], SRs are typically published in specialty journals.
There are some areas in which the quality of reporting has improved over time, such as the use of protocols [3
] and quality assessment [18
]. Similarly, very few reviews reported being funded by commercial sources, for which such funding has been associated with bias in the results of clinical trials [19
It is too early to comment on the significance of some of our findings. Few reviews are reported as updates of previously completed reviews. The utility of reviews likely diminishes over time as they become outdated. Likewise, no review had a formal registration number. Our study also provided some disappointing results. For example, publication bias was considered or assessed in only a minority of reports despite much evidence for its existence and potential influence on the results of reviews [20
However, for therapeutic reviews our comparison of Cochrane and non-Cochrane reviews provides the most discouraging results and suggests little improvement in the quality of reporting of non-Cochrane reviews over time [4
]. The Cochrane Collaboration has a strict set of policies and guidance as to how SRs should be conducted and reported. For example, it is Cochrane policy not to include the words “systematic review” or “meta-analysis” in the review title. This policy explains why we observed fewer Cochrane reviews reporting these terms, compared to non-Cochrane ones. Similarly, Cochrane reviews, because they are reported in an electronic medium, are not encumbered by word length restrictions as is the case for most paper-based journals. Electronic publication, along with mechanisms to ensure adherence to Cochrane policies, allows authors to provide more complete details of how they conducted their SRs, possibly explaining why we observed far superior reporting standards of Cochrane reviews compared to non-Cochrane therapeutic ones. Journal editors and/or readers, particularly of specialty journals, might be less interested in these details and/or not have the resources to monitor adherence to any set of policies and reporting guidelines.
Many non-Cochrane reviews did not report key aspects of SR methodology, thus impairing confidence in their results and conclusions. For example, only 11% of the authors reported working from a protocol to complete their review, showing little improvement over time [3
]. An examination of 47 Cochrane reviews, for which protocols almost always exist, revealed that 43 (91.5%) of them reported a major change such as the addition or deletion of outcomes between the protocol and the full publication [22
Strong evidence of outcome reporting bias was recently reported within clinical trials [23
]. Our results suggest that some aspect of selective outcome reporting bias might also exist within non-Cochrane reviews. Only about one-quarter of them reported a primary outcome, of which half report statistical significance in favour of this outcome (versus 14.4% for Cochrane reviews). This issue requires further investigation.
Results from SRs are most useful when they are up-to-date [25
]. Few reviews were reported as updates and there were large differences between Cochrane reviews and non-Cochrane reviews. This difference might reflect differing or nonexistent policies and/or practices across funders, but also may reflect the reluctance of journals to publish updates that are substantially the same as previous publications. If these SRs are to retain their currency, updating them needs to be a much higher priority. This issue is likely to become increasingly important in coming years.
Clinical trial registration was called for twenty years ago [26
]. In an attempt to minimize or avoid recent questionable behaviours such as hiding data [27
], it is now becoming widely endorsed by granting agencies, such as the Canadian Institutes of Health Research [1
], and editorial groups, such as the International Committee of Medical Journal Editors [28
]. Similar behaviours may well affect SRs, although there are currently few data to inform this belief. Although protocols of Cochrane reviews are published, no review in our sample was registered in the usual sense, or had any type of registration number. Systematic review registration options exist [29
] but are not well known. However, the reasons for registering SRs are likely different from that of registering clinical trials.
Providing reporting guidance is one way to improve the above-mentioned deficiencies. Real improvements have been seen in the reporting of RCTs [31
] since the introduction of the CONSORT Statement and endorsement by journals [32
] and editorial groups (for example, http://www.icmje.org
). The QUOROM Statement, which provides reporting guidance for meta-analysis of RCTs, was developed ten years ago in the hope of having a similar impact on meta-analysis [34
], and there are some indications that its use is associated with improved quality of reporting [35
]. Its uptake was much slower than was the CONSORT statement, possibly reflecting the belief that SRs were less frequently reported and not as important as RCTs. However, our study shows that currently about 2,500 such studies are published annually in English and indexed in Medline. QUOROM is currently being revised, and when published will provide journals and others with another opportunity to endorse its use as one way to improve the reporting of SRs. It is possible that the poor reporting of some SR categories reflects inadequate guidance available to authors. For example, we are unaware of any reporting guidance for SRs of prognostic studies.
Efforts directed toward improving reporting standards come at the end of the research process. Attempts to exert an influence much earlier in the research cycle might have a more profound and lasting effect. One option is to ensure that the evaluation process by granting agencies requires prospective applicants and grant peer reviewers to have some form of experience and expertise with SRs, for example by providing documentation of SR training. Additionally, journals might consider asking authors to include their SR protocols along with the completed review submissions.
Our cross-sectional analysis was limited in that we included just one month from a single database. It is unlikely, however, that sampling other months would have changed our results, as we verified that the month we chose was a typical entry month. While we included 300 reviews in our sample the majority of them were categorized as therapeutic. As such, our results concerning the other categories should be interpreted cautiously.
Our examination relied on what the authors reported. It is possible that the authors more completely conducted their SRs but omitted important details from their report, or the peer-review process resulted in the removal of key information we sought.
We have not provided here detailed information on several characteristics, such as the methods used to perform quantitative data synthesis or assess publication bias. More detailed information will be forthcoming in additional reports. Our intent here is to provide the reader with a broad overview of the reporting characteristics in recently published SRs.
Finally, we used a broad definition of what constitutes a SR. It is possible that a small proportion of our sample were intended to be literature reviews with very thorough descriptions of methods employed for the literature search. However, as such reviews would likely be used to the same extent as reports that are clearly defined as SRs or whose authors have quantitatively synthesized the information, we feel that they are subject to the same susceptibility to bias. Despite publishing SRs for about half a century, the National Library of Medicine does not yet index them as a publication type, at least in part because there is no agreed-upon definition of what constitutes a SR [36
], although there are initiatives to help identify them [37
]. A publication type for SRs on Medline is urgently needed.