The principal outcome of this study was the characterization of child-relevant SRs that are currently available in the Cochrane Database of Systematic Reviews. This descriptive analysis provides information on the extent of evidence synthesis on child-relevant topics as well as variation in methodological characteristics and rigour across the reviews. The register compiled through this effort provides a basis to identify gaps in the evidence base where reviews or updates are needed and to assist with prioritization for the production or updating of reviews within different topic areas. The register will also help identify priority areas for primary research in children. Further, ongoing identification and tagging of child-relevant SRs will facilitate access to pediatric-specific data.
Over the ten years studied, there has been a steady increase in the number and proportion of child-relevant reviews in the CDSR. This finding is inconsistent with the observation that the volume of adult trials is growing at a faster rate than pediatric trials [5
]. One of the key variables that we extracted was the number of trials within each child-relevant review that involved children only, adults only, or mixed adult and children populations. We found that less than half (47.4%) of studies included in child-relevant reviews were conducted solely in children. Moreover, only 25.2% of reviews that included both children and adults conducted separate analyses (11.4%) or subgroup analyses (13.8%) to distinguish between the results of the two groups. This figure may in part reflect the fact that separate or subgroup analyses were not deemed necessary in some situations (e.g., studies involving older adolescents and adults where there is no developmental or physiological basis for differences in response to a particular intervention). The extent of separate or subgroup analyses for children varied across Review Groups which also may reflect different situations where there may or may not be reason to expect a difference in effect. Nevertheless, previous work has suggested that there may be insufficient evidence specific to children for certain topic areas [6
]. This is particularly problematic in situations where the safety and efficacy of interventions for children may differ from adults due to variations in developmental physiology, disease pathophysiology, or developmental pharmacokinetics and pharmacodynamics [7
]. Systematic reviewers should be considering separate or subgroup analyses for children and discuss the applicability of the evidence for age groups that may show differential effects.
Several additional observations can be made regarding the extent of evidence available for child-relevant reviews. Overall, 9.3% of reviews found no relevant studies. This represents an important portion of child-relevant reviews and suggests that there is a need for primary research in a variety of areas. The proportion of reviews with no relevant studies varied across Review Groups and was particularly high (>20%) for the Cystic Fibrosis and Genetic Disorders and the Developmental, Psychosocial, and Learning Disorders Groups. Further, the median number of studies per review was 7 (IQR 3, 15) which is consistent with an analysis of Cochrane reviews published in 2004 [8
]. As many authors stated, this often provides insufficient data to attain significant results in meta-analyses, conduct subgroup analyses, or assess for publication bias. Finally, on average only half of relevant studies contributed to the largest meta-analysis in each review. Over-reliance on the results of meta-analyses is problematic as most meta-analyses will reflect only a portion of the available evidence. Research has demonstrated that the magnitude of effect decreases as the proportion of relevant trials contributing to the meta-analysis increases [9
One of the key goals of The Cochrane Collaboration is to ensure that available evidence is up-to-date http://www.cochrane.org
. To this end, The Cochrane Collaboration has implemented a policy that authors update their reviews every two years [10
]. We found a median date of 2007 for when reviews were 'Last Assessed as Up-to-Date,' suggesting that the majority of reviews have been updated according to Cochrane policy. However, 38% of the relevant reviews were last assessed as up-to-date prior to 2007 and are therefore considered out-of-date by Cochrane standards. This proportion varied across Review Groups. Additionally, 22 of the reviews that we had originally identified as relevant to the register were subsequently found to be withdrawn from The Cochrane Library and could not be included in our analysis. While there is ongoing work on the appropriate timing of updates [11
], mechanisms are needed to ensure that child-relevant reviews are in fact providing the most up-to-date accurate information for decision-makers. This presents a challenge for reviewers in terms of the time and resources required for updates, as well as the end-users of the reviews or those wanting to synthesize the reviews (e.g., in overviews of reviews) as a basis for decision-making.
A further goal of The Cochrane Collaboration is to be global in its scope and meet the health information needs of people worldwide. Consistent with a previous descriptive analysis of SRs published in 2004 [8
], we found that an overwhelming majority of studies had corresponding authors in countries with a high rating on indices of human development and income level. This variable is simply a proxy for the applicability of topics to countries with different indices of human development and income level. For instance, many of the reviews with a corresponding author from these countries may in fact be relevant to lower income/development countries and may also have co-authors from these countries. There was interesting variation across Review Groups. For example, authors from the Airways and the Cystic Fibrosis and Genetic Disorders groups were primarily from high income regions (UK, Australia, North America) whereas those in the Acute Respiratory Infections and especially the Infectious Diseases Groups were more evenly distributed worldwide, and more often included authors from Africa and Asia. This likely reflects the conditions of most interest, and perhaps the highest prevalence, in developing countries.
Approximately 40% of reviews examined non-pharmacological interventions, which is consistent with a sample of Cochrane reviews published in 2004 [8
]. The most frequently examined non-pharmacological interventions were educational, behavioural, psychological, policy or legislative interventions. This reflects the usefulness of reviews within the CDSR to a variety of end-users including a range of practitioners and policy-makers. The variety of interventions examined in reviews may also explain some of the variability observed in the types of study designs included.
One of the strengths of The Cochrane Collaboration is that it includes methodological experts worldwide and produces cutting-edge methods for the conduct of systematic reviews. The Cochrane Handbook represents state of the art systematic review methods [10
]. Further, Cochrane reviews have been found to have better reporting quality than paper-based reviews [8
]. Despite this, we found substantial variation in the conduct of reviews across the Review Groups. For example, over 27% of reviews did not specify a primary outcome and this proportion varied substantially across Review Groups, ranging from 10 to 73%. Specification of primary outcomes should be done at the protocol stage. Peer-reviewed, published protocols are required for Cochrane reviews; the PRISMA group that develops guidelines for the reporting of systematic reviews is now advocating registration of all SR protocols to enhance "transparency and accountability" [16
The approach to assessment of methodological quality also varied across reviews. Allocation concealment was most commonly assessed; this is consistent with the recommended Cochrane methods prior to 2008 after which time the Risk of Bias (RoB) tool was introduced. Despite long-standing caution against the use of scales for quality assessment [17
], the Jadad scale was often used, particularly within certain Review Groups. Finally, a very small proportion of reviews assessed for publication bias which is consistent with a previous sample [8
]. Most reviews did not mention publication bias at any point; however, some authors stated that an assessment was intended, but was not done, likely due to an insufficient number of included studies. Our findings suggest a potential for publication bias in the reviews, as we found that a high percentage of the included studies (94.6%) were published in peer-reviewed journals. Previous research suggests that published studies tend to present positive findings. For example, in a study of abstracts presented at a major pediatric research meeting, only 60% were subsequently published and those published were more likely to report positive findings [18
Approximately half of the reviews received no funding from external sources. This varied substantially across review groups, with over 90% of reviews in the Infectious Diseases Group receiving external funding compared to only 20% in the Epilepsy Group. The time and resources required to conduct a methodologically rigorous SR are substantial. Lack of funding will seriously hamper efforts to synthesize evidence in child health. While we did not collect data regarding funding for updates, it is likely that funding is less frequently available for updating reviews. This will be an important obstacle to achieving the Cochrane's mission of ensuring that evidence for decision-making is up-to-date.
This work provides a solid foundation for future research in two key areas. First, the data we have collected will serve as a baseline to examine changes and standardization in review methods over time, as well as the availability of child specific evidence. Second, this work provides a basis for methodological research investigating bias at the level of SRs, as well as trials (e.g., through meta-epidemiological methods). Empirical evidence of bias specific to child-relevant research will result in a better understanding of biases and how they operate in this context.
Limitations of this study
The main limitation we encountered resulted from inconsistent or lack of reporting by review authors for key variables, confirming previous findings that there is room for improvement in reporting of SRs [15
]. For example, the number of participants included in the reviews was inconsistently reported. Sometimes the overall number was reported in the abstract or beginning of the results section. Other times, the data extractors had to rely on the information in the Characteristics of Included Studies table. Review authors did not always specify whether the number cited corresponded to participants recruited, randomized, or analyzed. Due to these and other caveats, the total number of participants cited in our data is likely smaller than those that were actually included in all reviews. The other variable that was particularly problematic was whether the included studies included children-only, adult-only, or mixed populations. Often this detail was not reported for each included study. We used a strict cut-off of 18 years to differentiate study populations; therefore, some studies with older adolescents would be categorized as mixed populations but the actual participants may be relatively homogeneous in terms of physiology or development. We chose inclusion criteria that were consistent with the Cochrane Child Health Field Trials Register which excludes all pregnancy studies except those evaluating breastfeeding or nutritional supplements. There are numerous other interventions administered during pregnancy that may affect infant outcomes; however, these were not captured in this descriptive analysis and may be an interesting focus for future work. Moreover, we may have missed reviews of interventions that do not directly involve children but are intended to improve their health.