|Home | About | Journals | Submit | Contact Us | Français|
Conceived and designed the experiments: PG KW DM LH CH RP TK AH. Performed the experiments: PG KW DM CH AH. Analyzed the data: PG KW DM CH RP AH. Wrote the paper: PG KW DM LH CH RP TK AH. Provided input on the data analysis: KW DM LH CH RP TK AH. Provided clinical advice: KW DM CH TK AH. Provided methodological advice: LH CH TK. Provided critical revision of the manuscript for important intellectual content: KW DM LH CH RP TK AH.
As a first step in developing a framework to evaluate and improve the quality of care of children in primary care there is a need to identify the evidence base underpinning interventions relevant to child health. Our objective was to identify all Cochrane systematic reviews relevant to the management of childhood conditions in primary care and to assess the extent to which Cochrane reviews reflect the burden of childhood illness presenting in primary care.
We used the Cochrane Child Health Field register of child-relevant systematic reviews to complete an overview of Cochrane reviews related to the management of children in primary care. We compared the proportion of systematic reviews with the proportion of consultations in Australia, US, Dutch and UK general practice in children. We identified 396 relevant systematic reviews; 358 included primary studies on children while 251 undertook a meta-analysis. Most reviews (n=218, 55%) focused on chronic conditions and over half (n=216, 57%) evaluated drug interventions. Since 2000, the percentage of pediatric primary care relevant reviews only increased by 2% (7% to 9%) compared to 18% (10% to 28%) in all child relevant reviews. Almost a quarter of reviews (n=78, 23%) were published on asthma treatments which only account for 3–5% of consultations. Conversely, 15–23% of consultations are due to skin conditions yet they represent only 7% (n=23) of reviews.
Although Cochrane systematic reviews focus on clinical trials and do not provide a comprehensive picture of the evidence base underpinning the management of children in primary care, the mismatch between the focus of the published research and the focus of clinical activity is striking. Clinical trials are an important component of the evidence base and the lack of trial evidence to demonstrate intervention effectiveness in substantial areas of primary care for children should be addressed.
In both developed and developing countries, the clinical care of children accounts for a large proportion of primary care. For example, in the UK, over 95% of NHS clinical contacts are made in general practice with over 300 million taking place each year, and consultations with children under 15 years represent 18% of all primary care consultations in the UK, 17% in the US and 12% in Australia , , . A 10,000-patient general practice will expect to have 1,500 children under 16 years registered .
There are concerns about the quality of primary care delivered to children in the UK due to episodes highlighted by the confidential enquiry into child deaths and increases in Accident and Emergency presentations, short admissions and unplanned hospitalizations , . Similar concerns about the quality of ambulatory care for children have been documented in the US . Globally, more than 10.6 million children under 5 years of age die each year, mostly from preventable and treatable causes, including pneumonia, diarrhea, malaria, injuries, HIV/AIDS, measles and malnutrition , , , . The high level of child mortality in poor countries indicates that high quality evidence based provision of child healthcare is an even greater problem.
However, measuring the quality of care of children is challenging due to the lack of pediatric quality measures . The first step to developing a framework to evaluate and improve the quality of care of children is the identification of the evidence base for interventions relevant to child health and primary care. Systematic reviews are the most comprehensive syntheses of information in healthcare  and represent a logical starting point to identify high quality evidence relevant to a broad clinical topic.
The number of systematic reviews published annually has steadily increased over the past decade . In 2007 alone, 2,500 systematic reviews of research were published, with roughly 20% of them produced through the Cochrane Database of Systematic Reviews (CDSR) . The primary aim of the Cochrane Collaboration is to make available up-to-date syntheses of reliable evidence regarding the benefits and risks of healthcare interventions . Cochrane systematic reviews are widely regarded as methodologically rigorous and of higher quality than non-Cochrane systematic reviews , , , , . The CDSR therefore represents a sample of high quality studies to better understand the current state of the evidence base.
A previous review described the evidence available from child-relevant systematic reviews in the CDSR, yet it did not determine the evidence base relevant to primary care . In addition no previous studies have compared the scope and number of systematic reviews published with the burden of illness in general practice. Therefore, we completed an overview of systematic reviews in the CDSR relevant to the management of childhood conditions in primary care and compared the topics and number of systematic reviews with the burden of childhood illness in primary care. These efforts provide a framework for researchers and policy makers to identify avoidable waste in the production of research evidence  and will lead to further initiatives in the CDSR and beyond to improve the quality of evidence for children in primary care.
Data for this study was acquired through previously published work, no patient or hospital data was accessed. Therefore, written consent and institutional ethical review was not required for this research. The PRISMA checklist and flow diagram are available as supporting information; see PRISMA Checklist S1 and PRISMA Flow Diagram S1.
We identified eligible systematic reviews from the Cochrane Child Health Field Reviews Register, a register of all child health relevant systematic reviews published in the CDSR. The search terms used to develop the register are outlined in Appendix S1. The register includes all Cochrane systematic reviews that intended to include children (0–18 years of age) or studied an intervention intended to improve the health and wellbeing of children, consistent with those of the Cochrane Child Health Field Trials Register , including reviews that included both children and adult participants. The original register included all studies identified in Issue 2, 2009 of the CDSR and was updated in Issue 8, 2010. We did not include Cochrane protocols.
Three reviewers (PG, KW and AH) independently screened a sample of 100 abstracts to pilot and refine the inclusion criteria. Subsequently, two reviewers (PG and KW) independently screened all potentially relevant studies for inclusion. Disagreement was resolved by consensus or consultation with a third reviewer (AH or DM) at all stages. We included systematic reviews regardless of the study designs included. We excluded studies that focused only on children aged less than one month as we felt that the majority of neonatal interventions would be delivered in secondary and tertiary care.
Our definition of ‘primary care setting’ included general or family practice, ambulatory care, pediatric outpatient clinics, pediatric assessment units or emergency departments. We included systematic reviews that were relevant to the management of childhood conditions in primary care or which evaluated interventions that could be administered in the primary care setting by a healthcare provider (screening for early detection of disease, diagnosis of conditions, initiation of treatment, referral for secondary care and ongoing monitoring in primary care). To focus on topics specific to general practice, we excluded school based interventions, specialist nurses, interventions delivered during pregnancy, public health programs and orthodontic and specialist dental procedures.
The Cochrane Child Health Field provided data from systematic review register in a Microsoft Excel sheet. The data available in the register for each systematic review are described elsewhere  and include general review characteristics, characteristics of included studies and methodological approaches. We updated the register and classified included systematic reviews according to type of care (i.e. acute, chronic or preventative)  and applicability to developing or developed countries (defined by the United Nations, which includes Europe, Canada, the USA, Australia, New Zealand, and Japan). One reviewer (PG) extracted data which were checked by a second (KW). Disagreement was resolved by consensus or consultation with a third reviewer (AH or DM) at all stages.
We identified burden of illness data from publically available datasets for Australia, the Netherlands, the UK and the US which were classified according to the International Classification of Disease revision 9 (ICD-9) or the International Classification of Primary Care 2nd Edition (ICPC-2). In Australia, the Bettering the Evaluation and Care of Health (BEACH) Project provides an annual database of approximately 100,000 general practice records classified by ICPC-2 . The Royal College of General Practitioners Weekly Returns Service (WRS) is a sentinel general practice network located across England and Wales that routinely monitors disease episodes from every consultation on a weekly basis by ICD-9 , , . In the US, the Centers for Disease Control and Prevention and the National Center for Health Statistics publish periodic reports on ambulatory healthcare visits from specific time periods . Lastly, we obtained Dutch data on childhood morbidity in general practice through a literature search . The ages included in the dataset ranged from 0–15 to 0–17 years. Where possible, we used estimates over a 3-year period to reduce the possibility of year-to-year fluctuations. The codes reflect primary diagnosis (US), last diagnosis in consultation (the Netherlands) or any diagnosis (Australia and UK) , , , , . Estimates of the burden of illness data are approximate rather than precise due to the limitations of the disease coding systems and availability of data. Further details are outlined in Appendix S2.
ICD-9 has many limitations in providing useful clinical data in primary care due to the structure of the axes (e.g. diagnostic entities can with equal logic be classified in more than one chapter) ,. Several specific diagnostic codes must be combined to generate clinically meaningful data (see Appendix S2). ICPC-2 was developed for use in primary care and classifies clinical activity in the domains of General/Family Practice, taking into account the patient's reason for encounter, the problems/diagnosis managed and interventions . We categorized all systematic reviews by ICPC-2 and by ICD-9 Disease Related Groups. When comparing systematic reviews to burden of illness data, we excluded reviews relevant to developing countries only, as the burden of illness data is specific to developed countries. Due to the structural differences in coding systems, we present the findings separately by ICPC-2 chapters and by ICD-9 (specific diagnostic categories) where relevant. We also compared the number of systematic reviews relevant to low-income countries with the major causes of death in children under 5 years of age other than neonates in Africa and Southeast Asia . Data were analyzed descriptively in PASW Statistics Version 18.0 using frequencies and proportions.
From 1,183 systematic reviews we excluded 787 systematic reviews that did not fulfill the inclusion criteria (Figure 1). We included 396 systematic reviews; a full list of included systematic reviews is included in Table S1.
Of the 396 systematic reviews, 38 (10%) found no relevant studies (n=22) or only found studies whose participants did not include children (n=16) (Table 1). There were 216 systematic reviews (57%) of drug interventions and 14 (4%) which evaluated a drug and a complex or surgical intervention. Fifty-five (14%) systematic reviews focused exclusively on developing countries. Of the 6,022 primary studies included in the reviews, 3,385 included children only and 899 included children and adults.
A greater percentage of systematic reviews on complex interventions (n=9, 24%) and natural health products (n=8, 21%) found no relevant studies or children participants compared to included reviews (complex, n=43, 11%; natural health product, n=34, 9%). Of the 38 reviews that did not find any published studies, the main conditions studied were: respiratory (n=10), blood and immune mechanism (n=6), psychological (n=6), digestive (n=3) and neurological (n=3).
The first systematic review was published in 1996 and since 2003, the rate of new publications has remained relatively constant at a median of 37 per year (range: 28–44). Nearly half of the systematic reviews (44%) were updated in 2008 or more recently, up-to-date as defined by the Cochrane Collaboration. Of reviews considered out-of-date, two were last updated in 1998.
Figure 2 illustrates the increase in the number of systematic reviews in the CDSR relevant to pediatric primary care compared to the increase in all child relevant and all Cochrane reviews. In 2009, there were 4,005 Cochrane reviews, of which 1,110 (28%) were on children and 370 (9%) were relevant to pediatric primary care. Since 2000, the percentage of child relevant reviews nearly tripled (from 10% to 28%) compared to a much smaller increase of 2% (from 7% to 9%) in pediatric primary care relevant reviews.
Figure 3 illustrates systematic reviews categorized using the International Classification of Primary Care 2nd Edition. Of note, five conditions cover 76% of all systematic reviews: respiratory (37%, n=148), psychological (11%, n=42), general and unspecified (e.g. fever; 10%, n=41), digestive (10%, n=40), and neurological (7%, n=29).
Table 2 shows the most common reasons for childhood consultations in Australian and Dutch general practice, excluding systematic reviews relevant to developing countries only (n=55). Respiratory conditions are the most common reason for general practice consultations in children representing 23–32% of all consultations, of which 43% (n=147) of included systematic reviews are relevant. There was a discrepancy between the burden of illness and the corresponding number of systematic reviews. For instance, despite representing over 15% of general practice consultations, only 7% (n=23) of systematic reviews were relevant to skin conditions. In contrast, 12% (n=42) of systematic reviews were on psychological conditions even though these only represent a relatively small proportion of consultations in Australian and Dutch general practice (2–3%).
Table 3 evaluates specific diagnostic categories in greater detail and shows that despite asthma representing 3–5% of consultations in Australia, UK and US, it is the focus of 23% (n=78) of systematic reviews. However, only 2% (n=8) of systematic reviews were on injuries despite being the reason for 7–10% of consultations.
Table 4 compares the major causes of death in children younger than age 5 years in Africa and Southeast Asia and shows that while certain topics have a large proportion of systematic reviews, such as HIV/AIDS (22%, n=12) and diarrheal diseases (22%, n=12), childhood injuries and pneumonia have a relative lack of Cochrane reviews.
Our study is the first comprehensive mapping of the applicability of Cochrane systematic reviews to a large clinical topic – children in primary care – including approximately 10% of all reviews published in the Cochrane database. Although previous studies have shown that Cochrane systematic reviews only represent 20% of all published reviews , they are the principal international source of quality-assessed evidence on clinical effectiveness for health care workers and policy decision makers.
Our findings show some striking mismatches between the conditions for which the Cochrane database provides clinical evidence and the burden of childhood illness in primary care. For example, despite the increasing incidence of skin conditions (such as rash, eczema and impetigo) which precipitate up to 23% of all primary care consultations , , , only 7% of included Cochrane reviews are relevant to this topic. In contrast, 23% of reviews were on asthma which accounts for only 3–5% of clinical consultations. There are no similar primary care consultation data for low-income countries, but while the importance of HIV and malaria as important causes of childhood mortality is recognized in the Cochrane database, there is a relative lack of Cochrane review evidence on pneumonia and childhood injuries. Both conditions are commonly seen in primary care in low-income countries and are associated with high mortality .
There are several potential reasons for the observed discrepancy in the number of Cochrane reviews and the burden of consultations. First, systematic reviews are based on primary research and the deficiencies may reflect the lack of primary studies available for synthesis. For example, reviews of complex interventions and natural health products were more likely to identify studies with adult participants only or find no relevant studies. The lack of controlled trials in child health is well documented and due to numerous reasons . Second, clinicians and researchers identify topics for Cochrane systematic reviews based on clinical uncertainty, practice variation or research interests. These may not correlate with the number of consultations or burden of illness. Third, public funding of research is correlated only modestly with disease burden, if at all . Fourth, there may be more interventions (and comparisons) available for certain conditions than others. Fifth, certain areas may be broader in scope and represent more varied conditions (e.g. neurological disorders) that warrant a disproportionately greater volume of research relative to their consultation burden. Finally, the lack of pediatric training for general practitioners in many countries and therefore lack of academic interest in pediatric primary care is likely to contribute to the lack of research endeavor.
Previous studies have suggested that there may be insufficient evidence specific to children in certain topic areas , , , , especially non-drug interventions such as counseling and advice . The emphasis on drug interventions is reflected in the funding of both commercial research and non-commercial research  while 69% of all pediatric controlled trials assess drug products . In our study, we also found an over-representation of reviews relevant to drug interventions. Non-drug interventions are an important part of primary care and need to be reflected in evidence syntheses and research funding.
It is essential that systematic reviews address questions that are relevant to patients, clinicians and policy makers . Several research groups have attempted to quantify the evidence that is available and used by physicians making treatment decisions , . Even when reviews are applicable to children in general practice often authors fail to provide a description of how the intervention can be applied in clinical practice, reinforcing the findings of previous studies , . Current work is being completed to determine how primary care practitioners use and apply Cochrane reviews in practice.
Several of the systematic reviews in our study that used a meta-analysis included both children and adults; this observation is consistent with other studies , . Combining studies with adults and children does not identify potentially important differences in efficacy and safety between these population groups  and can result in ineffective or even unsafe medical care , . To derive relevant evidence for decision-making, systematic reviews should consider potentially important differences in effects across different populations. Guidance to inform subgroup analyses is currently being prepared through StaR Child Health (www.ifsrc.org), an initiative aimed at improving the design, conduct, and reporting of pediatric research .
Cochrane systematic reviews are expected to be updated every two years,  yet 56% of systematic reviews were last assessed as up-to-date prior to 2008 (as of August 2010), greater than 38% reported elsewhere . Several authors have reiterated the importance of developing mechanisms and channeling resources to ensure that systematic reviews are updated as new evidence emerges , .
Our definition of primary care setting included pediatric assessment units and emergency departments, which may have partially explained the discrepancy between consultations and reviews. However, we only included studies that could be administered in the primary care setting by a healthcare provider. Our analysis was restricted to Cochrane reviews of interventions; there may be systematic reviews published outside the CDSR that fill some of the clinical gaps identified, particularly screening, diagnostic and prognostic reviews. In addition, we did not assess the quality of included systematic reviews. Classification of the reviews by ICPC-2 and ICD-9 has several limitations and was completed to provide an approximate comparison rather than precise figures.
The study highlights the on-going challenges of evidence based practice in primary care and the importance of characterizing the current evidence base. Further steps are required to identify and prioritize which systematic reviews are needed in primary care and encourage Cochrane Review Groups, funders, and other relevant organizations to promote those topics among potential authors. The recently implemented initiative to register all systematic review protocols is a step forward , . There is a need for clinicians, researchers and policy makers in primary care to improve the evidence base in children by determining if the current evidence base adequately informs clinical practice.
Search terms used by the Cochrane Child Health Field. The search strategy used to develop the Cochrane Child Health Field Reviews Register, a register of all child health relevant systematic reviews published in the CDSR.
Methodology of classifying systematic reviews by burden of illness data.
List of all included Cochrane systematic reviews.
Ethics approval was not required for the conduct of this systematic review. We would like to thank Denise Thomson for providing access to the systematic reviews registry, Mike Clarke for assistance in developing the study question and Iain Chalmers for his invaluable input on the manuscript.
Competing Interests: The authors have declared that no competing interests exist.
Funding: The creation of the Cochrane Child Health Field Reviews Register was funded by the Canadian Institutes of Health Research (grant 13278), the Cochrane Child Health Field and the Alberta Research Centre for Health Evidence (ARCHE). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.