|Home | About | Journals | Submit | Contact Us | Français|
Are existing systematic reviews of interventions for age-related macular degeneration incorporated into clinical practice guidelines?
High-quality systematic reviews should be used to underpin evidence-based clinical practice guidelines and clinical care. We have examined the reliability of systematic reviews of interventions for age-related macular degeneration (AMD) and described the main findings of reliable reviews in relation to clinical practice guidelines.
Eligible publications are systematic reviews of the effectiveness of treatment interventions for AMD. We searched a database of systematic reviews in eyes and vision and employed no language or date restrictions; the database is up-to-date as of May 6, 2014. Two authors independently screened records for eligibility and abstracted and assessed the characteristics and methods of each review. We classified reviews as “reliable” when they reported eligibility criteria, comprehensive searches, appraisal of methodological quality of included studies, appropriate statistical methods for meta-analysis, and conclusions based on results. We mapped treatment recommendations from the American Academy of Ophthalmology Preferred Practice Patterns (AAO PPP) for AMD to the identified systematic reviews and assessed whether any reliable systematic review was cited or could have been cited to support each treatment recommendation.
Of 1,570 systematic reviews in our database, 47 met our inclusion criteria. Most of the systematic reviews targeted neovascular AMD and investigated anti-vascular endothelial growth factor (anti-VEGF) interventions, dietary supplements or photodynamic therapy. We classified over two-thirds (33/47) of the reports as reliable. The quality of reporting varied, with criteria for reliable reporting met more often for Cochrane reviews and for reviews whose authors disclosed conflicts of interest. Although most systematic reviews were reliable, anti-VEGF agents and photodynamic therapy were the only interventions identified as effective by reliable reviews. Of 35 treatment recommendations extracted from the AAO PPP, 15 could have been supported with reliable systematic reviews; however, only one recommendation had an accompanying intervention systematic review citation, which we assessed as a reliable systematic review. No reliable systematic review was identified for 20 treatment recommendations, highlighting areas of evidence gaps.
For AMD, reliable systematic reviews exist for many treatment recommendations in the AAO PPP and should be used to support these recommendations. We also identified areas where no high-level evidence exists. Mapping clinical practice guidelines to existing systematic reviews is one way to highlight areas where evidence generation or evidence synthesis is either available or needed.
Age-related macular degeneration (AMD) is the leading cause of severe vision loss in people over age 65 in industrialized countries.1,2 This disease can be divided into two basic subtypes: neovascular (“wet AMD”) and non-neovascular (“dry AMD”). Neovascular AMD is characterized by choroidal neovascularization (CNV), in which formation of abnormal blood vessels leads to sub- and intra-retinal macular edema, hemorrhage, and/or fibrosis causing rapid central vision loss. In non-neovascular AMD, because of the gradual loss of photoreceptors and development of geographic atrophy, vision decreases slowly over many years. With no effective treatment available, patients with non-neovascular AMD are usually followed to detect and treat complications, such as development of neovascular AMD.
For decades, laser photocoagulation was the only available treatment for neovascular AMD, yet other treatments have been the subject of research, including radiotherapy, interferon alpha, and photodynamic therapy, of which photodynamic therapy received regulatory approval in April 2000.3 More recently, treatments focusing on the neutralization of vascular endothelial growth factor (VEGF) by injecting antibodies (bevacizumab), antibody fragments (ranibizumab), or fusion proteins (aflibercept) into the vitreous of the eye have become the current standard of care for neovascular AMD.4
Systematic reviews are summaries of the best research evidence available to address a specific question and follow explicit eligibility criteria and methods.5 Because systematic reviews underpin evidence-based clinical practice guidelines, it is important that they are trustworthy and at low risk of bias, yet we know that this is not always the case.6 For example, an author who has a potential conflict of interest may influence research conclusions,7 or multiple reviews on the same topic may represent unnecessary duplication of effort and prove confusing if the review authors reach different conclusions. Some reasons for differing conclusions are understandable, for example when the studies synthesized in systematic reviews were conducted at dissimilar time periods or included different types of study designs.8 But sometimes differing conclusions can be ascribed to use of systematic review methods that are potentially subject to bias.9
Best practice for the development of clinical practice guidelines involves the integration of high quality systematic reviews.6 To accomplish this goal, guideline developers can elect to undertake a systematic review in-house, commission a third party to conduct a systematic review, use results from previously completed systematic reviews, or implement a combination of these methods.
The objectives of this study were to 1) identify all published systematic reviews in the area of eyes and vision that had examined the treatment of AMD, 2) assess the reliability of existing reviews, and 3) map clinical practice guideline recommendations to reliable systematic reviews in order to encourage the integration of reliable systematic reviews and clinical practice guideline recommendations.
The search strategies and definition used for systematic reviews have been published.10,11 Our searches employed no language or date restrictions and were up-to-date as of May 6, 2014. Systematic reviews eligible for the current study had examined interventions for AMD; we excluded reviews concerned only with AMD etiology, diagnosis, prognosis, and cost-effectiveness of treatment. Furthermore, to be eligible, reports of systematic reviews had to be full-text journal articles representing “a scientific investigation that addressed a focused question and used explicit, pre-specified scientific methods to identify, select, assess, and summarize similar but separate studies.”5,12 Systematic reviews were eligible regardless of whether meta-analyses were performed; however, we considered articles that described a meta-analysis only, without a systematic review component, ineligible, because we could not be sure they were based on a systematic review. For eligible reviews with multiple published versions, such as updated or co-published Cochrane reviews, we included the most recent publication.
We used a two-stage screening process to identify eligible systematic reviews. First, two individuals independently screened the titles and abstracts of all 1,570 reviews listed in our database of systematic reviews in eyes and vision as of May 6, 2014. Next, for all records classified as potentially relevant, two individuals reviewed each full-text report independently for eligibility. We resolved discrepancies at each stage through discussion.
For each eligible systematic review, two individuals independently abstracted data from the review onto an electronic data collection form developed, pilot-tested, and maintained in the Systematic Review Data Repository (SRDR).13 This form was adapted from components of the Critical Appraisal Skills Programme (CASP),14 the Assessment of Multiple Systematic Reviews (AMSTAR),15 and the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA);16 we have used it in other studies.9,17 We extracted data related to review objectives, populations, interventions, outcomes, methods (e.g., eligibility criteria for selection of studies for the systematic review, search strategies for eligible studies, assessment of risk of bias in included studies), results, conclusions, and financial support. When a meta-analysis was conducted, we also abstracted data on the statistical methods used. We resolved any discrepancy in data abstraction through discussion.
Based on previously published criteria,9 and standard systematic review methodology,5,6,14–16 we classified reviews as reliable when they reported (1) defined criteria for selection of studies, (2) comprehensive searches for eligible studies, (3) assessment of risk of bias in included studies, (4) appropriate statistical methods for meta-analysis, and (5) agreement between the results and conclusions. We considered searches to be comprehensive when three or more bibliographic databases were searched, at least one method of other searching was employed (e.g., handsearching conference abstracts, identifying ongoing trials, screening reference lists of included studies), and search results were not limited to English-language only.5 When one or more of these criteria were not met, we classified reviews as being unreliable.
We conducted descriptive analyses of review characteristics and estimated proportions of reliable reviews. We conducted a pre-specified subgroup analysis by whether the systematic review was a Cochrane review. Further, we explored characteristics of systematic reviews when more than one addressed the same research question.
We extracted treatment recommendations from the 2015 American Academy of Ophthalmology Preferred Practice Patterns (AAO PPP) on management of AMD.18 We included only recommendations related to the effectiveness of treatment interventions (i.e., recommendations related to diagnosis and follow-up were excluded) and recorded the section of the AAO PPP where we found each recommendation.
We mapped the treatment recommendations to systematic reviews identified by our study and assessed whether reliable systematic reviews were available to address each treatment recommendation and, if so, whether they were cited by the AAO PPP. We also assessed whether sources of evidence were provided with each treatment recommendation and, when provided, categorized each cited reference as a systematic review, randomized controlled trial, or other study type.
The earliest eligible AMD systematic review identified was published in 2001 (Table 1). More than half (26/47; 55%) of the AMD systematic reviews focused on neovascular disease. The most commonly investigated interventions were anti-VEGF agents (15/47; 32%), dietary supplements (9/47; 19%), and photodynamic therapy (6/47; 13%). A majority of systematic reviews examined the effect of treatment on visual acuity (32/47; 68%) and safety (37/47; 79%); almost half assessed quality of life as outcomes of interest (23/47; 49%).
About one-third (15/47; 32%) of AMD systematic reviews were published in The Cochrane Library,19–33 with 25/47 (53%) published in other journals,34–58 and 7/47 (15%) as agency reports (e.g. French National Authority for Health).59–65 Most systematic reviews had at least two authors (43/47; 91%). The median number of bibliographic databases searched for systematic reviews was four; 31/47 (66%) groups of authors searched all possible years of at least one database. Only 28/47 (60%) review groups searched for non-English language articles. The number of included intervention studies in each systematic review ranged from 0 to 88 (median 7). Review findings were synthesized qualitatively in most (38; 88%) and quantitatively (“meta-analyses”) in about half (22; 51%) of the 43 systematic reviews that included two or more studies.
Almost two-thirds of AMD systematic reviews provided information on funding (31/47; 66%), with government (18/31; 58%) and department or institution (10/31; 32%) as the most common funding sources. Less than half of systematic review author teams stated that they had no conflicts of interest (19/47; 40%), with 12/47 (26%) disclosing that at least one author had a potential conflict of interest; 16/47 (34%) did not report information on conflicts of interest.
We classified the majority (33/47; 70%) of AMD systematic reviews as reliable (Figure 2). The most common reason for classifying a review as unreliable was not reporting a comprehensive search for eligible studies (Table 2 available at http://aaojournal.org). Compared with unreliable systematic reviews, reliable systematic reviews were more likely to have been funded by departments or institutions and produced by review authors who explicitly stated they had no conflicts of interest; all four systematic reviews that reported industry funding were assessed as unreliable (Table 2). Areas needing improvement across all reviews were the need for explicit statements regarding 1) pre-specification of eligibility criteria for studies to be included and 2) limitations of the review. In addition, review authors seldom performed independent evaluation of study eligibility and methodological quality, or independent data abstraction, by two or more reviewers (Figure 2).
All 15 Cochrane systematic reviews were classified as reliable compared with 18/32 (56%) non-Cochrane systematic reviews (Figure 3 available at http://aaojournal.org). All 15 Cochrane systematic reviews specified pre-defined eligibility compared with 16/32 (50%) non-Cochrane systematic reviews, and were more likely to have reported independent selection of studies by two or more review authors, assessment of risk of bias, and extraction of data compared with non-Cochrane systematic reviews. However, fewer Cochrane systematic reviews (27%) discussed limitations at the review level (e.g., incomplete retrieval of relevant studies, the potential effect of reporting bias on the review findings) than non-Cochrane systematic reviews (53%).
Reliable AMD systematic reviews of anti-VEGF agents and photodynamic therapy reported favorable results (Table 3 available at http://aaojournal.org). For other interventions, including antioxidant vitamins and/or minerals, complement inhibitors, interferon alpha, laser photocoagulation, radiotherapy, rheophoresis, statins, submacular surgery, and steroids, reliable AMD systematic reviews reported findings that were either inconclusive or that demonstrated no evidence of an intervention effect.
Among reliable AMD systematic reviews that had addressed the same research question, the conclusions were in good agreement with the exception of the comparative effectiveness and safety of ranibizumab versus bevacizumab for neovascular AMD. Ten reliable systematic reviews published between 2007 and 2014 included 17 distinct randomized controlled trials published between 2004 and 201166–82 (Figure 4 available at http://aaojournal.org). Reasons for discordance among systematic reviews all related to the studies included which, in turn, were due to variations in search dates, eligibility criteria, and minimum lengths of follow-up time. Authors of earlier systematic reviews that had compared ranibizumab versus bevacizumab cautioned against using bevacizumab as an off-label alternative to ranibizumab,41–43 whereas the more recent reviews, which included additional randomized controlled trials, suggested no appreciable difference between the anti-VEGF agents in terms of effectiveness or safety.34,38 The eligibility criteria of the systematic reviews changed over time, in accordance with completion and publication of findings from new randomized controlled trials. For example, earlier systematic reviews evaluated pegaptanib or ranibizumab versus sham treatment, but more recent systematic reviews evaluated head-to-head comparison of bevacizumab versus ranibizumab.
We extracted 35 treatment recommendations from the 2015 AAO PPP for AMD (Table 4). Treatment recommendations appeared in five sections of the AAO PPP document: 1) Highlighted findings and recommendations for care table; 2) Background text; 3) Care Process text; 4) Treatment recommendations and follow-up for AMD (Table 4 of PPP); and 5) PPP recommendation grading (Appendix 3 of PPP). Twenty-five of 35 recommendations were reported within the section of the PPP specific to the management of AMD, and 4 of the 35 recommendations were stated in all five sections of the PPP that reported recommendations. Most evidence cited by the AAO PPP to support recommendations were RCTs rather than systematic reviews: 18/35 recommendations were accompanied by citations to randomized controlled trials, whereas 1/35 recommendations was accompanied by citation to a reliable systematic review (Table 5 available at http://aaojournal.org). The PPP cited one other reliable systematic review identified by our study, but it was cited in the background section and not in direct support of a recommendation. No citation was provided to support 12/35 recommendations and 4/35 recommendations cited other reference types (e.g., AAO policy statements, insurance company documents, non-AMD intervention systematic reviews).
We identified existing reliable systematic reviews of interventions for AMD for 15 of the 35 treatment recommendations (Table 4). For example, additional reliable systematic reviews of anti-VEGF agents, vitamins and minerals, photodynamic therapy, laser photocoagulation, submacular surgery, and radiotherapy could have been referenced by the AAO PPP guideline to inform their recommendations but were not (Table 4). There were 20 treatment recommendations for which we identified no existing reliable systematic review, which highlights evidence gaps. The treatment recommendations and findings from reliable systematic reviews were generally consistent (Table 3 available at http://aaojournal.org).
We classified 14 (30%) of 47 systematic reviews describing intervention effectiveness for AMD as unreliable according to standard methodological criteria. Lack of reporting a comprehensive search strategy was the most common reason for classifying a review as unreliable. We found that Cochrane reviews comprise about one-third of all AMD systematic reviews. We assessed all 15 Cochrane reviews as reliable compared with 18 (56%) of 32 non- Cochrane reviews. This finding is in keeping with other investigations that have shown the high quality of Cochrane reviews and methodology.83–90 Because we are affiliated with the Cochrane Eyes and Vision Group, the criteria we set for assessing review methods and reporting are Cochrane-oriented. Our perspectives may partially explain the judgements we made and the discrepancies we found.
Studies evaluating the reporting quality of systematic reviews of other topics have found systematic reviews to be of disappointing quality, many finding 20% to 65% of the systematic reviews as being poor or low quality.83,84,91–95 Yet with the availability and promotion of methodological and reporting standards for systematic reviews,16,96–98 we expect reliable conduct and reporting of systematic reviews published in the literature to increase. Well-reported methods may not accord with methods actually used to conduct the review, however. For example, an investigation of studies described as randomized controlled trials in Chinese-language journals found that 93% (95% confidence interval (CI) 92.3% to 94.1%) of the studies actually used non-random methods to allocate treatment groups.99 A limitation of our study is that we evaluated systematic review reporting and did not contact review authors for supplemental information when methods were not reported or were reported unclearly. Furthermore, authors of reports from studies included in systematic reviews may not report methods clearly and accurately.
The uncoordinated fashion in which many systematic reviews currently are conducted and reported appears to result in unnecessary duplication of effort and varying results.100,101 In some cases, existing reviews were unreliable because of the lack of adherence to reporting standards and use of systematic review methodology aimed at minimizing selection and reporting biases. Publication of unreliable reviews represents a waste of resources. Journal editors should set standards for systematic reviews they publish and refer authors and peer reviewers to the PRISMA reporting standards.96,97 To conserve resources, we recommend that future systematic reviews should address unanswered clinical questions. Further, systematic reviews should be undertaken by individuals trained in systematic review methodology. Manuscripts that report systematic reviews should be reviewed by editors and peer reviewers knowledgeable in methodological and reporting standards in order to produce reliable research that can be used by guideline developers, patients, clinicians, and others.
The risk of producing reviews that are not relevant to clinical users is made tangible by the fact that many treatments for AMD summarized in reliable reviews included in our study were not mentioned in the 2015 AAO PPP. Many systematic reviews, including Cochrane reviews, undergo a long publication process that on one hand ensures high quality, but on the other hand may render them out of date or unavailable to users and guidelines producers in a rapidly emerging therapeutic area, such as anti-VEGF therapy for neovascular AMD. Collaboration between systematic reviewers and guideline developers could facilitate relevancy of topics and communication of results in a timely manner.
Six types of treatments for AMD were evaluated by two or more systematic reviews. In the case of five types of interventions (antioxidants, omega-3 fatty acids, photodynamic therapy, laser photocoagulation, and submacular surgery), reviews addressing the same topic yielded the same conclusions and initially appear to indicate a waste of resources. However, in the case of anti-VEGF therapy, the research question and eligibility criteria addressed by the systematic reviews changed over time as treatment availability and potential outcomes changed. The first systematic reviews included only RCTS that had compared pegaptanib or ranibizumab with control. The more recent systematic reviews of anti-VEGF therapy also included case series and non-randomized studies, specifically to address the issue of effectiveness and safety of the off-label drug bevacizumab. Since the time the searches were conducted for the current study, Cochrane authors have updated an earlier review of anti-VEGF effectiveness and also have published a review comparing the systemic safety of ranibizumab versus bevacizumab.102,103 Unlike other research that has found duplication of systematic reviews on the same topic to be wasteful101,104 or lead to discordant findings,105,106 we conclude that sequential systematic reviews that at first glance appear to cover similar topics instead may represent evolution in the research question with increased clinical experience and serve as an indication of a rapidly developing field.
Despite summarizing the available evidence, systematic reviews may not meet the needs of clinicians, patients, and guideline panels. Reviews with narrow scopes, i.e., those that split a clinician’s “real world” question into answerable research questions, may not provide all information needed by guidelines panelists. Nor do traditional pairwise comparisons address the question of “what works best”? “Multiple treatment comparisons” utilize network meta-analysis methodology, and increasingly are used when head-to-head trials of multiple interventions are not available or are insufficient to address the research question.107
Literature searches for the 2015 AAO PPP on AMD were updated 11 June 2013. The AAO PPP cited two systematic reviews that were rated as reliable in our study, with many recommendations citing only evidence from individual studies or no citation at all; the AAO PPP did not cite any unreliable systematic review. However, evidence from 22 additional reliable systematic reviews underpinning 15 of the 35 recommendations could have been incorporated into the AAO PPP. There were nine existing Cochrane reviews that directly supported 12 of the treatment recommendations. In accordance with best practice standards outlined by the Institute of Medicine,6 we suggest that interaction between systematic review teams and clinical practice guideline groups be encouraged to provide a comprehensive view of the evidence at a point in time and to illuminate evidence gaps. For example the AAO PPP panel for AMD could collaborate with the Cochrane Eyes and Vision Group to identify existing Cochrane reviews for their guidelines and highlight evidence gaps where Cochrane reviews should be given high priority. Cochrane authors would need to act promptly to provide timely development or updating of reviews.
The majority of treatment recommendations in the AAO PPP for AMD were supported by evidence from only randomized controlled trials or non-randomized studies. We acknowledge that a number of studies supporting some recommendations on treatments for AMD were well-designed, landmark RCTs, and these studies may have been well known to experts preparing recommendations. However, by transparently filtering and summarizing evidence in one place, systematic reviews provide an evidence base more extensive and comprehensive than looking at individual studies alone; they include structured assessment of trial methodology and the overall certainty of the evidence, providing the opportunity to evaluate all the evidence addressing a question to determine the current best answer. Systematic reviews and meta-analyses also are likely to be more useful than individual studies for providing information about rare adverse events, as even large RCTs often are not adequately powered to detect differences between treatments for infrequently observed outcomes.108
Although systematic reviews are important underpinnings of trustworthy treatment recommendations, they are not intended to serve in place of clinical practice guidelines. Clinical practice guidelines should be clear in stating unambiguously what is recommended, or not recommended, and should provide the evidence in support of each recommendation. In fact, frameworks such as GRADE (www.gradeworkinggroup.org), have tools that use complementary methods and presentation graphics to support the work of both guideline developers and systematic reviewers. These are especially important for recommendations for which no high-quality evidence exists so that guideline developers must rely on lower level sources of evidence and clinical expertise. For clarity, when preparing clinical practice guidelines it would be helpful to have all recommendations with supporting citations clearly reported in one place in the guideline document.
Ideally, reliable systematic reviews underpin evidence-based clinical practice guidelines. For AMD,reliable systematic reviews exist for many treatment recommendations in the AAO PPP and should be used to support these recommendations. Mapping clinical practice guidelines to existing systematic reviews is a useful way to highlight areas where evidence generation or evidence synthesis is either available or needed.
The authors gratefully acknowledge Cesar Ugarte Gil, Daniela Bacherini, Andrew Law and Elizabeth Clearfield for assistance with screening reviews and extracting data. We also thank Stephan Ehrhardt, Xuan Hui, Xue Wang, Isabel Rodríguez-Barraquer, and Tsung Yu for extracting data from articles written in non-English languages.
National Eye Institute (Grant 1 U01 EY020522), National Institutes of Health, Department of Health and Human Services, Bethesda, Md, USA. The sponsor had no role in the design or conduct of this research.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of interest:
All authors are contributors to Cochrane and the Cochrane Eyes and Vision Group.
Part of this research (Figure 4) was presented as a conference abstract at the 22nd Annual Cochrane Colloquium in Hyderabad, India (2014).