Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Ann Intern Med. Author manuscript; available in PMC 2013 October 21.
Published in final edited form as:
PMCID: PMC3804310

What Comparative Effectiveness Research Is Needed? A Framework for Using Guidelines and Systematic Reviews to Identify Evidence Gaps and Research Priorities

Tianjing Li, MD, MHS, PhD, S. Swaroop Vedula, MD, MPH, Roberta Scherer, PhD, and Kay Dickersin, MA, PhD


The authors developed and tested a framework for identifying evidence gaps and prioritizing comparative effectiveness research by using a combination of clinical practice guidelines and systematic reviews. In phase 1 of the project, reported elsewhere, 45 clinical questions on the management of primary open-angle glaucoma were derived from practice guidelines and prioritized by using a 2-round Delphi survey of clinicians. On the basis of the clinicians′ responses, 9 questions were classified as high-priority. In phase 2, reported here, systematic reviews that addressed the 45 clinical questions were identified. The reviews were classified as at low, high, or unclear risk of bias, and evidence gaps (in which no systematic review was at low risk of bias) were identified. The following comparative effectiveness research agenda is proposed: Two of the 9 high-priority questions require new primary research (such as a randomized, controlled trial) and 4 require a new systematic review. The utility and limitations of the framework and future adaptations are discussed.

Comparative effectiveness research (CER) is a key element of current efforts in health care reform in the United States (1). Before initiating new research, investigators must ascertain gaps in the evidence by identifying the important clinical questions in a topic area and then determining whether existing research has answered these questions. Prioritizing new CER to address identified gaps presents an ongoing challenge (2). The Patient-Centered Outcomes Research Institute (PCORI) has committed funds to develop, test, refine, and evaluate methods that can inform the process of establishing and updating national priorities for CER and patient-centered outcomes research (3, 4).

We propose and assess a general framework for prioritizing CER, using primary open-angle glaucoma (POAG) as a topic area. Worldwide, glaucoma of all types is a major source of morbidity, decreased quality of life, and increased health care costs (5, 6). It has been estimated to be the second-leading cause of blindness and visual impairment in 2010 (5, 6). In the United States, open-angle glaucoma accounts for more than 90% of all glaucoma cases; it affected more than 2.25 million Americans aged 40 years or older in 2000, and this number is expected to increase to 3.36 million by 2020 because of the aging population (5, 6).

We aimed to develop, implement, and evaluate a framework for identifying evidence gaps and prioritizing CER. Because clinical practice guidelines (CPGs) reflect questions of interest to clinicians, we considered them a starting point for building our framework to prioritize CER. Once the clinical questions addressed in a CPG are identified, one can search for relevant systematic reviews to determine evidence gaps. For questions of intervention effectiveness, a systematic review of high-quality randomized, controlled trials (RCTs) can show whether research has answered the question; uncertainty reflected by systematic reviews indicates a potential candidate area for primary CER (2, 7, 8). Opinions on topics and outcomes can be sought from stakeholders, such as practicing clinicians, patients, and evidence users, and incorporated during the prioritization process (for example, when framing or ranking the research questions). Our objective for this report was to identify systematic reviews that addressed clinical questions derived from the CPGs, classify their methodological quality, propose a CER agenda, and assess the utility of our framework.


For practical purposes, we implemented and evaluated the utility of our framework in 2 phases (Figure 1). In phase 1, reported elsewhere (9), we used the 2005 American Academy of Ophthalmology (AAO) Preferred Practice Patterns for the management of POAG (10) to derive 45 answerable clinical questions that could be addressed by RCTs and systematic reviews of RCTs. We conducted a 2-round Delphi survey of 620 members of the American Glaucoma Society to prioritize the questions for research that would inform good patient care. Using responses from 169 participating clinicians, we classified 9 clinical questions as high priority (9).

Figure 1
Key steps and stakeholder involvement in a framework that combines practice guidelines and systematic reviews to identify evidence gaps and prioritize comparative effectiveness research.

In phase 2, we determined whether up-to-date systematic reviews were available for each of the 45 clinical questions derived from the AAO CPGs. To identify evidence gaps, we appraised the methodological quality of the reviews and mapped evidence from the reviews to the 45 questions. We derived the CER agenda for the management of POAG by combining the prioritized questions from phase 1 with the evidence gaps.

Identifying Existing Systematic Reviews

We prespecified the following criteria for including systematic reviews: full-text articles that described a systematic review, defined as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize similar but separate studies” (11, 12); inclusion of patients with POAG, as defined by the individual review authors; comparison of any intervention described in the 45 clinical questions with another intervention, placebo, or no intervention; and reporting of any outcome. We excluded reports that concerned only health economic evaluation and meta-analyses with no systematic review.

We worked with information specialists at the William H. Welch Medical Library, Johns Hopkins University School of Medicine, and developed a search strategy that combined eye and vision terms with a validated search filter designed to identify systematic reviews (13). We originally searched PubMed in November 2006; EMBASE in March 2007; and The Cochrane Library, Issue 4, 2006, with no language restrictions, to build a register of systematic reviews addressing eye and vision research questions (Appendix 1, available at, lists our search strategies). We updated our searches in September 2009.

We imported all citations from our electronic searches into a ProCite, version 5.0, database (Thomson Scientific, Philadelphia, Pennsylvania) and removed duplicate citations. Two investigators independently reviewed titles and abstracts, and we retrieved full-text articles of all records classified by at least 1 investigator as possibly eligible. Two investigators independently reviewed the full-text reports for final eligibility and resolved discrepancies through discussion.

Extracting Data and Appraising Methodological Quality

We extracted data from eligible full-text systematic reviews for study objective, population, interventions compared, outcomes examined, eligibility criteria for studies in the systematic review, search methods used to identify studies, and other study characteristics.

We assessed methodological quality by using a pilot-tested form that adapted items from related instruments (1418). Our assessment included the 13 quality criteria listed in Figure 2. When a meta-analysis was conducted as part of a systematic review, we abstracted data on the statistical methods used to calculate the meta-analytic estimate and its variance and on the assessment of statistical heterogeneity (the variability in the intervention effects across studies).

Figure 2
Systematic reviews on the management of primary open-angle glaucoma that satisfied each methodological quality criterion.

We selected 4 key deficiencies from the 13 quality items to classify the findings from a review as at low, high, or unclear risk of bias. Findings from a systematic review were classified as at high risk of bias if it contained a non-comprehensive literature search, did not assess the methodological quality of included studies, used inappropriate statistical methods for meta-analysis, or presented conclusions inconsistent with the review findings (for example, if the review finding was based on 1 statistically significant meta-analysis out of 26 meta-analyses conducted [19]). Findings were classified as at low risk of bias if none of these deficiencies was observed and as at unclear risk of bias if insufficient details were provided to permit judgment on 1 or more of the 4 deficiencies (Table 1) (20).

Table 1
Criteria for Classifying Systematic Review Findings as at Low Risk of Bias*

One reviewer abstracted data, assessed methodological quality, and classified risk of bias for the included systematic reviews. A second reviewer verified the abstracted data against the original publication; discrepancies were resolved through discussion.

Identifying Evidence Gaps and Prioritizing CER

We mapped systematic reviews that met our eligibility criteria to the 45 clinical questions; a systematic review could be associated with more than 1 question. We classified evidence cited in the AAO Preferred Practice Patterns to support the recommendations by study design (systematic review or RCT).

We compared the systematic reviews that were at low risk of bias with the results of the Delphi survey to identify evidence gaps and CER priorities. We considered evidence from systematic reviews to be most useful for decision making if it was at low risk of bias, up to date (the literature search was done within 2 years of publication of the review), and conclusive. We considered evidence to be conclusive if further research was unlikely to change our confidence in the estimate of effect and not conclusive if the reviews indicated uncertainty or unresolved questions and further research was likely to change our confidence in the estimate of effect (18).

Statistical Analysis

We summarized the characteristics of included systematic reviews and tabulated the number and proportion of reviews that fulfilled each methodological quality criterion. All data analyses were performed with Stata, version 10 (StataCorp, College Station, Texas). We estimated the resources required and staff needed to complete the project.

Role of the Funding Source

This phase of our study was funded by the National Eye Institute, National Institutes of Health. Phase 1 was supported by the Cochrane Prioritization Fund, The Cochrane Collaboration. The sponsors had no role in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript.


Searches and Study Selection

We identified 5990 unique records from our electronic searches, of which 547 systematic reviews addressed eye and vision research questions. Of these, 39 systematic reviews (42 articles) addressed questions related to management of POAG and met our eligibility criteria (Appendix Figure 1, available at (19, 2161). Eight of the 39 were Cochrane reviews, 18 were found in the Database of Abstracts of Reviews of Effects in The Cochrane Library, and the remaining 13 were found only in PubMed or EMBASE. The 39 systematic reviews were published between 2000 and 2009, with more than half published after 2007 (Appendix Table 1, available at The literature search was done within 2 years of publication of the review in 35 of 39 cases (89.7%).

Appendix Figure 1
Summary of evidence search and selection.
Appendix Table 1
Characteristics of the 39 Systematic Reviews on the Management of Primary Open-Angle Glaucoma

Figure 2 illustrates the findings from our appraisal of methodological quality of the 39 systematic reviews. Nearly 40% of the reviews did not report having prespecified eligibility criteria or performing a comprehensive literature search. For nearly half of the reviews, independent evaluation by more than 1 review author was not reported for eligibility assessment, data abstraction, or quality assessment of the included studies.

Thirteen of the 39 reviews (33.3%) included in this study were classified as at low risk of bias, 22 (56.4%) at high risk of bias, and 4 (10.3%) at unclear risk of bias. Reviews at high risk of bias were classified as such for 1 or more of the following reasons: noncomprehensive literature search (12 reviews), no assessment of the methodological quality of the included studies (11 reviews), inappropriate statistical analyses (13 reviews), or conclusions that were inconsistent with the review findings (1 review). Fifteen of 22 reviews were classified as at high risk of bias because of at least 2 reasons.

We identified 2 major types of inappropriate statistical analyses in 13 of 32 reviews (40.6%) with a meta-analysis (3 reviews had both types of inappropriate analyses) (Appendix 2, available at Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison (for example, data from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials were pooled separately and compared in a review [39]). In addition, 7 reviews that reported percentage change in intraocular pressure (IOP) from baseline as the effect estimate used an incorrect formula to calculate the variance of the effect estimate. Appendix 2 summarizes the incorrect formulas used in the reviews to calculate the variance for the percentage of change in IOP from baseline.

Identifying Evidence Gaps and Prioritizing CER

Table 2 shows the results of applying our method to identify evidence gaps in the context of the overall framework that we evaluated. We identified at least 1 systematic review for 23 of 45 clinical questions (51.1%) and at least 1 review at low risk of bias for 13 of 45 questions (28.8%) (Table 2 and Appendix Table 2, available at Five of the 9 questions identified by clinicians as high priority for research were associated with at least 1 systematic review that was at low risk of bias. Of these, 2 had reviews with findings that we classified as not conclusive; therefore, these 2 questions need new primary research (such as an RCT). The remaining 4 high-priority questions require a new systematic review because no systematic review at low risk of bias was found. Of the 36 lower-priority questions, 28 require a new systematic review (Appendix Table 2). The 2005 AAO Preferred Practice Patterns cited only 5 of 10 systematic reviews published before the guidelines were prepared (3235, 37, 49, 59, 60), and we classified 3 of the 5 as at low risk of bias (37, 59, 60).

Table 2
Results of Using Our Framework to Set the CER Agenda for High-Priority Clinical Questions
Appendix Table 2
Existence of and Findings From Systematic Reviews That Relate to Nonpriority Clinical Questions

Resource Requirements to Complete the Prioritization Project

We report the approximate time associated with each step and staff needed to complete the prioritization project (Figure 1 also lists stakeholder involvement).

For phase 1, deriving and refining clinical questions from CPGs took 80 hours. Systematic review methodologists, epidemiologists, and clinical trialists derived the questions from the CPG. Glaucoma specialists with expertise in clinical trials and systematic reviews and methodologists from the CPG development committee verified and refined the questions. Prioritizing the clinical questions took 200 hours, including the time for creating the survey questionnaire, conducting pilot testing, and administering the survey. In addition, we estimate that it took 30 minutes for each participating member of the American Glaucoma Society to complete our prioritization survey. The online survey interface and underlying database were created with donated service from a data management specialist. The survey was administered and data were managed by a graduate student.

For phase 2, we cannot estimate the time for identifying existing systematic reviews because this step was part of another project, for which we searched for and identified systematic reviews covering all eye and vision topics for a subject-specific database. Systematic review methodologists worked together with information specialists to complete this step. Extracting data and appraising the methodological quality of the systematic reviews took 3 hours per included review for data extraction, verification, and adjudication of discrepancies. A senior epidemiologist oversaw the process. Identifying the evidence gaps and prioritizing CER took 120 hours. Systematic review methodologists, epidemiologists, clinical trialists, and glaucoma specialists worked together in identifying evidence gaps. Finally, the time for drafting and revising the final report or manuscript will vary depending on the ultimate goal of the prioritization project, the sponsor’s requirements, and the publication process.


Our overall goal was to test a framework for prioritizing clinical questions and identifying evidence gaps by using existing systematic reviews and CPGs. The framework we tested includes the following steps: deriving clinical research questions from CPGs to reflect issues that clinicians encounter frequently, asking clinicians to prioritize the 45 questions for research to incorporate opinions from evidence users, and determining whether high-quality systematic reviews of all previous research exist for each clinical question and identifying evidence gaps. By mapping evidence gaps to clinicians’ priorities, we proposed a CER agenda.

Our experience demonstrated the feasibility of engaging key stakeholders in setting priorities at several junctures. We collaborated with practicing clinicians, researchers, CPG developers (AAO), and a relevant professional association (American Glaucoma Society) (9). In phase 1, we asked clinicians which questions they needed answered to practice effectively. Patient views could easily be incorporated into this phase. In phase 2, we identified the questions for which research is needed. Clinicians and patients may be in the best position to decide what is needed in clinical practice but may not be aware of or may discount existing research for various reasons (for example, because the patient population differs from the clinician’s own). Thus, a 2-phase framework that engages stakeholders with different perspectives and expertise is essential for setting CER priorities.

Our framework for setting CER priorities provides a clear guide for the type of research needed to address each clinical question (such as studies to synthesize existing evidence or to generate new evidence). We found that high-quality systematic reviews were needed to address 32 of the 45 clinical questions on the management of POAG, including 4 of the 9 high-priority clinical questions. We also identified 2 clinical questions for which evidence from new RCTs is needed because the existing reviews were not conclusive. However, a good practice would be to update the existing high-quality systematic reviews to include recent and ongoing research as a confirmatory step before initiating any new RCTs.

The framework we tested is a robust addition to existing approaches for setting CER priorities (2, 6270). It differs from nomination-based methods, such as those used by the Agency for Healthcare Research and Quality (62, 63) and the Institute of Medicine (64), in which research topics are suggested by clinicians, payers, members of the public, or others, and a set of rules are applied to determine the final priorities. These other methods are quite broad and have produced overall priorities for health care research (71). In contrast, our framework focuses on a specialty area and thus addresses the entire range of questions specialists face.

Our framework has limitations. When translating CPG recommendations into answerable clinical questions, we relied on interventions and outcome measures that were stated explicitly in the CPG. This may be of concern when guideline recommendations are out of date or do not address emerging research, techniques, practices, or outcomes considered important and relevant to clinicians and patients. We addressed this concern by asking stakeholders to supplement and refine the list of derived clinical questions as part of survey development. For our ongoing project in which we are applying our framework to diabetic retinopathy, a condition in which new technologies and drugs are developing rapidly, we are seeking the involvement of both clinical experts and patients to develop clinical questions to supplement those derived from CPGs. In contrast, when we applied our framework to interventions for POAG, we found no new interventions in the 2010 update of the 2005 guidelines by the AAO. In addition, we have involved stakeholders from regions where the condition is prevalent to ensure broad experience with the topic (Yu T, Li T, Friedman DS, Puhan MA, Dickersin K. Setting priorities for comparative effectiveness research on the management of primary angle closure: a survey of Asia-Pacific clinicians. In preparation). In a third example, we are planning to seek input from patients for research questions on dry eye, for which patient-reported outcomes are of primary interest. We encourage others to test our framework by extending our methods according to the topic and their needs.

Our framework relies on published systematic reviews; thus, the underlying trial data may not be completely up to date. We believe timeliness is a problem regardless of the prioritization method used; this is addressed to some extent when prioritized questions are first reported in a clinical journal because the community has an opportunity to assess the applicability of the question to current practice and knowledge. Our prioritization framework is less sensitive to timeliness than the questions themselves. For example, in our study of diabetic retinopathy, involving clinicians and patients will almost certainly result in adding clinical questions about the effectiveness of antivascular endothelial growth factor interventions for treating diabetic macular edema to those derived from CPGs. The trials on this topic are fairly recent, and existing systematic reviews may not yet include them. Thus, we suggest that CER researchers update existing systematic reviews before undertaking a new trial.

We focused on RCTs and systematic reviews of RCTs because we were interested in clinical questions of intervention effectiveness. Other study types may be equally or more appropriate in some instances, such as for questions about intervention harms. In addition, for questions that involve both effectiveness and harm, existing data may be conclusive for benefit but not for harm or conclusive for intervention effectiveness on some but not all outcomes. For example, we classified the main review findings about effectiveness as conclusive for our priority question on the effectiveness and safety of intraoperative mitomycin C; however, we also concluded that additional data may be needed to clarify certain serious but rare side effects, such as endophthalmitis (Table 2). The clinical and research community will need to decide when unresolved questions of harm are sufficiently important to warrant undertaking a new clinical trial or observational study.

Although we searched 3 major databases with a comprehensive and validated search strategy to identify systematic reviews, we did not search for unpublished systematic reviews, which may have a different methodological quality profile and findings from those published (72). In addition, our assessment of methodological quality relied on what the authors reported in the article, which may not reflect the actual methods used. Finally, if systematic review authors made errors when abstracting trial data, we could have underestimated the number of unreliable systematic reviews (7378).

Our findings show that although systematic reviews of RCTs are considered the highest level of evidence for questions about intervention effectiveness (7), their findings should not be accepted without critical evaluation of their methodological rigor. Consistent with others (7983), we identified a large proportion of systematic reviews that did not use rigorous methods, such as those suggested by the Institute of Medicine and others (84). We classified the findings of two thirds of the reviews as at high risk of bias for either having a noncomprehensive search for relevant studies or using inappropriate statistical methods. The negative impact of these methodological problems has been well-described in the literature (20, 8594).

The clinical questions we prioritized are restricted to pairwise comparisons and did not directly answer the question, “Which intervention works best?” because of the way clinical questions are addressed in most RCTs. Because this question must be answered to facilitate optimal health care decision making, a network meta-analysis should be considered once data from all high-quality systematic reviews and trials are identified for a given question (95). Our framework, which identifies all high-quality systematic reviews related to a health topic generally (96, 97), has an important scoping role in this regard.

We believe that our framework, which uses CPGs and systematic reviews to prioritize CER involving both new primary research and systematic reviews, is broadly applicable and could be applied in conjunction with other methods (for example, by PCORI) to formulate future research agendas.


The authors thank Claire Twose, MLIS, and Blair Anton, MLIS, MS, from the William H. Welch Medical Library for assistance in developing the search strategy; L. Susan Wieland, PhD, from the University of Maryland for assistance in screening the titles, abstracts, and full-text articles from the updated search; and Harini Sarathy, MBBS, MHS, for assistance in screening the full-text articles; and Isabel Rodriguez-Barraquer, MD, and Derek Ng, ScM, from Johns Hopkins Bloomberg School of Public Health and Yanfeng Chai, MD, MS, MPH, and Seryna Tamez from Jikei University Hospital, Tokyo, Japan, for assistance in retrieving and managing full-text articles for this project. They also thank Ann Margret-Ervin, MPH, PhD, and Dolly Chang, MD, MPH, from Johns Hopkins Bloomberg School of Public Health for their help in verifying the abstracted data and the risk of bias of included systematic reviews against the original publications.

Grant Support: By contract N01-EY2–1003 and grant U01-EY020522–01 from the National Eye Institute, National Institutes of Health. Phase 1 of the study was supported by the Cochrane Prioritization Fund, The Cochrane Collaboration.

Appendix 1: Search Strategies for Identifying Systematic Reviews on Eye and Vision Research Questions

PubMed Search Strategies

(ABNORMAL ACCOMMODATION[tiab] OR Abnormal color vision[tiab] OR ABNORMAL LACRIMATION[tiab] OR Abnormal vision [tiab] OR accommodative disorders [tiab] OR Amblyopia [tiab] OR Ametropia [tiab] OR ANISOCORIA-[tiab] OR ANOPHTHALMIA[tiab] OR Anterior CHAMBER hemorrhage [tiab] OR Aphakia[tiab] OR aqueous outflow obstruction [tiab] OR Asthenopia [tiab] OR Balint’s syndrome [tiab] OR Bilateral visual field constriction [tiab] OR Binocular Vision Disorder[tiab] OR BLEPHARITIS [tiab] OR BLEPHAROSPASM-[tiab] OR BLINDNESS [tiab] OR blurred vision [tiab] OR CATARACT [tiab] OR Cataracts [tiab] OR Chorioretinal disorder [tiab] OR Chorioretinitis [tiab] OR Choroid Diseases [tiab] OR Choroidal [tiab] OR Choroiditis [tiab] OR CHROMATOPSIA-[tiab] OR Color Blindness[tiab] OR Color Vision Defects [tiab] OR Color vision deficiency [tiab] OR Colour blindness [tiab] OR Conjunctival Diseases [tiab] OR CONJUNCTIVAL HAEMOR-RHAGE[tiab] OR Conjunctival Injury[tiab] OR CONJUNCTIVAL ULCERATION [tiab] OR CONJUNCTIVITIS [tiab] OR CORNEAL DEPOSITS [tiab] OR Corneal Diseases [tiab] OR Corneal Disorder [tiab] OR Corneal injuries [tiab] OR Corneal Injury[tiab] OR CORNEAL OEDEMA[tiab] OR CORNEAL OPACITY[tiab] OR CORNEAL ULCERATION [tiab] OR decreased Lacrimation [tiab] OR Decreased vision [tiab] OR defective vision [tiab] OR Delayed visual maturation [tiab] OR Difficulty seeing[tiab] OR difficulty with vision [tiab] OR Dim vision [tiab] OR Diminished Vision [tiab] OR DIPLOPIA[tiab] OR disturbed vision [tiab] OR ENDOPHTHALMITIS [tiab] OR EPIPHORA[tiab] OR Episcleritis [tiab] OR Equatorial staphyloma [tiab] OR EXOPHTHALMOS [tiab] OR Eye Abnormalities [tiab] OR EYE ABNORMALITY[tiab] OR Eye Burns-[tiab] OR Eye disease [tiab] OR Eye Diseases [tiab] OR Eye dis-order[tiab] OR Eye disorders [tiab] OR Eye edema [tiab] OR Eye Foreign Bodies[tiab] OR Eye Hemorrhage [tiab] OR EYE INFECTION [tiab] OR Eye Infections [tiab] OR Eye Injuries [tiab] OR EYE MALFORMATION[tiab] OR Eye Manifestations-[tiab] OR EYE MUSCLE PARALYSIS [tiab] OR Eye Neoplasm-[tiab] OR Eye Neoplasms [tiab] OR EYE PAIN[tiab] OR Eye swelling [tiab] OR Eyelid Disease[tiab] OR Eyelid Diseases [tiab] OR Eyelid Disorder[tiab] OR Eyelid Disorders [tiab] OR Eyelid pain[tiab] OR EYELID RETRACTION [tiab] OR FIXED PUPILS [tiab] OR GLAUCOMA[tiab] OR Glaucomas [tiab] OR Hazy vision [tiab] OR HEMIANOPIA[tiab] OR Hemianopsia-[tiab] OR Hepatolenticular Degeneration [tiab] OR Horner’s syndrome [tiab] OR HYPOPYON [tiab] OR Impaired vision-[tiab] OR impaired visual acuity[tiab] OR Interference with vision [tiab] OR IRITIS [tiab] OR KERATITIS [tiab] OR KERATOCONJUNCTIVITIS [tiab] OR Keratoconus[tiab] OR Lacrimal Apparatus Diseases [tiab] OR lacrimal disorder [tiab] OR LACRIMAL DUCT OBSTRUCTION [tiab] OR legally blind[tiab] OR Lens Diseases [tiab] OR Lens Disorder[tiab] OR Lens Opacities [tiab] OR Lens Subluxation [tiab] OR Localized anterior staphyloma [tiab] OR Low vision [tiab] OR MACULAR DEGENERATION [tiab] OR MEIBOMIANITIS[tiab] OR Metastases to eye[tiab] OR MIOSIS[tiab] OR MYDRIASIS-[tiab] OR MYOPIA[tiab] OR Night Blindness[tiab] OR Nystagmus [tiab] OR Ocular degeneration [tiab] OR Ocular discomfort [tiab] OR Ocular disease [tiab] OR OCULAR HAEMORRHAGE [tiab] OR OCULAR HEMORRHAGE [t-iab] OR OCULAR HERPES [tiab] OR Ocular Hypertension-[tiab] OR Ocular Hypotension [tiab] OR Ocular infections [tiab] OR Ocular inflammations[tiab] OR Ocular injuries [tiab] OR Ocular Injury[tiab] OR Ocular Motility Disorders[tiab] OR Ocular neoplasms [tiab] OR oculopathy [tiab] OR Open wound of ocular adnexa[tiab] OR OPHTHALMIC DISORDERS [tiab] OR Ophthalmological disorder [tiab] OR Ophthalmopathy-[tiab] OR OPTIC ATROPHY[tiab] OR Optic Nerve Diseases-[tiab] OR optic nerve disorder [tiab] OR Optic nerve injury[tiab] OR OPTIC NEURITIS [tiab] OR Orbital Diseases [tiab] OR PAPILLOEDEMA[tiab] OR Partial sight [tiab] OR Partial vision loss[tiab] OR Partially Sighted[tiab] OR Penetrating Eye Injuri-es[tiab] OR Periorbital fat herniation [tiab] OR Photalgia[tiab] OR PHOTOPHOBIA[tiab] OR PHOTOPSIA[tiab] OR PIGMENT PRECIPITATION[tiab] OR Poor vision[tiab] OR Posterior dislocation of lens[tiab] OR Posterior synechiae[tiab] OR Problem seeing [tiab] OR Proliferative Vitreoretinopathy[tiab] OR Pupil Disorders[tiab] OR Redness[All Fields] OR discharge of eye[tiab] OR Reduced ability to see[tiab] OR Reduced Vision-[tiab] OR Refraction Errors [tiab] OR Refractive disorders [tiab] OR Refractive Errors[tiab] OR Retinal defects [tiab] OR RETINAL DEPOSITS [tiab] OR RETINAL DETACHMENT [tiab] OR Retinal detachments [tiab] OR Retinal Disease[tiab] OR Retinal Diseases[tiab] OR RETINAL DISORDER[tiab] OR RETINAL EDEMA[tiab] OR RETINAL HAEMORRHAGE [tiab] OR RETINAL HEMORRHAGE [tiab] OR RETINAL OEDE-MA[tiab] OR RETINITIS [tiab] OR RETINOBLASTOMA-[tiab] OR retinopathy[tiab] OR RETROBULBAR NEURITIS-[tiab] OR Scleral Diseases [tiab] OR Scleral staphyloma [tiab] OR SCLERITIS[tiab] OR SCOTOMA[tiab] OR Sight impaired-[tiab] OR Staphyloma posticum[tiab] OR STRABISMUS [tiab] OR Subnormal Vision [tiab] OR SUNKEN EYES [tiab] OR SYMBLEPHARON[tiab] OR THROMBOSIS RETINAL VEIN [tiab] OR Tunnel Vision [tiab] OR Uveal Diseases [tiab] OR Uveal Disorder[tiab] OR UVEITIS [tiab] OR vision defects-[tiab] OR vision disorder [tiab] OR Vision Disorders [tiab] OR vision disturbance [tiab] OR vision impaired[tiab] OR VISION IMPAIRMENT[tiab] OR Vision problem[tiab] OR Visual Ag-nosia[tiab] OR visual defect[tiab] OR Visual difficulty[tiab] OR Visual disorders[tiab] OR Visual disturbance [tiab] OR VISUAL FIELD DEFECT[tiab] OR Visual Field Disorder[tiab] OR Visual field disorders [tiab] OR Visual impairment [tiab] OR Visual loss[tiab] OR Visual Pathway Disorder[tiab] OR Visual pathway disorders [tiab] OR Visual system disorder [tiab] OR VITREOUS DETACHMENT[tiab] OR VITREOUS HAEMORRHAGE-[tiab] OR Vitreous Hemorrhage [tiab] OR Vitreous membranes and strands [tiab] OR Vitreous prolapse [tiab] OR Vitreous sy-neresis[tiab] OR Wavefront aberration [tiab] OR Weak vision-[tiab] OR Wilson’s Disease[tiab] OR XEROPHTHALMIA[tiab]


((accommodation[tiab] OR aqueous humor[tiab] OR binocular movement [tiab] OR Capsule [tiab] OR choroid [tiab] OR ciliary body[tiab] OR conjunctiva[tiab] OR Cornea[tiab] OR (eye[tiab] AND adnexa[tiab]) OR eye[tiab] OR eyelid[tiab] OR globe[tiab] OR intraocular pressure [tiab] OR Iris[tiab] OR lacrimal passage [tiab] OR lacrimal system [tiab] OR lens [tiab] OR ocular adnexa[tiab] OR optic nerve[tiab] OR orbit[tiab] OR refraction [tiab] OR retina[tiab] OR sclera[tiab] OR trochlear nerve[tiab] OR uvea[tiab] OR Uveal Tract [tiab] OR vision [tiab] OR visual pathway[tiab] OR vitreous body[tiab] OR vitreous humor [tiab])


(“disease”[MeSH Terms] OR Disease[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Diseases [Text Word]) OR ((“disease” [TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Disorder[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease-”[MeSH Terms] OR Disorders [Text Word]) OR ((“contusion-s”[TIAB] NOT Medline[SB]) OR “contusions”[MeSH Terms] OR Contusion [Text Word]) OR (“contusions” [MeSH Terms] OR Contusions [Text Word]) OR (“syndrome” [MeSH Terms] OR Syndrome[Text Word]) OR ((“syndrome”[TIAB] NOT Med-line[SB]) OR “syndrome” [MeSH Terms] OR Syndromes [Text Word]) OR (“dislocations”[MeSH Terms] OR Dislocation-s[Text Word]) OR ((“dislocations”[TIAB] NOT Medline[SB]) OR “dislocations” [MeSH Terms] OR Dislocation [Text Word]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular [Text Word]) AND Occlusion [All Fields]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular [Text Word]) AND Occlusions [All Fields]) OR ((“wounds and injuries” [TIAB] NOT Medline [SB]) OR “wounds and injuries” [MeSH Terms] OR Injury[Text Word]) OR (“injuries”[Subheading] OR (“wounds and injuries”[TIAB] NOT Medline[SB]) OR “wounds and injuries”[MeSH Terms] OR Injuries[Text Word]) OR (“coloboma”[MeSH Terms] OR Coloboma[Text Word]))


“amaurosis fugax”[MeSH Terms] OR “amblyopia”[MeSH Terms] OR “asthenopia”[MeSH Terms] OR “blindness”[MeSH Terms] OR “blindness, cortical” [MeSH Terms] OR “color vision defects” [MeSH Terms] OR “conjunctival diseases” [MeSH Terms] OR “corneal diseases” [MeSH Terms] OR “diplopi-a”[MeSH Terms] OR “eye abnormalities”[MeSH Terms] OR “eye burns”[MeSH Terms] OR “eye diseases”[MeSH Terms] OR “eye diseases, hereditary” [MeSH Terms] OR “eye foreign bodies” [MeSH Terms] OR “eye hemorrhage” [MeSH Terms] OR “eye infections”[MeSH Terms] OR “eye injuries”[MeSH Terms] OR “eye injuries, penetrating” [MeSH Terms] OR “eye manifestations” [MeSH Terms] OR “eye neoplasms” [MeSH Terms] OR “eyelid diseases” [MeSH Terms] OR “hemianopsia” [MeSH Terms] OR “lacrimal apparatus diseases” [MeSH Terms] OR “lens diseases”[MeSH Terms] OR “night blindness”[MeSH Terms] OR “ocular hypertension”[MeSH Terms] OR “ocular hypotension”[MeSH Terms] OR “ocular motility disorders” [MeSH Terms] OR “optic nerve diseases”[MeSH Terms] OR “orbital diseases”[MeSH Terms] OR “photophobia”[MeSH Terms] OR “pupil disorders” [MeSH Terms] OR “refractive errors” [MeSH Terms] OR “retinal diseases” [MeSH Terms] OR “scleral diseases” [MeSH Terms] OR “scotoma” [MeSH Terms] OR “uveal diseases”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vitreoreti-nopathy, proliferative” [MeSH Terms] OR “vitreous detachment” [MeSH Terms])


(“therapy” [Subheading] OR (“therapeutics” [TIAB] NOT Medline[SB]) OR “therapeutics”[MeSH Terms] OR treatment-[Text Word]) OR ((“therapeutics”[TIAB] NOT Medline[SB]) OR “therapeutics” [MeSH Terms] OR treatments [Text Word]) OR (“diagnosis” [Subheading] OR “diagnosis” [MeSH Terms] OR diagnosis [Text Word]) OR intervention [All Fields] OR interventions [All Fields] OR (“prevention and control” [Subheading] OR prevention [Text Word])


(Humans[MeSH] OR (Humans[Mesh] NOT Animals [MeSH])


cochrane database syst rev[ta] OR search[tiab] OR metaanalysis [pt] OR MEDLINE [tiab] OR (systematic [tiab] AND re-view[tiab])

EMBASE Search Strategies

((*’asthenopia’*/de OR *’asthenopia’*) OR (*’cerebral blind-ness’*/de OR *’cerebral blindness’*) OR (*’diplopia’*/de OR *’diplopia’*) OR (*’eye malformation’*/de OR *’eye malformation’*) OR (*’eye burn’*/de OR *’eye burn’*) OR (*’eye injury’*/de OR *’eye injury’*) OR (*’perforating eye injury’*/de OR *’perforating eye injury’*) OR (*’hemianopia’*/de OR *’hemianopia’*) OR (*’lacrimal gland disease’*/de OR *’lacrimal gland disease’*) OR (*’intraocular hypotension’*/de OR *’intra-ocular hypotension’*) OR (*’photophobia’*/de OR *’photopho-bia’*) OR (*’pupil disease’*/de OR *’pupil disease’*) OR (*’sclera disease’*/de OR *’sclera disease’*) OR (*’scotoma’*/de OR *’scotoma’*) OR (*’vitreoretinopathy’*/de OR *’vitreoretinopathy’*) OR (*’vitreous body detachment’*/de OR *’itreous body detachment’*) OR (*’ransitional blindness’*/de OR *’ransitional blindness’*) OR (*’mblyopia’*/de OR *’mblyopia’*) OR (*’blindness’*/de OR *’blindness’*) OR (*’color vision defect’*/de OR *’color vision defect’*) OR (*’conjunctiva disease’*/de OR *’conjunctiva disease’*) OR (*’cornea disease’*/de OR *’cornea disease’*) OR (*’eye disease’*/de OR *’eye disease’*) OR (*’intraoc-ular foreign body’*/de OR *’intraocular foreign body’*) OR (*’intraocular hemorrhage’*/de OR *’intraocular hemorrhage’*) OR (*’eye infection’*/de OR *’eye infection’*) OR (*’eye tu-mor’*/de OR *’eye tumor’*) OR (*’eyelid disease’*/de OR *’eye-lid disease’*) OR (*’lens disease’*/de OR *’lens disease’*) OR (*’night blindness’*/de OR *’night blindness’*) OR (*’intraocular hypertension’*/de OR *’intraocular hypertension’*) OR (*’eye movement disorder’*/de OR *’eye movement disorder’*) OR (*’optic nerve disease’*/de OR *’optic nerve disease’*) OR (*’or-bit disease’*/de OR *’orbit disease’*) OR (*’refraction error’*/de OR *’refraction error’*) OR (*’retina disease’*/de OR *’retina disease’*) OR (*’uvea disease’*/de OR *’uvea disease’*) OR (*’visual disorder’*/de OR *’visual disorder’*)) AND ((*search*) OR (*’meta analysis’* OR *’systematic review’*))

The Cochrane Library Search Strategies

We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment database, and the United Kingdom National Health Service Economic Evaluation Database in the Cochrane Library by using the following Medical Subject Heading terms as terms and as keywords in “Title, Abstract, and Keyword”: amaurosis fugax; amblyopia; blindness; color vision defects; conjunctival diseases; corneal diseases; eye diseases; eye diseases, hereditary; eye foreign bodies; eye hemorrhage; eye infections; eye injuries; eye manifestations; eye neoplasms; eyelid diseases; lens diseases; night blindness; ocular hypertension; ocular motility disorders; optic nerve diseases; orbital diseases; refactive errors; retinal diseases; uveal diseases; and vision disorders.

We also searched the following non—Medical Subject Heading terms in “Title, Abstract, and Keyword”: vision, visual, eye diseases, eye disorders, macular degeneration, cataract, glaucoma, strabismus, hyphema, conjunctivitis, eye, optic nerve, choroiditis, chorioretinitis, blepharitis, blindness, ocular, lacrimal, iris, uveitis, pupil, and retina.

Appendix 2: Statistical Issues and Impact on Systematic Review Findings

Thirteen of the 32 systematic reviews with at least 1 metaanalysis that we included in our sample applied at least 1 inappropriate statistical analysis. We identified 2 major types of inappropriate analysis among the 13 reviews. Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison, and 7 used an incorrect formula to calculate the variance of the effect estimate (3 reviews had both types of inappropriate analyses).

Using a Naive Approach to Conduct an Indirect Comparison and Network Meta-analysis

Nine of the 32 systematic reviews with a meta-analysis that we included in our sample pooled data from treatment groups (for example, 1 review [39] pooled data separately from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials and compared them [Appendix Figure 2]). In all 9 cases, the reviews examined at least 3 interventions for a given condition. The analysis that pooled individual treatment groups may have been an attempt to indirectly compare interventions; that is, comparison of 2 interventions through a common comparator (20, 95). For example, indirect evidence on A versus C can be obtained from RCTs of either A or C versus a common comparator B.

Appendix Figure 2
Example of pooling data from treatment groups.

The consensus among methodologists is that pooling treatment groups is an incorrect way of conducting a meta-analysis (20, 98). Pooling treatment groups has been described by Glenny and colleagues (20) as the “naive method” for indirect comparison. This approach has been criticized for negating the randomized nature of the comparison between interventions and thereby increasing the likelihood of selection and confounding biases and falsely precise estimates. The correct way of comparing multiple treatments for the same condition is through a network metaanalysis (20, 95, 98, 99).

Using an Incorrect Formula to Calculate the Variance of the Effect Estimate

Many systematic reviews included in our sample described differences between the intervention and comparison groups in mean percentage of change in IOP from baseline to the end point as a measure of treatment effect and to gauge the success of IOP-lowering treatments. Mean percentage of change in IOP from baseline can be expressed as follows:

equation M1

Where i indicates individual participant and n indicates the total number of participants.

We found that incorrect formulas, in different forms, were used for estimating the SD for the percentage of change in IOP from baseline to the end point (Appendix Figures 3 and and4).4). Incorrect formulas for estimating the SD can invalidate results and conclusions from a review, because the SD directly relates to the SE through sample size. The SE is used to determine the weight of each study in the meta-analysis and to calculate the CIs for the pooled treatment effect.

Appendix Figure 3
Example of an incorrect formula used to estimate the SD of the percentage of change in IOP from baseline.
Appendix Figure 4
Example of transcription and use of an incorrect formula to estimate the SD of the percentage of change in IOP from baseline.

The formulas to calculate the SD of the absolute change in IOP from baseline to the end point were incorrect in Appendix Figures 3 (61) and and44 (19) because the analyses ignored the correlation between the repeated IOP measurements from the same individual at baseline and the end point. In Appendix Figure 3, the authors used the SE instead of the SD, which could be a notation error. The incorrect formulas used to calculate the SD of change in IOP from baseline to the end point were expressed as

equation M2


equation M3

in Appendix Figures 3 and and4,4, respectively. The correct formula could be written as

equation M4

in which p is the correlation between the repeated IOP measurements from the same individual at baseline and the end point.

When individual-patient data are available, that is, a data set with intervention, outcome, and covariate information for each patient, another way to compute SDIOPR is to take the differences (end point minus baseline) for each individual and compute the SD of these values.

The formulas to calculate the SD of the percentage of change in IOP from baseline to the end point were also incorrect in Appendix Figures 3 and and44 because the percentage of change in IOP from baseline to the end point is a ratio of 2 quantities, and estimating the SD for a ratio is mathematically complex (100). The incorrect formulas used to calculate the SD of the percentage of change in IOP from baseline to the end point were expressed as

equation M5


equation M6

in Appendix Figures 3 and and4,4, respectively.

If all trials included in the meta-analysis had reported the mean percentage of change in IOP from baseline to the end point and SD for each treatment group, or for between-group comparisons, then using these data to conduct a meta-analysis would have been straightforward. However, because 1 or more of the included trials did not report mean percentage of change in IOP from baseline and its SD, accurate estimation of these values requires individual-patient data.

Using percentage of change in IOP from baseline as an outcome in a meta-analysis is recommended only when individual-patient data are available and only with advice from an experienced statistician because of the myriad potential problems associated with the assumptions and the realities of available data. The clinical community should reach a consensus on whether percentage of change in IOP from baseline is an appropriate measure of treatment effect for IOP-lowering interventions. Such a consensus would inform design, analyses, and reporting for clinical trials of interventions that affect IOP. This discussion has been initiated through the National Eye Institute/U.S. Food and Drug Administration symposium on glaucoma clinical trial design and end points (101).


Author Contributions: Conception and design: T. Li, S.S. Vedula, K. Dickersin.

Analysis and interpretation of the data: T. Li, S.S. Vedula, K. Dickersin.

Drafting of the article: T. Li, S.S. Vedula.

Critical revision of the article for important intellectual content: T. Li, S.S. Vedula, R. Scherer, K. Dickersin.

Final approval of the article: T. Li, S.S. Vedula, R. Scherer, K. Dickersin. Statistical expertise: T. Li.

Obtaining of funding: K. Dickersin.

Administrative, technical, or logistic support: T. Li, S.S. Vedula.

Collection and assembly of data: T. Li, S.S. Vedula, R. Scherer, K. Dickersin.

Potential Conflicts of Interest: Disclosures can be viewed at−2461.

Current author addresses and author contributions are available at


1. American Recovery and Reinvestment Act of 2009. 2009;226 Pub. L. No. 111–5, 123 Stat.
2. Institute of Medicine. Knowing What Works in Health Care: A Roadmap for the Nation. Washington, DC: National Academies Pr; 2008.
3. Patient Protection and Affordable Care Act of 2010. 2010;119 Pub. L. No. 111–148, 24 Stat.
4. Patient-Centered Outcomes Research Institute. PCORI Funding Announcement: Pilot Projects Grants. Washington, DC: Patient-Centered Outcomes Research Institute; 2011. Accessed at on 11 January 2012.
5. Friedman DS, Wolfs RC, O’Colmain BJ, Klein BE, Taylor HR, West S, et al. Eye Diseases Prevalence Research Group. Prevalence of open-angle glaucoma among adults in the United States. Arch Ophthalmol. 2004;122:532–538. [PMID: 15078671] [PMC free article] [PubMed]
6. Quigley HA, Broman AT. The number of people with glaucoma worldwide in 2010 and 2020. Br J Ophthalmol. 2006;90:262–267. [PMID: 16488940] [PMC free article] [PubMed]
7. Guyatt G, Rennie D. User’s Guides to the Medical Literature: Essentials of Evidence-Based Clinical Practice. Chicago: American Medical Assoc; 2002.
8. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t [Editorial] BMJ. 1996;312:71–72. [PMID: 8555924] [PMC free article] [PubMed]
9. Li T, Ervin AM, Scherer R, Jampel H, Dickersin K. Setting priorities for comparative effectiveness research: a case study using primary open-angle glaucoma. Ophthalmology. 2010;117:1937–1945. [PMID: 20800896] [PMC free article] [PubMed]
10. American Academy of Ophthalmology. Primary Open-Angle Glaucoma Preferred Practice Patterns. San Francisco: American Acad Ophthalmology; 2010. Accessed at on 11 January 2012.
11. Haynes RB, Sackett DL, Guyatt GH, Tugwell P. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2005. Clinical Epidemiology: How to Do Clinical Practice Research.
12. Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.2 [updated March 2011] The Cochrane Collaboration. 2011 Accessed at on 11 August 2011.
13. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330:68. [PMID: 15619601] [PMC free article] [PubMed]
14. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. [PMID: 17302989] [PMC free article] [PubMed]
15. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–1020. [PMID: 19230606] [PubMed]
16. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health-care interventions: explanation and elaboration. Ann Intern Med. 2009;151:W65–W94. [PMID: 19622512] [PubMed]
17. Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–269. W64. [PMID: 19622511] [PubMed]
18. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–926. [PMID: 18436948] [PMC free article] [PubMed]
19. Cheng JW, Xi GL, Wei RL, Cai JP, Li Y. Efficacy and tolerability of latanoprost compared to dorzolamide combined with timolol in the treatment of patients with elevated intraocular pressure: a meta-analysis of randomized, controlled trials. J Ocul Pharmacol Ther. 2009;25:55–64. [PMID: 19232016] [PubMed]
20. Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, et al. International Stroke Trial Collaborative Group. Indirect comparisons of competing interventions. Health Technol Assess. 2005;9:1–134. iii-iv. [PMID: 16014203] [PubMed]
21. Abudu M, Kadir J, Abudukadir M, Chen XY, Junire, Arkin A. Nonpenetrating trabecular surgery versus trabeculectomy for open aangle glaucoma: a systematic review. Chinese J Evid-Bas Med. 2008;8:1094–1099.
22. Mei Li Ke Mu AS, Ju er ti HD. Dorzolamide/timolol combination versus latanoprost in the treatment of open-angle glaucoma: a systematic review. Chinese J Evid-Bas Med. 2009;9:350–354.
23. Aptel F, Cucherat M, Denis P. Efficacy and tolerability of prostaglandin analogs: a meta-analysis of randomized controlled clinical trials. J Glaucoma. 2008;17:667–673. [PMID: 19092464] [PubMed]
24. Burr J, Azuara-Blanco A, Avenell A. Medical versus surgical interventions for open angle glaucoma. Cochrane Database Syst Rev. 2005:CD004399. [PMID: 15846712] [PubMed]
25. Cheng JW, Ma XY, Wei RL. Efficacy of non-penetrating trabecular surgery for open angle glaucoma: a meta-analysis. Chin Med J (Engl) 2004;117:1006–1010. [PMID: 15265373] [PubMed]
26. Cheng JW, Wei RL. Meta-analysis of 13 randomized controlled trials comparing bimatoprost with latanoprost in patients with elevated intraocular pressure. Clin Ther. 2008;30:622–632. [PMID: 18498911] [PubMed]
27. Cheng JW, Li Y, Wei RL. Systematic review of intraocular pressure-lowering effects of adjunctive medications added to latanoprost. Ophthalmic Res. 2009;42:99–105. [PMID: 19546601] [PubMed]
28. Cox JA, Mollan SP, Bankart J, Robinson R. Efficacy of antiglaucoma fixed combination therapy versus unfixed components in reducing intraocular pressure: a systematic review. Br J Ophthalmol. 2008;92:729–734. [PMID: 18460539] [PubMed]
29. Denis P, Lafuma A, Khoshnood B, Mimaud V, Berdeaux G. A metaanalysis of topical prostaglandin analogues intra-ocular pressure lowering in glaucoma therapy. Curr Med Res Opin. 2007;23:601–608. [PMID: 17355741] [PubMed]
30. Einarson TR, Kulin NA, Tingey D, Iskedjian M. Meta-analysis of the effect of latanoprost and brimonidine on intraocular pressure in the treatment of glaucoma. Clin Ther. 2000;22:1502–1515. [PMID: 11192141] [PubMed]
31. Feyi-Waboso A, Ejere HO. Needling for encapsulated trabeculectomy filtering blebs. Cochrane Database Syst Rev. 2004 CD003658. [PMID: 15106211] [PubMed]
32. Friedman DS, Jampel HD, Lubomski LH, Kempen JH, Quigley H, Con-gdon N, et al. Surgical strategies for coexisting glaucoma and cataract: an evidence-based update. Ophthalmology. 2002;109:1902–1913. [PMID: 12359612] [PubMed]
33. Hylton C, Congdon N, Friedman D, Kempen J, Quigley H, Bass E, et al. Cataract after glaucoma filtration surgery. Am J Ophthalmol. 2003;135:231–232. [PMID: 12566033] [PubMed]
34. Jampel HD, Lubomski LH, Friedman DS. AHRQ Publication No. 03-E041. Rockville, MD: Agency for Healthcare Research and Quality; 2003. Treatment of Coexisting Cataract and Glaucoma. Evidence Report/Technology Assessment Number 38.
35. Jampel HD, Friedman DS, Lubomski LH, Kempen JH, Quigley H, Con-gdon N, et al. Effect of technique on intraocular pressure after combined cataract and glaucoma surgery: an evidence-based review. Ophthalmology. 2002;109:2215–24. quiz 2225, 2231. [PMID: 12466161] [PubMed]
36. Fung AT, Reid SE, Jones MP, Healey PR, McCluskey PJ, Craig JC. Meta-analysis of randomised controlled trials comparing latanoprost with bri-monidine in the treatment of open-angle glaucoma, ocular hypertension or normal-tension glaucoma. Br J Ophthalmol. 2007;91:62–68. [PMID: 16956912] [PMC free article] [PubMed]
37. Gray TA, Orton LC, Henson D, Harper R, Waterman H. Interventions for improving adherence to ocular hypotensive therapy. Cochrane Database Syst Rev. 2009 CD006132. [PMID: 19370627] [PubMed]
38. Hodge WG, Lachaine J, Steffensen I, Murray C, Barnes D, Foerster V, et al. The efficacy and harm of prostaglandin analogues for IOP reduction in glaucoma patients compared to dorzolamide and brimonidine: a systematic review. Br J Ophthalmol. 2008;92:7–12. [PMID: 18156371] [PubMed]
39. Holmstrom S, Buchholz P, Walt J, Wickstrøm J, Aagren M. Analytic review of bimatoprost, latanoprost and travoprost in primary open angle glaucoma. Curr Med Res Opin. 2005;21:1875–1883. [PMID: 16307709] [PubMed]
40. Hondur A, Onol M, Hasanreisoglu B. Nonpenetrating glaucoma surgery: meta-analysis of recent results. J Glaucoma. 2008;17:139–146. [PMID: 18344761] [PubMed]
41. Hong CH, Arosemena A, Zurakowski D, Ayyala RS. Glaucoma drainage devices: a systematic literature review and current controversies. Surv Ophthalmol. 2005;50:48–60. [PMID: 15621077] [PubMed]
42. Honrubia F, García-Sánchez J, Polo V, de la Casa JM, Soto J. Conjunctival hyperaemia with the use of latanoprost versus other prostaglandin analogues in patients with ocular hypertension or glaucoma a meta-analysis of randomised clinical trials. Br J Ophthalmol. 2009;93:316–321. [PMID: 19019922] [PMC free article] [PubMed]
43. Katz LJ, Simmons ST, Craven ER. Efficacy and safety of brimonidine and dorzolamide for intraocular pressure lowering in glaucoma and ocular hypertension. Curr Med Res Opin. 2007;23:2971–2983. [PMID: 17949534] [PubMed]
44. Li N, Chen XM, Zhou Y, Wei ML, Yao X. Travoprost compared with other prostaglandin analogues or timolol in patients with open-angle glaucoma or ocular hypertension: meta-analysis of randomized controlled trials. Clin Experiment Ophthalmol. 2006;34:755–764. [PMID: 17073898] [PubMed]
45. Loon SC, Liew G, Fung A, Reid SE, Craig JC. Meta-analysis of randomized controlled trials comparing timolol with brimonidine in the treatment of glaucoma. Clin Experiment Ophthalmol. 2008;36:281–289. [PMID: 18412600] [PubMed]
46. Maier PC, Funk J, Schwarzer G, Antes G, Falck-Ytter YT. Treatment of ocular hypertension and open angle glaucoma: meta-analysis of randomised controlled trials. BMJ. 2005;331:134. [PMID: 15994659] [PMC free article] [PubMed]
47. Minckler DS, Vedula SS, Li TJ, Mathew MC, Ayyala RS, Francis BA. Aqueous shunts for glaucoma. Cochrane Database Syst Rev. 2006 CD004918. [PMID: 16625616] [PubMed]
48. Netland PA; Ophthalmic Technology Assessment Committee Glaucoma Panel, American Academy of Ophthalmology. Nonpenetrating glaucoma surgery. Ophthalmology. 2001;108:416–21. [PMID: 11158823] [PubMed]
49. Pastor SA, Singh K, Lee DA, Juzych MS, Lin SC, Netland PA, et al. Cyclophotocoagulation: a report by the American Academy of Ophthalmology. Ophthalmology. 2001;108:2130–2138. [PMID: 11713091] [PubMed]
50. Rolim de Moura C, Paranhos A, Jr, Wormald R. Laser trabeculoplasty for open angle glaucoma. Cochrane Database Syst Rev. 2007 CD003919. [PMID: 17943806] [PubMed]
51. Simmons ST, Dirks MS, Noecker RJ. Bimatoprost versus latanoprost in lowering intraocular pressure in glaucoma and ocular hypertension: results from parallel-group comparison trials. Adv Ther. 2004;21:247–262. [PMID: 15605619] [PubMed]
52. Stewart WC, Kolker AE, Sharpe ED, Day DG, Konstas AG, Hollo G, et al. Long-term progression at individual mean intraocular pressure levels in primary open-angle and exfoliative glaucoma. Eur J Ophthalmol. 2008;18:765–770. [PMID: 18850556] [PubMed]
53. Sun W, Cai JP, Li Y, Cheng JW. Meta-analysis of nonpenetrating trabecular surgery combined with implant in treatment of open angle glaucoma. Int J Ophthalmol. 2008;8:749–752.
54. van der Valk R, Webers CA, Schouten JS, Zeegers MP, Hendrikse F, Prins MH. Intraocular pressure-lowering effects of all commonly used glaucoma drugs: a meta-analysis of randomized clinical trials. Ophthalmology. 2005;112:1177–1185. [PMID: 15921747] [PubMed]
55. Vass C, Hirn C, Sycha T, Findl O, Bauer P, Schmetterer L. Medical interventions for primary open angle glaucoma and ocular hypertension. Cochrane Database Syst Rev. 2007 CD003167. [PMID: 17943780] [PubMed]
56. Vinuesa-Silva JM, Vinuesa-Silva I, Pinazo-Durán MD, Soto-Alvarez J, Delgado-Ortega L, Díaz-Cerezo S. [Development of conjunctival hyperemia with the use of a fixed combination of latanoprost/timolol: systematic review and meta-analysis of clinical trials] Arch Soc Esp Oftalmol. 2009;84:199–207. [PMID: 19384760] [PubMed]
57. Webers CA, van der Valk R, Schouten JS, Zeegers MP, Prins MH, Hendrikse F. Intraocular pressure-lowering effect of adding dorzolamide or latanoprost to timolol: a meta-analysis of randomized clinical trials. Ophthalmology. 2007;114:40–46. [PMID: 17070584] [PubMed]
58. Webers CA, Beckers HJ, Nuijts RM, Schouten JS. Pharmacological management of primary open-angle glaucoma: second-line options and beyond. Drugs Aging. 2008;25:729–759. [PMID: 18729546] [PubMed]
59. Wilkins M, Indar A, Wormald R. Intra-operative mitomycin C for glaucoma surgery. Cochrane Database Syst Rev. 2005 CD002897. [PMID: 16235305] [PubMed]
60. Wormald R, Wilkins MR, Bunce C. Post-operative 5-fluorouracil for glaucoma surgery. Cochrane Database Syst Rev. 2001 CD001132. [PMID: 11686977] [PubMed]
61. Zhang WY, Po AL, Dua HS, Azuara-Blanco A. Meta-analysis of randomised controlled trials comparing latanoprost with timolol in the treatment of patients with open angle glaucoma or ocular hypertension. Br J Ophthalmol. 2001;85:983–990. [PMID: 11466259] [PMC free article] [PubMed]
62. Whitlock EP, Lopez SA, Chang S, Helfand M, Eder M, Floyd N. AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program. J Clin Epidemiol. 2010;63:491–501. [PMID: 19540721] [PubMed]
63. Chalkidou K, Whicher D, Kary W, Tunis S. Comparative effectiveness research priorities: identifying critical gaps in evidence for clinical and health policy decision making. Int J Technol Assess Health Care. 2009;25:241–248. [PMID: 19619341] [PubMed]
64. Iglehart JK. Prioritizing comparative-effectiveness research—IOM recommendations. N Engl J Med. 2009;361:325–328. [PMID: 19567828] [PubMed]
65. O’Leary TJ, Slutsky JR, Bernard MA. Comparative effectiveness research priorities at federal agencies: the view from the Department of Veterans Affairs, National Institute on Aging, and Agency for Healthcare Research and Quality. J Am Geriatr Soc. 2010;58:1187–1192. [PMID: 20936736] [PMC free article] [PubMed]
66. Smith N, Mitton C, Peacock S, Cornelissen E, MacLeod S. Identifying research priorities for health care priority setting: a collaborative effort between managers and researchers. BMC Health Serv Res. 2009;9:165. [PMID: 19754969] [PMC free article] [PubMed]
67. Maciosek MV, Coffield AB, McGinnis JM, Harris JR, Caldwell MB, Teutsch SM, et al. Methods for priority setting among clinical preventive services. Am J Prev Med. 2001;21:10–19. [PMID: 11418252] [PubMed]
68. Douw K, Vondeling H, Oortwijn W. Priority setting for horizon scanning of new health technologies in Denmark: views of health care stakeholders and health economists. Health Policy. 2006;76:334–345. [PMID: 16081185] [PubMed]
69. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23:310–315. [PMID: 17579932] [PubMed]
70. Husereau D, Boucher M, Noorani H. Priority setting for health technology assessment at CADTH. Int J Technol Assess Health Care. 2010;26:341–347. [PMID: 20584365] [PubMed]
71. Johnston SC, Hauser SL. IOM task impossible: set the agenda for neurological comparative effectiveness research [Editorial] Ann Neurol. 2009;66:A4–A6. [SPMID: 19938063] [PubMed]
72. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78. [PMID: 17388659] [PMC free article] [PubMed]
73. Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002;21:1635–1640. [PMID: 12111924] [PubMed]
74. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005;58:741–742. [PMID: 15939227] [PubMed]
75. Buscemi N, Harding L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59:697–703. [PMID: 16765272] [PubMed]
76. Gøtzsche PC, Hrobjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007;298:430–437. [PMID: 17652297] [PubMed]
77. Hróbjartsson A, Gøtzsche PC. Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies. J Clin Epidemiol. 2006;59:336–338. [PMID: 16549252] [PubMed]
78. Ford AC, Guyatt GH, Talley NJ, Moayyedi P. Errors in the conduct of systematic reviews of pharmacological interventions for irritable bowel syndrome. Am J Gastroenterol. 2010;105:280–288. [PMID: 19920807] [PubMed]
79. Bhandari M, Morrow F, Kulkarni AV, Tornetta P., 3rd Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am. 2001;83A:15–24. [PMID: 11205853] [PubMed]
80. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ. A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care. 2005;9:R575–R582. [PMID: 16277721] [PMC free article] [PubMed]
81. Glenny AM, Esposito M, Coulthard P, Worthington HV. The assessment of systematic reviews in dentistry. Eur J Oral Sci. 2003;111:85–92. [PMID: 12648258] [PubMed]
82. Lundh A, Knijnenburg SL, Jørgensen AW, van Dalen EC, Kremer LC. Quality of systematic reviews in pediatric oncology—a systematic review. Cancer Treat Rev. 2009;35:645–652. [PMID: 19836897] [PubMed]
83. Shea B, Moher D, Graham I, Pham B, Tugwell P. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof. 2002;25:116–129. [PMID: 11868441] [PubMed]
84. Institute of Medicine. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: National Academies Pr; 2011.
85. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081. [PMID: 18769481] [PMC free article] [PubMed]
86. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007:MR000001. [PMID: 17443625] [PubMed]
87. Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007:MR000011. [PMID: 17443632. [PubMed]
88. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in metaanalyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007 MR000010. [PMID: 17443631] [PubMed]
89. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009 MR000006. [PMID: 19160345] [PubMed]
90. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170. [PMID: 12775614] [PMC free article] [PubMed]
91. Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003;7:1–90. [PMID: 14670218] [PubMed]
92. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007 MR000005. [PMID: 17443628] [PubMed]
93. Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med. 2009;361:1963–1971. [PMID: 19907043] [PubMed]
94. Goodman S, Dickersin K. Metabias: a challenge for comparative effectiveness research [Editorial] Ann Intern Med. 2011;155:61–62. [PMID: 21727295] [PubMed]
95. Li T, Puhan MA, Vedula SS, Singh S, Dickersin K. Ad Hoc Network Meta-analysis Methods Meeting Working Group. Network meta-analysis-highly attractive but more methodological research is needed. BMC Med. 2011;9:79. [PMID: 21707969] [PMC free article] [PubMed]
96. Straus S, Moher D. Registering systematic reviews. CMAJ. 2010;182:13–14. PMID: 19620270] [PMC free article] [PubMed]
97. Li T. Register systematic reviews [Letter] CMAJ. 2010;182:805. [PMID: 20479061] [PMC free article] [PubMed]
98. Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009;338:bll47. [PMID: 19346285] [PMC free article] [PubMed]
99. Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med. 2002;21:2313–2324. [PMID: 12210616] [PubMed]
100. Raj D. A note on the variance of the ratio estimate. J Am Stat Assoc. 1964;59:895–898.
101. Weinreb RN, Kaufman PL. The glaucoma research community and FDA look to the future: a report from the NEI/FDA CDER Glaucoma Clinical Trial Design and Endpoints Symposium. Invest Ophthalmol Vis Sci. 2009;50:1497–1505. [PMID: 19321793] [PubMed]