|Home | About | Journals | Submit | Contact Us | Français|
To evaluate the accuracy of detecting clinically significant retinopathy of prematurity (ROP) using wide-angle digital retinal photography.
Literature searches of PubMed and the Cochrane Library databases were conducted last on December 7, 2010, and yielded 414 unique citations. The authors assessed these 414 citations and marked 82 that potentially met the inclusion criteria. These 82 studies were reviewed in full text; 28 studies met inclusion criteria. The authors extracted from these studies information about study design, interventions, outcomes, and study quality. After data abstraction, 18 were excluded for study deficiencies or because they were superseded by a more recent publication. The methodologist reviewed the remaining 10 studies and assigned ratings of evidence quality; 7 studies were rated level I evidence and 3 studies were rated level III evidence.
There is level I evidence from ≥5 studies demonstrating that digital retinal photography has high accuracy for detection of clinically significant ROP. Level III studies have reported high accuracy, without any detectable complications, from real-world operational programs intended to detect clinically significant ROP through remote site interpretation of wide-angle retinal photographs.
Wide-angle digital retinal photography has the potential to complement standard ROP care. It may provide advantages through objective documentation of clinical examination findings, improved recognition of disease progression by comparing previous photographs, and the creation of image libraries for education and research.
Proprietary or commercial disclosure may be found after the references.
The American Academy of Ophthalmology prepares Ophthalmic Technology Assessments to evaluate new and existing procedures, drugs, and diagnostic and screening tests. The goal of an Ophthalmic Technology Assessment is to evaluate the peer-reviewed published scientific literature to help refine the important questions to be answered by future investigations and define what is well established. After appropriate review by all contributors, including legal counsel, assessments are submitted to the Academy’s Board of Trustees for consideration as official Academy statements. This assessment evaluates the accuracy of detecting clinically significant retinopathy of prematurity (ROP) using wide-angle digital retinal photography.
Retinopathy of prematurity is a retinal ischemic disorder that affects low-birth-weight infants. Development of an international classification system has permitted standardization of diagnosis using parameters such as zone, stage, and presence of plus disease.1,2 Landmark multicenter randomized controlled trials, such as the Cryotherapy for ROP (CRYO-ROP) and Early Treatment for ROP studies, have established guide-lines for identifying treatment-requiring disease.1,3 Newer treatment methods, such as intravitreal bevacizumab, have shown promise as pharmacologic approaches to severe disease.4
Current management recommendations from these studies are that type 2 ROP (zone I, stage 1 or 2, without plus disease; or zone II, stage 3, without plus disease) should be observed very carefully. Type 1 ROP (zone I, any stage, with plus disease; zone I, stage 3, with or without plus disease; or zone II, stage 2 or 3, with plus disease) should be treated with laser photocoagulation or cryotherapy to decrease the likelihood of visual loss and blindness.1–3 This means that the presence of ROP in zone I or plus disease is the most important finding to guide management decisions. A joint statement by the American Academy of Pediatrics, American Academy of Ophthalmology, and American Association for Pediatric Ophthalmology and Strabismus describes criteria for identifying at-risk infants who require ROP examination.5
Retinopathy of prematurity continues to be a leading cause of childhood blindness throughout the world. Approximately 2100 infants in the United States are affected annually by long-term sequelae of ROP, such as retinal detachments, macular folds, and amblyopia, and 400 to 900 develop blindness each year.6,7 In most developed countries, ROP accounts for 6% to 18% of pediatric blindness. In middle-income countries in Latin America and Asia, this rate has been found to be 15% to 35%.8–12 A shortage of ophthalmologists with the capability and willingness to screen babies in neonatal intensive care units (NICUs) for ROP continues to be a major reason that blindness occurs at such a high rate in many middle-income countries.10
Retinopathy of prematurity examinations with binocular indirect ophthalmoscopy (BIO) have been effective at detecting disease. However, there are important limitations to this screening method: (1) BIO examinations are logistically difficult, and they require time and effort to travel to NICUs and to coordinate with neonatology staff; (2) the number of infants at risk for ROP is increasing because of higher premature birth rates and improved neonatal care, both of which increase the burden for ophthalmologists who continue to perform examinations13; (3) there is substantial medicolegal exposure associated with ROP care, which may be heightened by lack of objective documentation for BIO findings; and (4) documentation of BIO findings using traditional paper-based retinal drawings may be qualitative and imprecise.14 As a result of these factors, an academy survey found that only half of retinal specialists and pediatric ophthalmologists were managing ROP, and that >20% of them planned to stop in the near future (Ocular Surgery News U.S. Edition 2006. Survey: Physicians being driven away from ROP treatment. Available at http://www.osnsupersite.com/view.aspx?rid=18018. Accessed September 7, 2011).
Wide-angle digital retinal photography using a commercially available device has been performed for >10 years. Retinal photography may improve the objective documentation of disease findings, increase the accuracy and standardization of diagnosis, and create digital libraries for education and research.14,15 This may eventually improve the accessibility, cost, efficiency, and safety of ROP care through telemedicine.16–21 In the future, digital retinal photography may provide novel opportunities to obtain second opinions and consultations from remote experts, as well as education about uncommon variable clinical presentations such as aggressive-posterior ROP.
Digital retinal photography of premature infants requires a commercially available retinal camera and personnel in the NICU to capture and send retinal images. A telemedicine system based on digital retinal photography would also require capture of relevant clinical data; expert(s) at a remote site to receive images, interpret them, and provide follow-up recommendations; and a protocol for accepting and managing infants who were found to have clinically significant disease or images that were difficult to interpret. Two published studies have examined the cost-effectiveness of wide-angle photography for ROP management; however, economic analysis was not included in this evaluation.16,18
The focus of this assessment is to address the following question: What is the accuracy of wide-angle digital retinal photography to detect clinically significant ROP? For purposes of this assessment, the phrase “clinically significant ROP” is defined as treatment-requiring (e.g., type 1 or worse) or referral-warranted (e.g., type 2 or worse) disease based on current examination guidelines, as well as the CRYO-ROP and Early Treatment for ROP studies.1,3,5
PubMed and the Cochrane Library were searched on August 1, 2008, August 4, 2008, April 14, 2009, and December 7, 2010. The search strategy used the following MeSH and text terms. An asterisk represents a truncated term: (“Infant”[Mesh] OR pediatric* OR baby OR babies OR neonat* OR prematur* OR “Gestational Age”[Mesh] OR newborn*) AND (“Telemedicine”[ Mesh] OR telemedic* OR “retinal image*” OR “Telepathology”[ Mesh] OR telephotoscreen* OR “Image Processing, Computer-Assisted”[Mesh] OR “fundus image*” OR “RetCam” OR “Photography”[Mesh] OR “Remote Consultation”[ Mesh] OR “fundus camera*” OR “wide-angle camera*” OR “wide-angle contact photograph*” OR “Image Interpretation, Computer-Assisted”[Mesh] OR “Diagnosis, Computer- Assisted”[Mesh] OR “teleophthalmology” OR “retinal photograph*” OR “computer-based image analysis” OR “digital image*” OR “digital photo*” OR “wide-angle image*” OR “wide-angle retinal photo*” OR “clinical photo*” OR photo* OR camera*) AND (“Retinopathy of Prematurity”[Mesh] OR “retinopathy of prematurity” OR “ROP” OR “RoP” OR “Retinal Diseases/diagnosis”[Mesh]). The searches retrieved 414 references in all languages. Fifty of these were written in languages other than English, and these were not reviewed further. Ten retrieved citations were meeting abstracts and were not considered in the assessment.
The authors independently assessed the abstracts retrieved from the electronic searches and marked 82 that potentially met the following inclusion criteria: original research that evaluates clinical ROP diagnosis with digital retinal photography using a wide-angle camera. These 82 studies were reviewed in full text, and 28 met the inclusion criteria. The authors extracted from these 28 studies information about study design, interventions, outcomes, and study quality. After the data were abstracted, 8 studies were excluded for the following reasons: the reference standard was not indirect ophthalmoscopy (3 studies), the study was a case report (2 studies), the study reported outcomes other than ROP diagnosis (2 studies), and the study reported only study design/baseline characteristics (1 study). An additional 2 studies reported use of digital retinal photography for diagnosis of any ROP, but not for clinically significant ROP, and they were also excluded. Of the remaining 18 studies, 8 were superseded by a more recent publication that also was among the 18 studies, leaving 10 studies that were included in this assessment.
The methodologist (M.M.) reviewed the 10 studies and the data abstraction forms and assigned ratings based on the Oxford Center for Evidence-based Medicine Levels of Evidence. 22 Seven studies were rated as level I, and 3 were rated level III. No papers were given a level II rating.
The level I–rated studies all had an independent masked comparison of a cohort of consecutive subjects who were representative of the population requiring screening, and all subjects underwent both wide-angle digital retinal photography and the reference standard ophthalmoscopic examination. The level III studies were rated as such because of a lack of independence between the reference standard and digital retinal photography. In one of these studies, only infants already diagnosed with ROP by indirect ophthalmoscopy were photographed.23 In another, the gold standard indirect ophthalmoscopy examination was not performed on all infants.24 In the third study, indirect ophthalmoscopy was performed immediately if the photographs indicated referral-warranted ROP; otherwise, ophthalmoscopy was not performed until discharge.25
All studies described in this assessment evaluated detection of any ROP using wide-angle digital retinal photography (RetCam 120, RetCam II, or RetCam 3; Clarity Medical Systems, Inc., Pleasanton, CA). In general, all studies compared the accuracy of image-based diagnosis by remote readers with a reference standard of dilated ophthalmoscopic examination by an expert.
The study designs differed in 5 aspects: (1) The number of wide-angle retinal photographs taken, which ranged from 1 to 15 per eye examination; (2) the background of personnel, who included ophthalmologists, ophthalmic photographers, and trained NICU nurses, who captured retinal photographs; (3) the image readers, who included retinal specialists, pediatric ophthalmologists, and general ophthalmologists; (4) the diagnostic outcome measures, which included detection of moderate ROP (e.g., presence of type 2 or worse disease) and detection of severe ROP (e.g., presence of treatment-requiring disease); and (5) the metrics of accuracy, which include sensitivity (likelihood that a diseased patient, based on reference standard examination, is identified by digital photography), specificity (likelihood that a nondiseased patient, based on reference standard examination, is ruled out by digital photography), positive predictive value (likelihood that a patient identified by digital photography has disease based on reference standard examination), negative predictive value (likelihood that a patient ruled out for disease by digital photography does not have disease based on reference standard examination), absolute agreement (percentage of cases in which different graders agree on diagnosis), and kappa statistic (chance-corrected agreement among graders in which 1 represents perfect agreement and 0 represents agreement by pure chance).
Published studies have used several measures of accuracy. For purposes of cross-study comparison, the sensitivity, specificity, positive predictive value, negative predictive value, and corresponding 95% confidence intervals were abstracted directly from each paper or calculated by the methodologist based on data provided in the paper. When possible, 95% confidence intervals were calculated using the binomial exact method; otherwise, the normal approximation was used.
Table 1 summarizes the level I studies that evaluated detection of moderate and severe ROP using wide-angle digital retinal photography. Ells et al26 (371 examinations from 44 infants) examined detection of “referral-warranted ROP” (defined as any ROP in zone I, presence of plus disease, or presence of stage 3 ROP at any time during the infant’s hospital course) during longitudinal inpatient examinations. Digital photographs were taken after standard ophthalmoscopic examination by the same examiner. Hence, the technical execution of photography could conceivably have been influenced by knowledge of the severity of ROP. A masked independent pediatric ophthalmologist grader interpreted photographs, with a sensitivity of 100% and specificity of 96% compared with indirect ophthalmoscopy.
Chiang et al27 (163 examinations from 64 infants) examined a study cohort in which wide-angle retinal photographs were captured by an ophthalmic photographer. The accuracy of masked image interpretation was compared with a reference standard of dilated ophthalmoscopic examination by a pediatric ophthalmologist. Masked interpretation of wide-angle photographs by 3 image readers (1 general ophthalmologist and 2 retinal specialists) resulted in an average sensitivity of 77% and specificity of 96% for detection of type 2 or worse ROP. For detection of treatment-requiring ROP (defined as type 1 or worse disease), the image readers had an average sensitivity of 87% and specificity of 96%.
Wu et al28 (43 infants) examined the accuracy of wide-angle photography for detection of prethreshold or worse ROP in a longitudinal case series of infants meeting ROP-screening criteria. In this study, each infant was classified on the basis of serial examinations of both eyes. Images were taken by a pediatric ophthalmologist or ophthalmic photographer, and they were graded by a different masked pediatric ophthalmologist. No cases of prethreshold disease, threshold disease, or plus disease were missed by the reader, and digital photography had a sensitivity of 100% and specificity of 97% compared with ophthalmoscopic diagnosis.
In a different cohort, Chiang et al29 prospectively collected standardized sets of 3 to 5 wide-angle photographs, taken independently by a trained NICU nurse, of each eye of infants. The infants also underwent standard ophthalmoscopic examinations by a pediatric ophthalmologist. Examinations were performed at 31 to 33 weeks’ postmenstrual age (PMA) and subsequently at 35 to 37 weeks’ PMA (248 examinations from 67 infants), and masked photographic readings were performed by 3 pediatric retinal specialists using a secure website. For photographs taken at 31 to 33 weeks’ PMA, average sensitivity for detection of type 2 or worse ROP was 76% and specificity was 96%. At 35 to 37 weeks’ PMA, average sensitivity for detection of type 2 or worse ROP was 100% and specificity was 91%, and average sensitivity for detection of type 1 or worse ROP was 100% and specificity was 89%.29 In a separate study based on data from this cohort, Scott et al30 compared ophthalmoscopic examination findings with digital photographic interpretations in these 67 infants by the same graders. There was absolute agreement of 86% (178/206 eyes) and kappa values of 0.66 to 0.85 between ophthalmoscopic examinations and digital photographic interpretations. Among the 14% (28/206 eyes) discrepancies, some cases provided photographic documentation that ophthalmoscopy may have missed signs of mild ROP. In other cases, there were discrepancies between the presence of zone I ROP and the presence of plus disease, in which photography may have provided the theoretical advantages of allowing examiners to review their diagnoses, make more exact measurements of anatomic landmarks defining zone I of the retina, and directly compare images with the standard photograph for plus disease.1,2,30
The prospective, multicenter Photographic Screening for ROP study (300 examinations from 51 infants) evaluated detection of “clinically significant ROP” at any time during multiple longitudinal inpatient examinations.31 This outcome measure was defined as follows: (a) zone I, any ROP, without vascular dilation or tortuosity; (b) zone II, stage 2, with up to 1 quadrant of vascular dilation and tortuosity; (c) zone II, stage 3, with up to 1 quadrant of vascular dilation and tortuosity; (d) any vascular dilation and tortuosity noted in eyes for which ridge characteristics were not interpretable (not imaged or poor image quality); or (e) any ROP noted in eyes for which disc features (plus disease) were not interpretable (not imaged or poor image quality). Photographs were taken by an ophthalmologist and graded by consensus of 2 masked ROP specialists. This study found that “clinically significant ROP” was detected with sensitivity of 92% and specificity of 37%.31
Dhaliwal et al32 (245 examinations from 81 infants) conducted a masked, prospective, longitudinal case series. Two experienced pediatric ophthalmologists were randomized to perform examinations using either wide-angle retinal photography or standard ophthalmoscopy. Five to 15 images were captured from each eye of infants by the examining ophthalmologist, and almost all examinations were performed between 32 and 36 weeks’ PMA. Sensitivity of retinal photography for detection of stage 3 or worse ROP was 57%, and specificity was 68% compared with ophthalmoscopic examination. Sensitivity for diagnosis of plus disease was 80%, and specificity was 98% compared with ophthalmoscopy. Absolute agreement between ophthalmoscopy and photography was 96% for detection of stage 3 ROP, and 97% for detection of plus disease.
Dai et al33 (422 examinations from 108 infants) evaluated the effectiveness of wide-angle photography in a pilot telemedicine study, in which infants received serial digital photographs and concurrent standard ophthalmoscopic examinations by a pediatric ophthalmologist. Photographs were reviewed independently by a masked grader. Using ophthalmoscopic findings as the reference standard, the sensitivity of digital photographic reading for detecting treatment-requiring ROP (i.e., type 1 or worse) was 100% and the specificity was 98%. The positive predictive value of digital photographic reading for detecting treatment-requiring ROP was 85% and the negative predictive value was 100%.
Table 2 summarizes the 3 level III studies that evaluated detection of moderate and severe ROP using wide-angle digital retinal photography.
Schwartz et al23 (19 examinations from 10 infants) collected wide-angle photographs from a group of preselected cases with relatively severe ROP. Interpretation by 2 masked ophthalmologists revealed that 18 of 19 eyes (95%) showed agreement between photographic reading and ophthalmoscopy for plus disease diagnosis, and 17 of 19 eyes (89%) showed agreement between photographic reading and ophthalmoscopy for presence of prethreshold or worse ROP.
Lorenz et al24 (6460 examinations of 1222 infants) conducted a 6-year prospective study, in which a photographic reading center was incorporated into real-world ROP management at 5 NICUs in Germany. Local general ophthalmologists (4 NICUs) and pediatric ophthalmologists (1 NICU) were asked to continue standard ophthalmoscopic examinations while also taking wide-angle retinal photographs that were interpreted at a photographic reading center. Management decisions were made by the photographic reading center. All infants found to have “suspected treatment-requiring ROP” (defined as type 1 ROP or worse, or as anything else felt to represent possible treatment-requiring ROP that could not be reliably classified from retinal images) by the photographic reading center were referred for complete ophthalmoscopic examination. Based on findings from ophthalmoscopic evaluations and what was known from infants who were not referred, the sensitivity of telemedicine for detecting suspected treatment-requiring ROP was 100% and all treatment-requiring ROP was considered to have been detected appropriately.
Wide-angle retinal photography has been used for real-world telemedicine management at 4 NICUs in Northern California since 2005, in a program in which NICU nurses were trained to take serial images from all infants who meet criteria for ROP examination.25 This program is intended to represent a real-world telemedicine system, in which a retinal specialist at Stanford University manages infants remotely, based solely on photographic reading. Patients considered to have referral-warranted (i.e., type 2 or worse) or treatment-requiring (i.e., type 1 or worse) ROP based on photographic grading were referred for complete ophthalmoscopic evaluation by the same retinal specialist. Within 1 week of discharge from the NICU or before inpatient hospital discharge, all patients in this program underwent a mandatory ophthalmoscopic examination by the same retinal specialist at Stanford University. Outpatient ophthalmoscopic examinations were continued until screenings could be terminated according to published guidelines,5 and ophthalmoscopic examination findings were used as the reference standard for evaluation of photographic diagnosis in these studies. Among 230 infants (1059 examinations) who were managed longitudinally in that program, 10 were identified as having referral-warranted disease using remote photographic diagnosis, of whom 9 were found to have treatment-requiring ROP by ophthalmoscopic examination.25 The sensitivity of digital photography for identifying referral-warranted and treatment-requiring ROP was reported to be 100%, the positive predictive value of telemedicine for identifying treatment-requiring ROP was 90%, and the negative predictive value was 100%. No known cases of treatment-requiring ROP were missed, and there were no adverse outcomes such as retinal detachment, retrolental mass, or macular fold.25
In conclusion, there is level I evidence from ≥5 studies demonstrating that digital retinal photography has high accuracy for detection of clinically significant ROP.26,28,29,31,33 Exceptions are 1 study that showed sensitivity of 77% for detection of type 2 or worse ROP, 1 study that showed sensitivity of 76% for type 2 or worse ROP at 31 to 33 weeks’ and 1 study that showed sensitivity of 57% for detection of stage 3 disease.27,29,32 Level III studies have reported high accuracy, without any known complications, from real-world operational programs intended to detect clinically significant ROP through remote site interpretation of wide-angle retinal photographs.24,25,34 –36
The accuracy of wide-angle photography for detection of mild levels of ROP, particularly in infants at younger PMAs, is less clear. For example, 1 study showed that the sensitivity for detection of mild ROP among infants from 31 to 33 weeks’ PMA by 3 expert graders was 73% to 94%, whereas the specificity was 89% to 94%.29 The reasons for this may be that peripheral retinal findings are more difficult to visualize and that younger infants have smaller eyes with more media opacity, which creates difficulty for photography.
Wide-angle photography may provide advantages through objective documentation of clinical examination findings, improved recognition of disease progression by comparing with previous photographs, and better opportunities to communicate examination findings with families, pediatricians, and neonatal staff. In addition, wide-angle digital photography might provide benefits from more precise review of clinical findings and retinal morphology, with documentation of findings that could be missed during bedside ophthalmoscopic examination.30 Studies have suggested that there may be variability in ROP diagnosis, even among experts. For example, in the CRYO-ROP study, 12% of eyes diagnosed with threshold disease by 1 certified investigator performing standard ophthalmoscopy were diagnosed with nonthreshold disease when a second certified investigator was asked to perform confirmatory ophthalmoscopic examination.37 Other studies have suggested that experts diagnose plus disease and zone I disease inconsistently.38–41 However, it is possible that clinically significant ROP could be missed by wide-angle contact photography.42 In other disorders, such as diabetic retinopathy, studies have demonstrated that the accuracy of digital photography with remote reading center evaluation by dedicated, trained graders is sufficiently high to be considered the gold standard for detecting and classifying retinal disease characteristics.43,44 Future studies should characterize the precise benefits and roles of digital photography for supplementing standard clinical examinations.
An ongoing National Eye Institute–supported multicenter study is examining the validity, reliability, feasibility, safety, and relative cost-effectiveness of a telemedicine evaluation system to detect referral-warranted ROP in at-risk babies. The study includes a photographic reading center (Telemedicine approaches to evaluating acute-phase ROP. Available at: http://clinicaltrials.gov/ct2/show/NCT01264276. Accessed May 6, 2011). The ongoing clinical trial is based on the studies reviewed in this assessment, expanding them to a larger scale using a reading center. Expected results will enhance our understanding of the value and place of digital wide-angle photography in the evaluation of at-risk infants.
More broadly, effective screening for ROP requires high sensitivity for detecting clinically significant disease to avoid missed cases of potentially blinding disease, as well as high specificity to avoid excessive overreferral of cases to ophthalmologists who perform surgical management. The levels of sensitivity and specificity that are required to justify implementation of real-world ROP screening programs based on digital wide-angle photography must be established. If remote diagnosis using digital photography is used to substitute for standard ophthalmoscopic examination, guidelines for training ophthalmologists, neonatologists, and photographers must be developed. Standard photographic protocols that are analogous to standard image sets for other diseases such as diabetic retinopathy must be created. Clear rules and responsibilities must be defined for remote clinical management, for situations in which image quality is inadequate for accurate detection of clinically significant disease, and for cases in which digital retinal photography is impractical because of systemic comorbidities, infectious disease contact precautions, ergonomic restrictions from infant monitoring equipment, and other factors. Guidelines on medicolegal liability must be established. Reliable reading center software, which helps to optimize workflow and mitigate risk, needs to become widely available for ophthalmologists and hospitals (e.g., FocusROP; FocusROP, LLC, Wayne, PA. Available at: http://www.focusrop.com/. Accessed March 14, 2011). These challenges seem to have been addressed successfully by several ongoing programs in the United States25,34 –36 and internationally,24,33 but the development of larger-scale telemedicine programs for detection of clinically significant ROP will likely depend on the extent to which these solutions can be generalized, accepted locally, and implemented.
Funded without commercial support by the American Academy of Ophthalmology.
Prepared by the Ophthalmic Technology Assessment Committee Pediatric Ophthalmology/Strabismus Panel and Dr. Recchia from the Retina Panel and approved by the American Academy of Ophthalmology’s Board of Trustees October 21, 2011.
The authors have made the following disclosures: Dr. Chiang is an unpaid member of Scientific Advisory Board for Clarity Medical Systems (Pleasanton, CA) and is supported by EY19474 from the National Institutes of Health (Bethesda, MD), and by the Friends of Doernbecher Foundation (Portland, OR).