PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Ophthalmology. Author manuscript; available in PMC 2010 April 1.
Published in final edited form as:
PMCID: PMC2679961
NIHMSID: NIHMS105838

SCORE Study Report 2: Interobserver Agreement Between Investigator and Reading Center Classification of Retinal Vein Occlusion Type

Ingrid U. Scott, MD, MPH,1 Barbara A. Blodi, MD,2 Michael S. Ip, MD,2 Paul C. VanVeldhuisen, PhD,3 Neal L. Oden, PhD,3 Clement K. Chan, MD,4 and Victor Gonzalez MD5, the SCORE Study Investigator Group

Abstract

Objective

To evaluate interobserver agreement between investigator and Reading Center classification of retinal vein occlusion.

Design

The Standard Care versus Corticosteroid for Retinal Vein Occlusion (SCORE) Study includes two multicenter phase 3 randomized clinical trials evaluating efficacy and safety of intravitreal triamcinolone; one involving participants with central retinal vein occlusion (CRVO) and one involving participants with branch retinal vein occlusion (BRVO). Eyes with hemiretinal vein occlusion (HRVO) were classified as BRVO.

Participants

682 SCORE Study participants, including 271 with CRVO and 411 with BRVO.

Methods

Participants were enrolled into each trial based upon investigator classification of retinal vein occlusion. Definitions of CRVO, BRVO and HRVO based on clinical findings were specified in the study protocol. At baseline, seven-field color stereoscopic fundus photographs on the study eye of each participant were forwarded to the University of Wisconsin Fundus Photograph Reading Center (Reading Center) for evaluation. 28 eyes were excluded from the analysis.

Main Outcome Measures

Percent agreement and kappa statistics comparing investigator and Reading Center classification of retinal vein occlusion type.

Results

With HRVO categorized as BRVO (consistent with the SCORE Study design), there was 98.0% agreement between investigator and Reading Center classification (641/654 eyes; kappa = 0.96; 95% confidence interval: 0.94, 0.98). Disagreements were fairly evenly divided between classification of an eye as CRVO by the investigator and as BRVO by the Reading Center (7/13 disagreements) and vice versa (6 disagreements). When all three disease categories (CRVO, BRVO and HRVO) were compared, there was 92.2% agreement (603/654; unweighted kappa = 0.86; 95% confidence interval: 0.83, 0.90). Most disagreements (38/51, or 74.5%) were between BRVO and HRVO, with 9 between CRVO and BRVO, and 4 between HRVO and CRVO. In univariate analyses, baseline factors significantly associated with disagreement include larger area of capillary loss and black race; black race was the only significant factor in a multivariate analysis.

Conclusions

Investigator and Reading Center agreement was excellent in the SCORE Study; in only 13 (2.0%) eyes would disease classification have been altered had Reading Center eligibility been required. Interobserver agreement was least for classification between BRVO and HRVO.

INTRODUCTION

For decades, treatment of macular edema associated with retinal vein occlusion has been guided by the clinical trial results of the Central Vein Occlusion Study (CVOS)1 and Branch Vein Occlusion Study (BVOS).2 Since the results of these studies demonstrated a visual acuity benefit of grid photocoagulation treatment of macular edema in patients with branch retinal vein occlusion (BRVO)2 but not central retinal vein occlusion (CRVO),1 treatment of patients with macular edema associated with retinal vein occlusion has depended heavily on which type of retinal vein occlusion is diagnosed.

The Standard Care versus Corticosteroid for Retinal Vein Occlusion (SCORE) Study consists of two multicenter phase 3 randomized clinical trials; one trial evaluates the efficacy and safety of intravitreal triamcinolone acetonide in participants with central retinal vein occlusion (CRVO) and one trial evaluates the efficacy and safety of intravitreal triamcinolone acetonide in participants with branch retinal vein occlusion (BRVO). Based on investigator classification of disease at baseline, participants were either randomized into the CRVO trial or the BRVO trial. Within each trial, participants were randomized in a 1:1:1 allocation ratio to standard care therapy, intravitreal injection(s) of 4 mg of triamcinolone or intravitreal injection(s) of 1 mg of triamcinolone. Classification, by the investigator, of which type of retinal vein occlusion is present at baseline is important because standard care therapy differed in the 2 trials. In the CRVO trial standard care therapy was observation and in the BRVO trial standard care therapy was either immediate or deferred grid laser photocoagulation, depending on whether or not a dense macular hemorrhage was present. Investigators were not required to wait for confirmation from the University of Wisconsin Fundus Photograph Reading Center (Reading Center) regarding the classification of retinal vein occlusion (CRVO versus BRVO) before enrollment and randomization of study participants. The purpose of the current study report is to evaluate interobserver agreement between investigators and the Reading Center in the classification of retinal vein occlusion in the Standard Care versus COrticosteroid in REtinal Vein Occlusion (SCORE) Study.

METHODS

The design and methods of the SCORE Study, which consists of two phase 3 multicenter randomized clinical trials conducted at 84 clinical sites in the United States, are described in detail elsewhere.3, 4 The study enrolled 271 participants into the CRVO trial and 411 participants into the BRVO trial by 170 investigators, all board-certified ophthalmologists with at least 1 year of retina fellowship training. Two eyes diagnosed by the investigator as having CRVO, one eye diagnosed by the investigator as having BRVO, and one eye diagnosed by the investigator as having HRVO were subsequently realized by the enrolling investigator to have been enrolled by the site into the incorrect trial; for the purposes of this report, which aims to investigate agreement between investigators and the Reading Center, these 4 eyes were analyzed according to the investigator’s diagnosis.

Patients were enrolled into each trial based upon investigator diagnosis of retinal vein occlusion type. As directed by the SCORE Study protocol, participants diagnosed with hemiretinal vein occlusion (HRVO) were enrolled into the BRVO trial. Definitions of CRVO, BRVO and HRVO based on clinical examination findings were specified in the SCORE Study protocol (Table 1). All eyes with a visual acuity letter score of 34 or better were considered perfused and were enrolled if these eyes met all other inclusion and exclusion criteria. However, eyes with a visual acuity letter score of between 33 and 19 letters (inclusive) were considered eligible only if the investigator considered the eye to be perfused. With the exception of required mydriasis, no specific protocol for the clinical examinations was provided.

Table 1
Definitions of Retinal Vein Occlusion Type Provided to Clinicians in the Standard Care versus Corticosteroid for Retinal Vein Occlusion (SCORE) Study Protocol

At baseline, standard Early Treatment Diabetic Retinopathy Study (ETDRS) seven-field film color stereoscopic fundus photographs were obtained on the study eye of each study participant and forwarded to the Reading Center for grading by any two of six trained evaluators, who were masked to all clinical data except investigator assessment of retinal vein occlusion type. Each set of baseline photographs was evaluated for disease category classification (type of retinal vein occlusion) by two evaluators. Disagreements between evaluators were adjudicated by a retina specialist at the Reading Center. The evaluators used standardized definitions for CRVO, BRVO, and HRVO (Table 2).

Table 2
Reading Center Definitions of Retinal Vein Occlusion Type Employed in the Standard Care versus Corticosteroid for Retinal Vein Occlusion (SCORE) Study

The current report comprises an analysis of baseline data collected prospectively within the two phase 3 study trials. Photographs were not received at the Reading Center for 24 participants, leaving 658 participants with photographic assessment by the Reading Center. The Reading Center confirmed presence of a retinal vein occlusion in 654 of 658 eyes (99.4%) of randomized participants. These 654 eyes, for which presence of vein occlusion was documented both at the clinical site and the Reading Center, were the basis for analyses of agreement between the investigator at the SCORE Study site and the Reading Center. Of the four eyes excluded from analyses, 3 eyes had photographs that could not be graded—one due to a photographic artifact, the second due to the presence of only one photographic field, and the third due to poor image quality— the remaining eye had baseline photographs given the grade of “absent” for any retinal vein occlusion where the Reading Center noted non-proliferative diabetic retinopathy but no retinal vein occlusion.

To assess interobserver agreement between SCORE Study investigators and Reading Center classification of retinal vein occlusion type, both percent of agreement and the kappa statistic were calculated. Kappa values are interpreted as the proportion of agreement above and beyond what would be expected by chance, with a value of 1.00 indicating perfect agreement and a value of 0 reflecting complete chance agreement. Guidelines for interpreting kappa have been suggested,5 with 0.40 to 0.59 to indicate fair clinical significance, between 0.60 and 0.74 to indicate good clinical significance, and between 0.75 and 1.00 to indicate excellent clinical significance.

Univariate and multivariate logistic regression models were performed to assess whether any of the following baseline demographic, medical, and ocular factors were associated with investigator-Reading Center disagreement: age, race, gender, hypertension, coronary heart disease, diabetes mellitus, duration of macular edema, presence of dense macular hemorrhage according to the SCORE Study investigator, prior lens extraction, prior grid laser photocoagulation, best-corrected E-ETDRS visual acuity letter score, OCT assessment of center point thickness, total macular volume, presence of cystoid spaces, presence of subretinal fluid, fundus photographic assessments of disc areas of retinal thickening and retinal hemorrhage within the ETDRS macular grid (range: 0 – 16 disc areas), and fluorescein angiographic assessments of leakage within the grid (range 0 to 16 disc areas) and capillary loss within the eye (range 0 to 210 disc areas). SAS version 9.1.3 was used to conduct all statistical analyses.

All baseline photographs for which disease category classification by the investigator disagreed with classification by the Reading Center evaluators were reviewed by a Principal Investigator of the Reading Center (BAB) to identify potential reasons for the disagreements.

RESULTS

Table 3 summarizes results of investigator and Reading Center disease classification of CRVO and BRVO (with HRVO categorized as BRVO, consistent with the design of the study that included both a CRVO and a BRVO trial). The percent agreement was 98.0% (641/654) with a kappa value of 0.96 (95% confidence interval [0.94, 0.98]). The disagreements were fairly evenly split between CRVO classified by the investigator and BRVO classified by the Reading Center (7 of 13 disagreements) versus BRVO classified by the investigator and CRVO classified by the Reading Center (6 of 13 disagreements). Review of the photographs of the 7 eyes classified at baseline by the investigator as having CRVO and by the Reading Center as having BRVO demonstrated that the intraretinal hemorrhage did not involve all four quadrants on the baseline color photographs. Review of the photographs of the 6 eyes classified by the investigator as having BRVO and by the Reading Center as having CRVO showed intraretinal hemorrhages in all four quadrants, but the hemorrhages were dense in only 2 quadrants.

Table 3
Interobserver Agreement for Central Retinal Vein Occlusion (CRVO) and Branch Retinal Vein Occlusion (BRVO)

As the BRVO trial consisted of eyes diagnosed with either HRVO or BRVO, we also examined agreement between SCORE Study investigator and Reading Center disease classification across all three disease areas: CRVO, BRVO, and HRVO (Table 4). Figure 1A displays an example of a retinal vein occlusion which was classified as HRVO by both the SCORE Study investigator and the Reading Center. The percent agreement was 92.2% (603/654) with an unweighted kappa value of 0.86 (95% confidence limit [0.83, 0.90]). The highest proportion of disagreements (38 of 51 disagreements, or 74.5%) was between BRVO and HRVO. Review of these 38 cases demonstrates that in the 22 eyes classified by the investigator as having HRVO and by the Reading Center as having BRVO (not the HRVO subtype), the intraretinal hemorrhage involved more than one quadrant (as specified in the protocol definition of HRVO) but the extent of hemorrhage was not approximately equal in the nasal and temporal aspects of the fundus (as specified in the Reading Center definition of HRVO), as illustrated in Figure 1B. Review of the photographs of the 16 eyes classified by the investigator as having BRVO and by the Reading Center as having HRVO demonstrated that in these eyes, the extent of retinal hemorrhage was approximately equal on the nasal and temporal aspects of the fundus.

Figure 1
Figure 1A. Color fundus photograph of a left eye demonstrates an inferior retinal vein occlusion classified as a hemiretinal vein occlusion by both the Standard Care versus Corticosteroid for Retinal Vein Occlusion (SCORE) Study investigator and the Reading ...
Table 4
Interobserver Agreement for Central Retinal Vein Occlusion (CRVO), Branch Retinal Vein Occlusion (BRVO), and Hemiretinal Vein Occlusion (HRVO)

Based on univariate logistic regression analyses, a larger disc area of capillary loss (OR=1.05 per increase in 1 disc area; p=0.048) and black race (OR=3.27 compared with all other races, p=0.009) were associated with disagreement between investigator and Reading Center disease classification of CRVO versus BRVO versus HRVO. In a multivariate logistic regression analysis considering all the selected baseline factors, only black race (OR=3.71; p=0.007) was statistically significantly associated with disagreement between the investigator and Reading Center disease classification. For black participants, 20% (7/35) of eyes were misclassified compared with 7% (44/619) of eyes of participants of other races.

DISCUSSION

Classification of retinal vein occlusion type is important, as standard of care treatments differ between CRVO and BRVO. Therefore, how patients are treated depends upon which type of retinal vein occlusion is diagnosed. Similarly, in the SCORE Study, the trial into which a patient was enrolled and, therefore, which standard of care treatment was applied, depended on the type of retinal vein occlusion diagnosed. The finding in the SCORE Study that interobserver agreement was excellent between investigators and the Reading Center with regards to retinal vein occlusion classification suggests that appropriate standard of care treatments were applied, and the results that will emanate from the SCORE Study will be specific to CRVO and BRVO.

Review of the literature reveals that agreement in grading between clinical investigators and a centralized reading center has also been investigated in studies of diabetic retinopathy. In a study comparing assessment of diabetic retinopathy severity level performed by retina specialists and fundus photograph evaluations performed at a Reading Center (University of Wisconsin Fundus Photograph Reading Center) of the study eye of each of 118 individuals participating in a clinical trial of center-involved diabetic macular edema, 86% agreement within one step and a kappa value of 0.55 (95% confidence interval [0.41, 0.68) was reported.6 In this study, clinical assessments were made on a 4-step retinopathy severity scale without standardization of methods and compared with the ETDRS retinopathy severity scale used by trained evaluators at the Reading Center. Another study reported an agreement rate of 86% and 86% (weighted kappa, 0.56 and 0.62) in diabetic retinopathy grading in right and left eyes, respectively, with the retina specialist and the Reading Center using the same diabetic retinopathy severity scale.7

In only 13 of 654 (2.0%) eyes in the SCORE Study would participant enrollment by disease classification have been altered had Reading Center eligibility been required prior to enrollment into either the CRVO or the BRVO trial of the SCORE Study. This suggests that Reading Center eligibility determination for clinical trials based on retinal vein occlusion diagnosis and classification may not be needed for most trials.

Review of the photographs of the 7 eyes classified by the investigator as having CRVO and by the Reading Center as having BRVO demonstrated that the intraretinal hemorrhage did not involve all four quadrants at baseline. One hypothesis is that perhaps in these cases all 4 quadrants were involved with intraretinal hemorrhage prior to the photographs but that a substantial amount of hemorrhage had resolved prior to randomization. However, there was no significant difference in the mean number of days between screening and randomization in cases with (3.9 days) and without (4.3 days) agreement (p=0.67). With regard to these 7 eyes, 5 of them had photographs taken on the day of randomization and in the remaining 2 eyes, photographs were taken 7 days prior to randomization. Thus, a more likely explanation for the difference in classification between the investigator and the Reading Center is that intraretinal hemorrhage was present outside the photographic fields such that on clinical examination there was involvement of all 4 quadrants with hemorrhage.

Review of the photographs of the 6 eyes classified by the investigator as having BRVO and by the Reading Center as having CRVO showed that in these 6 patients, there were very few intraretinal hemorrhages in any of the quadrants, which may have made the distinction between CRVO and BRVO difficult. These small hemorrhages may have been missed on clinical examination by the investigator leading to classification as BRVO, but were present on the photographs leading to classification of the eye as CRVO by the Reading Center. In support of this hypothesis, the mean area of retinal hemorrhage in these 6 eyes was 1.78 disc areas (range, 0–5.6 disc areas) compared with a mean area of retinal hemorrhage of 3.36 disc areas (range, 0–15.1 disc areas) in all eyes classified by the Reading Center as having CRVO and 2.99 disc areas (range, 0–11.9 disc areas) in all eyes classified by the Reading Center as having BRVO.

When agreement across all three disease classifications (CRVO, BRVO and HRVO) was assessed, the highest proportion of disagreements (38 of 51 disagreements, or 74.5%) was between BRVO and HRVO and review of these 38 disagreements suggests that these disagreements may have been due to differences in the protocol-specified versus Reading Center definition of HRVO. While the protocol-specified definition states that HRVO is defined by retinal hemorrhage or other biomicrosopic evidence of retinal vein occlusion (e.g. telangiectatic capillary bed) and a dilated venous system (or previously dilated venous system) in more than 1 quadrant but less than all 4 quadrants (Table 1), the Reading Center definition specifies that the retinal hemorrhages involved are nearly equal in two altitudinal quadrants (the nasal and temporal aspects) of the involved hemisphere (Table 2).

Univariate logistic regression analyses demonstrated a significant association of larger area of capillary loss and black race with disagreement when analyses were performed univariately; however, only black race was statistically significant in multivariate analysis. Black race may be significantly associated with disagreement because of darker fundus pigmentation, rendering visualization of small hemorrhages or subtle vascular abnormalities more difficult. Photographic quality, with images categorized by the Reading Center into one of two groups: fair quality or better, or worse than fair quality, was also examined as a possible factor in explaining the finding that race was associated with disagreement. Our analysis revealed that photographic quality was neither associated with race (black compared with non-black; p=0.44) nor agreement status (p=0.23).

In summary, comparison of investigator and Reading Center classification of retinal vein occlusion type in the SCORE Study demonstrated high interobserver agreement as measured with kappa statistics. The differentiation between CRVO and BRVO showed the least disagreement. The greatest disagreement was differentiation between HRVO and BRVO, but much of the disagreement may be attributable to the small differences in the definition of HRVO and BRVO used by the investigators and the Reading Center. These findings suggest that clinical trials that require separation of HRVO from BRVO may benefit from centralized Reading Center eligibility determination. In contrast, clinical trials evaluating retinal vein occlusion that require only differentiation of CRVO from BRVO likely do not require centralized Reading Center eligibility determination.

Acknowledgments

Supported by the National Eye Institute (National Institutes of Health, Department of Health and Human Services) grants 5U10EY014351, 5U10EY014352, and 5U10EY014404. Support also provided in part by Allergan, Inc. through donation of investigational drug and partial funding of site monitoring visits and secondary data analyses.

Footnotes

None of the authors has a financial interest in the subject matter of this manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

1. Evaluation of grid pattern photocoagulation for macular edema in central vein occlusion: the Central Vein Occlusion Study Group M report. Ophthalmology. 1995;102:1425–33. [PubMed]
2. Branch Vein Occlusion Study Group. Argon laser photocoagulation for macular edema in branch vein occlusion. Am J Ophthalmol. 1984;98:271–82. [PubMed]
3. Manual of Policies and Procedures (MOPP) for the Standard Care vs. COrticosteroid for REtinal Vein Occlusion (SCORE) Study, version 4.0. Bethesda, MD: National Eye Institute; 2008. pp. 4.1–8. NTIS PB2008–106870.
4. Protocol for the Standard Care vs. COrticosteroid for REtinal Vein Occlusion (SCORE) Study, version 7.0. Bethesda, MD: National Eye Institute; 2008. pp. 24–65. NTIS PB2008-113743.
5. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6:284–90.
6. Scott IU, Bressler NM, Bressler SB, et al. Diabetic Retinopathy Clinical Research Network Study Group. Agreement between clinician and reading center gradings of diabetic retinopathy severity level at baseline in a phase 2 study of intravitreal bevacizumab for diabetic macular edema. Retina. 2008;28:36–40. [PMC free article] [PubMed]
7. Kinyoun JL, Martin DC, Fujimoto WY, Leonetti DL. Ophthalmoscopy versus fundus photographs for detecting and grading diabetic retinopathy. Invest Ophthalmol Vis Sci. 1992;33:1888–93. [PubMed]