|Home | About | Journals | Submit | Contact Us | Français|
To perform a Bayesian analysis of the Mycotic Ulcer Treatment Trial I (MUTT I) using expert opinion as a prior belief.
MUTT I was a randomized clinical trial comparing topical natamycin or voriconazole for treating filamentous fungal keratitis. A questionnaire elicited expert opinion on the best treatment of fungal keratitis before MUTT I results were available. A Bayesian analysis was performed using the questionnaire data as a prior belief and the MUTT I primary outcome (3-month visual acuity) by frequentist analysis as a likelihood.
Corneal experts had a 41.1% prior belief that natamycin improved 3-month visual acuity compared with voriconazole. The Bayesian analysis found a 98.4% belief for natamycin treatment compared with voriconazole treatment for filamentous cases as a group (mean improvement 1.1 Snellen lines, 95% credible interval 0.1–2.1). The Bayesian analysis estimated a smaller treatment effect than the MUTT I frequentist analysis result of 1.8-line improvement with natamycin versus voriconazole (95% confidence interval 0.5–3.0, P = 0.006). For Fusarium cases, the posterior demonstrated a 99.7% belief for natamycin treatment, whereas non-Fusarium cases had a 57.3% belief.
The Bayesian analysis suggests that natamycin is superior to voriconazole when filamentous cases are analyzed as a group. Subgroup analysis of Fusarium cases found improvement with natamycin compared with voriconazole, whereas there was almost no difference between treatments for non-Fusarium cases. These results were consistent with, though smaller in effect size than, the MUTT I primary outcome by frequentist analysis. The accordance between analyses further validates the trial results. (ClinicalTrials.gov number, NCT00996736.)
Infectious keratitis is one of the leading causes of monocular blindness worldwide.1 In South India, approximately half of all infectious corneal ulcers are of fungal etiology.2 For fungal keratitis, poor outcomes frequently result despite commonly used treatments, such as natamycin.2,3 The 2006 outbreak of Fusarium keratitis in the United States and Asia resulted in heightened concern over how to best treat fungal keratitis.4–6 Recently, our group completed the first of two Mycotic Ulcer Treatment Trials (MUTT I) to address this question. MUTT I is a multicenter, randomized, double-masked clinical trial investigating the efficacy of topical natamycin compared with topical voriconazole with regard to visual acuity in culture-positive fungal corneal ulcers.7
Although frequentist paradigms dominate statistical analysis of clinical trials, useful insight may be gained from Bayesian analysis, which permits the use of prior information derived from either expert beliefs (subjective prior) or observational results (objective prior).8,9 Currently, the clinical interpretation of data generated from randomized controlled trials often poses a challenge. The frequentist approach offers an estimate of the probability that, were there no difference between the two arms, we would have seen a difference at least as large as that observed (P value). This approach does not reveal how likely it is that one treatment is clinically better, whereas Bayesian analysis allows one to estimate the probability of clinical superiority.10,11 Since clinicians routinely use multiple sources of evidence when interpreting diagnostic tests in practice, the “Bayesian” approach of incorporating prior knowledge and addressing which treatment is superior may be more intuitive for clinicians.10,12–14
In this study, we obtained a subjective prior by using a questionnaire to elicit the beliefs of expert clinicians before the MUTT I results were available. As a comparison, we used results from the earlier Mycotic Ulcer Therapeutic Exploratory Trial to generate an objective prior.15 Using the subjective and objective prior distributions, we performed separate Bayesian analyses of the MUTT I study, thus allowing for comparison of the subjective and objective posteriors.
The Mycotic Ulcer Treatment Trials (MUTT) are multicenter, randomized, double-masked clinical trials studying the optimal antimicrobial treatment of filamentous fungal keratitis. MUTT I investigated the efficacy of topical natamycin compared with topical voriconazole on 3-month best spectacle-corrected visual acuity (BSCVA).7 MUTT II evaluates the efficacy of topical voriconazole with oral voriconazole compared with topical voriconazole with oral placebo, and is not analyzed here.
Detailed methods for MUTT I have been reported.7 Briefly, 323 fungal ulcer cases with enrollment visual acuity of 20/40 (0.3 logarithm of the minimum angle of resolution [logMAR]) to 20/400 (1.3 logMAR), presenting to the Aravind Eye Care System (Madurai, Pondicherry, and Coimbatore) in India, were randomized to receive 5% topical natamycin (Natacyn; Alcon, Fort Worth, TX) or 1% topical voriconazole (VFEND IV; Pfizer, New York, NY). The primary outcome for MUTT I was 3-month BSCVA in logMAR, using linear regression with baseline acuity and treatment arm as covariates. Complete details on baseline characteristics have been reported previously.7
The MUTT I trial was compliant with the Health Insurance Portability and Accountability Act of 1996, adhered to the Declaration of Helsinki, and received approval from the Institutional Review Boards (IRBs) at Aravind, Dartmouth, and University of California, San Francisco (UCSF). The questionnaire received IRB exemption at UCSF. Informed consent was obtained from all participants.
The subjective prior distributions were obtained by eliciting the opinions of international corneal experts through an online questionnaire (see Supplementary Material for questionnaire). Emails were sent to 14 corneal experts in the United States, United Kingdom, Australia, and India in August 2012. The emails stated the objective of the study and contained a link to the questionnaire. Experts were determined in one of two ways: well-known specialist in the field of fungal keratitis or corresponding author of one of the most commonly cited “fungal keratitis or fungal corneal ulcer” papers (search on 7/25/2012 on Web of Science, available in the public domain at http://www.isiknowledge.com). Participation was voluntary and all responses were anonymous. The online questionnaire was conducted through SurveyMonkey (SurveyMonkey.comSurveyMonkey.com, LLC, Palo Alto, CA).
The questionnaire elicited perceived best treatment of filamentous fungal keratitis and of Fusarium keratitis. Fusarium species are a subgroup of filamentous fungi and are the most common cause of fungal keratitis in South India.15,16 Two different methods were used to obtain a prior belief. Participants were asked to: (1) specify their estimated best effect and their 95% credible interval (CrI) (credible interval method) and (2) allocate percentage points of probability to discrete intervals (histogram method).12,17 We also asked participants to qualitatively select their belief (natamycin, voriconazole, or no difference) and used this response to verify the direction (natamycin or voriconazole better) of the credible interval and histogram methods.
For each individual response to the credible interval method, we created a prior distribution by minimizing the Fisher information that produced the desired mean and 95% CrI. The individual prior distributions were then summed and normalized to create a group prior distribution. Individual responses from the histogram method were also summed and normalized to obtain a group prior distribution. Other studies have used similar methods for eliciting and pooling subjective priors.12,14,17 For each expert, a non-Fusarium subjective prior was determined from the individual's prior beliefs for all filamentous and Fusarium cases (assuming 40% of all filamentous cases were Fusarium species as specified in the questionnaire). We also performed a sensitivity analysis for the credible interval method by excluding the most enthusiastic respondent (N = 1) for natamycin or voriconazole, based on mean belief.12,17,18 This allowed us to examine if our outcome had been biased by outliers.
For comparison, we determined the objective prior distribution from the results of the Mycotic Ulcer Therapeutic Exploratory Trial, performed in preparation for MUTT.15 The therapeutic exploratory study included 120 patients, of which 55 patients had enrollment BSCVA of 20/40 to 20/400. The study was also performed at the Aravind Eye Care System (Madurai and Pondicherry) in India, used the same personnel and drug dosing, and a similar study protocol. The primary outcome for the therapeutic exploratory study was also 3-month BSCVA. To standardize results, the therapeutic exploratory study data were reanalyzed using MUTT I methods for this study.7 We created an objective prior from the therapeutic exploratory study results by assuming a noninformative prior (a flat, improper prior). With this noninformative prior and a likelihood obtained from the therapeutic exploratory study's primary outcome, we derived a posterior using Bayes' theorem.19 This posterior then became the objective prior used in our analysis.
To describe the questionnaire responses, we performed frequentist statistics. The number of ulcers treated and the mean effect size were evaluated by the Student's t-test. Comparison of the credible interval and histogram group prior distributions was analyzed by a paired t-test. Conversion between Snellen line equivalents and logMAR units was approximated by 1 Snellen line = 0.1 logMAR. A negative change in logMAR indicates improvement in visual acuity and a positive logMAR indicates worsening. Statistical tests were performed using a commercial software package (Stata 10.0; StataCorp, College Station, TX).
Using Bayes' theorem, the posterior distribution was calculated by multiplying the likelihood of having obtained the MUTT I results by the prior distribution for each effect size and normalizing the result.19 The posterior distribution represents the updated belief or probability distribution, given the new experimental data (likelihood) and the prior distribution. To calculate the subjective posterior distribution, we used the credible interval prior since it was a continuous distribution. All graphics, prior distribution calculations, and posterior distribution calculations were performed using technical computing software (Mathematica 8; Wolfram Research, Champaign, IL).
Of the 14 respondents, 11 completed the questionnaire. Only the completed responses were analyzed. Two respondents had incomplete questionnaires, and another one indicated knowledge of the trial results; per the comment box, it was clear this respondent was referring to knowledge of already published Mycotic Ulcer Therapeutic Exploratory Trial results. All 11 respondents who completed the questionnaire were ophthalmologists and 10 (91%) were corneal specialists. Seven (64%) of the respondents had treated fewer than 100 ulcers in the past year, and four (36%) had treated more than 100 ulcers.
We examined the effect of reported number of ulcers treated on an individual's response to the credible interval method. There was no significant association between the mean estimated best effect for all filamentous cases and the number of ulcers treated (P = 0.15). Similarly, there was no significant association between an individual's uncertainty, measured by the width of the 95% CrI, and the number of ulcers treated (P = 0.30).
The individual responses to the questionnaire are portrayed in Figures 1A–C. For all filamentous cases (Fig. 1A), seven respondents (64%) expected no difference in 3-month BSCVA between the antifungals, three respondents (27%) expected an improvement in visual acuity with voriconazole, and one (9%) with natamycin. For Fusarium cases (Fig. 1B), only two respondents expected an alternate best treatment; of the 11 respondents, five (46%) expected no difference in acuity between the antifungals, four (36%) expected an improvement with voriconazole, and two (18%) with natamycin. Only one individual had discordant direction of belief between the qualitative and histogram methods.
Filamentous, Fusarium, and non-Fusarium cases (Figs. 1A–C) all had group prior distributions, with slight skewness of belief toward voriconazole. Using the credible interval response, natamycin treatment, compared with voriconazole treatment, was estimated to have approximately 0.5-line worse 3-month BSCVA for all filamentous (0.05 logMAR, 95% CrI −0.16 to 0.39), Fusarium (0.04 logMAR, 95% CrI −0.25 to 0.39), and non-Fusarium cases (0.05 logMAR, 95% CrI −0.14 to 0.39). Comparison of the credible interval and histogram group prior distributions showed no significant mean difference for all filamentous (P = 0.77), Fusarium (P = 0.90), and non-Fusarium cases (P = 0.29).
We generated a subjective posterior distribution using the credible interval response prior and the primary outcome of MUTT I (Fig. 2A). Frequentist analysis results for MUTT I and the therapeutic exploratory trial are shown in Table 1, with Bayesian analysis results listed in Table 2. For the subjective posterior distribution, there was a 98.4% belief (area under the curve to the right of 0) that natamycin improved 3-month BSCVA. The posterior distribution's mean was 1.1-line improvement with natamycin compared with voriconazole (Table 2), a smaller effect than the MUTT I frequentist estimate of 1.8-line improvement with natamycin (Table 1). Figure 2B shows the objective posterior distribution, which has a 97.6% probability (area under the curve to the right of 0) that natamycin improved 3-month BSCVA. In the sensitivity analysis, exclusion of the most enthusiastic respondent for natamycin or voriconazole from the group prior resulted in a subjective posterior distribution with a 97.4% or 98.4% belief, respectively, that natamycin improved 3-month BSCVA, similar to the 98.4% belief using all respondents.
Subjective analysis of Fusarium cases shows a 99.7% belief that natamycin improved 3-month BSCVA (Fig. 2C). In Figure 2D, the objective posterior for Fusarium cases shows a 99.97% probability that natamycin improved visual acuity.
Figures 2E, E,2F2F present the subjective and objective posterior distributions for non-Fusarium cases. The subjective posterior shows a 57.3% belief that natamycin improved BSCVA at 3 months, whereas the objective posterior shows a 38.9% probability. The mean BSCVA for both subjective and objective posterior distributions suggests essentially no difference between natamycin and voriconazole (Table 2).
MUTT I evaluated the efficacy of natamycin or voriconazole for treating filamentous fungal keratitis. The frequentist analysis found a significant improvement in 3-month visual acuity with natamycin compared with voriconazole.7 Here, we elicited the expert opinions of corneal specialists on treating filamentous ulcers, to perform a Bayesian analysis of MUTT I's primary outcome. Experts believed a priori that natamycin-treated cases would perform slightly worse than voriconazole-treated cases. However, subjective Bayesian analysis using the group prior distribution found a 98.4% belief that natamycin improved 3-month BSCVA for filamentous cases as a group compared with voriconazole. For Fusarium cases, the subjective posterior demonstrated a 99.7% belief in favor of natamycin treatment, whereas non-Fusarium cases had a largely neutral subjective posterior of 57.3% belief. Objective Bayesian analysis, using the Mycotic Ulcer Therapeutic Exploratory Trial results to form a prior, found similar results.
Bayesian analysis can be used as an alternative method to evaluate clinical trials. In brief, prior information is combined with experimental data to derive a posterior probability using Bayes' theorem.8,9,19 Compared with traditional frequentist statistics, Bayesian methods have the advantage of incorporating prior information and estimating the probability that one treatment is more effective than another.10–12 Frequentist analysis tests the hypothesis that the two treatment arms have the same efficacy; it offers an estimate of the probability of finding results as skewed or more skewed between the arms, if there was in truth no difference between them (P value). This finding is not directly related to the probability of treatment superiority. Using Bayesian analysis, we can directly estimate the probability that natamycin or voriconazole is more effective in treating fungal ulcers. However, Bayesian analysis can be highly dependent on the prior distribution and, in this way, has been deemed “subjective.”10 In this study, the prior distribution reflects the initial beliefs of fungal keratitis experts regarding treatment effect. If we had surveyed an alternate group of individuals, we would likely have a different prior distribution that may result in a different posterior distribution. Thus, the validity and reliability of prior elicitation are crucial.
For this reason, we elicited the subjective prior using two different methods: credible interval (mean and 95% CrI) and histogram. The two methods did not have a significant mean difference, although the distributions were inherently different—the histogram method used discrete intervals, whereas the credible interval was continuous. To have a standard of comparison for the subjective posterior, we used the therapeutic exploratory study to generate an objective posterior.15 We found both posteriors to be similar, although the objective suggested more uncertainty.
Overall, the Bayesian analysis results were consistent with, though more conservative than, the frequentist results. A Bayesian posterior consistent with the frequentist result can serve to confirm it, whereas substantial disagreement between Bayesian and frequentist results implies that previous knowledge and outcome are incompatible, and both should be reexamined.17 Our Bayesian analysis result suggested a similar, but smaller effect size than that of the frequentist result. This finding often occurs in Bayesian analysis: as the sample size of a study increases, the likelihood dominates the prior and the Bayesian posterior will asymptotically approach the likelihood.20,21 Although the 323-patient MUTT I study had similar frequentist and Bayesian results, the MUTT I frequentist analysis may have exaggerated the difference between treatment groups, and the more conservative Bayesian analysis may be more realistic in its estimate of effect size.
In our elicitation of the prior, we demonstrated that fungal keratitis experts marginally preferred voriconazole over natamycin, with a variation in belief consistent with community equipoise. Another recent survey of corneal specialists found that voriconazole was the preferred topical treatment over natamycin.22 Experts were likely optimistic about voriconazole use for several reasons. First, in vitro studies suggested that voriconazole could be effective for fungal keratitis in general and Fusarium and Aspergillus species in particular.23–25 Second, voriconazole had been shown to have better penetration through the corneal epithelium than natamycin.26–28 Third, voriconazole was a newer azole, with broad systemic coverage of invasive filamentous fungi and candidiasis.29,30 Finally, results from the Mycotic Ulcer Therapeutic Exploratory Trial found that voriconazole-treated patients had 1-line improvement in 3-month visual acuity compared with natamycin-treated patients (−0.098 logMAR, 95% confidence interval [CI] −0.28 to 0.083, P = 0.29).15 However, MUTT I frequentist and Bayesian analysis results were not consistent with prior beliefs or in vitro and exploratory study results.
There are limitations to in vitro susceptibility testing and exploratory studies, which may account for the discrepancy between prior studies and MUTT I results. Susceptibility testing is only one of several factors that predict treatment success.31,32 Exploratory studies are often small and are preliminary in nature. The 120-patient Mycotic Ulcer Therapeutic Exploratory Trial also did not limit enrollment by presentation visual acuity, whereas MUTT I enrolled only patients with visual acuity of 20/40 to 20/400. As described in our MUTT I primary paper, the success of natamycin treatment is largely attributable to improvement in Fusarium cases.7 When we examined the 22 Fusarium cases in the therapeutic exploratory trial with presentation visual acuity of 20/40 to 20/400, the natamycin-treated cases had marginally better acuity than that of voriconazole-treated cases, although not significantly so (−0.06 logMAR, 95% CI −0.42 to 0.30, P = 0.73).
There are several limitations to our elicitation method. First, our sample of 11 experts may not reflect the broad range of opinions of our target population. Although we included corneal experts from four different continents, most practiced in an academic environment. Although our questionnaire sample size was comparable to that of other Bayesian priors in the literature,14 further studies may require larger and more diverse sample populations. Second, since MUTT enrolled patients only in South India, the incidence of fungal keratitis and patient characteristics may differ from that of other regions. To address these differences in our questionnaire, we restricted the types of organisms and their relative proportions only to those found in South India. Third, our questionnaire may be susceptible to anchoring bias since we presented data in examples in an attempt to facilitate better understanding of statistical concepts. Anchoring bias occurs when a respondent's belief is influenced by how the data are presented. To limit anchoring bias, we used symmetric examples and three different methods (qualitative, credible interval, and histogram) to ask similar questions. Although there were more responses indicating no difference between antifungals, we found that all of the responses, except for one, were consistent among the three methods. This finding likely suggests that anchoring bias was limited.
In summary, we found that natamycin improved 3-month visual acuity using both Bayesian and frequentist analyses for all filamentous ulcers. For Fusarium ulcers, natamycin also performed better than voriconazole, but for non-Fusarium ulcers, there was equipoise regarding treatment. The Bayesian analysis suggested a smaller treatment effect than the frequentist result. Overall, Bayesian analysis may offer more intuitive results than frequentist analysis and could be considered as a supplement to the frequentist approach.10,12–14
The authors thank the members of the Mycotic Ulcer Treatment Trial Group and the MUTT Data and Safety Monitoring Committee for their help:
Clinical Centers: Aravind Eye Hospital, Madurai, Tamil Nadu, India: N. Venkatesh Prajna, MD (principal investigator), Lalitha Prajna, MD, Jeena Mascarenhas, MD, Muthiah Srinivasan, MD, Thirukkonda Subramanian Chandravathi, MA, R. Somu Saravanan, MA, Rajarathinam Karpagam, Malaiyandi Rajkumar, Rajendran Mahalakshmi, MSc, S.R. Sumithra, and Charles Sundar; Aravind Eye Hospital, Coimbatore, Tamil Nadu, India: Revathi Rajaraman, MD (site director), Anita Raghavan, MD, and P. Manikandan, MPhil; Aravind Eye Hospital, Pondicherry, Tamil Nadu, India: Tiruvengada Krishnan, MD (site director), and N. Shivananada, MD; Francis I. Proctor Foundation, University of California, San Francisco: Thomas M. Lietman, MD (principal investigator), Nisha R. Acharya, MD, MS (principal investigator), Stephen D. McLeod, MD, John P. Whitcher, MD, MPH, Salena Lee, OD, Vicky Cevallos, MT (ASCP), Catherine E. Oldenburg, MPH, Kieran S. O'Brien, MPH, and Kevin C. Hong, BA; Data and Safety Monitoring Committee: Marian Fisher, PhD (chair), Anthony Aldave, MD, Donald F. Everett, MA, Jacqueline Glover, PhD, K. Ananda Kannan, MD, Steven Kymes, PhD, and Ivan Schwab, MD; Resource Centers: Coordinating Center, Francis I. Proctor Foundation, University of California, San Francisco: Thomas M. Lietman, MD (principal investigator), Nisha R. Acharya, MD, MS (principal investigator), David Glidden, PhD, Stephen D. McLeod, MD, John P. Whitcher, MD, MPH, Salena Lee, OD, Kathryn J. Ray, MA, Vicky Cevallos, MT (ASCP), Catherine E. Oldenburg, MPH, Kevin C. Hong, BA, Kieran S. O'Brien, MPH; Project Office, National Eye Institute, Rockville, Maryland: Donald F. Everett, MA; Photography Reading Center, Dartmouth Medical School, Lebanon, New Hampshire: Michael E. Zegans, MD, and Christine M. Kidd, PhD.
Supported by National Eye Institute/National Institutes of Health Grants U10 EY018573 (TML) and K23 EY017897 (NRA) and by That Man May See (which supports the Department of Ophthalmology and the Francis I. Proctor Foundation for Research on Ophthalmology of UCSF); the Harper/Inglis Trust; The South Asia Research Foundation; Research to Prevent Blindness (TML, NRA); and UCSF Academic Senate Committee on Research (CQS). Natamycin and voriconazole were donated by Alcon and Pfizer, respectively. The sponsors did not have any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, and approval of the manuscript. The authors alone are responsible for the content and writing of the paper.
Disclosure: C.Q. Sun, None; N.V. Prajna, None; T. Krishnan, None; J. Mascarenhas, None; R. Rajaraman, None; M. Srinivasan, None; A. Raghavan, None; K.S. O'Brien, None; K.J. Ray, None; S.D. McLeod, None; T.C. Porco, None; N.R. Acharya, None; T.M. Lietman, None