|Home | About | Journals | Submit | Contact Us | Français|
The objective of the study was to examine the variation among institutional review boards (IRBs) in evaluation of the study design of a multicenter trial.
We assessed the first written response of local IRBs to each site investigator for a multicenter trial of vitamin A supplementation in extremely low birth weight (ELBW) infants performed by the National Institute of Child Health and Human Development Neonatal Research Network. Each author of this paper independently reviewed and categorized IRB concerns as major, minor or none, according to the predefined criteria.
Initially, 9 of 18 IRBs withheld approval because of at least one major concern. These concerns reflected difficulties in evaluating specific scientific issues for the design of the trial, including its justification, enrollment criteria, control and experimental therapies, co-interventions, toxicity assessment, outcome monitoring and informed consent.
The difficulty in assessing appropriate trial design for the specific hypothesis under investigation resulted in considerable variability in the evaluation by local IRBs.
Local institutional review boards (IRBs) vary considerably in their review of the same proposal for a multicenter study.1–11 Variation in the informed consent form has received considerable attention.8,12 However, little information is available regarding other differences in the review process and their importance, especially in assessing study design. Some variation may be appropriate because institutional IRBs reflect local values in assessing potential risks and benefits. However, less desirable reasons for variation may be due to the difficulty for local IRBs to include the expertise needed to review the full range of research questions that might be asked or study designs that might be used.
Variation among IRBs is of particular concern in large randomized multicenter trials, especially those involving high-risk, vulnerable populations in which the assessment of the risk–benefit ratio may be most important. To understand the causes contributing to variation in evaluating the design of such trials, we systematically examined the variation in IRB review of a multicenter randomized trial performed in extremely low birth weight (ELBW) newborns by the National Institute of Child Health and Human Development Neonatal Research Network.13,14 Although this analysis was completed during the trial, we present this information because of the increasing interest in promoting rigorous testing of interventions used in high-risk populations, because of the paucity of data regarding IRB review of trial design, and also because recent publications of IRB variation in the review of observational or minimal risk studies6–11 indicate that the type of difficulties in the IRB review illustrated by this trial continue today.
At the time the trial was proposed, vitamin A was not widely used in preterm infants, although a meta-analysis of several small trials identified an improved survival without chronic lung disease.15 The primary hypothesis was that supplementation with vitamin A would increase survival without chronic lung disease among ELBW infants who need early respiratory support.13,14 The eligible infants weighed 401 to 1000 g and required assisted ventilation or supplemental oxygen at 24 h of age. Exclusion criteria were major congenital anomaly, congenital nonbacterial infection, terminal illness, vitamin A administration in lipid emulsion or in amounts exceeding that in standard doses of multivitamins or parental refusal of consent. Infants were randomly assigned at 24 to 96 h of age to intervention or control groups after stratification by birth weight and center. Infants in the intervention group received 5000 IU (0.1 ml) vitamin A intramuscularly with a 29-gauge needle three times weekly for 4 weeks, a dose established by a preceding pilot study.15 Infants in the control group received simulated injections. To maintain masking, the research nurse administered treatment with a screen around the bed, used a pacifier to minimize crying and placed a band-aid over the actual or simulated injection site that the research nurse removed at the time of the next dose. Measurement of serum vitamin A levels and retinol dose-response testing were performed in the first 300 subjects, and masked investigators performed weekly physical examinations on all infants to assess potential clinical toxicity. An independent data safety and monitoring committee appointed by the National Institute of Child Health and Human Development reviewed data and adverse events at specified intervals. The final results indicated that the vitamin A regimen tested had no discernible toxicity and decreased the risk of chronic lung disease (relative risk 0.89, 95% confidence interval 0.80 to 0.99, number needed to treat 14 to 15).
Implementation of the trial followed a standardized process that involved a much more extensive critique of study design than for most studies submitted to IRBs.16 Network investigators evaluated the proposal and provided written critique to the study principal investigator (PI). A subcommittee was formed to assist the PI, and additional critique was obtained from the member neonatology divisions and from external reviewers with expertise in trial design, neonatal disease and treatment and vitamin A research. Before submission to any IRB, the protocol was approved by the entire network steering committee and an external advisory board.
Before implementation, site PIs at the 14 network centers submitted the protocol to 18 IRBs responsible for the research in participating hospitals. The protocol included an extensive discussion of the rationale for the study design related to background information about vitamin A, the results of previous neonatal studies of vitamin A, the need for an effectiveness trial, the selection of the primary and secondary outcomes, the basis for the vitamin A regimen, the assessment of safety and the avoidance of caregiver and assessor bias. The study PI prepared a general consent form describing the protocol and potential risks and benefits, which the investigators at each site modified to meet local requirements. However, the protocol was identical at all centers.
The authors evaluated the first written response addressed to each network site PI from their local IRB(s) after review of the protocol and consent forms. Two of the researchers (ARS, JET) categorized concerns raised by each IRB about the protocol a priori as major, minor or none. Major issues involved fundamental questions about study background and rationale, study design, inclusion or exclusion criteria, interventions, sample size, analysis, data safety monitoring or other issues that required further justification or clarification. We considered concerns about the protocol to be minor if they involved questions about administrative issues or further clarification of the protocol. We also reviewed changes requested in the consent form. To assess concordance, a third reviewer (PLH), who was not associated with the network and who was masked to the identity of the sites, evaluated the letters and classified responses using the same criteria. Any differences in the evaluation were resolved through discussion and consensus agreement.
In all, 7 of the 18 IRBs expressed no concerns about the study protocol. Initially, nine raised at least one major concern that needed to be addressed by the local investigator before the trial could be approved and initiated. Two IRBs had only minor concerns, requesting clarification about sample size or drug dosing. All except two IRBs required modifications of the consent form to address major or minor concerns. The protocol was subsequently approved at all sites without any changes in study design.
There was complete concordance among the three reviewers regarding classification of the concerns as major, minor or related to the consent form. The major IRB concerns were related to basic scientific issues in the design of clinical trials (see Table 1).
Despite extensive previous review, IRBs varied substantially in the concerns expressed about the study. Although we cannot entirely exclude the possibility that lack of clarity persisted despite this review and contributed to variation in responses, we think that these concerns may reflect the need for additional expertise to assist in evaluating the trial design for the specific intervention, disorder, population and outcome under investigation.
In considering whether the trial was justified, three IRBs required the site investigator to specify whether vitamin A supplementation was the standard of care for ELBW infants in that center. However, the central issue is the strength of the evidence supporting vitamin A supplementation, rather than the treatment preference of local neonatologists. As Freedman17 noted, ‘The ethics of medical practice grants no ethical or normative meaning to treatment preference, however powerful, that is based on a hunch or anything less than evidence publicly presented and convincing to the clinical community’.
In all areas of medicine,18,19 including neonatology,20,21 many commonly used therapies have been later shown to be harmful. Accordingly, the Institutional Review Guidebook of the National Institutes of Health defines experimental as a ‘term often used to denote a therapy (drug, device, procedure) that is unproven or not yet scientifically validated with respect to safety and efficacy’.22 The Guidebook also notes that a therapy may be considered experimental ‘without necessarily being part of a formal study (research) to evaluate its usefulness’. Using similar rationale, it would be preferable to refer to commonly used but unproven therapies as ‘conventional treatment’ rather than ‘standard of care’ in protocols submitted to the IRB. This may avoid any suggestion that such treatment is necessarily beneficial or even properly tested.
One IRB initially withheld approval because the evidence supporting routine vitamin A supplementation was considered too compelling to withhold its use. This judgment was based on a systematic review and meta-analysis of previously published trials.15,23 In contrast, a second IRB initially withheld approval because the evidence supporting routine supplementation was considered too weak to justify its use in the proposed trial. This IRB was particularly concerned about the possibility of unidentified toxicity. The first IRB eventually approved the protocol, in part because the dose in the proposed trial might be more effective than in most previous trials, the results of the meta-analysis were only marginally significant, no large trials had been performed and small trials with negative results are less likely to be published promptly or published at all than are trials with positive results.24 The second IRB approved the proposed trial after reconsidering the results of all previous trials.
In general, a clinical trial is warranted if there is sufficient but not definitive evidence that the intervention to be assessed would have a favorable risk–benefit ratio in the population to be enrolled. Justification to perform this trial was considered in depth by the Neonatal Research Network investigators during the development and extensive internal and external review of the protocol before submission to the IRBs at each site. Considerable training, skill and effort are often required to identify and evaluate the best relevant evidence25,26 and to assess and balance the potential hazards and benefits27,28 of interventions proposed for study. Routine involvement of a central body to provide an expert review of multicenter trials29–31 has been proposed to assure the availability of these skills for trials being assessed at local sites.
One IRB initially suggested using intramuscular injections of placebo rather than the sham procedure proposed in the protocol. Whether a placebo or sham procedure should be used in trials may involve complex clinical, ethical and methodological issues.32 In this case, reviewers with the relevant clinical and methodological expertise agreed that the sham procedure was preferred because placebo solutions previously thought to be innocuous might have harmful effects in small premature infants;33 the pain resulting from intramuscular injections of placebo might have important adverse effects in high-risk newborns;34 such injections would reduce the acceptability of the study to parents and caregivers; and the sham procedure would impose minimal, if any, risk, and assure that unmasking would be uncommon and unlikely to affect the primary outcome under investigation.35
Several IRBs were concerned that vitamin A provided by milk feedings should be controlled by the study protocol. This approach would have been inappropriate for several reasons. The study was designed as an effectiveness trial to assess the effects of an intramuscular regimen of vitamin A supplementation on ELBW infants whose feedings were regulated by their clinical team. Enteral intake of vitamin A would be difficult to determine and regulate because the concentration in human milk is variable, many ELBW infants have prolonged feeding intolerance and poor enteral absorption of vitamin A and intravenous administration is unreliable (a reason that vitamin A administration was intramuscular in the trial).13 Moreover, feeding tolerance and dietary intake of vitamin A and other nutrients were considered outcome variables that might be influenced by the effect of intramuscular vitamin A administration on the development of chronic lung disease, sepsis or necrotizing enterocolitis.13
Whether other aspects of clinical care should be controlled depends upon whether the clinical trial is being conducted as an efficacy or an effectiveness trial.19,36,37 Early randomized trials of new therapies are usually small efficacy trials (also known as explanatory trials). These trials are conducted to determine whether or how an intervention works under ideal or restricted circumstances. Thus, other aspects of care (co-interventions) are often closely controlled. Effectiveness trials (also known as management trials), such as the vitamin A trial, are conducted to determine whether an intervention is effective under usual clinical circumstances (‘real-world conditions’). Such trials are needed to determine whether a therapy should be recommended for routine clinical use.
One IRB initially questioned the enrollment of infants <500 g of birth weight on the grounds that their inclusion in the trial might prompt use of intensive care within or outside Network centers for infants too small or immature to benefit. We consider this concern inappropriate for several reasons. First, many centers have provided intensive care to selected infants with a birth weight <500 g. The majority of these infants develop chronic lung disease that might be ameliorated by vitamin A supplementation. The study was designed to be broadly generalizable to ELBW infants, including the smallest patients, who were given intensive care. Second, there was little basis to believe the enrollment criteria for the proposed trial would alter decisions to administer intensive care. Finally, the designated responsibilities of IRBs22 do not specify consideration of whether the enrollment criteria or results would be misused in clinical practice.
Two IRBs initially recommended obtaining serial assessments of serum vitamin A concentrations in all patients to distinguish those with abnormal or toxic values from those with normal values. Such a recommendation was problematic for several reasons. First, the IRB did not indicate what is meant by the term ‘abnormal values’. This term can have different meanings depending on whether it is defined in terms of values that are unusual (for example, values not within 2 s.d. of the mean), values associated with increased risk, values that cause adverse outcomes or values that warrant treatment (values for which the benefits of treatment exceed the hazards, a definition that may well be established only by conducting randomized trials as we proposed).38 A second problem is that there may not be adequate research to differentiate normal from abnormal values, regardless of the type of definition used. In premature infants, the values that would be considered abnormal by the last three definitions above are unclear, and cannot be extrapolated from term infants or older patients.13 For this reason, the identification of any vitamin A toxicity in the trial was based primarily on serial detailed assessments performed by masked and experienced evaluators. In this way, values obtained in any infants with signs of toxicity could be compared with values of infants without signs of toxicity. The most sensitive sign of vitamin A toxicity in infants is an increase in fontanelle tension.39–41 Fortunately, such signs were very uncommon and no more frequent among supplemented than control infants and a lower proportion of supplemented infants had blood levels considered to be low at older ages. In providing data for serum retinol at the end of treatment, the study helped define values that would be within the range of normal defined in terms of values that warrant treatment.
Modifications of the consent form to meet local institutional requirements are always appropriate. However, IRBs varied in their responses to the content of the form. Some IRBs required detailed descriptions of the randomization procedure, withdrawal criteria or the criteria used to evaluate the outcome of the study. One IRB initially disallowed a statement that the National Institutes of Health provided funding for the study. Several IRBs disallowed a statement that the potential benefits to participants were the additional examinations, monitoring and evaluation during the neonatal period and follow-up evaluations of health and development through 18 to 22 months corrected age. One IRB initially stipulated wording changes not appropriate for newborns (for example, noting ‘fainting’ as a hazard of collecting blood samples).
The value or justification for some of these requirements is debatable.42–45 Whatever the potential benefits of participating in our trial, further research is needed to determine whether or not patients enrolled in clinical trials generally have better overall outcomes than otherwise similar patients not enrolled in such trials.46 At least one study has shown that outcomes of sick newborns assigned to the placebo arm of a randomized trial were better than those of eligible infants who were not randomized.47 Additional study is required to define the information that is justified, understandable and beneficial in seeking consent, especially in highly stressful circumstances such as those experienced by parents of ELBW infants.42,48
One IRB planned to obtain interim outcome data for each treatment group in that study center, if not all study centers. These responses indicate the ambiguity in the role of the local IRB relative to that of the data safety and monitoring committee. There has been considerable dismay expressed about the requirement for local IRBs to monitor adverse events in their center in multicenter trials.49,50 These IRBs are often not staffed adequately or provided sufficient resources to evaluate those events. They may also lack the information required, particularly for masked trials, such as this one, in which the IRB does not know the treatment received by individual patients.51 These problems can be obviated if, as in this trial, a central data safety and monitoring committee is responsible for evaluating adverse events across all sites, conducting interim analyses and reporting their conclusions to the site PIs and their local IRBs.7,49,51,52
In conclusion, we observed considerable variation among IRBs in their concerns about scientific issues involved in the design of a multicenter trial. This variation might have been substantially greater had the design and description of the proposal been less thoroughly critiqued before submission or described in less detail in the protocol provided to each IRB. With their increasing work load, IRBs face increasing difficulty in recruiting members and accomplishing timely reviews. Given the extremely wide range of clinical problems and potential interventions, the complexity of the biological, clinical and design issues and the necessarily limited availability of expertise within any institution to address all these issues, the evaluation of multicenter clinical trials may be quite challenging and burdensome for individual IRBs. Furthermore, because large academic centers often have multiple IRBs, variability might worsen even within single institutions.
One potential solution applicable to multicenter trials is to involve a central body to provide scientific, and perhaps also ethical, review of the protocols.4,28,30,31,53,54 The extent to which the IRB chose to rely on this review body could vary, depending on local expertise and the specific study protocol. In this way, the local IRB might more efficiently use its limited personnel, time and resources to assure that high ethical standards were met in seeking consent and in performing the study while maintaining its authority.55,56
The National Institutes of Health and the National Institute of Child Health and Human Development provided grant support for recruitment and data analysis for the Neonatal Research Network’s Vitamin A Supplementation Study for 1996 and 1997. The funding agencies provided overall oversight for study conduct, but all data analyses and interpretation were independent of the funding agencies. We are indebted to our medical and nursing colleagues and the infants and their parents who agreed to take part in this study. The following investigators participated in this study: NRN Steering Committee Chair: Alan Jobe, University of Cincinnati; Brown University Women & Infants Hospital of Rhode Island (U10 HD27904)—William Oh and Angelita Hensman; Case Western Reserve University Rainbow Babies & Children’s Hospital (GCRC M01 RR80, U10 HD21364)—Avroy A Fanaroff, Michele C Walsh and Nancy S Newman; Emory University Children’s Healthcare of Atlanta, Grady Memorial Hospital, and Emory Crawford Long Hospital (GCRC M01 RR39, U10 HD27851)—Barbara J Stoll and Ellen Hale; Harvard Medical School Brigham and Women’s Hospital (GCRC M01 RR2172, GCRC M01 RR2635, U10 HD34167)—Ann R Stark and Kerri Fournier; Indiana University Indiana University Hospital, Methodist Hospital, Riley Hospital for Children, and Wishard Health Services (GCRC M01 RR750, U10 HD27856)—James A Lemons and Diana D Appel; National Institute of Child Health and Human Development—Linda L Wright and Elizabeth M McClure; Research Triangle Institute (U01 HD36790)—W Kenneth Poole, Abhik Das and Betty Hastings; Stanford University Lucile Packard Children’s Hospital (U10 HD27880)—David K Stevenson and M Bethany Ball; University of Alabama at Birmingham Health System and Children’s Hospital of Alabama (U10 HD34216)—Waldemar A Carlo and Monica V Collins; University of Cincinnati University Hospital, Cincinnati Children’s Hospital Medical Center, and Good Samaritan Hospital (GCRC M01 RR8084, U10 HD27853)—Edward F Donovan and Marcia Mersmann; University of Miami Holtz Children’s Hospital (U10 HD21397)—Charles Bauer and AM Worth; University of New Mexico Health Sciences Center (GCRC M01 RR997, U10 HD27881)—Lu-Ann Papile and Conra Backstrom Lacy; University of Tennessee (U10 HD21415)—Sheldon B Korones and Tina Hudson; University of Texas Southwestern Medical Center at Dallas Parkland Health & Hospital System and Children’s Medical Center Dallas (GCRC M01 RR633, U10 HD40689)—Jon E Tyson and Susie Madison; Wayne State University Hutzel Women’s Hospital and Children’s Hospital of Michigan (U10 HD21385)—Seetha Shankaran and Geraldine Muran; Yale University Yale-New Haven Children’s Hospital (GCRC M01 RR6022, U10 HD27871)—Richard A Ehrenkranz and Patricia Gettner.
Conflict of interest
The authors declare no conflict of interest.