|Home | About | Journals | Submit | Contact Us | Français|
To describe, qualitatively and quantitatively, the impact of a review by multiple institutional review boards (IRBs) on the conduct of a multisite observational health services research study.
Primary data collection during 2002, 2003, and 2004 at 43 United States Department of Veterans Affairs (VA) primary care clinics.
Explanatory sequential mixed methods design incorporating qualitative and quantitative elements in sequence.
Field notes and documents collected by research staff during a multisite observational health services research study were used in thematic analysis. Themes were quantified descriptively and merged with timeline data.
Approximately 4,680 hours of staff time over a 19-month period were devoted solely to the IRB process. Four categories of phenomena impacting research were observed:
Several features of the IRB system as currently configured impose costly burdens of administrative activity and delay on observational health services research studies, and paradoxically decrease protection of human subjects. Central review with local opt-out, cooperative review, or a system of peer review could reduce costs and improve protection of human subjects.
Ethical oversight of research involving human subjects is essential in order to insure that the values of respect for persons, beneficence, and social justice (United States Department of Health & Human Services 1978) are maintained. That function is currently served by the Institutional Review Board (IRB) system, based on the prospective and ongoing local review of the proposed research at every site involved in the conduct of a given project. Many papers critical of current IRB procedures have been written in the past decade. Criticisms include: that IRBs are generally ill equipped to review social science research (American Association of University Professors 2000), resulting in barriers to the effective conduct of such research; that IRB members do not use a systematic way of assessing the risk/benefit ratio when evaluating protocols (Reynolds 2002a); that IRB decisions may frequently be based more on institutional risk aversion than on subject risk and adequate protection (Rogers et al. 1999); that IRBs are more concerned with the content of the consent document than with the consent process (Lynn, Johnson, and Levine 1994); and that IRBs are typically made up of researchers and physicians who are biased toward quantitative research (Tod, Nicolson, and Allmark 2002). The high degree of inconsistency across IRBs, which delays and complicates multicenter studies, has long been observed (Benson 1989; Lux, Edwards, and Osborne 2000; Burman et al. 2001; Silverman, Hull, and Sugarman 2001; Stair et al. 2001; Hirshon et al. 2002).
While some have called for centralizing the IRB process to reduce variability, delays, and duplication of effort (Edgar and Rothman 1995; Christian et al. 2002), and to allow national-level discussion of difficult ethical issues (Lind 1992) and “moral consistency” (Moreno 1998), others focus on the advantages of local review (e.g., familiarity with locally relevant issues pertinent to human subjects) (Freedman 1994; Moreno 1998; Levine 2000; Reynolds 2002b).
Observational health services research is particularly sensitive to the issues arising from multiple IRB reviews. In order to be generalizable, research on health care delivery, physician practice patterns, and other health care systems issues must involve many and widely varying practice settings. As a result, observational health services research studies almost invariably undergo multiple reviews in the current local-IRB system. However, observational research budgets are typically very modest compared with clinical trials and are often unable to absorb the delays and unexpected expenses that can arise from multiple resubmissions and conflicting reviews. Wolf, Croughan, and Lo (2002) discuss the challenges of human subjects protection in multisite observational research, in the context of practice-based research networks. They point out that “… much of practice-based research has involved medical record review, interviews, or surveys. These types of research customarily present minimal risk provided that informed consent is appropriately obtained and confidentiality is protected. Such research therefore should require less scrutiny than multisite clinical trials of unproven interventions.” They recommend that articles should be published clarifying “how regulations developed for clinical intervention research may not fit practice-based research … and suggest[ing] how IRB policies or federal regulations need to be revised.”
Other studies have provided case examples of the variability and delays associated with multisite IRB reviews (While 1995; Lux, Edwards, and Osborne 2000; Silverman, Hull, and Sugarman 2001; Stair et al. 2001; Hirshon et al. 2002). Two of these involved randomized clinical trials (Silverman, Hull, and Sugarman 2001; Stair et al. 2001), two involved observational health services research (While 1995; Hirshon et al. 2002), and the type of research involved in the fifth was not described (Lux, Edwards, and Osborne 2000). Of the two involving observational health services research, only one discusses the reasons for the delays and the nature of the variable responses, and that study involved IRB review at only three sites. Studies of the IRB review process in multisite observational health services research using larger samples and providing more detailed enumeration of the components of delay and variation are needed in order to make informed recommendations for change. This study undertakes to do so.
This is an ad hoc, descriptive review of the process required to obtain IRB approval to conduct an observational health services research study in 43 Department of Veterans Affairs (VA) medical centers. To distinguish between the ad hoc study and the original multisite study, the former will be referred to as the “IRB study,” and the latter as the “multisite study.”
The descriptive analysis in the IRB study used a sequential explanatory mixed methods model (Creswell 2002), using qualitative analysis to explore initial quantitative findings, then quantifying the themes discovered in the exploration.
The initial quantitative observations were the elapsed times in the IRB approval process and the staff hours consumed. Extensive field notes were then reviewed to attempt to determine the procedures contributing to the observed delays. These field notes had been collected by the research team while carrying out the multisite study. The field notes consisted of entries made into a database designed for tracking the IRB process to ensure timely completion of steps, notes from phone conversations with staff at the participating sites, and e-mails with staff at the sites. The study team collated the field notes and performed a qualitative content analysis (Miller and Crabtree 1992) to identify major recurring themes and observed phenomena. The initial theme abstraction was performed individually by team members, then the results were merged in meetings and discussions to identify the final set of themes. Finally, descriptive statistics were compiled on the occurrences of those identified themes and phenomena.
The objective of the multisite observational study was to determine if a relationship existed between physicians' styles of acquiring new information about medical practice (Wyszewianski and Green 2000), the guidelines dissemination methods used at their clinical site, and their adherence to the recommendations of a clinical practice guideline on treatment of hypertension for patients with Type 2 diabetes.
The study consisted of three different phases of data collection: (1) two key informants at each site were interviewed by telephone to determine guideline dissemination methods used; (2) physicians filled out a one-page questionnaire to measure their cognitive style of handling new information (Green, Gorenflo, and Wyszewianski 2002), and (3) patient-level pharmacy dispensing data were used to measure the concordance between physicians' prescribing and guideline recommendations.
IRB approval for phase (1) was requested only at the investigators' site (the “central” site), because it was determined that this was not human subjects research—i.e., no data on human subjects were being requested, only administrative information about sites' guideline dissemination activities. Thus, IRB approval of this phase of data collection is not discussed any further in this paper. The survey data collected in phase (2) could not be anonymous because these data needed to be linked to data from phase (1) on guideline dissemination methods at each physician's site, as well as to each physician's prescription data from phase (3). Therefore, the investigators determined that written informed consent was needed for phases (2) and (3) of the study, requiring IRB review and approval from each of the 42 participating sites (in addition to the central site). Because of the observational nature of the study, the only risk described by the investigators in their IRB applications was that of potential loss of confidentiality of the data. The investigators explained in the consent form, “In the event of a breach of security, it is possible that someone could view your survey responses and assess your use of practice guidelines.” The consent form also included a detailed description of the security precautions that would be taken to try to minimize this risk, including the assertion that personal identifiers would not be included in any of the databases that included the survey or prescription data.
The process of obtaining IRB approval consisted of the following steps:
The first, quantitative, result was obtained by reviewing the activities and assignments of the study personnel to estimate the amount of time and effort needed to address IRB compliance in this study. Between May 2001 and December 2003 (19 months), 4,680 hours (or the equivalent of 1.4 full-time staff for that time period) were expended in addressing IRB-related issues in this study. The median time to IRB approval at the 43 sites was 286 days, with a minimum of 52 days and a maximum of 798 days. (The median time to approval for the 10 sites that granted expedited review was 289 days, ranging from 127 to 546; the median time from submission to first review was 1 week shorter for the 10 expedited review sites, but the difference was not statistically significant.) The total time requirement consisted of the following three major steps: (1) locating a site PI and preparing the application; (2) initial review of the application by the IRB; and (3) revisions to the IRB application. The time requirements for these three steps are shown in Table 1.
The qualitative analysis focused on identifying the processes involved in each of the three major steps, to better understand the reasons for the extensive time requirements. Four themes emerged from the qualitative content analysis of field notes as causes of that expenditure of staff time and work:
Recruiting and retaining local PIs required substantial time and effort. Changes in the local PI occurred three times in one site, and at least once in 21 percent of the sites. Multiple e-mails and calls to the local PI were often required to obtain all necessary information and signatures. On the other hand, local PIs at approximately 25 percent of the sites volunteered their time to assist with the application submission process, even though there was no expectation that they do so (by design there was no local data collection and all documents were prepared and submitted by the central site). They helped in ways such as submitting the forms after the central site had completed them, reviewing the forms to make certain no sections were overlooked, or sending correspondence from the IRB without solicitation from the central site.
Individual IRBs varied widely in the standards they applied to determine the level of review required. One exempted the study from IRB review, 10 granted expedited or brief reviews, 31 required full review, and one disapproved the study, citing the concern that if data confidentiality was breached, a physician's supervisor could use the data to evaluate his/her performance in adhering to practice guidelines.
Consent requirements for the physician survey varied as well. (Waivers of informed consent for patient data were requested and obtained from all sites.) Five sites waived the requirement for a written consent form, consistent with the established precedent that the return of a questionnaire that clearly states the purpose of the research and the voluntary nature of participation is acceptable as implied consent, provided the persons surveyed are competent adults in a population not vulnerable to potential coercion (Office of Human Research Protections 2003). Twenty-three IRBs required that certain sections of their standard consent forms be completed, even as they acknowledged that the sections did not apply to our particular survey. For example, many required that the consent document inform physicians responding to the questionnaire of how they could obtain health care for injuries suffered as a result of study participation. Five sites initially required that physicians complete HIPAA releases as well, even though no health information of any kind was asked of them; four of these sites eventually relented, but one did not.
Most IRBs returned applications for revision, requiring changes to consent procedures, study protocols, and forms. Especially in the beginning of the study, this feedback was often helpful in refining the protocol and clarifying the consent form. However, revisions continued to be requested at an undiminished rate even late in the recruitment phase of the study, when the study protocols and consent forms had been refined through revisions at multiple previous centers. At least one resubmission was required at 76 percent (31 of 41) of the sites, and three or more (up to six) resubmissions were needed to secure IRB approval from 15 percent (6 of 41) of the sites. The types of revisions required by the sites were categorized into editorial versus procedural revisions. Editorial revisions include changes in wording to the consent or clarifications in the protocol, while procedural revisions include changes in the actual procedures described in the protocol. All sites that requested revisions requested editorial revisions; indeed, nine sites each requested more than 10 editorial revisions apiece. Procedural revisions were required in only 12 percent (5 of 41) of the sites. Of the five procedural revisions requested, three were requests to eliminate the consent form and use a cover letter instead. No discernible patterns in the specifics of the editorial revisions emerged; they comprised a wide range of requests for deleting or adding sentences or paragraphs, phrasing, tense, and word choice.
IRBs varied widely in their response times to submissions and resubmissions. They also varied widely in their office procedures. Delays frequently occurred because the participating sites (including IRB staff or the local PI) did not respond in an accurate or timely manner to requests or questions from the central site. It appeared that many, if not most, of these delays occurred because local-IRB procedures are designed for communicating with local PIs, not with staff from an outside site. However, staff at the central site completed all forms and responded to all IRB concerns for the convenience of the local PIs, who had no substantive role in the research, nor any clerical or secretarial support resources. Direct communication with the central site staff would have reduced some of the delays, and was intended to alleviate the burden on the local PIs, but many IRBs insisted upon communicating only with the local PI, who then had to relay the information to the central site staff. Simple administrative errors at local IRBs added further delays. Two sites required forms to be accessed and filled out online, on servers inaccessible outside their sites. Others lost paperwork, or did not notify the central site of decisions. The online-only Appendix is a copy of the log of contacts with one site; it is not the longest-delayed site in our sample, but provides a rich qualitative sample having many of the elements described above.
Two anomalous phenomena were observed during the study, which merit particular mention. In one case, the study was reviewed twice by a single IRB acting on behalf of two different sites, being approved as minimal risk with waiver of consent in one case and rejected pending revision and with formal written consent required in the other.
The other anomaly encountered was IRBs imposing requirements that increased risk to participants. The only risk posed by the study was disclosure of participants' survey responses and guideline adherence data, which could in theory affect their employment if it reached their supervisors. The study protocol minimized this risk by ensuring that all data were kept in a secured form at the central site, inaccessible to those who might know or have administrative authority over participating physicians. For this reason, all IRB applications contained the following statement:
The [central site] Research team will send a recruitment letter to each of the medical center's primary care providers, as described in the preceding section. The list of primary care providers is not confidential information; but the names of those who choose to participate and those who choose not to participate will be kept confidential. Only the [central site] study team will know this information. It will be maintained in a password-protected computer file and will not be revealed to anyone outside of the [central site] study team.
Additionally, the survey booklet that was sent to all participants contained the following statement mandated by the central site IRB:
No one in management at your VAMC has requested your participation and we will not disclose to anyone in your medical center whether or not you have participated.
All the IRBs approved these statements and survey booklets. Of the 31 sites that have had a renewal submitted as of May 2004, 12 (39 percent) IRBs requested to see a list of the participating physicians. We responded to all 12 sites explaining why we were requesting this requirement be waived and two sites still insisted that we send the list, directly violating the terms under which study subjects consented to participate. One IRB insisted that copies of the consent forms with names be kept in the office of the very administrative personnel to whom physicians' identities were not to be provided.
Three structural features of the current local-IRB approach to human subjects protection appear to account for the recurring phenomena and the anomalies we observed: the requirement for a local PI; the practice of each site reviewing a study and requiring changes independently of all others; and regulations not designed for observational health services research.
In a multicenter clinical trial, the local PI is truly an investigator, playing a central role in fitting the trial into unique local conditions and being accountable for the conduct of the intervention; significant funds flow to each site, and the local PI is both accountable for them and supported by them. Observational health services research studies are very unlike clinical trials. No (or very limited) funds flow to sites, no intervention is being conducted that would need to be adapted to local conditions, and often there is nothing on site that needs supervision. It is unsurprising that recruiting and retaining local PIs is challenging when they are unlikely to be invested in the research and have so many other competing clinical and research concerns. The difficulty is magnified when local IRBs do not allow central staff in multicenter studies to manage tasks that could make the local PI's work easier. In addition, local PIs are placed in a difficult ethical position when they are asked to take responsibility for a research project over which they have minimal (if any) control. Perhaps most important of all, having a local PI offers no substantive protection for human subjects; the ultimate responsibility for observational health services research rests with the study team at the central site, who are responsible for designing and implementing the study protocol, and for maintaining and analyzing all study data.
The practice of each site reviewing our study and requiring changes independently of one another had the greatest consequences for us, due in part to the inconsistency of reviews across IRBs: the results ranged from waiver to outright rejection. Multiple review also resulted in added costs and delays associated with administrative errors. Every administrative process, no matter how conscientiously staffed, has an error rate. Inevitably the more people and offices involved, the more simple, administrative errors (lost forms, etc.) can be expected.
The costliest consequence of multiple review was because of the apparent irresistibility of editing. Our observations correspond to published observations from the U.K., where local IRBs continue to require global editorial changes even though directed by national governmental guidelines to request changes only to address local conditions (Burman et al. 2001). None of the editorial revisions requested by the IRBs in our study addressed any special local cultural or language issues. A number of the early ones did result in a clearer, more readable consent form in general. Yet the personnel time that had to be devoted to multiple revisions and resubmissions, even well into the project when the consent had been extensively edited already, consumed a substantial share of the project's budget and put it far behind schedule. Seeking—and incorporating—revisions from 43 different sites may not be the most efficient means of improving the readability of consent forms.
Enforcement of regulations not applicable to observational health services research was widely prevalent among this sample of IRBs, and was costly in staff time. The risk associated with observational health services research is that of disclosure of potentially sensitive health or other information. In contrast to the risks of adverse outcomes in experimental trials and the documented occurrences of abuses, the risk of observational health services research appears to be hypothetical: we are aware of no instances of harm from unauthorized disclosures in the course of observational health services research, and to date the Health Privacy Project at Georgetown University (http://www.healthprivacy.org) has documented none. Observational health services research not involving sensitive information (e.g., HIV status or substance abuse) or potentially psychologically traumatic events typically matches well the definition of minimal risk research, for which a brief review is appropriate under federal guidelines (http://www.dhhs.gov/ohrp/) (National Bioethics Advisory Commission 2001; Office of Human Research Protections 2003).
The multisite study in this case was specifically designed to meet the criterion of “no more than minimal risk” under CFR 46.110 (b) and 46.102 (i): physicians' practices were examined with identifiable data, but the results were kept confidential, so that the impact on physicians was markedly less than the routine administration of their clinics, where their practice profiles were regularly collected and provided to their supervisors. In other words, “the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life” (CFR 46.102 (i)). Nonetheless, only a minority of IRBs dealt with it under regulations and guidelines applicable to the review of minimal risk studies, in spite of guidelines and expert advice (National Bioethics Advisory Commission 2001; Annas 2002) urging IRBs to scale their review and monitoring activities to the risk and complexity of projects, and to focus on protecting participants rather than universally applying the most stringent rules. The majority of IRBs conducted full-scale reviews and many imposed all the usual regulatory requirements for clinical research, including requirements such as provision for medical care for injury during the study and HIPAA consent for physicians, even though the IRBs themselves acknowledged to us the irrelevancy to our study of what they were requiring.
Our findings raise issues not only of cost and delay because of imposition of unnecessary protections used in clinical research, but also of the quality of human subjects protection because of overlooking or compromising needed safeguards specific to health services research. One IRB granted the multisite study exemption, for which it did not qualify; 12 asked at renewal for names of responding physicians, which would have increased risk and violated the terms under which human subjects had consented to participate; and one abridged individual autonomy by deciding that physicians could not make the informed choice to participate.
As our study shows, the mismatch between current IRB practices and the characteristics of observational health services research can result in large additional costs, delays, and even in actions that impair rather than support protection of human subjects. These results are likely generalizable, as the majority of the IRBs involved were university based and not a part of the VA (24 versus 19). We observed no qualitative differences in procedures or requirements, nor statistically significant differences in timeliness or numbers of revisions, between academic and VA IRBs.
Several suggestions for modifications of the local-IRB system have been advanced (Wolf, Croughan, and Lo 2002), often involving education and dissemination of guidelines. However, the U.K. data (Burman et al. 2001) on IRBs and the large body of medical practice change literature (Davis 1998; Cabana et al. 1999) suggest that it is unlikely that educational and guidelines-dissemination efforts, by themselves, will have a significant impact on the costs or consistency of the multiple local-IRB system.
Either cooperative review between multiple IRBs or replacement of the current system of multiple independent local-IRB review with a central IRB plus local opt-out could, if properly implemented, remedy most of the cost and quality problems we observed. Under such a system local sites could agree or decline to participate based on their own review, but could not require modifications of the protocol or documents approved by the primary IRB, nor condition their approval on changes. For it to be effective the system would have to effectively prevent local IRBs from yielding to the temptation of requesting editorial revisions. The National Cancer Center's system of central primary review and facilitated local reviews using the primary review as a starting point, rather than de novo reviews at each site, may be a valuable model (Christian et al. 2002).
Further, having all sites take advantage of a designated IRB with expertise in health services research would bring to the process the experience and specific understanding to properly protect human subjects in such studies. An IRB with expertise in the specific form of research being evaluated could be expected to more often apply the correct level of review, and to avoid imposing requirements that, while needed to protect human subjects in clinical trials, are not applicable to observational studies and only add burdens that discourage observational research on health services. Such an IRB would also be likely to offer superior protection of human subjects in observational studies. (Such an IRB would not have to be nationally or regionally designated; it might well be simply selected democratically from among the involved sites for its specific expertise.) Finally, with the assistance of paid, qualified staff, such a committee could do a more efficient and effective job of revising consent forms to improve their readability, rather than delegating this responsibility to local committees, who may or may not have this expertise.
An intermediate alternative between the status quo and central review would be education and feedback for IRBs, especially regarding the assessment of risks and benefits of observational research. Education alone, even when quite explicit and directive, is unlikely to produce significant change (as seen in the U.K. example) (Burman et al. 2001). However, if IRBs submitted a regular random sample of their reviews to either a central organization or a set of their peer IRBs for feedback, unintended variation might be reduced without compromise of appropriate consideration of local conditions. Alternatively (or in addition), HHS could provide more extensive and concrete guidance on the interpretation of the definition of “minimal” risk research, perhaps through the use of case studies.
Finally, IRBs should restructure their procedures to eliminate the requirement for a local PI when their institution participates in multisite, observational health services research studies. The burden on busy local clinicians' time and the costs, logistical complications, and delays in the research bring no increment in the protection of human subjects.
The following supplementary material for this article is available online:
Case Illustration of Logistical Complications, “Site X”
All the authors have read the Conflict of Interest form. They have no conflict of interest. However, Leon Wyszewianski is a member of the Health Sciences Institutional Review Board and Lee Green is a member of the Medical School Institutional Review Board, both at the University of Michigan.