|Home | About | Journals | Submit | Contact Us | Français|
The following paper examines the issue of whether the current system for ethics review of multisite health services research protocols is adequate, or whether there exist alternative methods that should be considered.
(1) Investigators at different sites in a multisite project often have very different experiences with respect to the requirements and requests of the review board. Other problems include the waste of time and resources spent on document preparation for review boards, and delays in the commencement of research activities. (2) There are several possible reasons why there is variability in ethics review. These include the absence of standardized forms, differences in the background and experiences of board members, the influence of institutional or professional culture, and regional thinking. (3) Given the limited benefits derived from the variability in recommendations of multiple boards and the numerous problems encountered in seeking ethics approval from multiple boards suggest that some sort of reform is in order.
The increasing number of multisite, health services research studies calls for a centralized system of ethics review. The local review model is simply not conducive to multisite studies, and jeopardizes the integrity of the research process. Centralized multisite review boards, together with standardized documents and procedure, electronic access to documentation, and training for board members are all possible solutions. Changes to the current system are necessary not only to facilitate the conduct of multisite research, but also to preserve the integrity of the ethics approval process in general.
Research is changing. Recent years have seen a shift away from studies performed at single academic centers to larger, multisite projects involving numerous institutions in disparate geographic locations. The number of citations on PubMed for multicenter studies increased by 1.6–3 fold for each 5-year interval between 1985 and 1999 (McWilliams et al. 2003). While the shift is especially true in medicine, where clinical trials are often conducted as multicenter studies, the multisite design is of increasing importance in other fields, including health services research. A multisite study provides investigators with the opportunity to research different programs using a shared study design and methodology. Different sites investigate different services or programs, but have a common protocol. As a result, it becomes possible to measure “the same outcomes with the same instruments using the same timeframe across differing programs at multiple sites” (Dewa et al. 2002). Multisite studies are valuable, because the involvement of different centers means that the results obtained may be more generalizable than those of a single-center study. The existence of the common research protocol renders the outcomes of the various sites comparable.
Multisite protocols, like all research, must be approved by local institutional review boards (IRBs) before investigators can proceed. This means that ethics approval must be sought separately from research ethics boards at each site in a multisite study. As well, studies affiliated with both hospitals and universities must seek ethics approval at both institutions. Some have questioned the effectiveness of this system. Christian et al. (2002) state that the effectiveness of IRBs has been undermined because of the IRB system's failure to adapt to the changing research environment. Indeed, the current procedure for research ethics review, which involves seeking out ethics approval from each individual local committee, is not very conducive to collaborative, multisite research. The process of obtaining ethics review at multiple sites can be a daunting task, consuming time, money, and energy. One study in the U.K. indicated that extra funding had to be sought to cover the cost of applying to 125 local ethics review boards (Tully et al. 2000) The time, effort, and valuable research dollars spent on obtaining ethics review for a multisite project can be a source of irritation and resentment for many investigators (Personal communication 2003). They do not feel the extra administrative hurdles contribute toward the protection of research subjects, as requests from individual boards may differ considerably.
Some even go so far as to accuse the process of slowing the improvement and advancement of health care services by impeding project implementation. The inconsistency in ethics review, then, while a potential problem in all research, is especially problematic for multisite research. Inconsistency can result in subjects at different sites being afforded varying levels of protection. Without a standardized process for the interpretation of research ethics guidelines by IRBs, it is possible that subjects at Site A are better protected than their counterparts at Site B. Further, the money spent on securing multisite ethics approval does not represent the best use of research funds. There is concern, then, that large amounts of research funds—and therefore taxpayer dollars—are being squandered on repetitive, inconsistent, and unnecessary multiple ethics review processes. Money diverted to potentially unnecessary IRB review translates into less money spent on an actual study objective of improving health care services, raising questions regarding the ethics of such an opportunity cost. With the misdirected focus arising from the amount of money, manpower, and time required to seek multisite ethics review, some authors have pondered the question as to whether the system may even present an “unethical barrier” to potentially beneficial research activities (Tully et al. 2000).
The problems associated with multisite IRB review are of particular relevance to health services research, as multisite studies are of increasing importance in this field. Weinberger et al. (2001), for example, write that multisite randomized controlled trials in health services research offer “numerous advantages” over single-site studies. First, multisite studies enhance external validity, increasing the generalizability of research results. Second, when the subject of the research is a condition with a “low incidence or prevalence, small event rate in the outcome … and/or large variation in the distribution of cost,” a multisite study may be the only way to assure the requisite statistical power (Weinberger et al. 2001, p. 628). Finally, in a multisite study, the necessary sample size can be gathered faster. Weinberger et al. explain that this faster recruitment has a further advantage—since the study can be completed more rapidly, the “timeliness of the findings is enhanced.” They emphasize that this is of special importance in health services research, where investigators are frequently asked by health care organizations and policymakers to provide them with results as quickly as possible.
This last point, moreover, underscores why slow and inefficient ethics review for multisite research is of particular concern to health services researchers. The need for rapid dissemination of results to private and public entities for purposes of health care policy making makes speed a pressing factor for investigators on health services research projects. Unnecessary delays caused by a potentially ineffective multisite ethics review system can therefore affect the timeliness of health services research, with effects on subsequent health care policy and decision making.
The following paper examines this issue of whether the current system for ethics review of multisite research is adequate, or whether there exist alternative methods that should be considered. First, the available literature on this matter will be examined, including empirical studies describing the variability of, and problems with, multisite research ethics review. Second, reasons for variability in research ethics board review will be explored. Finally, potential solutions to the problem of multisite ethics review will be considered, including the implementation of special multisite review boards, mechanisms to facilitate communication and cooperation between ethics boards, and the notion of education or training and certification for ethics board members.
Several studies have looked at the problems arising from the review of multisite projects by multiple local ethics review boards. Recurring themes include delays and inconsistencies associated with multisite reviews that result in questionable benefits but extract high-opportunity costs in terms of knowledge, time, and money. Table 1 presents a summary of these studies.
In their investigation, Ah-See et al. (1998) sent letters to 19 local research ethics committees describing their multisite questionnaire survey designed to collect sociodemographic details and lifestyle information from individuals under 40 years with oral or oropharyngeal cancer. They received a variety of responses. One ethics committee out of the 19 approved the initial letter indicating that it was acceptable in the judgment of one group of reviewers. In contrast, 15 committees required completion of an application form unique to their committees; none offered electronic application forms. Upon review, 10 committees required changes, with most requiring more than one change. The majority of the requested revisions appeared to be capricious; none questioned the study's scientific merit nor whether its protocol was ethical. Instead, the majority of the revisions appeared to be stylistic in nature. For example, the types of changes the researchers were asked to make included changes to the wording of single paragraphs and the project's title. In addition, two committees required the researchers to resubmit their applications in their entirety incorporating the recommended changes on the new forms that the committee had developed in the meantime. It took over 3 months, on average, to secure final approval from all committees. Overall, approval time from the 19 committees ranged from 39 to 182 days (median 78 days).
In Middle et al.'s (1995) study on birth weight and child development, 118 of the 145 committees to which the investigators applied had unique application forms—that is, they all differed from one another. Three months after the applications were submitted to the 145 boards, only 78 percent (n=113) had responded. Almost three-quarters (n=82) approved the study with no objections. Although the majority of the boards agreed that there were no ethical concerns associated with the study and that it had scientific merit, there was a minority that expressed concern over a variety of different issues, including concern about the cost of the study, its objectives, confidentiality, consent, and the wording of the information sheets and the questionnaire itself.
As a result, the researchers were required to resubmit their protocol to almost a quarter of the committees and accommodate their 31 various application forms and questions. The committees' decisions to withhold approval and require re-submission were predicated on the belief that the recommendations were essential to ensure the quality of the study and to protect the study participants. This raises the question about the degree to which the research protocol was altered after all of the suggested revisions were made. If the revisions were indeed critical, there is an argument in favor of the protocols being re-reviewed by the 82 committees that originally gave their approval.
Yet, findings by Burman et al. (2003) question the extent to which simultaneous review from a number of IRBs improves the quality of the research and protects the rights of the study participants. They examined the effects of local ethics review at 25 different sites in a multisite project. They found a median of 46.5 changes were made per consent form. The majority of these were small changes of less than one sentence—many were related to spelling or grammar. Further, the changes requested by the IRBs often had the effect of increasing, rather than decreasing, the reading level of the consent forms, making them more difficult to read. As well, of 50 locally approved consent forms, approximately 40 percent had a reading level considered “high,” as measured by a Flesch–Kincaid reading grade level of greater than 8.0.1 In the end, the researchers observed that multisite ethics review was a time-consuming process that took more than 3 months to complete. They concluded that the revised consent forms that incorporated the ethics board-recommended changes “were generally longer, more complex (i.e., had an increased reading grade level), and often contained errors.” In other words, the language became less accessible to potential study participants.
Silverman, Hull, and Sugarman (2001) observed similar results. They examined whether there is variability among IRBs; decisions within the context of a multisite investigation. After the participating institutions received ethics approval for a multisite study protocol, the individual IRBs were contacted and asked to complete a survey on their approval process. The results indicated variability in the approval and decision making processes of the IRBs. One IRB waived the requirement for informed consent; five IRBs allowed consent to be given via the telephone; and three IRBs permitted the recruitment of prisoners as research subjects. Of the 16 IRBs who participated in the study, only three (ensured the consent form) contained the eight basic elements of informed consent required by U.S. federal regulations.2 With respect to the remaining 13, six forms were missing one element, four were missing two elements, two were missing four elements, and one was missing three elements. The most common missing element was Element 4, disclosure of alternatives to participation. Further, the four consent forms did not indicate that the subject may not receive any direct benefits from participation (Element 3). These results underscore the inconsistencies in the various reviews, and further questions the extent to which rights are protected.
Finally, McWilliams et al. (2003) documented the variability among local IRBs with respect to the review of a multicenter genetic epidemiology study. A seven-item survey inquiring about their experiences with their IRB was sent to the 42 sites participating in a multisite genetic epidemiological investigation on cystic fibrosis. Thirty-one (74 percent) of the sites replied. Their responses indicated the ethics review by local IRBs were highly variable. All the IRBs used different risk evaluation criteria. The number of consent forms required by the IRBs varied from one to four, with 15 IRBs (48 percent) requiring at least two-consent forms, and 10 (32 percent) not requiring consent to be obtained from children. Twenty-four (77 percent) of the sites required full IRB reviews while seven (23 percent) felt the project was suitable for expedited review. McWilliams et al. (2003, p. 360) concluded, “Lack of uniformity in the review process creates uneven human subjects protection and incurs considerable inefficiency.”
Inconsistency and delay in multisite ethics review is not merely frustrating, but may be hazardous with respect to improving health and preventing injury or illness. Jamrozik and Kolybaba (1999) describe their experience with respect to a multisite study examining the effects of widespread use of a prostate specific antigen test for diagnosing prostate cancer. The researchers were concerned that the sharp increase in incidence following the introduction of the antigen test had “little concomitant change in mortality, and major consequences for the many more men who had to live with the diagnosis and for the health services that had to be provided to them” (Jamrozik and Kolybaba 1999, p. 26). The study was delayed because of inconsistent ethics review. Jamrozik and Kolybaba assert that this lack of consistency in decisions reached by different research ethics review boards reviewing the same proposal risked the delay or obstructed the discovery of avoidable threats to health (Jamrozik and Kolybaba 1999, p. 26).
In the Middle et al. (1995) study, the authors estimated that, in total, preparing the documents for ethics review required 7–8 weeks of staff time. In Ah-See et al.'s (1998) project involving 19 sites, two committees required the investigators attend their research ethics meeting to discuss the protocol and answer questions. This required investigators to travel 156 miles to attend a 30-minute committee meeting. Burman et al. (2003) estimated that the process of getting local ethics approval from 25 different committees took a median of 30 hours of staff time. The evidence suggests that the preparation necessary for multiple ethics review can be labor-intensive, and is perhaps disproportionate to the benefits accrued from multisite ethics review. Indeed, delays at different sites of a multisite study can have a ripple effect. If each site has a different delay in its ethics review, sites may either begin data collection at different times with the effect of having varying finishing points, or else all sites can wait until all ethics review is complete to begin data collection. Either way, this has a potential impact on the timeliness of knowledge dissemination, as well as on research budgets with respect to the payment of staff unoccupied during delays. Moreover, as changes requested by IRBs can often be unrelated to ethics or research excellence, but instead may focus on minutiae such as mistakes or questions related to spelling or grammar (Burman et al. 2003), it is questionable whether the time and effort necessary to prepare for multisite ethics review is proportionate to its benefits.
In responding to their 145 research ethics committees, Middle et al. (1995) estimated that they were required to submit 1,095 copies of the protocol and 1,116 forms. The total cost of photocopying, staff time, and postage was approximately £4,606. In another British study (Tully et al. 2000), the total number of pages used for the applications was 105, 888.
The total cost of preparing the applications—including paper, photocopying, and postage—was £6,132.90, or $10,286.83 (USD). In fact, extra funding had to be sought to cover the cost of applying to 125 different ethics boards. Further, the authors of one Australian multisite investigation estimated that it cost them $20,000 (AUD) and a year's worth of work to obtain approval from 15 different ethics review boards (Smith et al. 1994).
The studies indicate the problems encountered with respect to ethics approval for a multisite investigation are fairly consistent. Substantive problems, related to variability in approval criteria and local review board interpretation, are prevalent. Often, investigators at the different sites in the multisite project have very different experiences with respect to the requirements and requests of the institutional review board. Other problems, include the waste of time and resources spent on photocopying and other means of preparation of documents for the ethics board, and delays in the commencement of research activities. This suggests the large sums of money spent on securing multisite ethics approval may not represent the best use of research funds, and raise concerns that large amounts of research funds—and thus taxpayer dollars—are being misspent on redundant multiple ethics review processes. While corners should not be cut when it comes to the ethics of human subjects research, the efficiency and cost effectiveness of such a system must be questioned. For example, Dunn, Arscott, and Mann (2000) suggest that many local research ethics committee applications could be dealt with by the committee chairperson or a subcommittee, which could reduce administrative costs. This evidence supporting this type of solution will be discussed in greater detail below.
There are several possible reasons for variability in the review process among ethics boards. Besides the absence of standard forms, there are a number of factors that may influence IRB decision making. First and foremost, every ethics board is made up of different individuals of varying backgrounds and experiences. These individual factors may have a large impact on how a protocol will proceed through review. While countries such as Canada, the U.S., the U.K., and Australia all have guidelines in place that describe the composition and functioning of research ethics committees, many of these standards, when operationalized, leave room for some interpretation. How standards or guidelines are actually put into play is likely determined in large part by the perceptions and experiences of the people sitting on a board. For example, members of one board may see their role as being primarily “micro-level”—that is, their duty is to carefully and meticulously review documentation and forms for errors or inconsistencies. Members of another board, however, may view their role at more of a “big picture” level, tending to focus more on adherence to general principles. As well, more basic individual characteristics or traits of members—even something as simple as temperament—may influence the process. Also important is the influence of institutional or professional culture. Who chairs or dominates a research ethics review board, for example, may effect how a review proceeds. Under U.S. law, an IRB must be composed of five members, including individuals with expertise-in law, individuals with scientific expertise, and members of the community (Code of Federal Regulations 1994). If the dominant board member is a research physician, the process and outcomes may be different than if the board were led by a lawyer or humanities-trained ethicist. Each profession represented on an ethics board brings to the table not only the expertise of that profession, but the culture, mores, and norms that accompany it. Lawyers may focus more on the wording of consent forms or potential liability issues in a given protocol, for example, whereas a physician may be more concerned with the clinical aspects of the protocol and its contribution to science and medicine. The institutional location of a board may also be an important determinant in how the review process is carried out. The ethics board of a hospital that carries out a considerable amount of cancer research, for example, may approach a protocol related to cancer differently than a board at a hospital where very little such research is conducted.
Regional thinking may also have an impact on the decision making processes of a research ethics board. Different provinces, states, or cities may have different approaches to, or perspectives on, issues that can affect research ethics boards' decision making. While this is important in some respects—protocols should reflect local differences and be sensitive to local issues—it should not be permitted to jeopardize the overall integrity of research ethics review, or the general principles of research ethics.
The numerous problems encountered in obtaining ethics approval from local research ethics boards in the context of a multisite investigation suggest that some sort of reform is in order. The local review model simply is not conducive to multisite studies, and jeopardizes the efficiency and integrity of the approval process. A possible solution for reform, such as the one outlined below, should include the following components: standardized multisite ethics review boards; better standards with respect to forms and procedure for local ethics review boards; and the creation of central databases for multisite projects that can be accessed by all local ethics review boards. Each component will be discussed here in turn.
The most frequently proposed solution to the problems related to multisite ethics review is the creation of central, multisite ethics review boards that would be in charge of approving projects involving more than one institution. The role of local research ethics boards would then be reduced to matters strictly local in nature. Such a system was established in the U.K. in 1997 (Alberti 2000). Responding to concerns about the ethics review process for multisite studies, “multicenter” research ethics committees were created on a regional basis throughout the country. The role of local committees was accordingly reduced to that of reviewing protocols for matters that might affect their acceptability from a local perspective only.
A similar model is being tested by the U.S. National Cancer Institute (NCI), in collaboration with the Office for Human Research Protection (Christian et al. 2002). A central review board first provides expert review of NCI sponsored trials at a national level. Following approval, protocols are passed on to local IRBs, who are then able to engage in a facilitated review process. A facilitated review may be carried out by a local IRB chairperson, or by a subcommittee after the central IRB documents have been reviewed. The local facilitated review committee is responsible for ensuring that the research is performed “safely and appropriately” (Christian et al. 2002). They must make sure that the local research environment is suited to the particular protocol and that it adheres to institutional standards with respect to research ethics and conduct. It is also their responsibility to review adverse events occurring at the local institution, and has in place a method for handing complaints. The facilitated review approach is intended to reduce the “vast amounts of duplication of effort,” and to permit local committees to focus on matters which are local in nature only (Christian et al. 2002). It is believed that this process will speed up the approval process considerably for multisite projects.
With respect to the advantages and disadvantages of such a model, the evidence from the U.K. provides mixed reviews (Alberti 2000). While studies suggest that fast-track reviewing by local review boards can, indeed, accelerate the approval process, it has also been demonstrated that protocols approved by the regional multicenter boards are in many cases still experiencing delays at the local review level. In a study by Tully et al. (2000), nine of 125 boards had not approved a regionally approved study after 6 months. As well, some local committees continued to ask for changes that were not local in nature.
However, it should be noted that many local boards did not share common application forms. Alberti (2000) asserts that a common form for multicenter ethics committees is essential for the new system to function properly. It would appear, then, that standardization with respect to forms and procedure must accompany any centralization of multisite ethics review. The British experience with multisite review seems to suggest that; overall, a system of centralized review with subsequent expedited local review can speed up the process of protocol approval. Writes Alberti: “So have multicenter research ethics committees worked? The answer must be a qualified yes, but further improvement is needed if we are to continue to perform timely and valuable multicenter research in the U.K.” (Alberti 2000, p. 1158).
With respect to the NCI pilot project, members of the central IRB were satisfied with the board's functioning during its first year. Christian et al. (2002) indicate that the expertise of the various members of the board “have resulted in a rich discussion…unmatched by many local IRBs.” The central board is also able to call on other experts at the NCI to ask questions during the review process—assistance not widely available to local IRBs. The pilot study has also led to the establishment of a “detailed communications plan” to facilitate necessary communication between the central ethics board and the local boards and investigators.
As indicated above, the lack of standardization with respect to application forms and review processes can prevent even a centralized multisite ethics review system from functioning optimally. In order to ensure the functioning of a multisite ethics review system, a single application form should be used uniformly by all local boards. Alberti (2000) suggests that a short form containing information that is relevant locally should be developed and sent electronically to the local ethics committees, thereby precluding the need to send “vast piles of papers.” In terms of procedure, Lux, Edwards, and Osborne (2000) make several suggestions they believe would enhance the U.K. system of multisite review, including the proposal that applications should be swiftly reviewed by executive committees of local boards, and that local boards should refrain from asking for minor changes that are not local in nature. Approval should not be stalled by a local research ethics board requesting revisions that do not address local matters.
Another promising means of handling multisite ethics review is to make use of new technology, including the Internet or databases. For example, all documents pertaining to ethics review could be submitted electronically to a central database. Then, committee members at particular research ethics boards could download whichever documents they need. For instance, if the proposal has already undergone extended ethics review at other institutions, the latest ethics review group might just need to have access to a few key documents rather than the entire submission. The NCI pilot study, for example, has developed a controlled-access website so that local investigators and ethics review boards can easily access documents produced by the central IRB (Christian et al., 2002).
As a complement to the above proposals for integrating multisite ethics review, standardized education or training and certification for members of IRB should be instituted to guarantee a minimum level of qualification among reviewers. Currently, most institutions do not require any sort of training to sit on a research ethics board, and reviewers are not tested as to their knowledge of the ethical and legal dimensions of research. Mandatory education sessions for potential board members would ensure that all reviewers be familiar with the ethical, legal, and regulatory standards for research in their jurisdiction, as well as the practical “how-to's” of research ethics review. Education and training would introduce a measure of uniformity to the ethics review process, allowing less room for variability. If all board members in a given jurisdiction are administered the same course in research ethics review, it should follow that there would be an increase in consistency in the review process and outcomes of different research ethics boards. Members will be guided by their standardized ethics training, leaving less room for the variability in interpretation and action that results from individual backgrounds, personal experiences, and local preferences. In addition to improving the efficiency of ethics review, ensuring IRBs operationalize research ethics guidelines in a uniform fashion decreases the potential for the differential protection of research subjects.
There are, of course, several challenges with respect to the reorganization of multisite ethics review. Local boards may be unwilling to relinquish their control over the review process—they may feel it is their duty to advocate for local patients and potential subjects.
Competitive attitudes may also become apparent should ethics review be centralized. It may be that local boards will feel the need to “outdo” the multisite boards, demonstrating that their ethics review process is more stringent. There may also be squabbling over what constitutes a “local” issue.
However, it is possible such territorialism can be mediated by emphasizing not only the potential for improved efficiency and quality of ethics review, but also by pointing out that such a system would result in a reduced workload for local review board members. Local, facilitated review would likely be carried out by the board Chair or by a subcommittee, thereby decreasing the amount of work and time members are required to put into multisite ethics review. This would also allow them to focus their attention on a few issues and giving them time to consider them in depth. As board members are often overwhelmed by the amount of work, this may prove to be an appealing proposition.
Another potential challenge to reforming the system is that it requires modifying the interaction between the researcher and the review committee. In the current system, the reviewer has the opportunity to directly interact with the committee to exchange thoughts and ideas as well as learn from the experience. A centralized review will necessitate making the review process one step removed from the researcher and potentially decrease the opportunity for interchange. However, current innovations in videoconferencing might be possible answers to preserving these face-to-face interactions.
The changing face of research suggests that reform is in order with respect to how research protocols are reviewed for ethics approval. The increasing number of multisite, health services research studies calls for a centralized system of ethics review, so as to minimize the time, money, and duplication of effort that may accompany a multisite ethics submission.
Centralized multisite review boards (either in the form of regional IRBs or a single central IRB like that developed by the NCI) together with standardized documents and procedure, electronic access to all necessary documentation, and training for board members are possible solutions to this growing problem. Changes to the current system are necessary not only to facilitate the conduct of multisite research, but also to preserve the integrity of the ethics approval process in general.
The authors are grateful for the helpful comments provided by Janet Durbin and two anonymous reviewers. This manuscript was written while Jennifer Gold was a graduate student trainee in the Centre for Addiction and Mental Health Health Systems Research and Consulting Unit's summer studentship program. Carolyn S. Dewa would like to acknowledge the support of the Ontario Ministry of Health and Long-Term Care's Career Scientist Award.
1This is a readability test designed to show how easy or difficult a text is to read. The Flesch–Kincaid Index uses the following formula: 0.39 × Average No. of words in sentences +11.8 × Average No. of syllables per word−15.59.
2The eight elements of informed consent as outlined in the Code of Federal Regulations are: (1) Disclosure of study purpose, expected duration, and procedures to be followed. (2) Description of any reasonably foreseeable risks or discomforts to the subject. (3) Description of benefits, to the subject and others. (4) Disclosures of alternatives to participation. (5) Assurances of confidentiality. (6) For research involving more than minimal risk, if research injury occurs, explanations as to whether compensation is available, and whether any medical treatments are available. (7) An explanation of whom to contact: (a) for answers to questions about the research (b) for questions about the research subjects' rights, and (c) in the event of a research-related injury. (8) Communication to potential subject of voluntariness of participation, and the ability to withdraw from the study.