|Home | About | Journals | Submit | Contact Us | Français|
We obtained data on Institutional Review Boards (IRBs) that review mental health–related applications (MHRAs) in a national survey of institutions with federally assured human research protection programs. Approximately 57% of IRBs review MHRAs, and among these a small percentage may not have mental health experts on their committees (5%). Moreover, mental health experts on IRB committees at high research volume institutions are carrying substantially greater workloads than their lower volume counterparts. In terms of committee demographics, more women (36%) are serving as IRB Chairs on committees that review MHRAs than expected from their representation on medical or university faculties; ethnic minority faculty have lower representation among Chairs than might be expected from their overall faculty representation. Our findings suggest the need for additional studies to (a) examine if the number of mental health experts on IRBs should be increased particularly among IRBs reviewing a high volume of MHRAs, (b) determine if the breadth of expertise among IRB mental health experts corresponds to the range of substantive and methodological approaches represented by the mental health protocols under review, and (c) examine if recruiting IRB scientific expertise from outside an institution, a more common practice among smaller research entities, impacts review quality.
In the U.S., empirical studies on research ethics typically concern the linkages among participants, research features, and ethical regulations; fewer studies examine larger systemic influences on review efficiency and quality, participant protection, and scientific outcomes (for example, Citro, Ilgen, & Marrett, 2003; Institute of Medicine, 2003, pp. 164–165). Systemic influences may include, for example, bureaucratic structure and management practices, workload (for example, volume and type), budget, and staff (for example, expertise and workforce size). In the present study, we assessed variation in IRB workload attributable to mental health–related applications (MHRAs).
The current report extends prior research reported in the accompanying article (Catania et al., 2008) on workload and institutional review board (IRB) composition to consider IRBs that review (MHRAs). Mental health–related research may place added demands on the IRB system. That is, MHRAs often concern research populations with multiple vulnerabilities, complex methodologies, and challenging ethical considerations (for example, DuBois, 2008). Consequently, MHRAs may contribute disproportionately to IRB workload and require expertise beyond that available on the typical IRB committee. These increased demands may decrease IRB review quality and contribute to delays in the review process that, in turn, impact research costs and quality.
Mental health investigations intersect with the broader areas of social-behavioral science and medical clinical research. Over the past 30 years, social and behavioral science investigators have expressed concerns about lengthy review processes and poor review quality (Gray, Cooke, & Tannenbaum, 1978; AAUP, 2006; Gunsalus et al., 2007; Bledsoe et al., 2007; Hamburger, 2005; Gunsalus, 2004; Citro et al., 2003). For instance, a number of investigators have noted the tendency for IRBs to establish a higher bar for social-behavioral science protocols than is required under the regulations, which in turn increases IRB workload and delays in review (AAUP, 2006; Bledsoe et al., 2007; Hamburger, 2005; Gunsalus, 2004). A comprehensive report by the National Research Council’s (Citro et al., 2003, pp. 39–43) panel on Institutional Review Boards, Surveys, and Social Science Research documents the problematic effects large IRB workloads may have on the quality of review for social-behavioral protocols, including MHRAs. The NRC report also notes that the type and level of experts needed for review of social-behavioral research may exceed the pool available to local IRBs (also see Hamburger, 2005). For example, an application requesting approval for an ethnographic study of people with mental health problems living in public housing would require both methodological and substantive expertise. While these issues apply broadly to social behavioral research, empirical studies focused specifically on mental health–related research are needed. Mental health studies in particular may heighten IRB methodological concerns and uncertainties because of the potentially greater vulnerabilities of the study populations.
IRBs in the U.S. are composed of members from the parent institution, typically faculty, one of whom is the IRB Chairperson. In addition, IRBs are required to include membership from outside the institution that in some manner represents the broader community of research participants [45 CFR 46.107(d)]. Further, IRBs may recruit scientific expertise from outside the institution to supplement key areas. In the present study, we determined the extent to which IRBs reviewing MHRAs include mental health experts and non-institutional community representatives as members. In addition, we examined the demographic characteristics of IRB members. Mental health research frequently involves diverse demographic populations (gender, race/ethnicity, age) and these populations may be better understood by IRB members with shared backgrounds.
The sample frame for the current study is described in the accompanying article (Catania et al., 2008). In brief, a random stratified sample was obtained of 2,070 Institutional Review Board Organizations (managing organizations) provided by the U.S. Department of Health and Human Services’ Office for Human Research Protections in 2004. The sample was divided into two stratum or tiers. Tier One consisted of 120 managing organizations representing the top 100 institutions in terms of NIH funding for FY 2002. Tier Two included a sample of the remaining managing organizations (N = 274). The sampled managing organizations represented approximately 400 IRBs. After identifying IRBs that review MHRAs, a subsample of IRB Chairs (N = 85), over-sampling women and ethnic minority Chairs, was obtained in order to conduct a separate research investigation. The subsample of IRB Chairs provided additional background details on IRB members (see below). A detailed sampling report is available from the authors. All materials and procedures were approved by IRBs for the University of California, San Francisco, Oregon State University (Dr. Catania), and for our survey subcontractor, the Henne Group-San Francisco/New York (Independent Review Consulting Inc., San Rafael, CA).
The survey was conducted between October 2005 and June 2006. Survey procedures are described in an accompanying report (Catania et al., 2008).
As described in the accompanying article (Catania et al., 2008), administrative interviews, among other topics, asked administrators to (a) describe the workload of their managing organization, (b) identify which of their IRBs reviewed MHRAs (if an institution had multiple managing organizations, we further determined which organizations had IRBs reviewing MHRAs), (c) enumerate for each IRB under their managing organization the types and volume of protocols received in the past year including the number of MHRAs reviewed, and (d) describe demographic characteristics of Chairs of IRBs conducting mental health–related reviews (gender, race/ethnicity, age), (e) provide the number of IRB members with mental health expertise on these IRBs, and (f) describe the composition of each IRB committee in terms of total members per committee, number of non-institutional members, number of non-institutional members without a science background. A person with mental health expertise was defined as being “trained in a mental health field including psychiatry, psychology, social work, psychiatric nursing, or psychiatric social work and behavioral pediatrics … [who] may be an M.D., Ph.D., nurse, or hold a Master’s degree in a mental health field.” Our subsample of IRB Chairs provided additional data on the number of women and ethnic minority members of their respective committees.
Counts of numbers of applications were measured using bounded categories (0, 1–25, 26–50, 51–75, 76–100, 101–150, 151–200, 201–250, 251–300, 301+). These assessments, therefore, are estimates (see “Statistical Methods,” below). Administrators and/or staff did not typically have the resources to produce an actual count.
Definitions of an MHRA supplied to administrators during this interview process were as follows:
A mental health–related application might concern research on any population that has, or is at risk for, a mental health disorder. It may also include research on a mental health outcome related to some other type of illness or problem. Mental health disorders are defined here as those listed in the Diagnostic and Statistical Manual of Mental Disorders [Interviewer: If asked, we are talking specifically about DSM IV.] [Interviewer: If needed, read the following: Examples would include research on depression or anxiety disorders or research on depression or anxiety as outcomes of physical health problems or some type of life-event such as divorce.] [Interviewer: In helping respondents define mental health research, we want you to be more inclusive than exclusive; if the respondent has a question about a particular type of research being mental health–related research, write the question down and call Dr. Dolcini or Dr. Catania for comment, but continue the interview as if the type of research was mental health–related.]
Definitions of a non-institutional IRB member, and a non-institutional non-science IRB member were as follows.
By non-institutional members, we mean people who are not connected to the university (or corresponding parent institution) through employment or unpaid adjunct faculty positions. Non-institutional members might be community advocates or members of community-based organizations (CBOs). [The administrators should not be included in this number.]
Non-institutional members might be community advocates or members of community-based organizations who do not have science backgrounds, meaning they do not have graduate or professional training in basic or applied sciences of any kind.
As described in our accompanying article (Catania et al., 2008), we constructed two a priori workload stratification variables based on: (a) research volume, as indexed by the number of NIH grants received in FY 2002 (top 100 vs. all others), and (b) three institutional designations (colleges or universities, non-university affiliated health institutions, and other types of research institutions: governmental departments, independent research institutes, independent IRBs).
Although sample weights were developed to adjust for unequal probabilities of selection, because of sample size limitations and sample complexities at the level of IRBs, weights were not used when computations were made within a given sample stratum, or when the unit of analysis was the IRB instead of the overall managing organization. Data were analyzed using SPSS and SVY procedures in STATA release 9 (to adjust standard errors when using weighted data). Estimating the number of applications processed was accomplished by utilizing three values from each predetermined category (minimum, midpoint, and maximum). The highest frequency category, which is open-ended (301+ applications), was truncated at 350 for estimating purposes. It should be noted, therefore, that the maximum frequency values might be somewhat larger than presented. The application frequency categories were constructed based on our pilot interviews. Midpoints of the categories provide moderately conservative estimates and are used in all computations unless indicated otherwise. All other counts were assessed as simple frequency counts.
The 394 managing organizations in our sample reported overseeing approximately 400 IRBs. Extrapolating from these data, we have estimated that nationally there were 2,728 IRBs among the 2,070 managing organizations in operation in 2005/06 (Catania et al., 2008). Based on data from managing administrators, we found that approximately 57% (weighted data) of IRBs reviewed MHRAs although not necessarily in the past year (i.e., 1,556 IRBs nationally review MHRAs). The proportion of IRBs reviewing MHRAs by institutional strata is presented in Table 1. Tier One managing organizations and universities have a larger proportion of IRBs reviewing MHRAs than Tier Two or other research settings.
In comparing IRBs who did, versus did not, review MHRAs, we found that those reviewing MHRAs had significantly larger committees (Ms = 14.8 vs. 12.1, p = .001). There were no differences between IRBs reviewing MHRAs (vs. not reviewing) with respect to the number of non-institutional members (Ms = 2.8 vs. 2.9, p >.10) and non-institutional community members without a science background (Ms = 1.9 vs. 1.7, p >.10). Thus, IRBs reviewing MHRAs have more members recruited from the institutional pool of scientists, which may bring greater scientific expertise to bear on MHRAs. In this regard, we found that 95% of IRBs reviewing MHRAs have mental health expertise, with an average of 2.4 mental health experts per committee (median = 2.0). (The mean number of mental health experts was similar across institutional strata; all p’s >.10.)
Based on our sample data showing that 5% of IRBs reviewing MHRAs have no mental health expertise, we would estimate that nationally approximately 78 (5%) of 1,556 IRBs may be reviewing mental health protocols without the perspective or oversight of a mental health expert. We estimate that these committees may, in total, have reviewed approximately 1,014 MHRAs in the past year (unweighted data), but at a relatively low level (1–25 applications per IRB/past year). IRBs reviewing MHRAs without mental health expertise were primarily found among non-university-based health institutions (9/12 cases, 75%). Tier One institutions contained 42% of the cases (5/12), which is surprising given the presumed availability of mental health experts at larger health-focused research institutions. Universities, however, which typically have social-behavioral science departments, were less likely to report IRBs to be without a mental health expert (2/12, 17%).
We assessed the number of MHRA applications received as categorical frequencies. The category boundaries can be understood to represent low-and high-end estimates of the actual frequencies. Midpoints of the categories are used in all computations unless otherwise indicated. The number of MHRAs reviewed in the past year (N= 255 IRBs) is highly skewed, with 7% of IRBs that occasionally review MHRAs (n = 14) reporting no reviews in the past year, and nearly 70% (n = 146) reporting 1–25 reviews of MHRAs. Because of this skewed distribution, we dichotomized the number of MHRAs reviewed (the top quartile vs. the lower three quartiles) and then stratified workload by institutional strata. Approximately 30% of university IRBs reviewing MHRAs are carrying a heavier workload of MHRAs/year (top quartile) compared to non-university-health (9%) and “other” types of institutions (18.8%) (X2 = 10.4; p =.005). Among Tier One institutions, approximately 38% are carrying a heavier workload of MHRAs/year compared to Tier Two institutions (8%) (X2 = 25.5, p = .0001). In accord, we found that Tier One vs. Tier Two institutions had proportionately more IRBs reviewing MHRAs (76% vs. 52%). IRBs with heavier MHRA/year workloads were not more likely to have more mental health experts on their committees (Ms = 2.3 vs. 2.7 members for, respectively, heavier vs. lighter workload IRBs; p > .10), suggesting that, for IRBs at the high-volume end of the distribution, the resident mental health experts may be carrying a disproportionately larger workload.
The demographic characteristics of Chairs overseeing IRBs that review MHRAs are reported in Table 2 (N = 255). The majority of these Chairs are white males over the age of 50. Comparing across tiers and types of institutions, we found white males over age 50 to hold the majority of all Chair positions. Moreover, male and female Chairs have similar racial/ethnic patterns. Among women (men) Chairs of IRBs reviewing MHRAs, 89% (93%) were non-Hispanic White, 3% (2.5%) Asian-Pacific Islanders, 5% (6%) mixed race/other, and <1% (<1%) either African American or Hispanic.
From our subsample of IRB Chairs who oversee IRBs reviewing MHRAs (N = 85), we obtained demographic details on IRB committee membership. Committee size on our subsample of IRBs was smaller than that for the overall sample of IRBs reviewing MHRAs (M = 12.3 non–staff members vs. M= 14.8 non–staff members). On average, approximately 50% of IRB members were women (≤ 10% of IRBs had fewer than 30% women IRB members), and 14% belong to an ethnic/racial minority group in the United States (24% had no ethnic minority members). We contrasted Tier One and Tier Two IRBs and those residing in universities, non-university health facilities, and other types of research institutions on gender and race/ethnicity of membership. We found no significant differences in committee composition by gender and race/ethnicity across types of institutions (data not presented, available from first author).
The 2003 Institute of Medicine’s report calls for studies on how different research areas fare within our system of ethical review (IOM, 2003, p. 165). For instance, problems in ethical review may be related to the wide variation in methodological, conceptual, and topical content of social behavioral science studies. In this respect, mental health–related research often involves difficult ethical issues and participant protection concerns because such studies often include vulnerable populations, address sensitive topics, and have complex research consent and design considerations. The present investigation provides an initial look at some of these matters with regard to mental health–related applications (MHRAs). We examined issues of mental health expertise, workload, and committee diversity at descriptive and comparative levels in a national survey of IRBs reviewing MHRAs.
It is particularly important that IRBs have relevant scientific expertise when reviewing research with vulnerable populations. Our findings suggest that a small minority of IRBs may lack mental health expertise, particularly IRBs of non-university health institutions. However, in general, IRBs reviewing MHRAs were found to have significantly more members on average and more members drawn from their respective institutional scientific pools than IRBs not reviewing MHRAs. Large, more scientifically diverse IRBs may bring greater scientific expertise to bear on the review of MHRAs. Moreover, the vast majority (95%) of IRBs reviewing MHRAs reportedly have one or more mental health experts on their committees. These findings are consistent with the goal of achieving necessary ethical oversight of mental health research.
While the present study documented the extent of mental health expertise on IRBs, it did not address the quality of review by mental health experts or their potential role as advocates. It is relevant to consider whether the breadth of expertise among IRB mental health experts corresponds to the range of substantive and methodological approaches represented by the mental health research protocols under review. For instance, an IRB mental health expert whose primary occupation is clinical practice may approach MHRAs with a different framework for judging potential risks than an experienced mental health research investigator. Each may bring valuable expertise to the review process. IRB mental health experts may also be called upon to play a dual role on their committees as both scientific experts and mental health advocates. Mental health experts, many with clinical training, provide a degree of advocacy for the populations being studied, but the potential for conflicting interests to arise is a concern. Future studies may want to obtain more detailed information related to mental health experts serving on IRBs.
Even if mental health experts advocate for vulnerable populations, others have argued for the importance of including community representation on IRBs to better serve the principles of respect for autonomy and justice (DuBois, 2008, p. 42). Indeed community representation on IRBs is an important goal and a regulatory requirement [45 CFR 46.107(d)]. Prior findings from our study suggest that virtually all IRBs include community representatives as members (Catania et al., 2008). Community representatives, as advocates in IRB review, provide many potential benefits. As summarized by DuBois (2008, pp. 42–43), major benefits may include insights into the risks and benefits of research for particular populations, added scientific value (for example, in identifying problems/solutions for participant recruitment and retention), and insights into the strengths and weaknesses of standard human protection practices. However, research has not addressed the ability of community representatives to adequately represent the diverse communities being studied.
There is no clear-cut method of improving review quality that involves simply adding one expert or a single community representative. Although this approach may be better than nothing, there are other approaches that may hold greater promise. For instance, IRBs constructed to focus on particular areas of research methodology with substantially more expertise in those areas (for example, social/behavioral science and ethnography) is a viable alternative given sufficient resources (IRB Advisor, 2007, pp. 31–34). A similar approach may be useful in addressing the diverse range of research applications in the field of mental health research (for example, child mental health and prevention).
Within limitations, demographic diversity on IRB committees may lend itself to a broader understanding of participant populations. In this regard, IRB Chairs in our survey were remarkably homogenous demographically (64% male, 91% white). However, gender and ethnic minority representation on the broader membership of IRBs conducting MHRA reviews appears on the surface to be fairly good (50% women, 14% ethnic minorities). These patterns were found to hold across types of research institutions.
Our data provide a point of comparison with earlier surveys, allowing us to examine the changing landscape of ethnic minority and gender representation on IRB committees. With regard to IRB Chairpersons, Bell et al. (1998) report data from 1995 on IRB Chairs in general and found the majority of Chairs to be male and white (58% male, 92% white). More recent data, based on a survey of medical school IRBs, also show IRB Chairs to be relatively homogenous (73% male, and 81% white; Campbell et al., 2003). In comparison to our data on IRB Chairs of committees that review MHRAs, these past studies suggest that the present findings may be more broadly generalizable, and that there has been little change in demographic diversity among Chairs since 1995.
To more fully understand these findings, it is important to consider the base population of faculty from which IRB Chairs and members are typically drawn. Relevant data from the Association of American Universities 2001 survey shows that, among tenured/tenure track (full-and part-time) faculty, approximately 76% are male, and 82% are white (32 AAU universities; N = 51,344 faculty). Thus, Chairs of IRBs reviewing MHRAs generally correspond to the base population from which they are drawn with a tendency towards more women Chairs (24% base population vs. 36% of Chairs of IRBs reviewing MHRAs), and fewer ethnic minority Chairs being selected from their base populations (18% base population vs. 9% of Chairs of IRBs reviewing MHRAs) on these committees.
In contrast, overall membership on IRBs reviewing MHRAs is more diverse than the faculty-at-large with respect to women (18% base population vs. 50% on average for IRBs reviewing MHRAs). However, ethnic minority representation among committee membership is lower than that for the faculty-at-large (18% base population vs. 14% on average for IRBs reviewing MHRAs). These results suggest that women are more successfully recruited than ethnic minority faculty to serve on IRBs. This fact is underscored by virtue of all IRBs in the present study reporting that they have at least one female member, while 24% had no ethnic minority members. These differences may reflect the higher demands on ethnic minority faculty, who form a smaller overall pool of faculty than women, to serve on a large array of institutional committees as ethnic minority representatives. The potential “over-utilization” of minority faculty by parent institutions is a relevant issue to consider in terms of the needs of IRB committees to provide broader representation on committees of scientists from ethnic minority populations.
Workload is a primary concern of investigators examining IRB function (see, for example, Citro et al., 2003; Wagner & Barnett, 2000). Heavy workload may decrease review quality, increase research costs, and negatively influence the willingness of faculty to participate on IRBs. However, evidence of direct links between workload and these outcomes is lacking or anecdotal in nature. We previously reported that the difference in average per-member workload between larger and smaller institutions is substantial with Tier One institutional IRB workloads consistent with the requirements of a part-time job (Catania et al., 2008). With respect to MHRA workload, we found that Tier One IRBs, relative to Tier Two IRBs, process significantly more MHRAs (Bell et al.  report that IRBs conducting a higher volume of reviews overall in 1995 also reviewed more clinical mental health protocols). Moreover, among IRBs reviewing MHRAs, those with heavier workloads did not have more mental health experts on their committees to cover the heavier load. This finding suggests that there may be a supply shortage of mental health experts on some IRB committees. Correspondingly, these experts may be handling a disproportionately large and diverse workload, particularly at larger institutions.
High-volume IRBs confront a substantially larger per person workload relative to smaller volume IRBs, and this is no less true of IRBs reviewing MHRAs. In particular, IRB mental health experts on committees receiving a high volume of MHRAs are conducting substantially more work than their low-volume counterparts. In terms of IRB membership, a small percentage of IRBs reviewing mental health–related protocols may not have mental health experts on their committees. In terms of demographic representation, women are more likely to be serving as Chairs or members of IRBs reviewing MHRAs than would be indicated by their representation among faculty. Ethnic minority faculty are, however, somewhat under-represented as Chairs or members.
Mental health investigators should consider the possibility that their protocols may be reviewed by committees that lack appropriate mental health expertise, and thus may need to explain the rationale for their approach in more detail. IRBs should consider the need for education regarding methods and ethical concerns that arise in MHRAs particularly if it is not feasible for the IRB to cover the range of MHRA protocols with their available mental health experts.
This work was supported by NIMH grant RO1-MH064696 to Dr. Catania and Dr. Lo. We wish to thank Caroline Fisher for manuscript preparation and Michael O’Grady for interview work.
Joseph A. Catania, Ph.D., is a developmental psychologist and Professor of Public Health at Oregon State University, specializing in research on sensitive topics and vulnerable populations, and research methods including the impact of subject protection requirements on survey quality.
Bernard Lo, M.D., is Professor of Medicine and Director of the Program in Medical Ethics at University of California, San Francisco, and is interested in ethical issues in clinical research.
Leslie Wolf, J.D., is Associate Professor in the School of Law at Georgia State University, where she works primarily on issues in research ethics and the law.
M. Margaret Dolcini, Ph.D., is Associate Professor of Public Health at the University of Oregon and conducts research on health problems of adolescents and ethical issues in research with adolescent populations.
Lance Pollack, Ph.D., is a research specialist and codirector of the cancer center’s behavioral methods core at the University of California, San Francisco.
Judith C. Barker, Ph.D., is Professor in Medical Anthropology at the University of California, San Francisco, and examines cultural issues in health.
Stacey Wertlieb, M.P.H., served as a research associate on this project at the University of California, San Francisco.
Jeff Henne heads the Henne Group, a medical and social research company.
Joseph A. Catania, Oregon State University.
Bernard Lo, University of California, San Francisco.
Leslie E. Wolf, University of California, San Francisco.
M. Margaret Dolcini, Oregon State University.
Lance M. Pollack, University of California, San Francisco.
Judith C. Barker, University of California, San Francisco.
Stacey Wertlieb, The Henne Group.
Jeff Henne, The Henne Group.