|Home | About | Journals | Submit | Contact Us | Français|
We investigated the association between a U.S. National Institutes of Health (NIH) R01 applicant’s self-identified race or ethnicity and the probability of receiving an award by using data from the NIH IMPAC II grant database, the Thomson Reuters Web of Science, and other sources. Although proposals with strong priority scores were equally likely to be funded regardless of race, we find that Asians are 4 percentage points and black or African-American applicants are 13 percentage points less likely to receive NIH investigator-initiated research funding compared with whites. After controlling for the applicant’s educational background, country of origin, training, previous research awards, publication record, and employer characteristics, we find that black or African-American applicants remain 10 percentage points less likely than whites to be awarded NIH research funding. Our results suggest some leverage points for policy intervention.
The U.S. National Institutes of Health (NIH) has a long history of working to increase the diversity of its intramural and extramural biomedical research workforce, especially through programs such as Minority Access to Research Careers, Minority Biomedical Research Support, Research Centers at Minority Institutions, and Diversity Supplements. However, the effects of these programs on the pool of funded NIH grants have not been reported.
In fact, there have been relatively few studies on the racial and ethnic composition of populations that apply for federal research funding. Studies of race and ethnicity in science generally focus on differences in representation (1–3). A recent National Academies study (4) emphasized the need to increase the participation of minorities in science and engineering. In this study, the terms employed for race and ethnicity denote commonly used sociocultural classifications.
We hypothesized that scientists of different races and ethnicities with similar research records and affiliations would have similar likelihoods of being awarded research grants. To test this, we used data from the NIH IMPAC II (Information for Management, Planning, Analysis, and Coordination) grants data system consisting of application and investigator data for Research Project Grants (RPGs) submitted between FY 2000 and FY 2006 (5, 6). During the application process, investigators self-identified their race and ethnicity. Our analysis sample contains (Type 1) R01 grant applications; the R01 is the oldest and most widely used investigator-initiated research project grant. Our sample is limited to Ph.D. investigators at U.S. institutions and includes 83,188 applications with data available for most of the explanatory variables. Because investigators can submit multiple grants for different projects, this represents 40,069 unique investigators.
To receive NIH funding, applications are evaluated by a peer-review process that considers the significance, innovation, and approach of the grant application, the investigator(s), and the research environment. Applications determined to be meritorious are discussed in detail and scored. About half of all applications are scored. Among those that are scored, relative merit score, budgets and NIH institute priorities, which vary by year and by institute, determine which applications are funded.
Award success frequently depends on an iterative process of commentary, revision, and review, and many applications are resubmitted as revised or amended applications. To capture this activity, we collapsed revised or related applications that were received within 2 years of the original submission into one application for the purposes of determining the award probability for the application. Information about an application and its review was derived from the last funded or unfunded application submitted. Because individuals could have submitted more than one grant application during our sample time frame, we estimated all standard errors used in test statistics by treating the data for each applicant as a cluster. We supplemented information from IMPAC II with institutional information from the Department of Education Integrated Postsecondary Education Data System (IPEDS); investigator information from the NIH Doctoral Record File (DRF), which is derived from the National Science Foundation Survey of Earned Doctorates (SED), a census of doctorates awarded in the U.S. since 1974; and faculty data from the Association of American Medical Colleges (AAMC) Faculty Roster. Of the investigators in the sample set, 57% were matched to the DRF. Race and ethnicity were identified by using a combination of self-reported responses in IMPAC II, the DRF (7), and the Faculty Roster.
Although applicants self-identify race, ethnicity, and gender, this information does not appear in the application and is not available to the review committee, staff, or council. However, information contained in the application biosketch, such as the undergraduate or doctoral institution attended and applicant names, may in some cases be used as a proxy for race/ethnicity (8). For those investigators for whom we could not identify race or ethnicity, we included a dummy variable set to equal one in our analysis to account for missing data. Applications from Asian, black, Hispanic, and Native American investigators together are 21% of the total for NIH research grant opportunities and are represented in similar proportion both to medical school faculty and biomedical Ph.D. matriculants (9). In our study sample, applications from Asian investigators were 16.2%, blacks were 1.4%, Hispanics were 3.2%, Native Americans were 0.05%, whites were 69.9%, and other/unknown were 9.2% of total applications. Due to the small number of applications from Native Americans in the sample (N = 41), the analysis focuses on Asian, black, Hispanic, and white investigators.
We examined the relationships among race, priority score, and award probability. Applications with good scores were more likely to be funded, regardless of race/ethnicity (table S1 and fig. S1). The relatively small number of applications for some of the racial and ethnic groups, coupled with the large number of NIH institutes, did not allow us to evaluate award probabilities by institute.
There were significant differences in award probability by race and ethnicity (Fig. 1) in our sample. Compared with NIH R01 applications from white investigators, applications from black investigators were 13.2 percentage points less likely to be awarded (P < .001), and those from Asian investigators were 3.9 percentage points less likely to be awarded (P < .001). Table S2 shows that the award probabilities in our analysis sample were very similar to those found in the entire RPG application pool. Thus, for the entire RPG pool, if blacks had the same award probabilities as whites (36.4% for RPGs and 29.3% for R01s) one would expect to see 1071 RPG awards instead of 585, and 337 R01 awards instead of 185 in our analysis sample.
Table S3 shows the distribution of applications submitted by year. We did not include new proposals submitted in 2007 and 2008 because we cannot observe them for the additional 2 years needed to account for resubmission. In addition, changes after 2008—including (i) the new NIH scoring system implemented in 2009 and (ii) the impact of funding from the American Recovery and Reinvestment Act (ARRA)—would introduce information that is not comparable to the rest of the sample. An analysis of success rates from FY 2000 to FY 2008 reveals only small year-to-year changes in award probabilities by race/ethnicity, suggesting that our study is representative of the entire period (fig. S2).
To measure productivity at the time of application, publication and citation information from Thomson Reuters Web of Science and Journal Citation Reports was matched to R01 application investigator information. We were able to match 84% of grant applications to publications with greater than 90% confidence. As described in more detail in the supporting online material, the matching process used conservative criteria and therefore may underreport publications for applicants with common names. This measurement error may have biased the coefficients in the model. The sign and size of the bias would depend on the relative magnitude of the average and variance of the underreporting, as well as the covariance between the underreported, and other variables in the model, and would be typically less than the omitted variable bias were these variables to be left out (10, 11).
We analyzed the probability of receiving an R01 award using probit models estimated through maximum likelihood. Our analysis progressed through five models that added explanatory variables most likely to explain the observed race/ethnicity differences (table S4). In place of reporting probit coefficients, we report the marginal effect of the variable on the award probability, which is the change in the award probability due to each predictor separately with other variables evaluated at their mean values. The resulting regression estimates are correlations between the covariate and the probability of receiving an R01 award and should not be interpreted as having a causal impact.
The race/ethnicity estimates of marginal effects in table S5 can be interpreted as the percentage point difference in the probability of receiving a NIH R01 award between applications from white investigators (the omitted category in the regressions) and applications from investigators of a given race/ethnicity. Model 1, which controlled for demographic characteristics, showed that applications from black investigators were 13.1 percentage points (P < .001) less likely to be awarded an R01 than white investigators, and applications from Asian and Hispanic investigators were 5.4 (P < .001) and 2.7 (P < .05) percentage points less likely to be awarded, respectively. When we added controls for education and NIH training in Model 2, the marginal effects did not change in size or significance. Model 3 added controls for employer characteristics, which reduced the significance of the marginal effects for Hispanics, but not for Asians or blacks (P < .001), compared with Model 1. Model 4 included controls for previous NIH grants, NIH review experience, and NIH institute, and while it reduced the award differential for blacks and Asians by 1 percentage point, the differential was still significant (P < .001). With the full set of covariates in Model 5, the award probabilities for applications from blacks were 10.4 percentage points lower, and for Asians were 4.2 percentage points lower, than for whites (P < .001). Our models fit the data well, correctly classifying R01 award outcomes for between 71 and 72% of the observations in the sample. In summary, Hispanic award probability differentials were explained by variables added in Models 4 and 5, but none of the observable characteristics in Models 1 to 5 fully explained the differential for Asians or blacks.
Next, we examined the average number of grants per person, the proportion of investigators submitting single and multiple grants, and the likelihood of application resubmission. On average, investigators had three to four Type 1 R01 grant applications each. We found that blacks and Asians resubmitted more times before being awarded an R01 (2.01, P < .06 and 1.85, P < 0.001, respectively) compared with whites (1.58), and at the same time blacks (45%) and Hispanics (56%) were significantly less likely to resubmit an unfunded application compared with white investigators (64%, P < 0.001) (table S6). We estimated Model 5 after introducing controls for the number of resubmissions and then estimated the model separately by the number of times a grant was submitted (table S7). Applications from black and Asian investigators were significantly less likely to receive R01 funding compared with whites for grants submitted once or twice. For grants submitted three or more times, we found no significant difference in award probability between blacks and whites; however, Asians remained almost 4 percentage points less likely to receive an R01 award (P < .05).
Together, these data indicate that black and Asian investigators are less likely to be awarded an R01 on the first or second attempt, blacks and Hispanics are less likely to resubmit a revised application, and black investigators that do resubmit have to do so more often to receive an award. Assistance with the grants submission and resubmission process may provide a policy lever for diversifying the scientific workforce.
Next, we examined the nativity of R01 applicants, because only U.S. citizens and permanent residents are eligible for NIH pre- and postdoctoral training programs. We used information from the DRF that allows us to identify citizenship at the time of Ph.D. receipt. For Ph.D. applicants that were not matched (15 to 22%), we manually reviewed their biosketch information to obtain information on the location of the school awarding the undergraduate and graduate degrees. If all degrees were received outside the United States, these individuals were classified as foreign-born and foreign-educated. More than 70 percent of these individuals had degrees from non-U.S. institutions. Applicants that we were unable to classify were categorized as having missing citizenship information, and we included a dummy variable in the model for those cases.
Figure S3 shows that 87% of Asian, 45% of black, 56% of Hispanic, and 25% of white applications were from non-U.S.-citizen investigators. When the analysis sample was restricted to include only those applicants who were U.S. citizens at the time of Ph.D. receipt, the difference in R01 award probability for Asian applications was cut in half and was no longer statistically significant (table S8). However, the 10 percentage point difference in award probability for blacks did not change (−0.107, P < 0.001) after including all covariates.
NIH pre- and postdoctoral training fellowships and traineeships serve as an intermediate step on the biomedical career path between degree completion and becoming an independent researcher. We expect training variables to be positively correlated with receiving an R01 award. Using the U.S. citizen and permanent resident sample, we explored the impact of NIH pre- and postdoctoral fellowships (F), NIH pre- or postdoctoral traineeships (T), and NIH career development awards (K), which are largely awarded to early career investigators as grant funding for research.
Participation in these programs varied by race, ethnicity, and program (table S9). For R01 applications from U.S. citizens, 69% from Asian investigators, 54% from blacks, 62% from Hispanics, and 62% from whites were associated with previous NIH F, T, or K support. More applications from Asians were associated with previous T support (58%) compared with blacks (44%), Hispanics (45%), and whites (43%), whereas fewer applications from black investigators were associated with previous F awards (16%) compared with whites (27%), Hispanics (22%), and Asians (22%). Previous K support was associated with 17% of applications from Asian investigators, 10% from blacks, 16% from Hispanics, and 11% from whites.
Early scientific training is first included as a covariate in Model 2, which omits post-training variables such as current institution. After controlling for demographic characteristics and educational background, fellowships were associated with a 2.5 percentage point increase in the probability of R01 award (P < .001), traineeships with an increased award probability of 2.2 percentage points (P < .001), and career development awards with an increased award probability of 4.8 percentage points (P < .001) relative to R01 applicants who had no previous participation in these NIH training programs. The estimated impact of training is reduced once the full set of covariates is included in Model 5 (table S8).
Participation in training programs significantly improved subsequent R01 award probability for both applications and applicants (Fig. 2Atable S10). However, when we examined the effect of race/ethnicity on R01 award probability for all applications and applicants that received F or T training, we found that training did not mitigate differences in award probability (Fig. 2B and table S10). Compared with R01 applications from white U.S. citizens or permanent resident investigators with previous NIH training experience, applications from black investigators were 13.5 percentage points less likely to be funded (P < .001). For all applicants who received F or T training, blacks were 27.4 percentage points (P < .001), Asians were 6.9 percentage points (P < .01), and Hispanics were 9.5 percentage points (P < .01) less likely to ever receive an R01 award compared with whites. A closer investigation of the impact of training by race/ethnicity may provide insight into differences in R01 award probability and perhaps provide a policy lever for diversifying the scientific workforce.
Research has established that the perception of scientific merit is affected by past performance— such as association with high-ranking departments or institutions and previous funding and publication records—and by access to organizational resources (12). If this is the case, and racial and ethnic groups do not have the same distribution of these characteristics, then including controls for these effects might reduce or eliminate differences in award probability.
There were fewer total applications from blacks (27%) at institutions receiving the most NIH funding (the top 1 to 30) compared with whites (33%, P < .05) but a similar number at institutions ranked 31 to 100 in amount of NIH funding awarded (table S11) (13). Applications from white investigators were more likely to be associated with a previous NIH RPG or K award (78%) compared with blacks (69%), Asians (73%), and Hispanics (70%) (P < .001). Average number of publications and citations at the time of application varied significantly by race and ethnicity. Black R01 applicants published a similar number of articles compared with white applicants (13.7 compared with 14.3, respectively), whereas Hispanic and Asian applicants on average published more articles than white applicants (17.8, P < .01 and 28.8, P < .001, respectively). In biomedical sciences, a last-author position indicates responsibility for managing the group carrying out the research described in the publication. Blacks had a lower percentage of papers that were last-authored (22.4%, P < .001) compared with whites (30.4%), Asians (34.2%), and Hispanics (30.3%) (tables S11 to S13). The largest observable difference was in the number of citations at the time of the R01 application. On average, white applicants had 78 citations to previous work, blacks had 40 (P < .001), and, as with publications, Asians (143, P < .001) and Hispanics (90, P < .01) had more citations than white applicants. However, even after controlling for these differences there were significant differences in R01 award probability between applications from blacks and whites (table S5, Model 5).
We examined the marginal effects of these characteristics on R01 award probability for the full sample. Working at a nonacademic research organization increased the probability of receiving an R01 award by 4.2 percentage points (P < .01), whereas working at an institution with the most NIH funding (ranked 1 to 30 in total grant funding) increased the R01 award probability by 9.7 percentage points (P < .001), and those with substantial NIH funding (ranked 31 to 100) increased R01 award probability by 6.1 percentage points (P < .001) compared with all institutions ranked below 200 in NIH funding (Fig. 3 and table S5). Previous research awards were associated with increased subsequent award probability. Previous receipt of NIH RPG or K grants increased the probability of R01 funding by 8.1 (P < .001) percentage points. Serving on a NIH review committee (itself an indication of receiving NIH funding) increased R01 funding by 8.2 percentage points (P < .001). Publications and citations also were significant contributors to R01 funding. An application from an investigator with more last-authored publications relative to total publications had a 2.1 percentage point greater chance of receiving R01 funding (P < .05). In addition, investigators with citations above the median (more than 24 citations) at the time of application were 3.6 percentage points (P < .001) more likely to receive an R01 award (14). The number of first-authored papers by the applicant had no significant effect on the award probability, regardless of race or ethnicity. We tested whether these marginal effects varied by race and found no significant differences.
Next, we estimated the effect of our model variables on the probability of receiving a priority score during the review process (table S14). Negative marginal effects indicate that the application was more likely to be unscored, whereas positive marginal effects indicate the application was more likely to be scored. In the full sample, all of the variables associated with increased award probability were also significantly associated with increased likelihood of an application being scored (P < .001). Marginal effects for whites, Asians, and Hispanics are not different from the full sample. However, marginal effects for applications from blacks were significantly different from the full sample (P < .05): For blacks only, NIH review committee experience (P < .001) and citation count (P < .01) were significantly correlated with receiving a priority score. Together, these results suggest that previous research and affiliation do not have the same impact across racial and ethnic applicant groups.
Throughout the education pipeline, blacks are less likely to graduate from high school, attend college and major in biomedical science, and obtain a Ph.D. in biomedical science. Given our previous results, we expected to find that black scientists who made it to the stage of principal investigator would have similar chances of obtaining NIH funding, all other things being equal. We find it troubling that the typical measures of scientific achievement –NIH training, previous grants, publications, and citations—do not translate to the same level of application success across race and ethnic groups. Our models controlled for demographics, education and training, employer characteristics, NIH experience, and research productivity, yet they did not explain why blacks are 10 percentage points less likely to receive R01 funding compared with whites.
Although our models do not fully explain the funding gap, the greatest differences between blacks and whites that we observed were in the effect of previous training and the probability of receiving a priority score. Although more research is needed to discern the basis for the award differences, it is possible that cumulative advantage may be involved (15). Small differences in access to research resources and mentoring during training or at the beginning of a career may accumulate to become large between-group differences. This suggests that more analysis on the impact of NIH training may be warranted. In addition, further research into the review process could help to understand why variables that increased the likelihood of an application receiving a priority score for the full sample did not have the same impact for applications from black investigators.
D.K.G. acknowledges financial support from NSF grant SES-0353703 and NIH grant 1R01AG36820-01. D.K.G. and Discovery Logic, a Thomson Reuters company, acknowledge support from NIH contracts HHSN276200800458U and HHSN276200900100U, made possible through an NIH Evaluation Set-Aside Award to W.T.S. (07-6008 OD OER). We thank C. Bollinger, S. Kahn, and W. McGarvey for helpful discussions. The analytical files used in the studies described here contain personal information from individuals who have submitted applications and in some cases have received awards from the NIH. Many of these application records have been matched to records included in the Survey of Earned Doctorates as maintained by the National Science Foundation and to records included in the Faculty Roster maintained by the Association of American Medical Colleges. The information is therefore protected by the Privacy Act of 1974 as amended (5 U.S.C. 552a) and the National Science Foundation Act of 1950 as amended (42 U.S.C. 1873(i). More complete information can be found in the NSF/SRS Restricted-Use Data Procedures Guide available at www.nsf.gov/statistics/license/forms/pdf/srs_license_guide_august_2008.pdf and the NSF Data and Tools website at http://www.nsf.gov/statistics/database.cfm. Researchers interested in access should call W.T.S. at 301-402-2725 to discuss the security clearance requirements necessary for using the data. A de-identified version of the data files will be posted at http://report.nih.gov/investigators_and_trainees/index.aspx and archived by Science.
Supporting Online Material
Materials and Methods
Figs. S1 to S4
Tables S1 to S15