Characteristics of Participants
Of the 205 individuals who were asked to participate, 79 did not reply, 8 refused (usually because of lack of time), 1 was unavailable for participation, and 117 were included. Of those, 98 were interviewed, including 38 internal reviewers, 27 external reviewers, and 33 grant applicants; none declined participation after receiving oral information on the study. The remaining 19 individuals (2 internal reviewers, 9 external reviewers, and 8 applicants) either canceled or failed to attend the interview appointment. shows the participant characteristics. Most participants were male (71%) and worked in the Paris region (66%). Among the 107 non-participants, 7 were internal reviewers, 56 external reviewers, and 44 grant applicants. About half of the non-participants came from the Paris region (52%) and 79 (74%) were. Of the 34 applicants who refused to participate, 26 (77%) had submitted PHRC grant applications that were rejected in 2009.
Characteristics of participants.
Perception of Conflicts of Interest that Might Affect Grant-application Peer Review
During the interviews, most participants (79/98) spontaneously voiced concerns about non-financial CoIs and listed them ahead of all other biases such as those related to scoring, expertise, or notoriety. Industrial or financial CoIs were rarely mentioned by participants and were often viewed as minor or nonexistent in the PHRC review process: “Normally, [industrial conflicts of interest] shouldn’t arise in the kind of proposals submitted to the national or regional PHRCs"
(External Reviewer 8 [see Table S1
for external reviewers’ characteristics]). In addition to financial CoIs, four types of non-financial CoIs were identified ().
Non-financial conflicts of interests in grant-application peer review: perception, experience and management.
Experience with Non-financial CoIs
Applicants could not formally prove the existence of non-financial CoIs in the grant-application peer review process, but one-third of them (13/38) reported having personal experience with such CoIs. Their suspicion that non-financial CoIs had affected the review process originated occasionally in personal convictions, interpretations, and hearsay and more often in discordances between reviewers’ reports. “So it is a small world, we know everyone within the disciplines, and it is human, so there are true scientific reviews, and then politics, conflicts of interest, rivalries. jealousies but like any review involving scientific experts, I think we cannot avoid that" (Applicant 22);“I don’t have any proof of what I say! I don’t know for sure, I am just guessing" (Applicant 28); and “What is a little weird sometimes too, is the gap between two reviewers […], here we will never know." (Applicant 19). Most applicants were fatalistic about this situation and did not complain despite their suspicions: “Of course, we always hear about applicants who may be well-connected. because. you know. because the internal reviewers, well the external reviewers who. who are chosen know the applicants or there are conflicts of interest. It is possible, isn’t it? Yes, we hear about that but. what can we do?" (Applicant 10).
Applicant 29 suspected that an idea was stolen from a previous application he had submitted: “That can happen, and according to me… I made a proposal about a gene and … I saw a database [about that gene] two years later! It could be a coincidence but it is weird! […] They looked for the gene I had proposed in a cohort of patients. […] Now I don’t know for sure, but I have my suspicions."
Prevention of Non-financial Cois
While non-financial CoIs were considered either unacceptable or unavoidable by the various stakeholders, opinions about the feasibility of preventing CoIs were more contrasted. Some interviewees were fatalistic (“It is human […] I think we cannot avoid it", Applicant 22), while others were quite satisfied with the current peer review system (“I don’t have any criticisms to make about the peer review process", External Reviewer 22). Moreover, no participants suggested the peer review system should be reconsidered. “Is there a better system?" (External Reviewer 16) and “Who else do you want as reviewers?" (External Reviewer 23).
Other participants considered that CoIs were too variable in nature to be properly managed: “It is absolutely unfeasible, because there are fifty different levels of conflicting interests, disciplinary, geographic, personal, you see what I mean…. All kind of networks, in every way, so we can’t manage that… and it goes in all directions, you see what I mean… there are positive conflicts of interests, negative ones […]. For example, something that happens all the time is that people trash others’ proposals in order to open the way for theirs, you see?" (Internal Reviewer 30).
Interestingly, even when CoIs were suspected, they were not always perceived as important by the internal reviewers. “I don’t think it matters that much. In practice, it may explain 15% of the variance […], that’s all" (Internal Reviewer 16). Finally, the internal reviewers felt that, despite rare exceptions, the best applications were selected: “So, after that, from a pragmatic viewpoint, when all is said and done, we have the feeling that the best proposals are funded" (Internal Reviewer 12).
In addition, external reviewers had no knowledge of the reporting and management of CoIs during the grant-application review process. More generally, most of them were unaware of how their reviews were considered in the final assessment: “We don’t have the list of the funded proposals and neither do we get feedback about the reasons for rejections. So, I don’t know in the end, after providing my expertise, how my review was used in the process." (External Reviewer 16).
Current Regulation Mechanisms
We found that several mechanisms were used to limit CoIs, although they were not explicitly described in an official policy statement. Grant applicants could list the names of experts they did not want as reviewers of their projects. Experts could, but were not mandated to, refuse to review projects they felt might involve CoIs. Each grant application was reviewed by three (national PHRC) or five (Paris regional PHRC) internal and external reviewers, whose names were masked to the applicants. Moreover, the panel members were chosen from a variety of geographic areas and specialties to ensure that the panel represented the diversity of the grant applications. Grant applications were discussed collectively during the panel meeting, and panel members were free to voice their opinions, although the discussions were influenced by individual factors such as effectiveness in public speaking, scientific expertise, desire to share personal convictions, and willingness to risk expressing disagreement. The panel president ensured that internal reviewer(s) who were involved with an application as investigator were not present when the application was discussed. During the panel meeting, the president played an important role in identifying and managing CoIs, for example by ensuring that internal reviewers did not place excessive emphasis on applications in their own disciplines to the expense of those in other disciplines: “These proposals are certainly excellent, but all the same, we must consider the others…" (observed during the national PHRC committee meeting in March 2009).
Management of Non-financial Conflicts of Interest
Personal experience with non-financial CoIs was reported by 26 reviewers. Based on the interviews of these reviewers, approaches to CoI management were divided into three evenly represented categories.
First, some reviewers routinely refused to review grant applications if they felt they might be biased in favor of or against the applicant: “I have already refused to review [a proposal] because of conflicts of interest" (External Reviewer 21) and “Well, I refused when I received the first letter about [the proposal]… although I was itching to do it…" (External Reviewer 15). This concern about non-financial CoIs was often based on the existence of personal relationships – positive or negative – with the applicant: “It happened to me once, no, sorry, twice, to send back a proposal because of conflicts of interest. Twice, because the person who sent me the proposal didn’t realize that I was part of a team that was involved in the research project." (Internal Reviewer 14).
Second, some reviewers felt that non-financial CoIs were unavoidable and should be managed by conducting the reviews in a strictly impartial manner. They only refused to review applications for which they felt unable to remain impartial: “I have already reviewed an application for which I had [a CoI]., and I tried to separate myself from any influence of that" (External Reviewer 12). The problem is recognizing the non-financial CoI: “Where does it begin? Where does it stop?" (External Reviewer 11).
The third group of reviewers adopted a case-by-case approach to decide whether or not to review each application according to their subjective understanding of potential non-financial CoIs. For example, two reviewers said that they refused reviews if they were biased against the applicant or project, but not if their bias was positive: “I am not perfectly honest, because I am too positive, but… in any case, I do not batter a project for reasons that are not purely scientific." (External Reviewer 14).
Suggestions for Improvement
Among the numerous suggestions for improving the peer review process (), masking of applicants was listed most often. “Obviously, if [my name] had been masked… that would have changed things…" (Applicant 16) and “It [applicant name masking] would result in the application being evaluated independently from the research group, its financial resources, whether it received a PHRC grant last year (…) and so the review would be based only on scientific quality. I think it would be better" (Applicant 19). However, some of the reviewers believed this method would fail in many cases: “We can guess who it is. We don’t know for sure, but we guess or we believe we know" (External Reviewer 13). In addition, masking may prevent a valid assessment of the feasibility of the research project: “I don’t think anonymity matters that much, but it can be harmful, because in the reviewing process, you must know the team (…) if it is blinded, I don’t know who will carry out the project (.) Sometimes a team can write a good proposal but does not have the resources to carry it out! Knowing the clinical research network helps me to say ‘if it is this team or that team, OK, I know it can work’. But if it is blinded, you cannot do that." (External Reviewer 8).
Participants’ suggestions to minimize non-financial conflicts of interests (CoIs).
Regarding the overall reviewer selection process, many interviewees voiced major concerns about reviewer selection, especially for specific specialties or research topics: “Genuine conflicts of interests result from the choice of reviewers who will assess the applications and on their relationships, if any, with the applicant" (External Reviewer 8). Selecting reviewers from other countries was suggested as a possible solution by some participants: “We must stop using self-assessment and confining ourselves to the French community. We must avoid the consequences of having French people assess French projects. Other organizations require that the applications be written in English and send them to international reviewers. The [peer review] process is influenced by interpersonal factors. We must steer clear of all relationships that can result in conflicts of interest." (Internal Reviewer 21). For small disciplines at greater risk for non-financial CoIs, some of the internal reviewers suggested the selection of non-clinical peer reviewers, such as methodologists: “You will have an external reviewer who is not a specialist, who is not competing with you, and who will give a more objective opinion" (Internal Reviewer 7).
Another suggestion was to give the applicants the opportunity to challenge the report of the reviewers: “[We should] have a process for applicants to acknowledge that a reviewer was objective […] or to refuse that a given reviewer assesses their work [as happens with manuscript peer review]" (Applicant 7).
Improving transparency was also suggested: “[The proposal] is discussed by the committee, [but] we do not have much transparency about the discussion, the results, how they talk among themselves, and how they rate the applications …" (Applicant 19). “It is not transparent at all. When someone submits an application, he or she doesn’t know what will happen! Of course, some of us (the reviewers) can tell the applicant [what happened], because we are members of the committee [….] but that is not official policy." (Internal Reviewer 12). Identifying the reviewers might improve transparency: “By conviction and for transparency, I believe it would be better if the reviewers were identified" (External Reviewer 12). However, reviewer anonymity may also have advantages, as explained by the same external reviewer: “Well, nevertheless, I would prefer it to remain blinded because one can express oneself more easily.[…] And if reviewers were unmasked, well, we might not provide the full extent of our opinions, to keep from offending or hurting someone". Similarly, according to Applicant 15, “That’s a good question! Would I like to know my reviewers? No, I don’t think so, it must stay impersonal. No, no, (…) I think it would bring nothing but trouble, particularly in the medical community, where we all … you know… perform favors for one another. I would be embarrassed [to know] that a colleague refused my application."
Other suggestions were made, such as interactions between applicants and reviewers: “We should consider, I don’t know, a hearing of the applicant for example, because there can be things that are easier to discuss or to talk about face-to-face" (External Reviewer 12). “Peer review by correspondence, or just by talking and asking questions of the applicant who would reply ‘No, you misunderstood, I said that and not that’, [would allow us to] make a more objective review" (External Reviewer 9).
Disclosure of CoIs, particularly of a non-financial nature, was mentioned by many interviewees as an important transparency procedure that did not appear to be part of PHRC policy: “I think [disclosure] is left to the morality of reviewers. Well, we chose a system where we trust one another, but we should be able to be more objective, it would not be completely crazy" (Internal Reviewer 4). “Conflicts of interest should be disclosed formally, as for articles in high impact factor journals, where we must routinely disclose the presence or the absence of conflicts of interest, and it is something that may be challenged. I mean that if someone complains or something else, or if someone has not disclosed a relevant conflict of interest, it should matter. Here, it should be done routinely. I don’t think it is done at present… But for the reviews, I think conflicts of interest should be disclosed routinely for each applicant, and applicants should be able to challenge the selection of a reviewer in the event of a conflict of interest." (Internal Reviewer 11).
Finally, one reviewer suggested training of peer reviewers in the identification and management of CoIs and improved uniformity of the peer review process:
“Maybe we should have training sessions for reviewers? […] To see what makes a good application. I think it could be really useful to invite peer reviewers to a few training sessions; this might be a good idea? To increase uniformity of reviewers’ work" (External Reviewer 1).