The results are reported according to RATS qualitative research review guidelines 
Survey of Procedures and Criteria used to Review Grant Applications
Overall description of grant application review procedures
Fourteen calls for proposals (five international and nine national) were investigated. All funding organizations used a two-step assessment process for all calls: the applications were first reviewed by internal and/or external reviewers then discussed by a committee. For three international calls, the review procedure included specific recommendations to take into account the applicant’s replies to reviewer comments during the assessment. Details on each procedure are provided in and . Additional information on French PHRC procedures have been reported previously 
Guidelines for peer review of grant applications issued by the eight French Academic Hospital Research Grant Agencies.
Guidelines for peer review of grant applications issued by one French national and five international funding organizations.
Description of reviewer assessment practices
The assessment procedure included three parts: global scoring of the application, detailed assessment of specific criteria, and a written report. Global scoring was required for all 14 calls. The score was numerical for 10 calls, qualitative for 3 calls, and either numerical or qualitative for 1 call. and show the scoring guidelines. An evaluation of specific criteria was required for all 14 calls, and the French PHRC guidelines involved completing a checklist of criteria. The criteria were scored in 12 of the 14 calls, using various methods and weighting procedures ( and ). For eight calls, the criteria assessment was used to compute the global score. A written report was required for 12 of the 14 calls, although guidelines about the structure of the report were provided for a single call.
Description of assessment criteria
and list the criteria listed in the review guidelines for each call. The median number of criteria was five per call (range, 3–8). reports our overall typology of assessment domains and criteria. We identified nine assessment domains.
Criteria for grant application assessment recommended by French funding organizations.
Criteria recommended for grant application assessment by one French and five international funding organizations.
Typology of assessment criteria derived from our sample of calls for proposals.
- Relevance of the research project to the call for proposals, amount of funding requested, and characteristics of the applicant. Information on this domain was required by only three funding organizations.
- Scientific relevance of the research project or study question. Information on this domain was required for all 14 calls. Reviewers were asked to broadly assess the usefulness of the research project or to provide specific information on previous studies in the field and on the literature review supplied by the applicant.
- Originality was to be assessed for 13 of the 14 calls, based on the potential impact of the research project as assessed by the reviewer, in particular based on the potential for publication.
- Innovativeness referred to the technological, technical, or methodological innovations used or investigated in the research project. Information on this domain was required for only 6 calls. The innovativeness domain was sometimes included in the originality domain.
- Methodology was a domain on which information was required for all 14 calls. In weighted scoring systems, a high weight was given to this domain. The review guidelines included specific questions about numerous methodological issues such as sample size estimation and quality of the study design ().
- Feasibility. This domain encompassed a number of issues pertaining to the research project, characteristics of the applicant (e.g., previous publications and collaborations), and scientific context (e.g., competing research projects). In the guidelines for some of the calls, the feasibility domain included methodological issues (e.g., required sample size), adequacy of the requested funds, and ethical aspects (e.g., about patient consent to participation) ().
- Financial considerations and requested funds. This domain included specific questions on the planning of the project and description of necessary resources. Scoring was not always required.
- Ethical considerations, including potential risks to patients. In some cases, this domain included methodological issues such as the management of missing data or of patients lost to follow-up (). It was often assessed qualitatively, as opposed to scored ( and ).
- Writing or readability of the application. Only two calls requested information on this domain, which was usually assessed subjectively. The guidelines for one call included a question on how well the application could be understood by nonscientists ().
Qualitative Study on Reviewer Practices and Perceptions
Characteristics of the interviewees
We invited 128 reviewers (45 internal and 83 external reviewers), of whom 76 (40 internal and 36 external reviewers) accepted to be interviewed and 65 accepted to participate in the qualitative study; 11 reviewers were interviewed and consented to the study but finally were not available for the study interviews. The interviews began after the committee meetings, in June 2009, and ended in November 2010. Thirty-six (37%) interviews were conducted by telephone. Two interviewees refused to be recorded during the interview, and two recordings were of insufficient quality to allow transcription; the written notes taken during the interviews allowed us to use these four interviews. The saturation point was reached after 38 interviews of internal reviewers and 27 of external reviewers. reports the main characteristics of the 65 participants. Interview length ranged from 15 to 91 minutes (median, 31 minutes). Most participants were pleased to participate and to discuss the grant application review process. The main reason for refusing to participate was lack of time.
Characteristics of the 65 reviewers who participated in the qualitative study.
Three themes emerged from our analysis and are detailed below: practices of external reviewers, practices of internal reviewers and the assessment process during the committee meetings.
Practices of external reviewers
We evaluated the practices of external reviewers based on time spent on the review, whether reviewers looked at previously published studies, whether reviewers used funding organization checklists, and writing of the report.
• Time Spent on Reviewing Grant Applications
The interviews showed wide variations in the time spent reviewing applications, from a few hours to several days: “We read a little … I would say … adding it all up … it must take a good ten hours I think. […] And I will not spend more than ten hours – I can’t anyway!” (external reviewer 17) and “It depends on the project, but one or two days” (external reviewer 15). Most reviewers spent several work sessions on each application: “I take notes as I go along, I often need time, well I don’t know what a decent time would be […], but sometimes I spend quite some time … I read through the application once, to get an overall idea of project, its goals and approach, the methodology, and then I read it a second time more carefully, and I usually make a few comments. So first I try to get a broad picture of the research project and then I focus on the details […]” (external reviewer 12). More rarely, reviewers processed each application in a single session: “When I have enough time, I focus and I read the entire application in one session – so I arrange to have enough time, an afternoon, or whatever time I need to read the application, and as I read I make notes.” (external reviewer 10). Some reviewers also complained about lack of time and short deadlines: “The deadlines are always very short; when you apply for a grant, you always find that getting the answer takes a very long time … […] but for me … every time I’m given only ten days to send in my report!” (external reviewer 26). However, a few reviewers felt that time was not a problem because they always reviewed the applications at the last minute: “Anyway, it’s true that the deadline is always too short, but you know, even if we received the applications one month before the deadline, we would wait until the last minute to review them, or at least I would (smile).” (external reviewer 10). Finally, some reviewers felt that the ability to meet the deadline was chiefly dependent on reviewer behaviors: “It’s not a real concern; there are people who put in the work and people who don’t. Some people miss the deadline regardless of the circumstances.”(external reviewer 8).
• Looking at Previously Published Studies
Reviewers varied in their practices regarding referral to previously published studies. Most external reviewers performed a literature search, mainly to assess the scientific relevance of the proposal: “I read the proposal and, when I have time, I read the literature, at least… I always try to take a quick look at the literature to assess the relevance [of the proposal]…” (external reviewer 25). Some reviewers searched the literature only on a case-by-case basis, to confirm an opinion or to explore specific issues: “I rarely search for articles. Except on matters that puzzle me, or if I feel the proposal is incomplete – then, I write a note and I check on PubMed to see whether it is correct it is correct… But not routinely, I must admit.” (external reviewer 24).
• Use of the Assessment Checklists Recommended by the Funding Organizations
For the 2009 national and regional PHRCs, assessment checklists were provided to all reviewers as an aid to reviewing and scoring the applications. Most of the external reviewers found these checklists helpful: “The checklists clarify the way in which we see the project. They help us become aware that our approach to assessing projects is sometimes a bit fuzzy. They give us a clearer picture of the overall project” (external reviewer 16). The checklists were also perceived as providing information on the points that were important to the funding organization: “The checklists help us to understand the committee’s point of view … the hierarchy of the assessment parameters, and they are important to help us determine how to write the final report.” (external reviewer 16). The reviewers felt the checklists might help them write their own applications in the future: “I wrote a proposal just before phoning you. The checklist is very helpful because we can find out right away what is missing…” (external reviewer 20). However, some reviewers felt the checklists were difficult to complete: “It is not always easy, is it? […] Some of the items may not be relevant to an individual proposal and are therefore difficult to answer. When I review several proposals, it is obvious to me that there are differences in the usefulness of the checklist, depending on the specific features of each proposal. In general, the checklist is not too difficult to complete.” (external reviewer 11). Another criticism related to the broad nature of the assessment criteria: “The items are good, but I think that for each item there should be a list of sub-items and response options. For example, for assessing the methodology, in the checklist that was given to me, the item was just “methodology”. The reviewer has to provide details on the method chosen, its appropriateness to the study question, whether the nature of the data allows the statistical analysis, whether the statistical methods chosen are appropriate, and whether the sample size is large enough.” (external reviewer 15). A few reviewers strongly criticized the scoring of proposals: “You can give scores from 0 to 20, it’s the same thing, there will be scores of 18, 12, 4 […] It makes no sense! First, because we have no control over the quality of the reviewers or their scoring practices. I am not even sure that all the reviewers read the long list of explanations on the scoring procedure. They don’t even read it. So it is useless. Now, it makes everyone happy … and it rationalizes rejections: “Here, you see, you got a bad score, so we will not not fund [your proposal]” (external reviewer 8). In practice, most of the reviewers completed the checklists at the end of the review process, as a means of supporting rather than of developing their opinion: “I always used the checklists at the end […]. I formed my own opinion of the proposal, by making a critical appraisal of the proposal on my own, and when that was done I matched my comments to the checklist.” (external reviewer 22). Nevertheless, reviewers felt the checklists served a purpose: “It has never happened to me that, after having reviewing each point of the proposal, my final score was very different from what I expected. […]. I think the checklists are useful – clearly, they can be very useful when the proposal is rejected and returned to the applicant […], and also for helping to rank proposals.” (external reviewer 9).
• Writing the Final Report
Few details were given about the writing of the final report. One reviewer felt that the report should only provide a scientific opinion, without assessing whether the proposal should be funded: “I don’t think it it is the case for French PHRCs, but sometimes other organizations ask us to make the final decision about funding, and I don’t think this is an appropriate request to make to external reviewers, […] who have not seen all the proposals and consequently cannot rank them.” (external reviewer 16). Most reviewers felt that their report should be designed to help the applicants improve their proposals: “If our report only says “oh! your proposal is bad”, that’s not interesting, not constructive, not useful.” (external reviewer 10); and “In my opinion, one of the most important aspects of the peer review is the opportunity to improve [the proposal].” (external reviewer 12). Thus, the review process was sometimes perceived as a way to help rather than to judge the applicants: “less like a judgment and more like help” (external reviewer 18).
Practices of Internal Reviewers
We assessed three components of the practices of internal reviewers: the material conditions of the reviews- in particular, time spent and literature search-, the use of assessment checklists, and the use of external reviewer reports.
• Material Conditions of the Reviews
The time spent on each application varied less among internal than among external reviewers. Internal reviewers usually spent 1 to 2 hours on each application: “I think that now I spend one hour …no, two hours. One hour the first time I read it, then one hour to read it again, so two full hours.” (internal reviewer 33). The time spent on each application was perceived by the internal reviewers as dependent on the quality of the external reviewers’ assessments, on whether the external reviewers met the deadline, on the number of external reviewers, on the existence of disagreements among external reviewers regarding the application, and on the level of expertise of the internal reviewer in the field relevant to the application. Most internal reviewers did not perform routine literature searches, instead using previously published data only to support the opinion they had already formed (“We read the proposal and we check the references if necessary. We do not check whether they exist or not, but we check them if we disagree or if we believe that new data have been published.”, internal reviewer 1) or to assess the applicant’s reputation and ability to publish (“I check the publications on Medline or in the proposal, and I see if the applicant has been able to produce papers that were sound.”, internal reviewer 10).
• Use of the Assessment Checklists Recommended by the Funding Organizations
Most of the internal reviewers had opinions similar to those of the external reviewers regarding assessment checklists. Thus, checklists were usually perceived as helpful, although a few internal reviewers criticized the scoring method: “[…] I distrust numbers: you know that book on statistics that says “There are three kinds of lies, lies, damned lies, and statistics”! We can make the numbers say what we want them to say.” (internal reviewer 18); and “Summing to get a global score does not provide a global opinion – this point has been convincingly demonstrated. In general, the opposite happens and there is a “halo effect”. In general, reviewers form an overall opinion about the proposal and then they assign the scores and subscores based on that opinion.” (internal reviewer 16).
• Use of External Reviewers’ Reports
The internal reviewers relied heavily on the reports by the external reviewers. Some internal reviewers read the external reviewer reports before reading the application: “ I take the report that is on the top of the stack and, since my role as an internal reviewer is that of a rapporteur, I read the experts’ reports before reading the proposal. So I read the two or three reports that I have. Then I form an opinion, since my job is to create a synthetic overview of the reports – I form a global opinion of the external reviewers’ perceptions and of the differences that may exist among them.” (internal reviewer 18). Other internal reviewers read the applications first: “As the internal reviewer, I read the proposal first to form my own opinion, then I read the two external reviewer reports; if they support my opinion, I don’t have much more work to do if I believe the reports are correct; if the two reports differ widely, I go into the details of the proposal; and if the reports do not support my opinion, I determine who is right, and sometimes I realize I had missed something.” (internal reviewer 17). The quality of the external reviewer reports was perceived as crucial by the internal reviewers, who gave great importance to point-by-point analysis and discussion: “When an expert writes ‘Excellent project that must be funded’ with a four-line assessment, the report goes straight to the wastebasket – it is not useful at all. A review is useful only if it analyzes and discusses each of the important points relevant to the funding decision. ” (internal reviewer 20).
We identified various strategies used by internal reviewers to write their reports. Some internal reviewers wrote a synthetic overview of the external reviewer reports, usually without giving their own opinion: “ I always try to restate what the external reviewers wrote, because I do not want to act as an ‘additional reviewer’ giving an opinion that would prevail over the opinions of others.” (internal reviewer 19); and “The job of the external reviewers is to give their opinion, whereas the internal reviewers act as rapporteurs, whose job is to assess whether these opinions are… founded or not, subjective or not… and whether their impact is limited or major.” (internal reviewer 5). The internal reviewers sometimes sought to compare the detailed analysis in the external reviewer report with the score assigned by the external reviewer: “When the report provides a detailed analysis, I try to look at each point to see whether I agree with the external reviewer […] and whether there is a discrepancy between the analysis and the score. […]. Scoring is relative, and my job is to try to find a balance.” (internal reviewer 20). When external reviews were lacking or of poor quality, the internal reviewers sometimes acted as external reviewers. Furthermore, some internal reviewers sought to reconcile differences between external reviewer reports: “Sometimes, when there were discrepancies, I had to make a choice.” (internal reviewer 26). Internal reviewers who were thoroughly conversant with the relevant field sometimes gave precedence to their own opinions, rather than to those of the external reviewers: “When I feel the field is one in which I have considerable expertise, I put my score in the final report, and I discuss the external reviewers’ opinions based on my interpretation – so I answer the concerns raised by the reviewers. […] So in this situation I act as a ‘super reviewer’” (internal reviewer 14).
Actual Perceptions and use of Assessment Criteria by Internal and External Reviewers
When the reviewers were asked about the criteria they used to assess proposals, most of them said they used the criteria in the national and regional PHRC checklists: scientific relevance of the proposal, originality of the study, methodology, feasibility, ethics, and financial considerations. Many reviewers felt that one or a few items were particularly important, whereas a few of them placed all the criteria on the same level: “The report is only useful if it contains a detailed analysis of all the important points relevant to the funding decision.” (internal reviewer 20). lists the perceptions of criteria by internal and external reviewers.
Reviewers’ perceptions about assessment criteria.
- Originality of the study was perceived as the most important criterion (): “Good projects are based on original ideas. […] If the proposal offers a sound rationale, and says ‘this is what we know, this is the current state of science in this field, and there are absolutely no data about this point, so this is the point we will investigate’ […]” (internal reviewer 7). Although originality was rarely defined by the reviewers, some reviewers felt it was relevant to the potential impact of future publications: “My main interest is in the originality and usefulness of the study – that is, in its scientific originality […] – the results the study will provide, that is the main point in my opinion.” (external reviewer 10). Many reviewers perceived originality as deserving priority, despite the risk involved, as opposed to more pragmatic considerations such as feasibility: “I feel that originality is very important. […] In my opinion, feasibility is less important than originality for a research proposal. […] When there is a flawed but very original idea, this idea can then be refined and improved as the process of research unfolds.” (external reviewer 16).
- Methodology was the second most important criterion according to the reviewers. However, a few reviewers pointed out that advice from the reviewers can result in improvements in the methodology and that, consequently, applications should not be rejected based on methodology alone: “[…] the methodology can be improved, and [what really matters] remains the relevance of the project itself.” (internal reviewer 25).
- Scientific relevance was ranked third in importance among assessment criteria. However, most reviewers did not explain in detail how they decided that a study question was scientifically relevant. This criterion seemed to be perceived as reflecting the usefulness of the proposal for the scientific community or for patients: “Does the proposal address a real issue? […] what is its relevance to patient management?” (external reviewer 15).
- Feasibility was an important criterion for many reviewers: “Clearly, you can have the best idea in the world, if you have only one-tenth of the patients needed to investigate it, there is no point in carrying out the project. Especially when it comes to funding, the money would be wasted.” (external reviewer 9). However, most of the reviewers felt that feasibility was difficult to assess, either because the necessary information was not available (“Well for feasibility, there is no doubt that we often lack the necessary information. We would have to know whether the applicant’s research group has other projects that compete with this one, the size of the population, the size of the group, and we don’t have information on any of these points.”, external reviewer 25) or because the reviewers felt they lacked the necessary expertise (“I am not capable of assessing feasibility. If someone tells me that 200 marijuana addicts are needed for an upcoming trial, then how can I know whether obtaining that number is feasible?” (internal reviewer 1). We identified several strategies used in practice by the reviewers to assess feasibility. One of these strategies consisted in relying on personal experience: “I think this assessment relies mostly on personal experience and on our knowledge about the topic…” (external reviewer 12). Another strategy involved considering the reputation of the applicant and his or her scientific environment: “What matters regarding feasibility is the applicant’s reputation, not practical feasibility. If an applicant previously conducted a project to term then submits another project, then this new project will probably also be completed. Feasibility is based on the individual, not on practical considerations.” (internal reviewer 3); and “In my opinion, a project does not come out of thin air! The project is developed within a research group that knows how to do a number of things … it’s not a castle in the air! … […] So we need to know … I don’t know, it is like when you buy a car, you are more confident if you buy it from someone whose car you know is reliable…” (external reviewer 23). To assess reputation, the reviewers relied on their personal acquaintances with the applicants (“I take a look at the team, and in general I know them so I know if they are able to do it or not”, internal reviewer 10) or on previous publications by the applicants (“They have already carried out projects, so they will be able to carry out this one. If they haven’t completed any projects, then they won’t be able to complete a new project. […]. It is always the same logic. If you are a researcher, you must publish.”, external reviewer 18). Third, reviewers sometimes assessed feasibility based on expectations regarding patient enrollment. However, assessment of this point was perceived as very difficult in some cases: “It can be difficult to predict […]. It is impossible to know how much energy the research team will put into enrolling patients, or how the study will be managed.” (internal reviewer 2). Reviewers had to check the sample size estimations and the recruitment rates: “They have to prove to us that they can enroll the necessary number of patients, for instance by stating that they see X cases each year, and given the inclusion and exclusion criteria, we therefore expect to enroll x% of those patients […]” (internal reviewer 19). The study methodology and, more specifically, the sample size and patient recruitment issues were considered relevant to the feasibility assessment: “In clinical research, the sinews of war are the patients. So there has to be a sample size estimation … then proof that the available number of patients will be sufficient to reach that size. If the number of patients is inadequate, the study will never be done; there is a famous law, I forget its name, saying that when you expect to recruit 100 patients then you actually enroll only 50, with difficulty: it’s always half the expected number.” (internal reviewer 11).
- Ethical aspects were rarely viewed as important by the reviewers. The ethical acceptability of the study was often perceived as easily assessed and not critical for the review: “I rarely encounter major ethical dilemmas and I am not sure that the reviewers are the best people from whom to seek advice on this point. If ethical obstacles exist, then that may have a very small impact [on the assessment], but I am not sure that the ethics of the project should be given the same weight as the other criteria or that ethical aspects should be assessed by the reviewers” (external reviewer 14).
- Financial considerations were considered very important by many reviewers: “[…] If the funding requirements are properly described, then they know how to manage a PHRC grant.” (external reviewer 8). However, most reviewers felt that the appropriateness of the funds requested was too difficult to assess: “We don’t know how to evaluate this. I am always very puzzled about this point. It’s guesswork, isn’t it? We look at things and I think that we don’t have the necessary training, or maybe we should have points of reference […].” (external reviewer 4). Some reviewers felt they lacked the necessary skills to assess financial issues: “I am not competent to give advice about the budget. […] This is not my area of expertise. I prefer to assess scientific issues […]. I often write that for this part of the assessment, I don’t know.” (external reviewer 10).
- Finally, the quality of the writing was rarely mentioned by the reviewers, probably because quality was considered good overall:“[Bad proposals] are very few and their number is decreasing over time, because quality is improving gradually.” (internal reviewer 30).
The Assessment Process during the Committee Meetings
The assessment process of the PHRCs committee meetings was explored during the observation sessions: during the committee meeting, each internal reviewer summarized the application and subsequently the reviewers’ reports then finally gave his or her own assessment. The committee then discussed the funding decision. All committee members participated in the discussion
and had the opportunity to ask questions of the internal reviewers. In practice, the main assessment criteria discussed during the committee meetings were those on the PHRC assessment checklists: methodology, originality, and relevance. Financial considerations were discussed when the funds requested were felt to represent an excessively large percentage of the total funds available for the call. Feasibility issues were also explored, in particular based on the applicants’ résumés and previous applications. As the time available for discussing each proposal was short, the internal reviewer reports and the articulateness of the internal reviewers had a substantial impact on the discussions. Internal reviewers who delivered clear and well-reasoned presentations usually had their opinions accepted by the committee. In contrast, a more lengthy discussion was likely to unfold in response to presentations delivered in a hesitant manner or marked by inconsistencies. Finally, the funding decision was made by developing a consensus and not by majority vote.