|Home | About | Journals | Submit | Contact Us | Français|
Objective. To fill the gap in grant writing training in pharmacology graduate education using an active-learning strategy.
Design. Graduate students wrote subsections of a grant according to NIH guidelines. Students revised their applications based on multiple rounds of critiques from professors and peers throughout a semester-long scientific writing course.
Assessment. Prerevision and postrevision grant drafts were graded. Students were provided with questionnaires assessing their perception of the process. To determine the impact of feedback on the proposals, the quality of the pre/postrevision drafts was assessed by professors who were blinded and unaffiliated with the course.
Conclusion. Student grades improved significantly upon resubmission. Perceptions of the proposals by blinded faculty members favored revised submissions based on multiple criteria. Survey feedback indicated an increase in student confidence in grant writing ability. The results of 3 independent measures demonstrate that intensive feedback on scientific writing improved the quality of student proposals.
Writing is essential for academic and scientific success and the dissemination of scientific findings. Writing and revising a grant application or manuscript helps students learn the research process,1 solidifies the goals of future experiments, enhances awareness of recent research breakthroughs in the field, and places pilot data in a broader context. Manuscript writing, characterized by theme-centered, expository, dispassionate prose of scholarly pursuit, is markedly different from successful grant writing. The latter is characterized by a project-centered approach and a persuasive, personal tone conducive to addressing the service goals of the sponsor.2
Graduate students need critical feedback during this process as they can struggle to frame their research plan and findings in a manner that highlights the relevance and contribution of their work. Junior scientists are often encouraged to provide drafts of grant applications to senior colleagues so flaws in scientific rationale, interdependent specific aims, lack of critical preliminary findings, and unclear prose can be addressed before submission. Generally, the only grant writing training graduate students or postdoctoral fellows receive is from assisting their mentor with his or her application, or possibly in writing their own fellowship proposal. The introduction of an explicit grant writing course or training program would, therefore, benefit students in transition to independent research careers.
The goal of the present study was to incorporate grant writing into a pre-existing research seminar course for students enrolled in the Graduate School of Pharmaceutical Sciences at Duquesne University. Many of these students may depend on scientific writing in their future careers, whether in industry, academia, or government research settings. Learning grant writing skills in graduate school may even bear on the student’s accessibility to a faculty position.3 One approach to improving the opportunity for successful funding before entering the profession is to write and then rewrite mock grant applications in graduate school.4 Thus, a priority in the pharmacology graduate curriculum at Duquesne is to prepare graduate students to write scientifically for an audience of scientists in their field, as well as for the broader scientific community.
A lack of grant writing instruction is evident at many institutions of higher education.5 Guidance from the literature regarding writing a National Institutes of Health (NIH) starter grant proposal is relatively sparse.6 While there is a need for including grant writing as a requirement in biomedical graduate coursework,7, 8 detailed and specific writing instruction is typically not part of formal graduate coursework at research-intensive universities. This is surprising because writing skills are a key competency for a successful academic or biomedical career.9,10 Indeed, students interested in biological research identify grant writing as their weakest skill.11 This gap may be prevalent at research-intensive institutions because their faculty members are largely paid from grants and not for teaching.11 Yet, the need for graduate training in effective scientific writing is evident. The American Association of Colleges of Pharmacy (AACP) Graduate Education Special Interest Group has advocated graduate training in communication and grantsmanship to prepare students for careers in academia and industry, recommending the development of foundational courses in scientific communication within pharmaceutical sciences graduate curricula.12
In the present study, we hypothesized that graduate student rewriting, guided by instructor feedback, would significantly improve grantsmanship and the quality of scientific prose. Pharmacology graduate students were trained in grant writing via lecture and class discussions and then asked to write an R03 grant proposal according to guidelines published by the NIH. An R03 grant is designed to fund a small-scale 2-year pilot study for up to $100 000 and requires little if any preliminary data; the research strategy section is limited to 6 pages. According to the NIH website, “discrete, well-defined projects that realistically can be completed in 2 years and that require limited levels of funding” are suitable for the R03.Thus, this NIH format was an appropriate choice for a grant writing course that would include multiple rounds of submission and feedback. The R03 assignment served multiple roles. First, it honed scientific writing and grantsmanship skills. Second, it served as practice for the department’s comprehensive examination, part of which includes a mock R03 research proposal on a topic that the students are not directly researching in the laboratory. Third, students were encouraged to write proposals regarding their own research to provide a glimpse into the workday of a pharmacy faculty scholar and to serve as the nucleus for pre/postdoctoral applications for NIH National Research Service Awards.
“Oral/Written Presentations Skills in Pharmacology” is a mandatory course taken every fall semester by pharmacology PhD and MS students. All PhD students must enroll in the course for 4 consecutive years, and all MS students must enroll for 2 consecutive years. Repercussions for poor performance include failing and repeating the course. Furthermore, a minimum grade point average (GPA) of 3.0 is required to remain enrolled in the graduate program; thus, a “C” grade for this course may jeopardize the student’s PhD candidacy. The course has 2 foci: preparing and delivering research seminars and training in research grant writing.
After appropriate training in the course, students were asked to propose an original research idea and to write several subsections of an R03 grant application, following NIH formatting and instructions.13 The R03 mechanism affords the full NIH grant writing experience, but on a scale compatible with a course assignment. The R03 was also chosen for its training potential: The pharmacology PhD students at Duquesne must successfully propose, write, and defend an R03 application before progressing to the dissertation defense, and the R03 is a likely starter grant for an assistant professor in a pharmacy school that employs a faculty “teacher/scholar” model. Although not assessed here, all students enrolled in the course were also expected to deliver a research seminar to the pharmacology students and faculty members and a pharmacology-based lay presentation to eighth graders at a local middle school.
The course directors first instructed students on the basic structure of an R03 proposal per the NIH format, which includes specific aims, significance, innovation, and approach sections. In accordance with NIH instructions, the course directors also instructed the students that the grant must include a description of the importance of the scientific problem, or “critical barriers to progress in the field,” and an explanation of how the proposed project will improve “scientific knowledge, technical capability, and/or clinical practice.” In addition, the innovation section had to explain how the application “challenges and seeks to shift current research paradigms.” This section also had to describe any “novel theoretical concepts or methodologies or any refinements” and “improvements to current concepts.”
The approach section (experimental design) had to describe the overall methodological strategy and plans for data analysis. An important component of this section was the discussion of “potential pitfalls, alternative strategies, and benchmarks for success.” Students were also encouraged to describe strategies that addressed “feasibility and the management of risky projects.” Multiple examples of successful, well-structured grants were presented as guides, and the students were encouraged to discuss pros and cons of each example with each other and with the professors. Websites and resources that offered grant writing tips also were reviewed in class.14
Student discussion on how much scientific background to cover was encouraged, given that there is no section devoted to background information in the new NIH format. As an example, students were encouraged not to provide extensive coverage of background literature or to discuss the economic and social impact of the diseases for more than a few introductory sentences. Rather, students focused on how the proposed project would add new and important information to the field.
Students submitted grant sections over the course of the semester at biweekly intervals. In the first iteration of the course, course directors provided extensive feedback on first submissions and a preliminary grade was assigned for each effort. At the end of the semester, students submitted all revised sections as one combined document for final grading. As a result, they were graded twice on the same grant application. In the second offering of the course, students were paired in order to provide peer feedback on the original document before submission for the first round of grading. After the students incorporated peer comments, the course directors provided extensive written feedback on the assignments. Specific feedback was provided as comments within the document and as a general summary at the end of the grant application to force students to rewrite the proposal themselves, as opposed to only accepting changes that were inserted directly into the text.
The effectiveness of these exercises was assessed in the course over 2 years with 3 mechanisms. First, grades for the first and second grant drafts were compared by the paired, 2-tailed Student’s t test (paired grades for each individual student). Data were only deemed significantly different if p≤0.05. Second, a 5-point Likert scale questionnaire with a comments section was administered to the students after the end of each semester. The questionnaire, approved by the Duquesne University Institutional Review Board, was anonymous and voluntary. Students were requested to return the forms to a departmental administrative assistant in the envelope provided. Linear regression analyses were performed to test for correlations across directly or indirectly related questions. A 2-tailed determination of significance was set at p≤0.05. Data were analyzed with SPSS, v19 (IBM, Armonk, NY).
For the third assessment mechanism, faculty members not directly involved in the course were asked to voluntarily grade the original and revised grant abstract submissions. Rubrics were left in mailboxes of faculty members who had a history of receiving external funding. The faculty members were asked to grade each abstract according to clarity of writing, strength of rationale, rigor of experimental plans, and quality of scientific language (see Table 1 for rubric). These faculty members were blinded—they were not made aware of the student name or which abstract was the original or revised version. Seven faculty members unaffiliated with the course participated in the blinded grading. Grades for the original and revised abstracts were contrasted by the one-tailed Student’s t test (pairing for this measure is described in figure/table legends). The one-tailed t test was chosen because the first assessment (student grades) had already showed a robust effect of instructor feedback and editing on student performance.
The efficacy of instructor feedback was assessed in 3 independent ways. First, grades for the original and revised submission were contrasted statistically. Second, faculty members not directly involved with the course graded the original and revised abstracts from Year 2 in a blinded manner. Third, a feedback survey was administered to the students. As shown below, detailed instructions and formal feedback were found to elicit measurable improvements in grant writing, as originally hypothesized. Multiple independent statistical measures suggested that this course accomplished its major goals of preparing students to develop a grant proposal, incorporate scientific feedback and criticism into their research, and improve the lucidity of their scientific writing.
In the first year, 9 students were enrolled in the course. One of these students graduated immediately after completion of the course and was not included in the student feedback survey, so that 8 students were included for year 1. However, this student’s grades could still be analyzed (ie, 9 students for year 1 for grade analysis). In the second year, 10 students were enrolled in the course. One of these students left the program (n=9 students for year 2 for both the survey data and the grade analysis). For both years, the course had 2 instructors, rendering student-to-faculty ratios of 4.5:1 for year 1, and 5:1 for year 2. Only 3 students from year 1 were also enrolled in year 2.
Students largely followed the directions for the outline and format of an R03 grant in the first round of submissions. Peer review mostly addressed issues of clarity and background on the initial draft. After editing based on peer comments, course directors provided scientific feedback, mirroring an actual NIH critique. As identified by Crowe and colleagues, student writing was strongest when reflecting knowledge or comprehension; synthesis of new ideas or critical evaluations of a technique or body of knowledge were more difficult.8 For example, potential pitfalls, technical difficulties, or unexpected outcomes that required higher-order thinking were often underdeveloped or absent from the first submissions. The instructors commented on these weaknesses and requested students to provide alternative interpretations of both expected and unexpected outcomes. In addition, students were requested to add appropriate controls to the experiments and assays to avoid “falsely positive” or “falsely negative” conclusions. Extensive feedback on basic grammar and sentence structure was also provided. For instances in which a student made repeated mistakes in sentence structure or style, an example sentence from the grant was rewritten to demonstrate a more logical flow.
Using a 4-point scale, the average grades for initial and final abstract drafts were 3.2 (0.3) and 3.8 (0.3), respectively (n=9 entries from year 1, n=9 entries from year 2, for a total of 18 grade entries, with 3 students repeating the course). The difference between initial and final submission averages was significant (p≤0.001). The robustness of the effect suggests that student writing did improve with feedback as expected.
Significant improvements in the revised abstracts were noted regarding clarity of writing, strength of rationale, and rigor of experimental plans (Table 2). The largest impact of grant revision appeared to be on clarity of writing. There was no significant improvement with respect to scientific language, the highest-ranked domain for the original abstract versions, possibly because students continued to use undefined acronyms and scientific jargon (Table 1).
When considering the averages of all scores in all domains, all but 2 students (3 and 5) showed significant improvement or trends towards improvement in the revised abstracts (Table 2). Student 3 was already a strong writer; the performance of Student 5 declined in all respects over the semester, resulting in the lowest grade in the course. An analysis of average scores for all 9 students (including the 2 who did not show individual improvements) demonstrated that a significant improvement overall occurred in the revised submissions. Comments from faculty members regarding the abstracts supported these quantitative data.
For the original versions of the grant abstracts, faculty members agreed that the explanations of the experiments were “considerably worse,” not “adequately defined,” that the grants contained “wide, sweeping statements” and “vague approaches.” Furthermore, multiple reviewers noted that the originally submitted grants were “too wordy,” thereby sacrificing clarity and were generally “lacking in detail.” In contrast, faculty members wrote that the revised versions were “very clear in terms of the goals of the study,” had “clarified rationale,” “clearer and more comprehensive Aims,” “better flow,” and a “more complete story,” and exhibited fewer “unnecessary words.”
Three students repeated the course in the second offering. This allowed assessment of whether or not repeating the class affected the quality of abstract revisions. To this end, changes in the original and revised abstract scores for these students (see fourth column in Table 2B for this measure) were compared to those for the more junior students (n=6). The senior students exhibited the same overall degree of improvement as the junior students (p=0.834). Thus, repeater students continued to respond positively to grant writing feedback.
The survey response rate for both years was 100% (8 responses for year 1, 9 responses for year 2). Overall, students acknowledged improvements in their grant writing and general scientific writing skills (Table 3). The highest scores (at or above 4.5 out of 5) were elicited by the statements, “This course taught me how to organize and develop the subsections of a grant,” and, “The feedback on my grant proposal helped me to improve the quality of the English in my revised document.” The latter finding was not surprising given that approximately half of each class consisted of ESL students. The comments section question, “Which aspect of the course had the greatest impact on your learning?” elicited the response, “…grant writing assignments helped for a better understanding of scientific writing and communication.”
Several students responded that the grant writing portion had the greatest impact on their learning and that regular writing encouraged improvement of writing skills. For the comments section question, “What suggestions can you offer to improve the course?” one student wrote, “Keep the grant-writing section – it’s very practical and I really appreciated [the instructors’] input/comments.” This feedback suggests that the students recognized the added value of repeated grant writing assignments and feedback in the course. All evaluation scores improved from year 1 to year 2 regarding how well the course prepared students for the mock research proposal, how much the feedback helped in rethinking experimental design, and how much the feedback stimulated new research ideas. Student confidence in producing future research grants also increased in the second year (eg, item 9).
A major goal of the grant writing exercises was to help students prepare for the original research proposal (ORP), a mock grant proposal component of the graduate school’s comprehensive examination typically attempted by third-year or fourth-year PhD students. Some year 1 student comments for Item 3 reflected anxiety over a recent revision of the ORP rules, but the class in year 2 felt more strongly that this course would help prepare them for the ORP examination.
Students on average did not anticipate a career that required regular grant writing (Item 6), perhaps because pharmaceutical science graduate students primarily consider career paths in teaching, consulting, or industry, though any of those career paths could include grant writing.
Students disinterested in future career paths with a direct grant writing component might be less inclined to favorably view a time-consuming class exercise devoted to grantsmanship, as tested in the Pearson correlation analysis. The finding that the year 1 students did not feel much more capable of producing similar grants in the future (Item 9) suggests a general sense of insecurity in grantsmanship. Regarding this statement, one student wrote, “I might need some more practice and examples of what a finished product should look like.” This comment strongly supports the need for further instruction in grant writing. In the comments section another student wrote, “I think we should spend more class time devoted to explaining the sections of a grant and how to write them.”
Concerns include the responses to items 5, 10, and 11 of the questionnaire. Fewer year 2 students disagreed with the Item 5 statement that the course should not be required (ie, more year 2 students thought that the course should not be mandatory). This may be attributable to the seniority of 3 students in year 2—three students had just experienced 3-4 consecutive years of this course. The negative phrasing of Item 5 also may have caused confusion in the respondents to account for the lower scores. The Item 10 score indicated that students found less useful the peer reviews of their grant proposals, an exercise new to year 2. Students also reported being unable to gain deeper insights into the strengths/weaknesses of their own writing by being exposed to other grants via peer review. The scientific and critical-thinking skills of the students likely had not reached an adequate level of maturity. The student feedback score (2.9) for this item contrasted significantly with student opinion on the usefulness of instructor feedback (4.5).
A Pearson correlation analysis of individual student responses (Table 3) was performed to test if the scores for any particular question predicted the scores for any of the other questions (Table 4, 2-tailed analysis of significance). This was done to test the hypothesis that students who anticipated grant writing in the near future (eg, comprehensive examination, postdoctoral applications) or in their ideal future careers would be more likely to find that the writing exercises were worthwhile. This analysis also allowed determination of which type of student felt the course should be mandatory or elective. This was important to assess because several of these students were more interested in careers in industry, where scientists write fewer grant applications than in academia.
Data were gathered separately for both years, and the domains that were correlated differed from year to year (Table 4). For example, in year 1, agreement with the statement that the student’s ideal career would involve writing similar grant proposals (item 6) also predicted agreement with the statement that the course prepared students for the graduate school’s ORP examination (item 3).Thus, students who wished to write grants in the future and perhaps stay on the academic career track were more likely to appreciate grant writing practice before the ORP examination. In turn, students who felt the course was preparative for the ORP were also more likely to agree that the feedback challenged them to think of new research ideas (item 8) and that they would be able to produce similar grants in the future (item 9).
For year 2, there was a negative correlation between agreement that this course should not be mandatory (item 5) and: (a) whether the course taught them how to organize and develop a grant proposal (item 2), or (b) whether or not this course helped them prepare for the comprehensive examination (Item 3). In other words, students in favor of the course remaining mandatory were more likely to agree that the course prepared them for grant writing and the ORP examination.
There was also a negative correlation between the statements that the feedback made students rethink some of their experiments (item 4) and that their ideal careers would involve writing similar grant proposals (item 6).This may suggest that students who already had their own insights into the limitations and caveats of scientific experimentation were more likely to strive for an academic career. Finally, there was a negative correlation between finding the peer reviews valuable (item 10) and agreement with the statement that the course should not be mandatory (item 5).This suggests that students who found the peer reviews to be helpful also felt that the course was valuable enough to remain a requirement.
The results of this study indicate that student grant writing can be taught effectively if extensive critique from instructors is provided. Thus, this active-learning approach may be preferable to traditional “sage on the stage” lectures. In general, the students reported feeling that the exercises in grant writing were useful. For both years that the course was offered, students indicated that feedback provided for their proposals improved the quality of their scientific writing and that they learned how to develop the subsections of a research grant. A majority of students agreed that this course should remain a prerequisite for graduation.
Evidence for a culture change among students was found in the responses to the survey’s comments section question, “What advice do you have for future students in this course?” One student responded, “Read the comments you receive on … all written work and be willing to change your writing style based on the feedback. Don’t be stubborn because what sounds good to you may not come across to your audience the way you had hoped.” Another student wrote, “Write regularly, revise, revise, revise.”
Because students are so familiar with their own research, they often cannot place themselves in the reader’s shoes. There is also a general tendency towards turgid prose in scientific writing that graduate students naturally absorb from their reading of the scientific literature. To appreciate the reader’s dilemma, students were asked to read each other’s proposals in the year 2 class. This was expected to help the students place their own proposals in the proper context, and encourage communication of complex scientific ideas in a straightforward, economical manner.15,16 Even before the responses to this questionnaire were collected, the instructors had observed a general lack of quality in the weekly peer reviews. In retrospect, the fact that the peer reviews were not anonymous was a major design flaw in the course. This aspect undoubtedly discouraged frankness, as peers were asked to provide unflinching critique of each other’s work. An alternative proven successful in the classroom would be to create small peer review groups (mock NIH study sections) among the students, and have a given “study section” review proposals of classmates outside of that group.1,17,18 Future iterations of the course will include further instruction from the professors on the extent and focus expected for peer review.
Although the students in our study did not appear to use the peer feedback in their resubmissions, the limited room allotted for most grant proposals (as opposed to a dissertation) and the highly competitive nature of grantsmanship make it essential that students learn to write with brevity and take into account the reader’s perspective.4,19 Students also initially struggled to place their experimental design and hypotheses into a clear, concise presentation. After the peer and instructor feedback, the instructors observed better-defined justifications of rationale for experiments and greater attention to alternative outcomes and interpretations. A reduction in grammatical errors and verbosity was seen in student resubmissions, even in the sentences not present in the original submission.
In general, the language became simpler and the sentences easier to comprehend following instructor feedback. The improvement in the submissions was independently and clearly noted by faculty members outside of the course. The Pearson correlation analysis suggested that students who felt the course helped prepare them for their comprehensive examination also agreed that they would be able to produce similar grants in the future and would use grant writing in their ideal future careers. These findings suggest that intensive grant writing instruction can help prepare students for careers in the biomedical sciences by improving clarity of prose, strength of rationale, and quality of experimental plans. All of these improvements may boost student self-confidence in tackling major writing tasks in the future.
Scientific writing is a key though underdeveloped skill for graduate students in the pharmaceutical sciences. We hypothesized that the application of scientific writing in grant proposals could be improved through repeated rounds of instructor and peer feedback, discussion, and editing of student grant applications. The results demonstrated that student confidence increased after participating in the course, and that students reported greater insight into the grant writing process. Quantitative assessment of student proposals by faculty members unaffiliated with the course showed that the student proposal abstracts improved in clarity, description of scientific rationale, and explanation of experimental plans. This course design could be adapted to pharmaceutical science graduate schools or other biomedical programs in order to provide a mechanism for developing grantsmanship skills among junior scientists.
We are grateful for the outstanding administrative support of Mary Caruso, Deborah Willson, and Jackie Farrer. We thank the graduate students for their eagerness to participate in this exercise and provide survey data. There are no conflicts of interest to declare.