|Home | About | Journals | Submit | Contact Us | Français|
Statistics is an essential training component for a career in clinical and translational science (CTS). Given the increasing complexity of statistics, learners may have difficulty selecting appropriate courses. Our question was: what depth of statistical knowledge do different CTS learners require?
For three types of CTS learners (principal investigator, co‐investigator, informed reader of the literature), each with different backgrounds in research (no previous research experience, reader of the research literature, previous research experience), 18 experts in biostatistics, epidemiology, and research design proposed levels for 21 statistical competencies.
Statistical competencies were categorized as fundamental, intermediate, or specialized. CTS learners who intend to become independent principal investigators require more specialized training, while those intending to become informed consumers of the medical literature require more fundamental education. For most competencies, less training was proposed for those with more research background.
When selecting statistical coursework, the learner's research background and career goal should guide the decision. Some statistical competencies are considered to be more important than others. Baseline knowledge assessments may help learners identify appropriate coursework.
Rather than one size fits all, tailoring education to baseline knowledge, learner background, and future goals increases learning potential while minimizing classroom time.
Clinical and Translational Science (CTS) is an emerging field that is drawing together researchers from a wide variety of disciplines. The goal of CTS is to move foundational discoveries into testing for clinical safety and efficacy in controlled environments, to test effectiveness in a real‐world setting, and finally to enhance implementation to improve the health of the population. The National Institutes of Health (NIH) has invested significantly in training for careers in CTS, including through the Clinical and Translational Science Award (CTSA) mechanism. To become an expert in CTS requires grounding not only in medicine and the biological sciences, but also in quantitative research methods.
Statistics are increasingly reported in the medical literature.1, 2, 3 Unfortunately, statistical mistakes are common in work published without the benefit of statistical expertise.4, 5, 6 Just as Zelen7 and Begg and Vaughan8 argue that graduate students in biostatistics should be better trained to work with investigators in the biomedical sciences by incorporating fundamental knowledge of biomedical specialty areas, a similar argument can be made for training CTS learners in biostatistics. Consequently, training in CTS typically includes methodological coursework in statistics for learners who typically have little prior formal training.
Because CTS is a relatively new domain, a variety of educational programs, courses, and curricula are being developed. Core competencies for CTS learners have been developed to include introductory topics in biostatistics and epidemiology.9, 10, 11 Although core competencies may provide a fundamental framework, learners will be challenged to keep pace with the expanding field of statistics. In particular, current coursework may be driven by preexisting degree programs rather than the needs of CTS learners. Courses often include statistical theory and statistical programming designed for graduate students in statistics and biostatistics, and not for CTS learners. Lesser and Parker12 argue that biostatisticians should develop statistics courses specifically for biomedical investigators. Windish et al.,13, 14 Novack et al.,15 and Rao16 discuss the challenges that physicians and others who receive CTS training have with traditional statistical coursework. O'Brien et al.17 discuss that when developing statistics courses for CTS learners, it is important to recognize that “conducting clinical research requires a fundamental understanding of the terminology and concepts of biostatistics,” but that “few clinical scientists need to know the mathematical and computing technicalities as covered in traditional academic courses.”
One of the challenges related to developing courses in statistics for CTS learners is the wide range of experience among those seeking further training. Ambrosius and Manatunga18 suggest that while many students prefer to spend time on specialized topics related to their focus area, others need to learn or relearn the fundamentals first. In the field of CTS, one question that has yet to be addressed is the level and breadth of statistical competencies required by scientists with different clinical research experiences and future goals.
The national CTSA consortium has provided a framework for Biostatistics, Epidemiology, and Research Design (BERD) practitioners to collaborate to develop best practices for integrating BERD into CTS. Key metrics by which the BERD community measures itself include the training of CTS learners.19 Indeed, a central tenet of the BERD community is to “ensure that the biostatistics education they provide to investigators meets high standards” and that “such education is continuous and occurs at many junctures—from one‐hour consultations to formal classes to multiyear degree programs to ongoing mentoring and collaborations.”19 With this mandate, leading members of the BERD community came together to answer the question of what statistical training is most appropriate for which CTS investigators given their individual level of experience and future goals.
At the 2012 CTSA BERD Key Function Committee Face‐to‐Face Meeting in Washington, DC, moderated small group discussions were held to explore “teaching biostatistics, epidemiology, and research design: who, what, when, where, and how.” After the meeting concluded, invitations were issued to BERD members to participate in a formal assessment of recommendations on statistical education for CTS learners. We used a two‐part convenience sampling strategy. The active members of two committees (the Education Working Group and its parent committee, the Online Resources Task Force), consisting of 22 members anticipated to be most interested in education, were targeted with an opt‐out strategy. In addition, an opt‐in invitation was extended to 67 additional individuals participating in the 2013 CTSA BERD Key Function Committee Face‐to‐Face Meeting in Washington, DC. Twenty individuals (13 opt‐out committee members and 7 opt‐in individuals) agreed to receive the assessment; of these, 18 (all 13 that were sent to the opt‐out committee members and 5 that were sent to the opt‐in individuals) were completed and returned. Participants also provided information about their background and expertise to help contextualize the recommendations.
Our assessment was developed around a previously published set of statistical competencies.11 Each participating BERD member was also solicited for additional topic areas of importance; four new competencies especially salient to CTS were included to reflect recent developments in statistical practice: reproducible research, diagnostic testing, data presentation, and data quality and management. The full assessment is available in Appendix 1.
In the final assessment distributed to BERD members, respondents were asked for each topic what level of learner competency was needed (none, some, or high) for each topic given the learners career goals (principal investigator, co‐investigator, reader of the literature) and research backgrounds (no prior research experience, reader of the research literature, has been involved in research).
Data analysis is descriptive. We summarized the proportions of respondents who indicated a learner should achieve a certain level of competency given the learner's anticipated career goal and prior research experience. It should be noted that for learners who had a goal of only reading the literature, we presumed their background was little or no prior research experience.
We grouped the competencies into three categories—fundamental, intermediate, and specialized—based on how frequently they were considered important. Fundamental competencies were identified as those where a large majority (≥70%) of BERD members felt CTS learners in the group would require training. Intermediate competencies were those for which between 60% and 69% of BERD members felt CTS learners in the group would require training. When less than 60% of BERD members indicated CTS learners would require training in a topic, it was considered a specialized competency. Cut points were determined post hoc. The fundamental, intermediate, and specialized categories ended up containing 9, 6, and 6 competencies, respectively.
For each statistical competency, we examined the association between career goal and the need for training by comparing the percentage of respondents who felt that there was a high need for training between learners with different goals. Due to the descriptive nature of this research, no formal statistical tests were performed. Differences in percentages of at least 22%, which corresponds to 4/18 respondents, were considered meaningful. This cut point was determined by surveying the manuscript writing group using the Delphi Method.20 Data management and descriptive analyses were performed using SAS, version 9.3 (SAS Institute, Inc., Cary, NC, USA).
Of the 18 respondents, 17 had earned a doctoral degree and one was nearing completion of a doctoral degree, with 78% (14/18) of these degrees in biostatistics or statistics, and 22% (4/18) in epidemiology. Ten (56%) respondents were male. The mean (standard deviation) number of years of teaching experience since earning the doctoral degree was 18.9 (8.5). The entire group of experts regularly teaches learners including pre‐ and postdoctoral medical and allied health students, residents, fellows, and nonstatistical health sciences faculty.
Results for fundamental, intermediate‐level, and specialized competencies appear in Tables 1 , 2 , and 3 , respectively. The fundamental competencies include concepts traditionally taught in introductory biostatistics and epidemiology courses. There was only one competency that is not (yet) traditionally taught at the introductory level: understanding the reasons for performing reproducible research. This competency was not consistently considered important for learners whose goal is only to read the literature. However, training in this area was highly valued for learners who intended to become principal investigators and at least moderately important for co‐investigators. We also noted that there was only one competency, assess sources of bias and variation, for which every respondent felt that at least some training was needed for all learners with no previous research experience, regardless of CTS aspirations.
The intermediate level competencies comprised less general statistical topics, such as interpreting results in light of multiple comparisons and distinguishing among different measurement scales. These were considered valuable for learners intending to become Co‐Is or PIs, but of less value for learners whose goal was to read the literature.
The specialized competencies comprised statistical topics that are usually represented in upper‐level courses, such as stopping rules in clinical trials, meta‐analysis, statistical methods to address missing data, and diagnostic testing. However, there were two additional specialized competencies that are typically taught in introductory coursework or applied coursework: sample size and power calculations and data quality and data management procedures. The former is often taught at a lower level in introductory courses but may require more complex knowledge when implemented in the real world, and the latter requires real‐world experience to understand the level of complexity involved. The specialized competencies were not considered to be of high value for learners whose goal was to read the literature or become a Co‐I, but were considered important for learners who intended to become PIs. This likely reflects the importance of these topics for leaders of complex clinical and translational studies.
Learners with no CTS background who wished to read the literature were consistently determined to need less training in the fundamentals than learners who wished to become PIs. Among the fundamental competencies, the smallest difference was in the need to assess sources of bias and variation. The largest difference was in the need to propose study designs. For the intermediate competencies, the smallest difference was in the need to interpret results in light of multiple comparisons. For the specialized competencies, the largest difference was in the need to describe appropriate data quality and data management procedures. When comparing competencies for learners who wished to learn to read the literature to learners who intended to become Co‐Is, the only competency items which differed meaningfully were the ability to communicate research findings, understand the reasons for performing reproducible research, distinguish among the measurement scales, understand general simple descriptive and inferential statistics, and describe appropriate data quality and data management procedures. Overall, responders indicated that future PIs required increased training when compared with future Co‐Is across all competencies. Interestingly, only diagnostic testing did not differ perhaps because only 50% of respondents felt that future PIs with no prior research background had high training needs in this domain.
For learners who already read the literature, we considered training goals of becoming either a Co‐I or PI. For the fundamental competencies, respondents felt that all but two were more important for future PIs than Co‐Is. Describing concepts and implications of reliability and validity was the topic most often considered to be of more importance for future PIs than future Co‐Is. Similarly, all but three of the intermediate and specialized competencies were considered more important for future PIs than future Co‐Is. Scrutinizing assumptions and limitations, and characterization of diagnostic testing were the topics with the greatest difference.
When comparing learners' goals for those with prior research experience, only six of 15 fundamental and intermediate competencies differed between future Co‐Is and future PIs. Of these, the largest differences were observed for scrutinizing assumptions and limitations and identifying inferential methods appropriate for clustered, matched, paired, or longitudinal studies. In contrast, most of the meaningful differences between future PIs and future Co‐Is were among the specialized competencies. Only one competency failed to reach our threshold for a meaningful difference; need for training in computing sample size, power, and precision for comparisons was similar for both groups.
For most competency areas, there was a trend toward lower expectations for training as the learner's prior research background increased. However, for the competencies regarding paired data, adjustment and interaction, and data quality and data management there was an increase in expectations for those with a greater research background, especially among future Co‐Is (Figure 1 ). This suggests that priorities for certain competencies may require greater emphasis once fundamental concepts are mastered.
Figure 2 focuses upon four competency areas for which the assessed needs for Co‐Is and PIs differed as the learner's research background increased. For all four competencies, there was a decrease in the expectations for PIs with greater research background but increasing expectations for Co‐Is who had more research experience. These differences suggest that training needs may vary by learners' goals and backgrounds.
We identified three levels of statistical competencies, not all of which are appropriate for the goals and perspectives of all learners. Institutions should account for learners' multifaceted needs rather than providing a one‐size‐fits‐all curriculum. As might be expected, we found the greatest competency expectations for future PIs and lower expectations for Co‐Is and readers of the literature. Furthermore, learners with the same goals may have subtle differences in curricular needs based upon their prior research experience. For most competencies, there was a reduction in curricular expectations for learners with more research background. For a few of the more challenging competencies (paired data, adjustment and interaction, and data quality and data management), our results suggest that those with more research experience should be trained in these concepts so they can maintain data integrity within the context of team science. Importantly, our work indicates a potential need to revise the CTSA biostatistical competencies.
Curriculum design varies by institution, and is not included in this manuscript. However, we believe that the fundamental statistical competencies are likely included in nearly every introductory statistics course for CTS learners, with one critical exception. Reproducible research is not typically included in statistical curricula for nonstatisticians. Reproducible research has come to public attention in recent years because of a plethora of preventable research mistakes and even flagrant violations.21 Consequently, as a field we are learning the importance of training nonstatisticians in understanding the critical nature of reproducible research methods to support published research. However, statistical curricula may need to expand beyond the fundamental statistical competencies to better address this area.
The intermediate and specialized statistical competencies we identified may be taught sporadically, if at all. For example, meta‐analysis may be introduced only briefly in a required course, with a higher level elective option potentially available for interested learners. Our results are consistent with this approach, suggesting that all learners should get a good grounding in the fundamental statistical competencies, but that the learner's goal should help drive the choice of specialized material.
A common concern in any graduate program is how to cover emerging methodology while still ensuring coverage of the classical topics. This concern also applies to CTS learners. Our results suggest that some topics are not appropriate for all learners, and that these could be removed from introductory curricula. Instead, learners should be encouraged to work with the statistician on their research team for such topics. These include computing sample size, power, and precision, and describing methods for loss to follow‐up. Removing these could help create curricular space for reproducible research and other fundamental statistical competencies. In addition, self‐paced, just‐in‐time learning could help meet students' curricular needs. This may allow flexibility with new topics and emerging research areas.
We observed trends across the learner's goal (reader of the literature, Co‐I, and PI) with increasing needs for learners with more advanced goals. In practice, this could drive tailored coursework with perhaps add‐on materials or separate courses for the different learner goals. Topics that were considered to be needed differently across learner goals included scrutinizing the assumptions and limitations of different statistical methods, early stopping rules for clinical trials, meta‐analysis, communicating research findings, effect size, describing the study sample and sampling methods, multiple comparisons, repeated measures, and adjustment and interaction. Moving these more challenging topics out of the classroom for learners who do not need them may make statistics more appealing with regard to the critical fundamental statistical competencies. Learners who need to use these topics in their research would then be candidates for within‐consultation training with a statistician on their research team.22, 23
We believe the CTSAs can help achieve the goal of improved statistical training for CTS learners. Not every institution needs to develop all the coursework. Indeed, not all institutions have the resources to develop coursework on every topic. Instead, we can use these ideas to leverage the CTSA network to compile training materials from across the nation, and make them available through media such as CTSpedia (http://www.ctspedia.org).
Both course evaluation and program evaluation are growing in popularity as educators realize the importance of evaluation for updating course materials. We propose that the statistical competencies identified be used as a basis for program evaluation. In particular, evaluation should differentiate between the fundamental, intermediate, and specialized competencies and permit an individualized approach tailored to each learner's needs based upon their background and intended role.
In addition to program evaluation, we believe learners would benefit from baseline knowledge assessment to help them identify statistical competencies upon which to focus their coursework. Learners searching for education in statistical topics may not know which topics they need training for. Assessment instruments closely aligned with the statistical competency areas could help learners to identify needed and targeted training. These tools might be used at the start of a program, the start of a course, or on an ad hoc basis for just‐in‐time learning. This might even challenge the concept of “required coursework” in a program. Pretesting is frequently done, but it rarely impacts course or program content delivered. Pretesting could be used to steer learners toward the most appropriate topics to ensure a rounded knowledge base. This may also decrease heterogeneity of learner knowledge levels in a class, facilitating content delivery.
Our initial competency list was derived from combining competencies in CTS and Public Health.11 During our process, four additional competencies were added: reproducible research, diagnostic testing, data presentation, and data quality and data management. These competencies were proposed by multiple respondents, and were then agreed upon as being important by the full group.
However, our results highlight the potential need for further change. Appendix 2 lists the many suggestions that arose during this collaboration. Some of the fundamental and intermediate level competency areas have verbs associated with higher cognitive function (Bloom's taxonomy). For instance, in scrutinize the assumptions and limitations behind different statistical methods the term “scrutinize” ignores the team science aspect of CTS in which learners work with an interdisciplinary group that should include a statistician. “Scrutinize” suggests a higher level of statistical knowledge to compare and contrast different statistical methods with regard to the study design, type of variables and their statistical distributions in the study, missing data, transformations of current variables, impact of censoring, and so on. We concluded that the competency would be better phrased with the term “assess,” which would imply that the goal is to simply understand the general assumptions of each statistical method, which would provide learners with the tools to identify when they need to seek statistical help. In general, our edits to the phrasing for the fundamental and intermediate level competencies were intended to reduce the required competency levels.
For the specialized competencies, on the other hand, it appeared that learners often needed to attain a greater level of understanding in order to serve as the PI for a CTS study. For example, loss to follow‐up was associated with the verb “describe,” but we felt it would be better aligned with the verb “understand.” Loss to follow‐up is defined as incomplete ascertainment of the primary outcome for participants randomized in a controlled clinical trial. Learners taught to “describe” loss to follow‐up are asked to estimate and report the percentage of participants with this issue when reporting the trial. However, learners instructed to “understand” loss to follow‐up will be expected to classify reasons for this, such as (1) nonadherence to the intervention, (2) cross‐over to the other intervention, (3) loss of contact because of study‐related illness, (4) loss of contact because subject moved out of the geographical area or other nonstudy related reason. In addition, learners instructed to “understand” loss to follow‐up will need be fully aware of its consequences for statistical methods and inferences. For the specialized competencies, our proposed phrasing edits were intended to increase the required competency level for PIs.
To our knowledge, this is the first assessment in which learners' individual background and goals were considered when anticipating needs for statistical training. Our evaluation was performed by a limited sample of BERD educators who are engaged in CTS at academic research institutions that are part of the CTSA consortium. The nature of our sample (all BERD members with an interest in biostatistics education) makes our results applicable primarily to CTS learners, though results may be useful in related populations such as public health students. A few other minor concerns arose during our discussions of the results. The term “Co‐I” was not consistently defined by all respondents. We decided to retain the results for this learner goal, as the group felt this inconsistency was reflective of the nature of CTS training for this intermediate population. Furthermore, our scale of none, some, or high did not provide much opportunity to distinguish learners for individual respondents. We retained this scale because the aggregate results provided additional detail.
Experienced statisticians believe learners with different backgrounds and especially learners with different goals have disparate needs for statistical training. Our results suggest statistical coursework should be individually tailored to meet learners' needs. This can be achieved through a baseline knowledge assessment linked to the statistical competency areas that will help learners select appropriate coursework. Overall, these changes would permit a truly individualized statistical academic experience for CTS learners, guiding them to more efficiently realize their intended goals with the best possible knowledge base.
The authors wish to thank Laura Lee Johnson, PhD, of the NIH National Center for Complementary and Alternative Medicine, for her critical review of this work.
This work was funded by the NIH CTSA Program in collaboration with the University of Alabama at Birmingham (UL1 TR000165), University of Cincinnati (UL1 TR000077), Northwestern University (UL1 TR000150), Weill Cornell Medical College (UL1 TR000457), University of Rochester (UL1 TR000042), University of Texas Health Science Center at Houston (UL1 TR000371), Mayo Clinic (UL1 TR000135), University of Texas Health Science Center at San Antonio (UL1 RR025767), and University of Pennsylvania (UL1 TR000003). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NCATS or the NIH.
|No research experience||Reader of the research literature||Has been involved in research|
|Goal||Read the literature||A|
Depth of research: 0 = none, 1 = some, 2 = high.
|1. Assess sources of bias and variation in published studies and threats to study validity (bias) including problems with sampling, recruitment, randomization, and comparability of study groups|
|2. Propose study designs for addressing a clinical or translational research question|
|3. Describe the basic principles and practical importance of probability, random variation, systematic error, sampling error, measurement error, commonly used statistical probability distributions, hypothesis testing, type I and type II errors, and confidence limits|
|4. Compute sample size, power, and precision for comparisons of two independent samples with respect to continuous and binary outcomes|
|5. Explain the uses, importance, and limitations of early stopping rules in clinical trials|
|6. Describe the concepts and implications of reliability and validity of study measurements and evaluate the reliability and validity of measures|
|7. Scrutinize the assumptions behind different statistical methods and their corresponding limitations and describe preferred methodologic alternatives to commonly used statistical methods when assumptions are not met|
|8. Distinguish among the different measurement scales and the implications for selection of statistical methods to be used on the basis of these distinctions|
|9. Generate simple descriptive and inferential statistics that fit the study design chosen and answer research question|
|10. Describe the uses of meta‐analytic methods|
|11. Communicate research findings for scientific and lay audiences|
|12. Describe size of the effect with a measure of precision|
|13. Describe the study sample, including sampling methods, the amount and type of missing data, and the implications for generalizability|
|14. Interpret results in light of multiple comparisons|
|15. Identify inferential methods appropriate for clustered, matched, paired, or longitudinal studies|
|16. Identify adjusted inferential methods appropriate for the study design, including examination of interaction|
|17. Describe statistical methods appropriate to address loss to follow‐up|
|18. Understand the reasons for performing research that is reproducible from data collection through publication of results|
|19. Understand appropriate methods for data presentation, especially effective statistical graphs and tables|
|20. Characterization of diagnostic testing, including sensitivity, specificity, and ROC curves|
|21. Describe appropriate data quality and data management procedures|
|Original Wording||Abbreviated Wording||Proposed Changes to Wording|
|Assess sources of bias and variation in published studies and threats to study validity (bias) including problems with sampling, recruitment, randomization, and comparability of study groups||Assess sources of bias and variation|
|Propose study designs for addressing a clinical or translational research question||Propose study designs||Change “propose” to “assess”|
|Describe the basic principles and practical importance of probability, random variation, systematic error, sampling error, measurement error, commonly used statistical probability distributions, hypothesis testing, type I and type II errors, and confidence limits||Describe basic statistical principles and their practical importance||Change “describe” to “assess”|
|Describe the concepts and implications of reliability and validity of study measurements and evaluate the reliability and validity of measures||Describe concepts and implications of reliability and validity||Change “describe” to “assess”|
|Communicate research findings for scientific and lay audiences||Communicate research findings|
|Describe size of the effect with a measure of precision||Describe size of the effect with a measure of precision||Change “describe” to “assess”|
|Describe the study sample, including sampling methods, the amount and type of missing data, and the implications for generalizability||Describe the study sample, including sampling methods||Change “describe” to “assess”|
|Understand the reasons for performing research that is reproducible from data collection through publication of results||Understand the reasons for performing reproducible research|
|Understand appropriate methods for data presentation, especially effective statistical graphs and tables||Understand appropriate methods for data presentation|
|Scrutinize the assumptions behind different statistical methods and their corresponding limitations and describe preferred methodologic alternatives to commonly used statistical methods when assumptions are not met||Scrutinize the assumptions and limitations behind different statistical methods||Change “scrutinize” to “assess”|
|Distinguish among the different measurement scales and the implications for selection of statistical methods to be used on the basis of these distinctions||Distinguish among the different measurement scales||Change “distinguish among” to “assess” and “distinctions” to “measurement scales”|
|Generate simple descriptive and inferential statistics that fit the study design chosen and answer research question||Generate simple descriptive and inferential statistics||Change “generate” to “assess”|
|Interpret results in light of multiple comparisons||Interpret results in light of multiple comparisons||Change “interpret” to “assess”|
|Identify inferential methods appropriate for clustered, matched, paired, or longitudinal studies||Identify inferential methods appropriate for clustered, matched, paired, or longitudinal studies|
|Identify adjusted inferential methods appropriate for the study design, including examination of interaction||Identify adjusted inferential methods appropriate for the study design|
|Compute sample size, power, and precision for comparisons of two independent samples with respect to continuous and binary outcomes||Compute sample size, power, and precision for comparisons||Change “compute” to “understand how to determine”|
|Explain the uses, importance, and limitations of early stopping rules in clinical trials||Explain the uses, importance, and limitations of early stopping rules in clinical trials|
|Describe the uses of meta‐analytic methods||Describe the uses of meta‐analytic methods||Change “describe” to “understand”|
|Describe statistical methods appropriate to address loss to follow‐up||Describe statistical methods appropriate to address loss to follow‐up||Change “describe” to “understand”|
|Characterization of diagnostic testing, including sensitivity, specificity, and ROC curves||Characterization of diagnostic testing, including sensitivity, specificity, and ROC curves|
|Describe appropriate data quality and data management procedures||Describe appropriate data quality and data management procedures||Change “describe” to “understand”|