This analysis shows the need for a new instrument to assess biostatistical competencies for medical researchers. Neither the existing instruments nor a set of questions taken across these instruments sufficiently addressed the competencies required for medical research using statistics. Further research will be required to identify and validate a brief but complete set but of questions addressing the competency areas appropriately.
The Comprehensive Assessment of Outcomes in Statistics (CAOS) test by delMas et al13
was specifically designed to assess students’ understanding of variability, and this is reflected in the topic areas of the questions. While appropriate for undergraduates in introductory statistics, the difference in distribution of question topics corroborates prior work demonstrates the quantifiable baseline differences of graduate students in biostatistics21
. While variability remains a critical topic for the foundation of biostatistics, the competency expectations for this group are far more complex. However, the CAOS test is the only measure designed to assess students who are learning statistics in the classroom, rather than those who need only use statistics to read journal articles. As such, it provides an important indication of some topics which may be important for the learning process, such as understanding study designs and the central limit theorem.
The instruments designed to assess practicing physicians fit more closely with the goals of this paper. However, practicing physicians need only to be able to read and understand the research literature. They need not understand the intricacies of the statistical methods or perform the methods. The questions designed for this population reflect the need for physicians to understand pretest and posttest probability well in order to treat their patients appropriately22
; there are numerous questions on diagnostic testing. Nevertheless, as a whole the instruments for this group seem somewhat inadequate even for their designated purpose, as there is little attention paid to assessing whether the appropriate method has been used or to interpreting statistical results. Both of these topics are required to robustly critique or defend a research manuscript. Similarly, very few questions pertain to confounding, yet a thorough grounding in this topic is essential for understanding and assessing observational studies. More research may be required to develop or refine an assessment instrument more appropriate for statistical methods in evidence based medicine.
The statistical competencies for clinical and translational research seem more mature than those from public health. This likely reflects the timing of competency development; the clinical and translational research group benefitted from the prior work of their public health colleagues. Both disciplines’ documents appear to have some gaps (). These are presumably oversights; competencies such as proposing appropriate study designs and performing sample size and power calculations are routinely included in most graduate level curricula for both disciplines. However, as competency documents are increasingly used to guide program evaluation, it is to be hoped that these gaps will be considered in future revisions.
There are several statistical topics required by the competencies but not included in any of these assessment instruments. Perhaps the most egregious of these omissions is the lack of items to assess dependent data, such as paired, matched, or longitudinal data. Such data are frequently used in observational studies, as matching is one way to minimize the effect of anticipated confounders. Indeed, Horton and Switzer (2005) found that 12% of New England Journal of Medicine’s original articles in 2004-2005 used repeated measures methods, one of the more advanced techniques for coping with dependent data23
. Together, these gaps may prove as a starting point for an effort to develop a new assessment instrument targeted to graduate students in CTS and PH.
In developing a new assessment instrument for researchers, may help guide instrument development as it shows both a comprehensive list of statistical competencies and the corresponding statistical methods upon which specific questions can be based. Most importantly, includes an assessment of what topics are not included within this set of questions, but this list should not be considered exhaustive. A review of individual items for quality and comprehensiveness will be needed to ensure further questions are not needed for topic areas with existing questions.
It is likely that two instruments will be required, one a full assessment of statistical competency and another an assessment designed for topics taught in introductory biostatistics courses. This distinction is aided by the wording of the competencies, which vary in their level of expectation. Students are expected to perform lower level statistical methods themselves but work with a statistician for more complex methods. Working with a statistician requires some knowledge of appropriate methods, sufficient common language for collaboration, and the ability to interpret statistical results. Few of the previously developed items assess this second level of understanding.
A new assessment instrument would ideally be developed and validated by a group of inter-institutional experts in biostatistics as used in clinical research. Fortunately, such a group has been formed by the National Institutes of Health through the CTSA award mechanism. Researchers from CTSA institutions collaborate in a variety of areas, including education and evaluation24
. Indeed, the biostatistics courses taught within the CTSA come from both public health and clinical and translational research program, so this collaboration could also provide an initial step toward unifying the statistical competencies from these disciplines. A new publicly-available instrument would open the door for further intra- and inter-institutional research on knowledge and skill levels before and after biostatistics coursework, before and after research degree programs, and among practicing researchers. This research would hopefully aid both statistics coursework and program development to improve graduate outcomes.