PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Pediatr Res. Author manuscript; available in PMC Feb 26, 2013.
Published in final edited form as:
PMCID: PMC3582389
NIHMSID: NIHMS440648
Tracking the Impact of the National Institutes of Health Clinical and Translational Science Awards on Child Health Research: Developing and Evaluating a Measurement Strategy
W. Charles Huskins,* Clare D. Sullivan, Janey Wang, Mary Aitken, Steven R. Alexander, Leon Epstein, Alejandro Hoberman, Ellis Neufeld, Anthony Philipps, Thomas P. Shanley, Peter Szilagyi, Mary Purucker, and Shari L. Barkin
W. Charles Huskins, Division of Pediatric Infectious Diseases, Mayo Clinic, Rochester, Minnesota 55905;
*Corresponding author: Mayo Clinic 200 First Avenue SW Rochester, MN 55905 Tel: 507-255-8464; Fax: 507-255-7767 ; huskins.charles/at/mayo.edu
Since 2006, the National Institutes of Health has provided institutional infrastructure grants, called Clinical and Translational Science Awards (CTSAs), to support adult and pediatric clinical and translational research in United States institutions. A CTSA Consortium Child Health Oversight Committee workgroup developed metrics to measure the impact of CTSAs on child health (CH) research. A cross-sectional survey to collect metric data was distributed to the 46 institutions that received CTSAs during 2006-09. Thirty-seven (80%) institutions responded to the survey. Data regarding 7 metrics were reported by >70% of responding institutions: the proportion of overall funding (median, interquartile range; 0.12, 0.06–0.19) and pilot grants (0.15, 0.11–0.21) supporting CH research; the proportion of active clinical research center studies involving children (0.23, 0.15–0.35); the proportion of IRB-approved (0.24, 0.16–0.30) and funded (0.22, 0.18–0.30) studies involving children; the proportion of mentored research training awards to CH investigators (0.18, 0.11–0.28); and, the proportion of CTSA leadership positions held by pediatricians (0.18, 0.12–0.28). CTSAs provide substantial support for CH research, although additional investment in CH research is needed to improve the health of children. These metrics provide an initial means to track the impact of CTSAs on CH research.
The National Institutes of Health (NIH) has an extensive history of supporting child health research through a variety of infrastructure grants, including funding for clinical research centers (CRC), institutional clinical research curricula and training and career development programs for child health researchers, pediatric clinical research networks, and education and meeting activities (1, 2). In 2006, the NIH launched the Clinical and Translational Science Award (CTSA) program with the vision that these enhanced institutional infrastructure grants would better catalyze the translation of new scientific discoveries into improved health of human populations (3, 4). Existing pediatric CRCs and some institutional training and career development programs were subsumed into the CTSAs as linked awards; there was no requirement for a separate child health component. The NIH Reform Act of 2006 changed this framework in order to preserve independent funding and infrastructure for pediatric CRCs (5). Subsequent CTSA Request for Applications (RFA) have allowed for the appointment of a pediatric co-principal investigator under a single CTSA who would have direct authority over a separate budget or would otherwise secure institutional independence of pediatric CRCs with respect to finances, infrastructure, resources, and research agenda. As of 2011, 58 of the 60 CTSAs include child health components and 9 have principal investigators who are themselves pediatricians (6). However, none of the CTSAs have chosen to create a separate budget for child health, which makes it difficult to directly identify the resources devoted to support of child health research in individual institutions and to measure the effectiveness of the CTSA program as a whole in supporting child health research over time.
Concurrent with the funding of the first group of 12 CTSAs in 2006, the CTSA Consortium-Child Health Oversight Committee (CC-CHOC) was established as a part of the broader CTSA Consortium governance structure and was charged with the responsibility of facilitating child health research throughout the Consortium (7, 8). A CC-CHOC workgroup was formed to develop and test a measurement strategy to: 1) enable CTSA institutions to uniformly assess their current levels of support, research activity, training of new investigators, and multicenter collaborations related to child health research; 2) track changes in these parameters over time within institutions; and 3) eventually estimate the impact of the CTSA Consortium, as a whole, on child health research. This report describes the initial development and evaluation of the measurement strategy and presents data on specific metrics from CTSA institutions receiving CTSAs during 2006-2009.
Definitions
The workgroup used the NIH definitions for clinical and translational research (9) and for inclusion of children as participants in research involving human subjects (10) to define child health research as, “all clinical and translational research involving subjects less than 21 years of age that impacts children’s health or addresses childhood diseases, including maternal-fetal research, as well as studies involving infants, children and adolescents.” Research that primarily focused on adults but in which a few older adolescents were enrolled was excluded.
Development of the Metrics
From 2008-2009, workgroup members representing 12 CTSAs participated in monthly teleconferences during which measurement domains and specific metrics were proposed, discussed, and selected. Decisions were made by consensus during all phases of this process. The workgroup reviewed the CTSA Consortium Strategic Goals (11) and chose the following measurement domains, which aligned with these Strategic Goals: I) funding, infrastructure support, and research activity; II) career development of new investigators; III) collaborative research; and, IV) leadership. Workgroup members were then asked to propose specific metrics in each of these domains and then to rank prospective metrics using the following criteria: 1) capacity to utilize current systems to capture existing data; 2) generalizability across most institutions; and 3) ability to identify “new value” attributable the CTSA infrastructure. Metrics that were ranked highly across these criteria were selected for further development. No metrics were added or deleted thereafter. Several members of the workgroup (S.L.B., C.D.S., W.C.H.) collaborated to draft a written description of the metrics with specific definitions of the numerator and denominator for each metric and instructions for data collection. Other workgroup members provided critique of the draft and comments were incorporated into a revised version which was presented to the workgroup in the spring of 2009. During the summer of 2009, members of the workgroup pilot-tested procedures for collecting the required data in 6 of their own institutions and the metric definitions and data collection procedures were revised based on their recommendations. The proposal to implement the data collection strategy for the metrics was reviewed and approved by the CTSA Consortium Steering Committee, a body that consists of all participating CTSA principal investigators and NIH CTSA representatives.
Collection of Metric Data
In November 2009, the final version of the metrics (Table 1) and a worksheet for data collection with instructions were distributed by e-mail to the CC-CHOC member or other relevant contact at each CTSA institution. For each metric, the institution recorded the numeric data and provided qualitative comments, as needed, about how the metric was interpreted and data were collected, as well as any barriers or difficulties encountered in this process. An open-ended field allowed for additional qualitative descriptions of institutional enhancements to promote the conduct of child health research. Given the need to assess administrative financial and IRB databases, data were collected primarily by administrative personnel under the supervision of the CC-CHOC member and either entered directly by each institution using a confidential access code into a Research Electronic Data Capture (REDCap, Vanderbilt Institute for Clinical and Translational Research, VICTR, Nashville, Tennessee) database (12) or transmitted confidentially to VICTR and subsequently entered into the database. Four follow-up e-mails reminders were sent to CC-CHOC members at all institutions, but non-responding institutions were not contacted individually either by e-mail or telephone.
Table 1
Table 1
Measurement Domains and Description of Metrics
Data and Statistical Analyses
REDCap and SPSS, version 18 (IBM SPSS Statistics, formerly Predictive Analytics SoftWare Statistics, Armonk, New York) were used to generate descriptive statistics. Qualitative data were abstracted and synthesized by members of the workgroup (W.C.H., C.D.S., J.W., S.L.B.) and reviewed by the other members. Fisher exact tests were used to compare institutional characteristics between responding and non-responding CTSA institutions.
All 46 institutions that received CTSAs from 2006-2009 received an invitation to participate; 37 (80%) institutions provided responses. Comparing responding and non-responding institutions, there was no evidence of a difference in geographic distribution by region (P=0.96), year of award (P=0.75), and the amount of the CTSA in dollars by quartiles of percentile rank (P=0.33). A children’s hospital was listed as a partner institution and a pediatrician was the CTSA Principal Investigator for 34 (92%) and 5 (14%) responding institutions, respectively, vs. for 6 (67%, P=0.08) and 0 (P=0.57) of non-responding institutions, respectively.
The numeric data needed to calculate each metric were provided by most of the responding institutions (Table 2). Except for the collaborative research metric, data were provided similarly for child health research, investigators, or pediatric subjects (the numerators) and for the CTSA or institution as a whole (the denominators). Data regarding the number of multicenter studies involving more than one CTSA were the most difficult for institutions to provide, both for studies involving pediatric subjects and for studies involving subjects of any age.
Table 2
Table 2
Institutions Providing Data and Descriptive Data Regarding Metrics
The numeric data values reported by individual institutions ranged widely (Table 2). Likewise, there was wide variation in the calculated metrics (Figure 1), most notably in the metrics related to active CRC studies, mentored research training awards, and CTSA leadership positions involving child health researchers. There was diversity among the institutions that represented the upper bounds of the ranges, with 6 different institutions represented (the upper bound was represented by 1 institution for the metrics related to overall funding, active CRC studies, and mentor research training awards, 1 institution for both metrics related to research activity, and 4 different institutions for the other metrics). Nonetheless, despite the overall variation, the medians of the metrics were generally consistent (Figure 1, range of medians, 0.11-0.24) and the central portions of their distributions, as represented by the interquartile ranges, were relatively narrow.
Figure 1
Figure 1
Distribution of Metric Proportions across CTSA Institutions
Challenges in collecting the numeric data were due primarily to issues related to the interpretation of the definitions and the data sources of the metrics (Table 3). Differences between NIH and IRB definitions of child health research and whether to include other sources of support (e.g., supplemental awards, institutional matching funds) were identified commonly. Less common, but more fundamental, were issues of whether studies that relate to child health but do not involve children as research subjects directly should be viewed as child health research, such as healthcare services research and maternal health studies. Inadequate sources of data were a major challenge due to the lack of variables that clearly identify child health research studies and to difficulties synchronizing the records of separate institutional databases, such as institutional review board (IRB) and grant funding databases, to perform queries using the combined data fields from each database.
Table 3
Table 3
Description of Issues Identified in Collecting Data Regarding the Metrics a
Institutions provided additional qualitative statements of how the CTSA prompted the development of new institutional strategies and initiatives to enhance child health research, including: establishing multidisciplinary child health research teams or committees to better integrate child health research into the CTSA, enhancing linkages with basic science researchers, and developing specific foci of expertise in child health research; using evidence of CTSA support to obtain additional matching funds to support child health research and new child health investigators; using indirect funds to increase investment in child health research; recruiting child health researchers to compete for pilot grants; establishing a community-based child health research network; and inaugurating an annual pediatric research day event.
The metrics presented in this cross-sectional survey illustrate that institutions that received CTSAs from 2006-09 provide substantial support for child health research through overall funding and pilot study grants, CRC utilization, and mentored research training awards for child health investigators. In these institutions, roughly a quarter of all IRB-approved studies include pediatric subjects and nearly a fifth of CTSA leadership positions are filled by pediatricians. These findings may be positively skewed by the trend toward a higher proportion of institutions with a children’s hospital listed as a partner institution among responding vs. non-responding institutions.
Although these findings are generally positive, additional investment by CTSAs in child health research will be required to achieve the major breakthroughs necessary to improve the health of children and those children into adulthood. As reported in this study, the proportions of overall funding and pilot study grants devoted to child health research in CTSA institutions are consistent with the proportion of NIH funding targeted toward child health research reported recently (13). However, the proportion of overall NIH funding for child health research has declined by nearly a third over the past two decades (13). Moreover, leaders in the pediatric research community have identified major unmet needs for child health research that must be addressed in order to improve the future health and well-being of children in the United States (13, 14).
These results should also be viewed in the context of the limitations of this first attempt to assess the impact of the CTSA program on child health research. The lack of pre-award data and the cross-sectional design of this study, which occurred 1 to 4 years after the receipt of the CTSA in the cohorts of institutions receiving awards during 2006-09, do not allow us to assess trends in the support for child health research in relation to the CTSA in individual institutions or across the Consortium as a whole. Although we provided instructions to standardize data collection, issues related to metric definitions and data sources (Table 3) may have resulted in underestimates or greater variability with respect to the metric data reported by individual institutions. We did not perform formal assessments of the reliability of data collection at individual CTSA institutions or the validity of the data in relation to externally verified measures, although we plan to do so in the future. The difficulties we encountered with definitions and data sources for the metrics illustrate general challenges inherent in quantifying the impact of the CTSAs on clinical and translational research in general, and on child health research in particular. Some of these barriers can be addressed easily. For instance, we have already refined the definitions of the numerators and denominators to be more detailed and specific. Other barriers, such as those involving data sources, are likely to require new or revised databases with additional data fields and improved linkages. The difficulty we encountered in capturing information about research collaborations across CTSA institutions from existing grants management databases deserves particular attention because this metric was one of the most difficult to quantitate and yet it uniquely reflects the power of the Consortium.
Nonetheless, the metrics presented in this study provide an important starting point for evaluating the impact of CTSAs on child health research. First, the metrics were developed intentionally to align with the strategic goals of the CTSA Consortium. Therefore, over time, they should reflect enhancements to the clinical and translational research infrastructure that occur in individual institutions and throughout the Consortium. Second, they provide an opportunity for CTSA institutions to assess their current support for child health research in relation to the Consortium as a whole. Third, the metrics provide a standardized mechanism for individual institutions to track the effect of their efforts to enhance child health research over time through changes in either the numeric data values (i.e., the numerators for the metrics), the calculated proportions, or both. Institution-specific tracking of these metrics is important since factors that limit the validity of comparisons across institutions—particularly those related to sources of data—are likely to be less variable within institutions over time. Moreover, by creating an institutional process for measuring child health research efforts, CTSA institutions can examine if they are achieving the child health research goals they set. Fourth, the high response rate indicates there is support for this effort across the Consortium.
To better understand the impact of CTSAs on child health research, we also need better approaches for describing child health research portfolios, including the spectrum of translational research (i.e., T1 through T4) conducted in CTSA institutions(15), research that crosses populations (e.g., women’s health research with outcomes relevant to child health, studies of adult diseases that have their antecedents in conditions affecting children), and research that has an important impact on child health, but does not involve pediatric subjects directly.
As the CTSA Consortium matures, its strategic goals evolve, and the definition of child health research is refined, the metrics used to describe the impact of CTSAs on child health research may need to be adapted. We hope these data will also inform a discussion of what may be considered adequate (or optimal) performance with respect to these metrics. Finally, one of the aspirations of this effort is to develop metrics to evaluate the impact of CTSAs on child health, not just child health research. Indeed, the process of developing such a measurement strategy, in and of itself, demonstrates one of the potential strengths of the CTSA Consortium in facilitating child health research (8).
Acknowledgments
The authors acknowledge Tomaso Tempesti for his assistance with statistical analysis, Ashley Harris for her assistance with administering the survey, and the CTSA Consortium Publications Committee for their critical review of the manuscript.
Statement of Financial Support: This publication was supported by National Institutes of Health/National Center for Research Resources Clinical and Translational Science Award Grant Numbers UL1 RR024150 (Mayo Center for Translational Science Activities, Mayo Clinic), UL1 RR024975 (Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University); UL1 RR029884 (Arkansas Center for Clinical and Translational Research, University of Arkansas for Medical Sciences); UL1 RR025744 (The Stanford Center for Clinical and Translational Education and Research, Stanford University; UL1 RR025741 (Northwestern University Clinical and Translational Sciences Institute, Northwestern University); UL1 RR024153 (University of Pittsburgh Clinical and Translational Science Institute, University of Pittsburgh); UL1 RR025758 (Harvard Catalyst: The Harvard Clinical and Translational Science Center, Harvard University); UL1 RR024146 (UC Davis Clinical and Translational Science Center, University of California, Davis); UL1 RR024986 (Michigan Institute for Clinical and Health Research, University of Michigan at Ann Arbor); UL1 RR024160 (University of Rochester Clinical and Translational Sciences Institute, University of Rochester School of Medicine and Dentistry). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.
Abbreviations
CC-CHOCClinical and Translational Science Award Consortium-Child Health Oversight Committee
CHchild heath
CRCClinical Research Center
CTSAClinical and Translational Science Award
IRBInstitutional Review Board
NCRRNational Center for Research Resources
NIHNational Institutes of Health
RFARequest for Applications
VICTRVanderbilt Institute for Clinical and Translational Research

Footnotes
The authors have no relevant financial disclosures or other potential conflicts of interest to report.
Contributor Information
W. Charles Huskins, Division of Pediatric Infectious Diseases, Mayo Clinic, Rochester, Minnesota 55905.
Clare D. Sullivan, Vanderbilt Institute for Clinical and Translational Research Nashville, Tennessee 37203.
Janey Wang, Vanderbilt Institute for Clinical and Translational Research Nashville, Tennessee 37203.
Mary Aitken, Arkansas Children’s Hospital Research Institute, Little Rock, Arkansas 72202.
Steven R. Alexander, Department of Pediatrics, Stanford University School of Medicine, Stanford, California 94305.
Leon Epstein, Northwestern University Feinberg School of Medicine, Chicago, Illinois 60611.
Alejandro Hoberman, Department of Pediatrics, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania 15224.
Ellis Neufeld, Children’s Hospital Boston and Harvard Medical School, Boston, Massachusetts 02115.
Anthony Philipps, Department of Pediatrics, University of California Davis School of Medicine, Sacramento, California 95817.
Thomas P. Shanley, University of Michigan Medical School, Ann Arbor, Michigan 48103.
Peter Szilagyi, Department of Pediatrics, University of Rochester School of Medicine and Dentistry; Rochester, New York 14642.
Mary Purucker, National Center for Research Resources, National Institutes of Health, Bethesda, Maryland 20892.
Shari L. Barkin, Department of Pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee 37232.
1. Winer KK, Levin E, Grave GD, Alexander DF. The Child Health Research Centers program: 1990-2000. J Pediatr. 2001;139:757–8. [PubMed]
2. McCabe LL. National Institutes of Health support for research and training: future of pediatrician scientists. Arch Pediatr Adolesc Med. 1998;152:839–42. [PubMed]
3. Zerhouni EA. Translational and clinical science--time for a new vision. N Engl J Med. 2005;353:1621–3. [PubMed]
4. Zerhouni EA. US biomedical research: basic, translational, and clinical sciences. JAMA. 2005;294:1352–8. [PubMed]
5. Office of Legislative Policy and Analysis [21 April 2011];Legislative updates: National Institutes of Health Reform Act of 2006. http://olpa.od.nih.gov/legislation/109/publiclaws/reformact06.asp.
6. CTSA Consortium [29 November 2011];NCRR Fact Sheet: Clinical and Translational Science Awards. https://www.ctsacentral.org/documents/Communication_Toolkit/CTSA_FactSheet.pdf.
7. CTSA Consortium [29 November 2011];CTSA Consortium Child Health Oversight Committee. https://www.ctsacentral.org/committee/ctsa-consortium-child-health-oversight-committee.
8. Heubi JE. Child health research and the Clinical Translational Science Awards: where have we been and where are we going? Clin Transl Sci. 2010;3:67–68. [PubMed]
9. Office of Extramural Research, National Insitutes of Health [14 May 14 2011];Glossary & acronym list. http://grants.nih.gov/grants/glossary.htm.
10. NIH Guide [14 May 2011];NIH policy and guideines on the inclusion of children as participants in research involving human subjects. http://grants.nih.gov/grants/guide/notice-files/not98-024.html.
11. Clinical and Translational Science Awards [29 November 2011];Strategic Goal Committees. https://www.ctsacentral.org/content/strategic-goal-committees.
12. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81. [PMC free article] [PubMed]
13. Hay WW, Jr, Gitterman DP, Williams DA, Dover GJ, Sectish TC, Schleiss MR. Child health research funding and policy: imperatives and investments for a healthier world. Pediatrics. 2010;125:1259–65. [PubMed]
14. Genel M, McCaffree MA, Hendricks K, Dennery PA, Hay WW, Jr., Stanton B, et al. Pediatrics; A National Agenda for America’s Children and Adolescents in 2008: recommendations from the 15th Annual Public Policy Plenary Symposium, Annual Meeting of the Pediatric Academic Societies; May 3, 2008; 2008. pp. 843–9. [PubMed]
15. Szilagyi PG. Translational research and pediatrics. Acad Pediatr. 2009;9:71–80. [PubMed]