|Home | About | Journals | Submit | Contact Us | Français|
This study reports the use of exploratory factor analysis to describe essential skills and knowledge for an important segment of the domestic public health workforce—Centers for Disease Control and Prevention (CDC) project officers—using an evidence-based approach to competency development and validation.
A multicomponent survey was conducted. Exploratory factor analysis was used to examine the underlying domains and relationships between competency domains and key behaviors. The Cronbach α coefficient determined the reliability of the overall scale and identified factors.
All domestic (US state, tribe, local, and territorial) grantees who received funding from the CDC during fiscal year 2011 to implement nonresearch prevention or intervention programs were invited to participate in a Web-based questionnaire.
A total of 34 key behaviors representing knowledge, skills, and abilities, grouped in 7 domains—communication, grant administration and management, public health applied science and knowledge, program planning and development, program management, program monitoring and improvement, and organizational consultation—were examined.
There were 795 responses (58% response rate). A total of 6 factors were identified with loadings of 0.40 or more for all 34 behavioral items. The Cronbach α coefficient was 0.95 overall and ranged between 0.73 and 0.91 for the factors.
This study provides empirical evidence for the construct validity of 6 competencies and 34 key behaviors important for CDC project officers and serves as an important first step to evidence-driven workforce development efforts in public health.
The public health infrastructure has been defined as the nerve center of the public health system1 and provides the capacity to effectively deliver the 10 essential public health services.2 The backbone of this infrastructure is the public health workforce. As such, workforce competency is a fundamental component of the public health system and critical to optimal system performance.3 Defined as a measureable human capability that is required for effective performance,4 competencies are used by organizations to recruit the right people for specific positions and jobs, clarify performance expectations, appraise performance, inform training programs, and align workforce behavior with organizational strategies and values.5
A common theme throughout the public health literature is a call to action to better prepare the public health workforce.6–8 The development and application of public health competencies are recognized as essential components of a sound workforce development strategy.5,9 While the topic of workforce competencies has a long history in the public health literature,8,10,11 a majority of the available literature is descriptive and suppositional in approach.11 The methods used to define and validate competencies for the public health workforce rely heavily on the Delphi method, a consensus building technique that uses a series of structured questionnaires (commonly referred to as rounds) to gather information from experts, resulting in a group opinion or consensus decision.12 While the Delphi method has clearly advanced public health and health care competency development, significant concerns about its methodological rigor and deficiencies in its application are well documented in the literature.12–14
Relatedly, reviews of public health workforce research have identified limitations in the evidence base for the development and application of public health workforce competency and recommend that empirical research methods from the social sciences be adopted for use in public health workforce research.3,11,15,16
In response to the dearth of evidence and lack of empirical research methods used to guide public health workforce development efforts, the field of Public Health Systems and Services Research (PHSSR) has focused attention on the following public health work-force research questions:
In the area of PHSSR, there is growing interest in evidence-based practices for public health administration. According to Brownson et al, “The National Public Health Performance Standards Program, Public Health Accreditation Board standards and measures, and local quality improvement and accreditation processes are drawing increasing attention to administrative practices.”18 One area of administrative practice particularly important to the Centers for Disease Control and Prevention (CDC) and its state, local, tribal, and territorial grantees is the role and competency of the CDC project officers in providing technical, scientific, and programmatic guidance and support to CDC-funded public health programs (from this point forward referred to as grantees) for a particular grant, cooperative agreement, or contract to ensure programmatic success. They serve as CDC’s counterpart to the principal investigator or program director within the grantee organization implementing the funded prevention or control program. In 2012, the CDC conducted an assessment of the technical assistance and customer support provided to grantees implementing public health programs. One of the purposes of the assessment was to define and validate a set of competencies that were essential to the successful administration and implementation of domestic prevention and control programs and central to effective CDC project officer performance.
This article advances the dialogue around the aforementioned research questions by presenting the results of a competency development study that uses an evidence-based approach to define and validate the requisite knowledge and skills of the federal workforce whose functional role and responsibilities include the administration of public health prevention and control cooperative agreements and grants.
A survey of domestic public health programs receiving CDC funding was undertaken in fall 2012. All principal investigators or program managers for domestic public health programs funded during fiscal year (FY) 2011 through a cooperative agreement or grant to implement nonresearch public health prevention or intervention programs were invited to participate in a Web-based survey. In those instances where 1 person was the primary point of contact for 2 or more cooperative agreements or grants, one was randomly selected for inclusion. This process ensured that each response represented a single cooperative agreement or grant and reflected a single project officer, allowing for comparisons across programs. Each invitation to participate included the questionnaire and a cover letter explaining the survey. The Office of Management and Budget granted approval for the survey (OMB no. 0920-0879; expiration date: March 31, 2014).
Because this survey represents the first attempt to define a set of competencies for federal public health workers serving in the role of project officer, a process to identify all potential competencies and associated behaviors, knowledge, and skills began in fall 2011. The first step in identifying behaviors, commonly viewed as important, was to conduct focus group interviews with CDC project officers and key informant interviews with their supervisors based at CDC headquarters in Atlanta, Georgia. The staff interviewed represented a cross section of 9 CDC program areas (HIV/AIDS prevention, TB elimination, immunization services, cancer prevention and control, smoking and health services, injury response, health assessment and consultation [Agency for Toxic Substances and Disease Registry], public health preparedness and response, and public health infrastructure). A total of 9 focus groups with 35 project officers were conducted. In addition, 9 key informant interviews with a total of 26 supervisors were completed. Topical areas of discussion included the roles and responsibilities of project officers and essential skills, knowledge, and characteristics required of project officers to demonstrate in providing technical assistance and guidance to public health programs. These interviews identified a broad scope of concepts and constructs important for inclusion in the survey, including items measuring (1) aspects of technical assistance provided through all phases of a federally funded program, (2) expectations and support provided by project officers for continuous improvement, and (3) the skills and characteristics necessary to provide quality technical assistance and capacity-building guidance to grantees. Once key behaviors were fully described and categorized into themes, the underlying constructs were defined through an examination of existing and familiar public health competency sets and training resources for public health and CDC staff. The materials reviewed and used to inform the development of the behavioral constructs are listed as follows:
This process resulted in a questionnaire ready for a survey of nationwide principal investigators/program managers of domestic public health prevention and control programs. The survey tool contained a total of 34 key behaviors representing knowledge, skills, and abilities, grouped into 7 domains representing distinct constructs for competencies: communication, grant administration and management, public health applied sciences and knowledge, program planning and development, program management, program monitoring and improvement, and organizational consultation. Participants rated the importance of each behavior on a 4-point option Likert scale from 0 to 3, where 3 = very important, 2 = important, 1 = somewhat important, and 0 = not important or not applicable (see Table, Supplemental Digital Content 1, available at: http://links.lww.com/JPHMP/A117, which provides the items presented to survey participants).
The study population included all domestic grantees who received CDC funding (FY2011) through a cooperative agreement or grant to implement nonresearch prevention or intervention programs. The total population of potential respondents was surveyed (1) to address possible bias posed by a stratified sampling strategy due to significant variability in the size and organization of CDC centers, institutes, and offices and (2) to achieve the desired level of accuracy using the rule of 20 subjects per item27 for conducting exploratory factor analysis.
A total of 1365 principal investigators or program managers for 61 unique cooperative agreements and grants funded by the CDC during FY2011 were identified. However, no single database or source of information providing contact information was available. Therefore, several sources of information were used to develop the distribution list, including CDC’s FY2011 grant funding files and the Information for Management, Planning, Analysis, and Coordination (IMPAC II) system. In addition, any missing contact information was solicited from the CDC programs included in the survey.
The analyses were conducted using STATA (version 11.2) software. Demographic data were analyzed using means and standard deviations and other descriptive analysis. These demographic variables were further examined for possible nonresponse bias using the χ2 test of independence with a significance level of .001. Since this study represents the first attempt to explore the construct validity of the defined behaviors and competencies for the federal project officer, exploratory factor analysis was deemed appropriate to uncover the underlying domains and relationships between the selected key behaviors.28 Using the Kaiser criterion, all factors with eigenvalues exceeding 1 were extracted.28,29 An orthogonal rotation (Varimax) was used to explore the degree of correlation between the factors and variables.28,30 A cutoff point for factor loading of 0.40 was used. Reliability was tested by means of the Cronbach α coefficient, where a value of 0.70 is considered acceptable.31
As with any study, there are limitations inherent to this study. Although CDC project officers and their supervisors provided input to the competencies and behaviors examined in this study, the participants of this survey include only grantees. While this perspective is critically important, it should be noted that this population provides the perspective of 1 of 3 key constituent groups. Therefore, these findings relate to the essential knowledge and skills necessary for project officers to demonstrate in providing technical and scientific guidance to grantees and does not necessarily reflect the knowledge and skills required to successfully meet the expectations or requirements of other duties or responsibilities of their position within the CDC.
A total of 875 responses were received. Of these, 80 were incomplete and excluded from the analysis, giving an adjusted response rate of 58.07%. Respondent characteristics are outlined in Table 1. Approximately one-third of the respondents worked in infectious disease programs and 28.68% worked in chronic disease prevention and control programs. The mean length of experience within the program was 6.92 years (SD = 6.78 years). The majority of respondents were from state public health departments (84.91%). Responses that made up “other” STLT (state, tribe, local, and territorial)-type settings included nonprofit, private, non-governmental organizations (n = 4) and private for- profit organizations (n = 1). A comparison of responders with nonresponders on these characteristics revealed that respondents (n = 795) and nonrespondents (n = 570) were from similar program areas, χ2 (4, N = 795) = 22.545, P >.001, and STLT type, χ2 (4, N = 795) = 15.125, P >.001.
Given the total number of respondents of 795, the sample-to-item ratio exceeded the rule of 20:1 sample to variables. The data were considered appropriate for factor analysis, with 0.95 Kaiser-Meyer-Olkin measure of sampling adequacy.32 Results yielded a 6-factor solution, accounting for 63% of the total variance.
The 6 factors were defined using the 7 domains in the survey tool, where 2 domains merged into a single factor. Using a cutoff level of 0.4, all 34 items loaded on a factor. Items 19–20, 22–27, and 34 loaded on factor 1 (program management and improvement); items 15–18, 28, and 32–33 loaded on factor 2 (program planning and development); items 9–14 loaded on factor 3 (public health applied sciences and knowledge); items 21 and 29–31 loaded on factor 4 (organizational consultation); items 5–8 loaded on factor 5 (grant administration and management); and items 1–4 loaded on factor 6 (communication). Six of the 34 items loaded on 2 different factors. Each of these items was examined and deemed appropriately categorized for the factor with the stronger loading value. Table 2 displays the 6 factors defined with each item’s factor loading score.
Internal consistency using the Cronbach α coefficient was observed to be high (α = 0.95) overall as well as for each of the factors: program management and improvement (α = 0.91), program planning and development (α = 0.90), public health applied sciences and knowledge (α = 0.86), organizational consultation (α = 0.75), grant administration and management (α = 0.78), and communication (α = 0.73), indicating good internal consistency.
Further analysis was conducted to determine whether certain domains were of greater importance to certain programs that to others. While all 6 domains were found to be important to all programs, the χ2 and Fischer exact tests (when n of respondents <5 in any cell) indicated that certain domains are of slightly greater importance for certain types of programs:
No association by program type was observed for factor 5 (grant administration and management) or factor 6 (communication).
The results of this study have implications both for public health practitioners responsible for the provision of technical assistance and guidance to awardees and for public health services and systems researchers. For practitioners, this study provides empirical evidence for the construct validity of 6 competencies and 34 key behaviors important for federal public health workers to demonstrate when supporting a federally funded public health program implemented by an STLT health department. While the competencies in question target a very specific segment of the public health workforce—CDC project officers—their role in the public health system cannot be overlooked. These CDC project officers serve as a prominent resource for public health agencies implementing federally funded programs aimed at preventing health problems and improving and protecting community public health. Therefore, the identification of a valid set of competencies for federal employees serving in the role of project officers is essential to evidence-driven workforce development efforts, including, but not limited to, recruitment and selection, training and curricula development, and employee performance appraisals.
For researchers, this study contributes to the PHSSR agenda by advancing the evidence base and dialogue related to workforce competencies in 2 key ways. First, the establishment of the Public Health Accreditation Board has focused national attention on administrative and management capacity and public health operations, organization, and governance. Relatedly, the national research agenda for PHSSR poses the question, “What forms of decision support, guidance, and technical assistance for governmental public health agencies are most effective in improving the effectiveness, efficiency, and outcomes of public health strategies delivered at local, state, and national levels?”17 While the answer to this research question goes beyond the scope of this study, the findings presented in this article begin to lay the measurement foundation and evidence required for future inquiry on the impact of specific forms of technical assistance on governmental public health agency effectiveness, efficiency, and outcomes. That is, this study confirms the importance of specific types of technical and management skills to governmental public health practice and provides evidence-based measures of human capability related to the delivery of technical assistance to STLT governmental public health agencies for purposes of improving public health programs and services. While the direct application of these competencies is limited to the federal public health workforce, the results of this study bolster the evidence in an underdeveloped area of public health research18 and provide a framework for further study in the area of technical assistance.
Finally, this study demonstrates how empirical research methods from the social sciences can be adapted for use in public health workforce research, thus responding to concerns about the lack of rigor and quantifiable evidence used in public health workforce development research.15,33 Factor analysis, as used in this study, offers those engaged in the development of public health competencies a methodological alternative to consensus-based approaches such as the Delphi technique.
In summary, this study addresses an important theme from the national research agenda for public health services and systems related to workforce competencies. More specifically, the results from this study contribute to the body of evidence related to the rigor of workforce competency development methods and effective forms of guidance and technical assistance for governmental public health agencies. Furthermore, the competencies themselves provide a valid framework for the development and evaluation of workforce training and certification programs critical to advancing and strengthening the efficacy of public health programs overall.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Supplemental digital content is available for this article. Direct URL citation appears in the printed text and is provided in the HTML and PDF versions of this article on the journal’s Web site (http://www.JPHMP.com).
The authors declare no conflicts of interest and no source of funding.