|Home | About | Journals | Submit | Contact Us | Français|
There is a growing emphasis on the role of organizations as settings for dissemination and implementation. Only recently has the field begun to consider features of organizations that impact on dissemination and implementation of evidence-based interventions. This manuscript identifies and evaluates available measures for 5 key organizational-level constructs: (1) leadership; (2) vision; (3) managerial relations; (4) climate; and (5) absorptive capacity. Overall the picture was the same across the five constructs—no measure was used in more than one study, many studies did not report the psychometric properties of the measures, some assessments were based on a single response per unit, and the level of the instrument and analysis did not always match. We must seriously consider the development and evaluation of a robust set of measures that will serve as the basis of building the field, allow for comparisons across organizational types and intervention topics, and allow a robust area of dissemination and implementation research to develop.
Over the past several years researchers and practitioners alike have recognized the need for more research focused on dissemination and implementation (D & I) of evidence-based programs to promote health and manage chronic disease. Organizations (e.g. schools, workplaces, hospitals) are considered important settings for delivering health promotion interventions (Brownson, Haire-Joshu, & Luke, 2006; Fielding, 1984; Katz, 2009). There is a reasonably robust literature across organizational settings on the delivery of health promotion interventions. However, only recently has the field begun to consider features of organizations that facilitate or inhibit the D & I of evidence-based interventions.
Some of the earlier studies examining the role of organizations in the delivery of evidence-based interventions have considered primarily structural features, such as organization size, complexity, and formalization (Drazin & Schoonhoven, 1996; Emmons & Biener, 1993; Emmons et al., 2000; Emont & Cummings, 1989). Some of these features may in fact reflect less tangible but perhaps more important characteristics of organizations in influencing D & I decisions, such as organizational readiness (Weiner, 2009; Weiner, Amick, & Lee, 2008), leadership, climate(Helfrich, Weiner, McKinney, & Minasian, 2007), and organizational culture(Barnsley, Lemieux-Charles, & McKinney, 1998; Ferlie, Gabbay, Fitzgerald, Locock, & Dopson, 2001; Kanter, 1988; Van de Ven, Polley, Garud, & Venkataraman, 1999). For example, although organizational size has been well-studied, it is likely a proxy for other determinants, such as extent of resources available and functional differentiation or specialization of roles (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Much of the literature at this point is conceptual, with a call for increased research examining the role of these factors in D & I.
A key challenge in the transition from research focused on evidence generation to that focused on D& I is the unit of analysis. The very nature of dissemination efforts often requires an organizational perspective, moving beyond the individual as the unit of analysis and exploring how organizational factors impact on dissemination efforts. Although such an approach is relatively new in the health field, other fields have historically focused on organizations as a key intervention target (e.g. organizational behavior and theory, public policy, education) and have extensively utilized organizational-level measures to assess factors influencing organizational behavior and outcomes.
To build the field of D & I research, we need reliable and valid measures. Recent reviews have reported a dearth of such measures even when considering the broader literature. For example, Weiner recently developed a conceptual framework of organizational readiness to change (Weiner, 2009) and completed an extensive review examining how organizational readiness for change has been defined and measured in health services research and other fields(Weiner, Amick, & Lee, 2008). Analysis of 106 peer-reviewed articles revealed conceptual ambiguities and disagreements, and limited evidence of reliability or validity for most publicly available readiness measures.
As health promotion research increasingly examines organizational-level factors, the need for good operational definitions and measures of key organizational characteristics becomes clearer. The purpose of the proposed manuscript is to identify available measures for key organizational-level constructs that are important for D & I research, to evaluate the measures’ psychometric properties, and to determine if additional measures are needed. A key goal from the outset was to recommend measures that appear to have sound psychometric properties so that a larger body of research using common measures could develop.
The starting point was to identify and evaluate the extant measures related to organizational factors. To guide this work, we selected Greenhalgh and colleagues’ (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004) conceptual framework for considering the determinants of D & I within health service and delivery organizations (see Figure 1). This framework identifies three categories of system antecedents for innovation that relate to the organizational-level factors that may influence dissemination outcomes: (1) structure; (2) absorptive capacity for new knowledge; and (3) receptive context for change. Since there has been considerable attention in the literature to structural variables, for this review we focused on absorptive capacity and four features of receptive context (leadership, vision, managerial relations, and climate). We provide a brief review of the literature for each of these 5 constructs below.
Although leadership has been studied since the 1930s, it is a difficult construct to define and measure. Aditya (Aditya, 2004) argues that leadership centers on both the creation and achievement of goals. The Global Leadership and Organizational Behavior Effectiveness program (GLOBE), a network of researchers that seek to identify effective “universal and culture-specific” leader behaviors and organizational practices, has defined leadership as: “the ability of an individual to influence, motivate, and enable others to contribute toward the effectiveness and success of organizations of which they are members”(House et al., 1999).
Leadership is thought to be an important component of organizational change, and may be particularly important in terms of encouraging members of an organization to think in new ways (Van de Ven, Polley, Garud, & Venkataraman, 1999). Leadership style also has important implications for power balances, social relationships within the organization and across systems, and institutional attitude and approach towards risk-taking (Kanter, 1988; Van de Ven, Polley, Garud, & Venkataraman, 1999).
Although vision is often seen as a leader’s ability to create and communicate clear direction and rationale for the organization (Alexander, Zakocs, Earp, & French, 2006), it can also be conceptualized as a separate construct that cuts across multiple levels within a given organization. Vision has been defined as a guiding theme or over-riding principle that steers the organization, articulating the desired direction and intentions for the future (Reh).
West (West, 1990) defines vision as “an idea of a valued outcome which represents a higher order goal and a motivating force at work” (Anderson & West, 1998). From a theoretical perspective, we would expect that having a clear vision that values innovation sets the stage for adoption and implementation of new health programs. We might also expect that a clear vision would allow members of an organization to adequately assess whether or not an innovation matches with the existing organizational values and goals of the organization, a concept known as compatibility, as described in Roger’s Diffusion of Innovation Theory (Rogers, 1995).
A simple definition for good managerial relations is the positive alliances between groups within an organization to promote change (S. Shortell, Morrison, & Friedman, 1990). (Pettigrew, Ferlie, & McKee, 1992). In clinical settings, Shortell, Morrison and Friedman (S. Shortell, Morrison, & Friedman, 1990) emphasized the significance of: (1) looking for common ground, 2) involving selected physicians early on in planning, 3) carefully identifying the needs and interests of key physicians, and 4) working on a daily basis to build a climate of trust, honesty and effective communications. Pettigrew and colleagues (Pettigrew, Ferlie, & McKee, 1992) also proposed that manager-clinician relations are easier when negative stereotypes have been broken down, and that it is important to understand what clinicians value. In fact, managers who were best at promoting change had semi-immersed themselves in the world of clinicians, understood the implications of medical workflow, and perhaps even helped clinicians to do their own planning as a way of earning trust.
Climate describes organizational members’ perceptions of their work environment. Climate captures the meaning or significance that organizational members perceive in organizational policies, procedures, and practices. These perceptions enable organizational members to interpret current events, predict possible outcomes, and gauge the appropriateness of their own actions and those of others (Jones & James, 1979).
There is some debate as to whether climate is a homologous multi-level construct. Some argue that the construct’s meaning, measurement, and relations with other variables differ across levels of analysis. Reflecting this position, scholars distinguish between psychological climate, a property of individuals that refers to personal perceptions of work environment, and organizational climate, which is a property of the collective and refers to shared perceptions of work environment.
Climate is also a multifaceted construct. In a classic formulation, James and his colleagues identified five primary work environment facets: (1) job characteristics; (2) role characteristics; (3) leadership characteristics; (4) work group and social environment characteristics; and (5) organizational and subsystem attributes (LR James & Sells, 1981; Jones & James, 1979). More recently Patterson and his colleagues (Patterson et al., 2005) identified 17 facets of work environment. Confirmatory factor analysis provides support for both approaches (LA James & James, 1989; Patterson et al., 2005).
An important albeit quite under-studied construct is absorptive capacity, or an organization’s ability to access and effectively use information. Absorptive capacity was originally conceptualized as an organization’s cognitive structures and prior related knowledge that underline learning (Cohen & Levinthal, 1990). Zahra & George (Zahra & George, 2002) expanded this definition to include: (1) a broad set of skills needed to deal with and modify transferred knowledge; and (2) the organization’s capacity to learn and solve problems. This construct reflects the process by which ideas from outside of the organization are captured, circulated internally, adapted for the organizational culture/environment, internalized, and integrated into organizational routines. Absorptive capacity reflects an organization’s ability to find and utilize knowledge. In the theoretical literature, there is a discussion of potential vs. realized absorptive capacity, with the former focusing on organizational capability for knowledge acquisition/assimilation, and the latter focused on use of absorbed knowledge (Cohen & Levinthal, 1990; Daghfous, 2004) suggest that an organization’s absorptive capacity depends on that of its individual members, but is not simply a summation of individual capacity. As Caccia-Bava and colleagues (Caccia-Bava, Guimaraes, & Harrington, 2006) point out, organizational absorptive capacity is best understood in terms of the structures that allow multiple organizational members to gather, communicate, apply, and exploit diverse knowledge within the organization.
In order to identify measures of these five constructs, we searched the peer-reviewed literature using PubMed, CINAHL®, and ISI Web of ScienceSM. With assistance from a reference librarian, we formulated a set of search strings that combined each of the target organizational constructs (organization, climate; managerial relations; leadership; vision; absorptive capacity) and the following terms: innovation, diffusion, adoption, implementation, and health. All search strings included the target construct and “health”, and were applied to the title, abstract, and keyword list. We added wildcard characters to all terms in order to capture potentially important permutations (e.g., innovate, innovating, innovative, and innovation). We restricted the search to English-only articles. Overall, we identified 1178 articles in PubMed, 946 articles in CINAHL, and 1698 articles in ISI Web of ScienceSM . The distribution of these articles by construct is outlined in Table 1.
Of articles identified, we then reviewed the abstracts using four predetermined inclusion/exclusion criteria. First, articles had to report original empirical research and appear in a peer-reviewed journal. Second, articles had to mention the organizational construct or a close synonym in the abstract and employ a quantitative measure of the construct. Third, articles had to include a measured outcome relevant to innovation, diffusion, adoption, or implementation. Examples of such outcomes include implementation of an innovation, defined as a new program, product, service, practice, or technology; the spread of an innovation through an organization or social system; the decision to use an innovation; the process of putting an innovation into practice; and the level of initial use of an innovation, or the sustained use of an innovation beyond an initial trial period. Finally, articles had to focus on a health service or health-related intervention or outcome. Research studies involving schools, businesses, or other organizational settings were included if they focused on a health-related intervention, such an employee wellness program or a tobacco cessation program.
We assigned each article retained after full-text review a unique identification number and used a structured data abstraction form to extract key information from each article (e.g., study setting, construct name, construct dimensionality, number of items, and construct level). Table 2 shows the “dictionary” that we used to structure the data abstraction form. We then created tables to display, categorize, and analyze the information we extracted.
We used Trochim’s (Trochim & Donnelly, 2007) classification of validity and reliability types (see Table 3), in which construct validity is regarded as an umbrella term that includes translational validity and criterion-related validity. Translational validity includes both face and content validity. Criterion-related validity includes predictive, concurrent, convergent, and discriminant validity. Reliability includes inter-rater or inter-observer reliability, parallel forms reliability, test-retest reliability, and inter-item reliability (e.g., Cronbach’s alpha). In this review, we combined face and content validity into a single category. We did the same for the various forms of reliability assessment.
Twelve articles examined the association of organizational leadership and health innovation dissemination. Among these, six (50%) focused on health care organizations, three (25%) on public health organizations, and two (16%) on schools; one article (8%) focused on multiple settings. Table 4 shows the diversity of dimensions included in the leadership construct and Table 5 summarizes results for leadership and the other organizational constructs.
The articles examined various leadership dimensions, including leadership: commitment, clarity, champion, transformational leadership, style, competence, and administrative support, among others. The measures of leadership included one to forty-five items. Several articles measured more than one dimension of leadership; in all, there were 16 leadership measures identified across the 12 studies reviewed.
Five articles (42%) reported measurement of the leadership construct at the organizational level of analysis. One article (8%) measured the construct at the team level of analysis, and two articles (16%) assessed leadership at the individual level of analysis; one article (8%) measured leadership at both the individual level and organizational level of analysis; one article (8%) measured leadership at the team and organizational levels; and two articles (16%) measured the construct at the individual level and then combined responses to create a team level variable. Among the articles that described leadership measured at either the team or organizational level, two used assessments of leadership based on a single response per unit, eight studies based the assessment on multiple respondents per unit, and one article did not report the number of responses per unit used.
Organizational leadership was examined in the context of several dissemination-related outcomes. Nine (75%) focused on innovation implementation, one (8%) focused on innovation sustainability (Evashwick & Ory, 2003), and two (16%) focused on innovation adoption and innovation sustainability (Thaker et al., 2008) (Weiner, Alexander, & Shortell, 1996). In all but three cases (Kegler, Steckler, McLeroy, & Malek, 1998; McFadden, Henagan, & Gowen, 2009)(West et al., 2003), the level of the outcome variable matched the level of the leadership variable. Thaker (Thaker et al., 2008) assessed leadership at the team and organizational level and outcomes on the organizational level, and Bin Saeed (Bin Saeed, 1995) (looked at leadership at the individual and organizational levels and outcomes at the organizational level.
There was no agreement across the articles about how leadership should be assessed; our review identified 16 different measures to assess organizational leadership in dissemination-related studies, with no measures used in more than one study.
Psychometric properties of instruments were reported in some of the studies. Four studies (33% of articles, representing five measures) reported face/content validity (Bin Saeed, 1995; McFadden, Henagan, & Gowen, 2009; Roberts-Gray, Gingiss, & Boerm, 2007; Somech, 2006). Five studies (42%) reported reliability (Bin Saeed, 1995; Livet, Courser, & Wandersman, 2008; Marchionni & Ritchie, 2008; McFadden, Henagan, & Gowen, 2009; Somech, 2006). Ten of the leadership constructs measured either reported on or exhibited predictive validity for dissemination related outcomes. For example, Bin Saeed (Bin Saeed, 1995) collected data from 202 physicians across three hospitals in Saudi Arabia to determine factors associated with implementation of a hospital based quality assurance (QA) program. Using factor analysis to identify dimensions related to implementing QA programs, an 11-item scale was created for “leadership commitment.” In a subsequent analysis using multiple regression, this leadership construct was significantly associated with implementation of the QA program. West (West et al., 2003) measured leadership clarity among 3447 respondents from health care teams in the UK. Leadership clarity predicted levels of innovation among community mental health teams and breast cancer care teams, but not among primary health care teams. Table 6 summarizes the psychometric properties assessed in each study.
Of the two articles examining the association of vision and innovation dissemination, one took place in a health care organization and the other in a public health organization (see Table 4 for study description).
Both studies examined the association of organizational vision with innovation implementation. Livet, et al. (Livet, Courser, & Wandersman, 2008) measured shared vision and Inamdar (Inamdar, Kaplan, & Bower, 2002) measured presence of a well-defined vision as well as the extent to which vision barriers were overcome by program implementation. Shared vision was measured with four items, and was based on a single response per unit. Presence of a well-defined vision was measured with a one item question. Inamdar (Inamdar, Kaplan, & Bower, 2002) also assessed the extent to which barriers were overcome by successful implementation of the innovative strategy and had interviewees rate (from 1-100%) the extent to which the vision barrier (lack of vision related to adoption and implementation) was overcome. Neither study reported psychometric properties of instruments (see Table 6).
The Livet study (Livet, Courser, & Wandersman, 2008), which aimed to identify organizational mechanisms or characteristics that influence implementation of comprehensive programming frameworks, demonstrated that shared vision was correlated with use of program planning, implementation, and maintenance processes. In the Inamdar study (Inamdar, Kaplan, & Bower, 2002) interviewers asked executives of healthcare organizations that were early adopters of an organizational strategy framework (Kaplan & Norton, 1992) whether or not their organization had a well-defined vision. All but one of these organizations reported having a well-defined vision. These same respondents later rated the extent to which the barriers of having “a lack of vision” was overcome (i.e. the strategy is understood by most of the organization); the average rating was 77%.
Only one article examined the association of managerial relations with implementation (Lukas, Mohr, & Meterko, 2009). This article examined management support in the context of the national implementation of an Advanced Clinical Access initiative in 78 VA medical centers, and included the dimension of managerial support. We did not find any articles that studied the association of managerial relations and innovation adoption, implementation, or dissemination (see Table 4 for study description).
The single article measuring managerial relations (Lukas, Mohr, & Meterko, 2009) examined 2 dimensions of management support: personal leadership support and practical management support. Personal leadership support was measured using 7 items (Cronbach’s alpha = .89) adapted from scales designed to measure effectiveness of work teams (Lemieux-Charles, Murray, & Baker, 2002; S. M. Shortell et al., 2004). Personal leadership support was measured at the individual level. Practical management support, a facility-level variable, was measured using a summary index created from 8 dichotomous items about the presence of specific practical expressions of management support for the Advanced Clinic Access (ACA) initiative at the facility (Cronbach’s alpha not reported). Contrary to their hypotheses, only personal leadership support was significantly associated with greater ACA implementation (see Table 6). Of note, these 2 dimensions just included items on communication (“..talking about the ACA ” at facility “town meeting” events and staff meetings as well as presentations by clinical staff at managerial meetings) and did not encompass the other characteristics of positive alliances described by Shortell et al (Pettigrew, Ferlie, & McKee, 1992; S. Shortell, Morrison, & Friedman, 1990), which include: 1) looking for common ground, 2) involving selected physicians early in planning, 3) identifying the needs and interests of key physicians, 4) daily efforts to build a climate of trust and honesty; 5) breaking down negative stereotypes; and 6) understanding what physicians valued.
Of the 14 articles that examined the association of organizational climate and innovation dissemination in health, four (29%) focused on health care organizations, one (7%) focused on public health organizations, four (29%) focused on schools, three (21%) focused on mental health or substance abuse organizations, and two (14%) focused on more than one type of organization (see Table 4). Six articles measured organizational climate as a one-dimensional construct; these measures of climate included from one to thirty-two items. Eight articles measured organizational climate as a multi-dimensional construct; these measures included sixteen to thirty-nine items.
The majority of articles (n=11) assessed organizational climate at the organization-level of analysis. In all but one case (Brownson et al., 2007), team-level or organization-level assessments of climate were based on the perceptions of multiple respondents per unit. The articles also examined the association of organizational climate with a variety of dissemination-related outcomes, although the majority focused on adoption (5 articles (36%); e.g., presence of innovative medical imaging technologies), and implementation (6 articles (43%); e.g., therapist adherence to multi-systemic therapy). Two articles (14%) focused on sustained innovation use (e.g., institutionalization of obesity prevention interventions in schools). In all but one case, the level of the outcome variable matched the level of the climate variable. Only Schoenwald and colleagues (Schoenwald, Carter, Chapman, & Sheidow, 2008) examined the possibility of cross-level effects—that is, the effects of organizational climate on aggregate (i.e., organization-level) and individual-level therapist adherence to multi-systemic therapy.
Of the 14 articles reviewed, there were 14 instruments used to assess organizational climate. In other words, each article used a different instrument.
Nine (64%) of the studies reviewed used in whole or in part well-researched, standardized instruments to assess organizational climate. Examples include the Team Climate Inventory, Creative Climate Questionnaire (CCQ), the Organizational Social Context instrument, the Charles F. Kettering School Climate Scale, the TCU Organizational Readiness for Change instrument, and the Organizational Climate Questionnaire (OCQ). Not surprisingly, the psychometric properties of these instruments are better understood than those of locally adapted, exploratory instruments. Even for these well-researched instruments, however, questions remain about their validity and reliability. Mathisen and Einarsen (Mathisen & Einarsen, 2004) raised questions about the translational validity of the CCQ. Although the instrument’s developer defined climate as an organizational attribute, the CCQ measured individual perceptions of the work environment. These authors also note that instrument’s developer has not presented sufficient information to support his claim that the CCQ has adequate psychometric properties. Similarly, several authors have raised doubts—based on empirical study—about the factor structure of the OCQ (Hellriegel & Slocum, 1974; Patterson et al., 2005; Simms & Lafollette, 1975). It remains unclear whether these scales developed by Litwin and Stringer (Litwin & Stringer, 1968) and subsequently used in two of the studies included in this review (Nystrom, Ramamurthy, & Wilson, 2002; Wilson & Ramamurthy, 1999), actually measure what they purport to measure. Likewise, the climate scales in the TCU instrument have displayed variable levels of reliability across types of respondents and settings. For example (Lehman, Greener, & Simpson, 2002), computed alpha coefficients for each of the six climate scales for administrators, staff, and programs. Six of the eighteen alpha coefficients were below the accepted .70 threshold. Other studies also report variable levels of reliability for climate scales, with some also reporting alpha coefficients well below .70 (Rampazzo, De Angeli, Serpelloni, Simpson, & Flynn, 2006; Saldana, Chapman, Henggeler, & Rowland, 2007).
The climate instruments included in this review exhibited mixed results in terms of predictive validity (see Table 6). Three (21%) instruments showed statistically significant associations with dissemination-related outcomes in health. Allen and colleagues (Allen, Lehrner, Mattison, Miles, & Russell, 2007) observed a positive, significant relationship between organizational climate—which they measured as perceived organizational support—and the frequency with which health care providers screened patients for possible domestic violence. In an organizational sample that included medical research and services firms, Choi and colleagues (Choi, Andersen, & Veillette, 2009) found that an unsupportive organizational climate inhibited employee creativity. Finally, Wilson and Ramamurthy (Wilson & Ramamurthy, 1999) observed in correlation analysis that organizations with more risk-oriented climates tended to adopt more radical innovations and innovations that provide greater relative advantage.
Eight (57%) instruments exhibited only partial evidence of predictive validity for dissemination-related outcomes in health. In most cases, some of the dimensions of organizational climate assessed by the instrument were significantly associated with some of the dissemination-related outcomes examined in the study. For example, Anderson and West (Anderson & West, 1998) found that one dimension of team climate, support for innovation, emerged as the only significant predictor of overall innovation and innovation novelty. Another dimension of team climate, participative safety, emerged as the best predictor of the number of innovations. It also emerged as the best predictor of team self-reports of innovativeness. A third dimension of team climate, task orientation, predicted innovations’ anticipated administrative efficiency. Finally, as Table 6 shows, well-researched climate instruments exhibit no greater predictive validity than locally developed, exploratory ones, at least with regard to dissemination-related outcomes in health. It is difficult to interpret the mixed results observed in these and other studies included in this review because the statistically significant relationships between climate dimensions and dissemination-related outcomes displayed no obvious pattern.
Of the 3 articles that examined the association of absorptive capacity and dissemination in health, one focused on health care organizations, one focused on private substance abuse organizations, and one focused on governmental health ministries and hospitals (see Table 4).
There was no agreement across the three studies reviewed in terms of how Absorptive Capacity (AC) was measured. Caccia-Bava (Caccia-Bava, Guimaraes, & Harrington, 2006) defined and measured two dimensions of AC: managerial knowledge and communication. The communication dimension encompassed communication channels, cross-function teams for knowledge integration, and communication boundary spanners. Both dimensions were related to hospitals’ IT adoption, the dependent variable. This study also examined the relationship between absorptive capacity and organizational culture; strong developmental and rational cultures and weaker hierarchical culture were significantly related to both dimensions of absorptive capacity (see Table 6).
Knudsen and Roman (Knudsen & Roman, 2004) defined and assessed AC across three different dimensions, including workforce professionalism, environmental scanning, and satisfaction among client referral sources and third party payers. The dependent variable was organizational adoption of treatment innovations. They found partial support for workforce professionalism as a predictor of innovation adoption; both environmental scanning and satisfaction were associated with innovation adoption.
Belkhodja and colleagues (Belkhodja, Amanr, Landry, & Ouimet, 2007) defined absorptive capacity as the size of the unit, as an estimate of its capacity to acquire/absorb knowledge; and whether there were people in the unit who were paid to do research. The dependent variable was whether or not governmental health service organizations used research, assessed for seven types of activities: (1) received research results for areas of responsibility; (2) understood research results; (3) referenced research evidence; (4) adapted research results to provide information to decision makers; (5) promoted adoption of research evidence; (6) made professional decisions based on evidence; and (7) made concrete changes in services provided based on use of research evidence. Absorptive capacity was associated with research utilization. However, the individual components of absorptive capacity had different relationships with research utilization in different kinds of health organizations. Being a medium size unit was related with research utilization in health ministries and hospitals, but not in regional health authorities; being a smaller unit was associated with higher research utilization in regional authorities, as did having paid research staff.
There is currently a growing emphasis on the D & I of evidence-based strategies at the organizational level. The standard methodologies used in the health promotion and disease prevention literature to create evidence (e.g. randomized trials) have serious limitations for research focused on how to disseminate evidence-based interventions (Mercer, DeVinney, Fine, Green, & Dougherty, 2007). Attention to measurement issues is critical if we are to make progress in both understanding the factors that influence dissemination and for developing approaches to accelerate the adoption and use of evidence based strategies in practice and community settings (Bowen et al., 2009; Weiner, 2009). The development of this area of research will benefit from use of good measures with adequate psychometric properties.
The purpose of this paper was to identify available measures for 5 key systems antecedents hypothesized to be important for D & I research (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004), to report on the measures’ psychometric properties, and to develop a compendium of measures that could be recommended for use. We limited our review to peer-reviewed empirical research that focused on outcomes related to innovation, diffusion, adoption, or implementation, that assessed the construct at the organizational level, and that employed a quantitative measure of the construct. Some of the constructs we studied have a larger quantitative literature (e.g. climate, leadership) than others (e.g. managerial relations, absorptive capacity), and thus we expected that we might be able to make more recommendations for measures for those constructs. Overall, however, the picture was the same across the five constructs—no measure of a specific construct was used in more than one study, many studies did not report the psychometric properties of the measures, some assessments were based on a single response per unit, and the level of the instrument did not always match the level of analysis. Thus, we are unable to make any recommendations for specific instruments in any of the five constructs evaluated.
A key issue with the majority of the constructs measured is that they might not be homologous multi-level constructs. That is, the construct’s meaning, measurement, and relationships with other variables differ across levels of analysis. Construct validity issues arise when an organizational characteristic is measured at the individual-level of analysis when such measures do not convey the construct’s emphasis on shared perceptions of the organization. Construct validity issues also arise when data regarding organizational characteristics are obtained from only one individual. One cannot ascertain from a single respondent the extent to which other organizational members share his or her perceptions across the organization. As more research is done using these constructs, it would significantly benefit the field if extant measures were evaluated and built upon, rather than new measures being created for each inquiry. Across all of the constructs, there is a significant need for strong psychometric evaluations of the measures’ properties, and in particular a focus on predictive validity across a range of D & I outcomes. Of particular concern is the considerable diversity in how the constructs are measured, which suggests a lack of theoretical agreement on the constructs’ meaning. This highlights the need for more theoretical work, particularly for construct that do not seem homologous. The need for more careful attention to measurement practices was also apparent in this review. When theory suggests that consensus is important, then it is critical to have more than one respondent. In addition, when multiple respondents are utilized, it is important to conduct the appropriate statistical tests (e.g., within-group agreement statistics) before aggregating to higher levels of analysis.
Three limitations merit discussion. First, our review only covered those articles published in peer-reviewed journals. Our review could be subject to a publication bias if the “gray literature” contains reliable, valid measures of system antecedents for dissemination and implementation that do not also appear in peer-reviewed articles. Second, our review only included articles that contained the term “health” in the title, abstract, or keyword list. Restricting the search in this way facilitated the review by eliminating articles that did not focus on health-related dissemination and implementation issues, our principal concern. However, our search might have missed articles that included disease-specific terms like diabetes, cancer, stroke, or HIV rather than the broader term “health.” Finally, our review only includes articles that contained a measured outcome relevant to innovation, diffusion, adoption, or implementation. Although we accepted a broad range of outcomes within these domains, we excluded articles that measured outcomes in terms of organizational members’ knowledge, attitudes, or behavioral intentions. Researchers interested in reliable, valid measures of system antecedents relevant to these important precursors to dissemination and implementation should exercise care in interpreting and using the results of our review.
It is likely that the concerns raised in this review of measures of systems antecedents of D & I are relevant to other organizational constructs, as was noted in Weiner’s evaluation of measures of organizational readiness (Weiner, Amick, & Lee, 2008). It is important for health promotion researchers to recognize that there is a large literature in other fields focusing on organizational assessment beyond the health sector. Inclusion of this literature was beyond the scope of this review, but may include measures that would be useful in the health context. If we are to build a literature that addresses how to effectively disseminate and implement evidence-based health interventions, we must consider the development and evaluation of a robust set of measures that will serve as the basis of building the field, allowing for comparisons across organizational types and intervention topics, and will allow a robust area of research to develop.
This manuscript was supported in part by the Centers for Disease Control and Prevention and the National Cancer Institute through the Cancer Prevention and Control Research Network, a network within the CDC’s Prevention Research Centers Program (U48 – DP001911, -U48-DP000057, U48-DP001946, U48/DP000059 , U48/DP001944) and by grants from the National Cancer Institute (1K05 CA124415-01A1, R01 CA124402, R01 CA124397, R01 CA123228). The authors would like to acknowledge Glenna Dawson for her work on the literature reviews, and Nancy Klockson for participation in the manuscript preparation.
The authors have no conflict of interest to report.