|Home | About | Journals | Submit | Contact Us | Français|
Community-based coalitions are commonly formed to plan and to carry out public health interventions. The literature includes evaluations of coalition structure, composition, and functioning; evaluations of community-level changes achieved through coalition activities; and the association between coalition characteristics and various indicators of success. Little information is available on the comparative advantage or “added value” of conducting public health interventions through coalitions as opposed to less structured collaborative mechanisms. This paper describes a qualitative, iterative process carried out with site representatives of the Controlling Asthma in American Cities Project (CAACP) to identify outcomes directly attributable to coalitions. The process yielded 2 complementary sets of results. The first were criteria that articulated and limited the concept of “added value of coalitions”. The criteria included consensus definitions, an organizing figure, a logic model, and inclusion/exclusion criteria. The second set of results identified site-specific activities that met the definitional criteria and were, by agreement, examples of CAACP coalitions’ added value. Beyond the specific findings relevant to the added value of coalitions in this project, the use of a social ecological model to identify the components of added value and the placement of those components within a logic model specific to coalitions should provide useful tools for those planning and assessing coalition-based projects.
The online version of this article (doi:10.1007/s11524-010-9520-y) contains supplementary material, which is available to authorized users.
Agencies that fund comprehensive, community-based public health projects often encourage or require the formation of community coalitions to design, plan, and carry out those projects.1 The underlying assumption is that a formal collaboration of diverse stakeholders, including community members, will either achieve goals beyond the reach of individuals or organizations, or that collaboration will reach those goals in a more efficient, effective, and sustainable way.2 Proponents of community coalitions reason that coalitions (1) create a “critical mass” of interested persons and necessary resources to effect change, (2) reduce competition between members and avoid duplication of effort through improved communication and trust building, and (3) promote multi-level, multifaceted interventions that work synergistically to address complex public health problems.3 Proponents further claim that community-based coalitions are likely to achieve durable change because they create new linkages and modes of collaboration between groups and organizations, negotiate broad support from community members and other stakeholders, and advocate for supportive policies and legislation.4 The cost of forming, developing, and maintaining coalitions, however, is substantial,5 and the benefits—as opposed to alternative means of interaction among stakeholders—have not been well defined.
Community-based coalitions are usually complex constructs that wrestle with complex projects. Evaluating coalition performance, then, must involve multiple layers of assessment, including both process and outcome evaluation. Butterfoss and Francisco1 describe 3 levels of coalition evaluation:
Many published evaluations of community-based coalitions focus on the first of these levels and report on coalition composition, processing, and functioning, for which numerous tools are available.6 The literature includes analyses of the relationship between coalition characteristics (e.g., leadership, decision making, organizational climate) and member satisfaction,7 determinations of how perceived benefits and costs to members relate to participation,8 and measurements of how coalition characteristics effect coalition activities and interventions.9 Coalitions can now use electronic monitoring systems to track the association between those activities or interventions and community changes.10 Studies focused on outcome evaluation have analyzed the extent to which coalitions have been successful in leveraging funds, strengthening community leadership, and achieving policy and systems changes at the community and organization levels.4,11–13
As more distal outcomes were evaluated, however, it became increasingly difficult to identify the added value of implementing projects through coalitions over more traditional approaches.11 The coalitions evaluated in the literature received considerable external funding to support their interventions. There were few, if any, comparison communities carrying out the same projects through non-coalition mechanisms. Thus, it was problematic to differentiate between outcomes resulting from interventions or activities funded through the coalitions, and outcomes attributable to the coalition itself—that is, the added value of the coalition.
The National Asthma Program of the Centers for Disease Control and Prevention (CDC) funded seven sites of the Controlling Asthma in American Cities Project (CAACP) to implement comprehensive asthma projects in communities of need through community-based coalitions. The coalitions were charged with providing services and carrying out interventions, as well as with achieving policy and systems changes. Other papers in this journal supplement describe some of the interventions accomplished through these projects and their outcomes. This paper describes the process used to distinguish between accomplishments achieved through the project and its funding, and accomplishments dependent on or markedly enhanced by the formal collaboration of diverse partners through a coalition structure. It attempts to identify outcomes that, in the absence of coalitions, would likely not have occurred and to describe the issues encountered when attempting to determine a coalition’s added value.
The purpose, design, and theoretical framework underlying the CAACP, the characteristics of the coalitions, and the main interventions carried out by the sites are described elsewhere in this supplement.14,15 The 7 CAACP sites were funded for 2 years of planning and 5 years of implementation between 2001 and 2008. The average total funding was $5 million per site. The application did not exclude applicants without a preexisting coalition, but funded recipients were required to develop or enhance a collaborative group with broad community and organizational representation.
All 7 coalitions were formed from groups and individuals who had worked together previously, either on asthma or on other child health issues. The size of the coalitions ranged from approximately 40 to over 100 members. The membership varied, but generally included academic institutions, local and state education and public health agencies, non-governmental agencies (e.g., the American Lung Association as well as local grass roots organizations), school districts, faith-based organizations, health plans, hospital systems, health care provider groups, home-health agencies, and public health departments. Local faith-based, ethnic, and environmental grass roots organizations represented community interests on an ongoing basis; the coalitions sought input from individuals in the community through focus groups to assess needs and through participation in committees, work groups, or advisory panels.
For 4 of the sites (Chicago, Minneapolis/St Paul, Northern Manhattan, and Philadelphia), the coalition’s activities were originally limited to the project. As the Minneapolis/St Paul coalition leveraged additional funds, however, it gradually expanded its scope over the CAACP time frame. By the end of the project period, the Philadelphia, Northern Manhattan, and Chicago coalitions also transitioned to become part of asthma coalitions serving larger geographic areas. From the onset, the coalitions in Oakland, Richmond, and St Louis were part of larger collaboratives that focused on asthma but had other funding and were involved in other projects.
Data were collected both prospectively and retrospectively using mixed methods. This iterative, interactive process involved CDC staff, as main stakeholders in the evaluation, and representatives from each of the 7 sites. The idea of describing the added value of coalitions was introduced at the end of implementation year 2 when CDC gathered information about the CAACP coalitions through structured telephone interviews with each site’s project manager. Responses to questions about the benefits of implementing the project through a coalition varied, with some respondents attributing all activities and outcomes to the coalition, and others identifying the development of very specific relationships or products. Beginning in implementation year 3, CDC introduced a new section in the annual report template for the purpose of capturing outcomes related to added value in a standardized way. The CDC leads adapted the operational definition of systems change proposed by Cheadle et al.11 in the evaluation of the California Wellness Foundation’s community coalitions to propose 3 categories of achievement that might reflect added value:
Cheadle’s fourth systems change activity, finance/budget reform, was later adapted to reflect resources generated by the coalition.
An extensive review of the literature, begun in 2006 and updated periodically through 2009, did not identify an appropriate tool for assessing the added value of CAACP coalitions. Information reported by the sites and the experience of the sites’ representatives helped guide a work group of representatives from all the sites to develop further the concept of “added value” and to design an approach to describing it. During face-to-face sessions conducted over 2 days of an annual meeting in 2006, the work group reached consensus on initial definitions of coalition and of “added value,” and refined a logic model to reflect accurately the sites’ shared assumptions and experiences of how their coalitions functioned and what they achieved. The group drew on the definitions and logic model to expand the original 3 added-value categories into the 7 components listed in Table 1. They also nominated 3 people (2 site representatives and 1 CDC lead) to continue the process and propose a set of inclusion and exclusion criteria for each added value component.
The development and refinement of the inclusion and exclusion criteria was an iterative process first among the 3 designated members and then among representatives from each site. Using site input provided during the annual meeting and relevant literature as a guide, 1 person proposed a set of inclusion and exclusion criteria. These criteria were critiqued by the committee members, revised and critiqued again, and so on, until consensus was reached. At that point, the process of critiquing and revising was repeated with the full work group. This process identified ambiguities and inconsistency in wording or content and raised challenges to specific criteria that were resolved by consensus.
CDC leads drew from site progress reports for the last 3 implementation years to identify outcomes that potentially reflected the added value of the coalition approach. The same individuals assigned those outcomes to one of the 7 previously defined components of added value and identified inclusion and exclusion criteria that applied to each outcome. A combined spreadsheet of all the sites’ entries was then forwarded for review by the site’s coalition for the purposes of (1) generating ideas regarding additional outcomes and (2) responding to the inclusion or exclusion criteria suggested by the CDC leads for each proposed outcome.
Each coalition selected 2 or more representatives to further revise their site’s spreadsheet in collaboration with the CDC leads. Selection was based on knowledge of the project. Both representatives from 5 of the sites and one of the representatives from the other 2 (both principle investigators) had been in a leadership position since the project’s planning phase. During conference calls between each site’s representatives and the CDC authors, all outcomes were reviewed to clarify whether items
Although many outcomes met the criteria for several components, each outcome was included only once, under the component of its main effect. Throughout this process, attention was also paid to the inclusion/exclusion criteria, with consistent re-examination of appropriateness. In particular, it was during this period that some criteria were identified as applicable to all components while others were designated as specific to one or more components.
After the first set of calls, CDC leads reviewed the revised spreadsheets from all the sites to ensure that outcomes were consistently defined and catalogued across sites. They also identified outcomes over which disagreement about inclusion arose between the CDC leads and site representatives, and specified the inclusion/exclusion criteria leading to that disagreement. In the course of a second round of conference calls, the CDC leads and site representatives discussed those items needing clarification or about which disagreement arose. During these discussions, the site representatives identified and documented the mechanisms through which, in their opinion, the coalitions achieved the disputed outcomes. If the CDC and site representatives could not reach agreement on inclusion, that was documented in the spreadsheet.
The CDC leads revised and returned spreadsheets for a final review by the site representatives approximately 1 year after conclusion of funding, and sites were given the opportunity to provide additional justification for outcomes excluded by the CDC leads.
As no attempt was made to score the scope or effect of each outcome, the number of outcomes did not necessarily reflect the effort or effectiveness of a particular coalition or site. Thus, the work group felt it inappropriate to attempt to analyze the number of outcomes by site or coalition characteristics.
Project documentation and CDC site visits confirmed that all coalitions functioned well with respect to infrastructure, governance, and procedures over the time frame of the project. Six of the 7 conducted formal self-assessments, all of which yielded positive feedback. Structure and governance variations did arise between sites; 5 of the projects had bylaws or formal decision making procedures, the other 2 reported informal procedures and consensus decision-making.
The work group’s methods to assess added value yielded 2 complementary sets of results. The first were products that defined and limited the concept of “added value of coalitions.” These included definitions of coalitions and of added value, an organizing figure, a logic model, and inclusion/exclusion criteria. The second set of results identified site-specific activities that met the definitional criteria and were, by agreement, examples of the added value of the CAACP coalitions.
The working group considered a number of published and proposed definitions of an asthma coalition, and agreed on one adapted from previous discussion of community partnerships:16
A professional and/or community-based, ongoing, organized collaborative effort by people with expertise, resources, passion, and/or influence to address asthma as a community health problem.
The group proposed and adopted a definition of the added value of CAACP coalitions that included
…a variety of outcomes attributable to the work of the coalition, individual participants or participating organizations, which would not have occurred in the absence of the coalition, or were significantly enhanced due to the existence of the coalition.
Two models were created to identify the relationship among the various components of added value and the relationship of the components to inputs, outputs, and long-term outcomes. For the first (Figure 1), the group modeled the components of added value after the social–ecological model of influence,17 beginning with intrapersonal influences and proceeding outward to include changes at the interpersonal, institutional, community, and policy levels. The design was similar to the concentric circles of influence in asthma control proposed by Clark et al.18 Drawing on their collective experience, workgroup members specified the following correlates to the ecological levels:
These multilayered components of added value then became the core around which the second model—the common logic model initially proposed by CDC—was refined and finalized (Figure 2). The outcomes in the shaded central column represent those the authors identified as most closely and uniquely linked to the added value of a coalition approach. No doubt, the coalition process could also influence many of the outputs and more distal outcomes listed in the model, such as intervention design or increased access to diverse populations. The group consensus was, however, that with data available to the sites, the coalitions’ contribution to those outputs and outcomes—as distinguished from the interventions’ effect—could not be credibly measured or described.
The development of explicit inclusion and exclusion criteria guided the work of populating each added value component with specific outcomes. Box 1 lists the one inclusion and 6 exclusion criteria that applied to all components. Additional criteria for individual components were also developed and applied. For example, one inclusion criterion for Increased coordination/linkages among participating organizations was the signing of contracts or memoranda of agreement between organizations that had not collaborated before coalition formation but that were directly attributed to the coalition. One exclusion criterion was a relationship or interaction between organizations that depended on specific persons and thus was not institutionalized. The criteria for the other components are available electronically (e-Table 3).
Table 1 contains a simple count, by site, of the number of outcomes in each component. Table 2 lists examples of the types of outcomes identified. Most sites identified the majority of outcomes in the components: “Institutional changes within” and “Linkages between” member organizations.
Entries in the “resources generated” component indicated that 3 of the sites had won competitive awards for multiyear funding. In the other sites, interventions had been sustained through a combination of institutionalization by member organizations and funding generated through the coalition. One site (B) was particularly effective in marketing the coalition and its accomplishments. Site B supported its interventions through multiple donations from foundations, health plans, non-government organizations, community groups, and individuals. One year after the funding period, major project-related interventions continued at all sites, and all 7 coalitions continued in some form.
Site representatives were not systematically asked about the mechanism by which added value outcomes were generated. Nevertheless, during the conference calls, information about mechanisms or processes emerged. Some of the outcomes resulted from the meeting-based contacts of diverse members who had not previously communicated with each other. Others resulted from group members refining their knowledge and understanding of community-based asthma interventions by participation in multiple cycles of planning, implementing, evaluating, and revising plans, a process related to the “communities of practice” concept.19 Some outcomes were dependent on an increased motivation or willingness to change that developed from accountability to the group or a “shared passion.” Certain interventions were simply too big or too diverse for a single member to implement and required the expertise, coordination, and burden-sharing of multiple coalition partners. The power and credibility of a large collaborative body speaking with one voice advanced legislative and policy initiatives in ways that individual organizations could not.
CDC and 4 sites reached final agreement on all outcomes. Still, on 52 (22%) of the 240 added-value outcomes, the site representatives from 3 sites and the CDC leads did not agree. Most disagreements straddled the fine line between (1) outcomes that resulted from the coalitions and (2) outcomes that resulted from interventions funded and implemented through the coalitions, thus mapping to the sixth exclusion criterion (Box 1). Five sites and the CDC leads considered asthma educator training and certification as an intervention only paid for by the coalition—thus not directly attributable to coalition work. Representatives from 2 sites felt the coalition increased members’ awareness of and motivation to take the preparatory course, which resulted in raising the competence and skills of community professionals. All sites worked with primary care practices to improve procedures and systems to promote the best asthma care. Six of the sites considered the resultant changes in these primary care practices as intervention products (and not a coalition outcome). One site considered the primary care practices’ willingness to participate and change to be an outcome of the coalition and reasoned that should be listed as such. In fact, most (38) of the differences between the site representatives and the CDC leads reflected a single CAACP site’s classification of institutional improvements in primary-care practices as coalition outcomes.
Identifying outcomes representing the added value of implementing projects through coalitions depends on perspective, interpretation, judgment, and familiarity with project details. Without an organizing principle, coalitions are likely to measure and summarize added value very differently. This paper outlines an attempt to reach a consensus definition of and criteria for assigning added value through a negotiated, iterative process. Although knowledgeable persons can debate the proposed definitions or the inclusion of specific entries, the process focuses and operationalizes an idea that was previously conceptual and unbounded. Beyond the specific findings defining the added value of coalitions in this project, the use of a social ecological model to identify the components of added value and the placement of those components within a logic model specific to coalitions should provide useful tools for those planning and assessing coalition-based projects.
The methods and results described in this paper contribute to the coalition literature in several ways. Directly, they model a process through which funders and project managers can describe and document the benefits of coalitions so as to weigh the benefits against the cost of coalition development and maintenance. Indirectly, they provide additional tools for describing and evaluating what, in the Community Coalition Action Theory,5 is labeled “synergy.” If the function of coalitions extends beyond serving as conduits of funding, they will demonstrate the types of interaction, institutional change, pooling of resources, and collaboration demonstrated and reported by the CAACP coalitions. The components of added value, as defined here, may be critical to the transition between coalition maintenance and institutionalization of change, and thus worthy of further study. As noted in the Introduction, Butterfoss and Francisco1 describe 3 levels of coalition evaluation: measures of coalition infrastructure, function, and procedures; indicators of the extent to which interventions and activities are carried out and reach the target populations; and outcomes involving health and community change. The process described in this paper addresses an intermediate level, which conceptually falls between the first 2. That level includes the processes and activities that are intrinsic to a coalition and distinct from the evaluation of a project implemented through a coalition.
An ecological assessment of community coalitions as proposed by Goodman20 requires understanding the effect of coalitions on policy development. It is, however, difficult to isolate the effect of the coalitions from contextual factors and secular trends. The use of the inclusion and exclusion criteria developed by the CAACP sites, and the identification of added value mechanisms can help to support or refute coalitions’ contributions to policy and other distal outcomes. These tools should assist planners in identifying the types of changes, and level of change, that are most directly attributable to a collaborative coalition process. This information, in turn, should help to define the primary goals, memberships, and operational priorities of a coalition.
The interpretation of these results is subject to several limitations. Because the CDC required that the CAACP projects operate through coalitions, those communities that applied most likely represented a subset of groups with either a history of collaboration or a commitment to the coalition approach—possibly not characteristic of all communities. Moreover, CAACP coalitions had substantial financial support for administrative functions and for interventions not generally available to other coalitions. Much of the funding for the CAACP was subcontracted to partners, possibly promoting cooperation and collaboration. Although the continuation of the coalitions and their interventions a year beyond project funding suggests they were able to achieve sustained change in the short term, measuring the durability of that change is an important future project.
The outcomes listed were self-reported by the site representatives and were not independently validated. CDC leads did, however, have knowledge of the level of accomplishment through routine site visits and semi-annual written reports from each site. While the methodology captured specific and concrete outcomes, because of variations in the nature and potential impact of different outcomes, it did not determine the relationship between the number of added-value outcomes and coalition size, structure, governance, membership, length of existence, or other factors as reported in other publications.21 It also did not determine which types of projects or activities were best implemented through coalitions as opposed to small, project-oriented partnerships. An additional limitation is the non-systematic collection of information regarding the mechanisms through which added value was achieved. Future applications of this methodology should consider the literature on social capital which is concerned with understanding how social–structural relationships within collaborating groups effect what can be accomplished by the group.22
The work group used inclusion and exclusion criteria to standardize the language and apply consistent terms to the process of attributing outcomes to coalitions. These criteria and the definition of 7 added-value components may have limited thinking about components of added value falling outside the framework. The criteria themselves are subject to interpretation. They should be considered an imperfect, initial step in a process that needs to be validated and further refined through appropriate research strategies similar to those used in the development of inclusion and exclusion criteria for psychiatric diagnoses.23
The methodology presented here is capable of further development to advance the knowledge base about coalitions. Defining and tracking costs would, for example, be an important next step. Project costs include staff time for developing policies and procedures, organizing coalition and committee meetings, and communicating with members, as well as administrative overhead and material costs associated with meetings. The costs can be substantial to participants preparing for, travelling to, and attending meetings. Opportunity costs also confront individuals and organizations in a coalition, in that activities that might have been undertaken independently are delayed or deferred.
Ideally, added-value outcome data and coalition costs should be collected prospectively. This would require funders and program managers to agree on the procedures by which to collect data, including a system for tracking costs, during a project’s planning phase. Funders and program managers would also have to reach consensus regarding the definition, components, terms, and criteria for documenting added value. In addition to creating a credible database, such advance planning would articulate and communicate the coalition’s goals at the institutional, community, and policy levels. Although the nature of the inputs, outputs, and outcomes may preclude a formal cost-effectiveness analysis, a “real time” tracking of costs and added value outcomes would permit funders and other stakeholders to assess whether they have struck a reasonable balance between costs incurred through the coalition approach and the outcomes uniquely attributable to the coalition. Further, the routine identification and documentation of the mechanisms through which added value is achieved—or perceived to be achieved—can provide helpful feedback to the coalition members and useful guidance for future efforts.
Below is the link to the electronic supplementary material.
Inclusion and exclusion criteria used to assess the added value of Controlling Asthma in American Cities Project coalitions (DOC 41 kb)
The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.The authors gratefully acknowledge Gail Brottman, Cizely Curian, Vicky Persky, Gloria Thomas, Dolores Weems, and Kristin Wilson for reviewing their sites’ outcome spreadsheets, and Maureen Wilce for her valuable comments on early drafts of the manuscript.
This project was supported through a cooperative agreement with the Centers for Disease Control and Prevention, US Department of Health and Human Services, under program announcement 03030. The findings and conclusions of this manuscript are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention