Identifying outcomes representing the added value of implementing projects through coalitions depends on perspective, interpretation, judgment, and familiarity with project details. Without an organizing principle, coalitions are likely to measure and summarize added value very differently. This paper outlines an attempt to reach a consensus definition of and criteria for assigning added value through a negotiated, iterative process. Although knowledgeable persons can debate the proposed definitions or the inclusion of specific entries, the process focuses and operationalizes an idea that was previously conceptual and unbounded. Beyond the specific findings defining the added value of coalitions in this project, the use of a social ecological model to identify the components of added value and the placement of those components within a logic model specific to coalitions should provide useful tools for those planning and assessing coalition-based projects.
The methods and results described in this paper contribute to the coalition literature in several ways. Directly, they model a process through which funders and project managers can describe and document the benefits of coalitions so as to weigh the benefits against the cost of coalition development and maintenance. Indirectly, they provide additional tools for describing and evaluating what, in the Community Coalition Action Theory,5
is labeled “synergy.” If the function of coalitions extends beyond serving as conduits of funding, they will demonstrate the types of interaction, institutional change, pooling of resources, and collaboration demonstrated and reported by the CAACP coalitions. The components of added value, as defined here, may be critical to the transition between coalition maintenance and institutionalization of change, and thus worthy of further study. As noted in the Introduction
, Butterfoss and Francisco1
describe 3 levels of coalition evaluation: measures of coalition infrastructure, function, and procedures; indicators of the extent to which interventions and activities are carried out and reach the target populations; and outcomes involving health and community change. The process described in this paper addresses an intermediate level, which conceptually falls between the first 2. That level includes the processes and activities that are intrinsic to a coalition and distinct from the evaluation of a project implemented through a coalition.
An ecological assessment of community coalitions as proposed by Goodman20
requires understanding the effect of coalitions on policy development. It is, however, difficult to isolate the effect of the coalitions from contextual factors and secular trends. The use of the inclusion and exclusion criteria developed by the CAACP sites, and the identification of added value mechanisms can help to support or refute coalitions’ contributions to policy and other distal outcomes. These tools should assist planners in identifying the types of changes, and level of change, that are most directly attributable to a collaborative coalition process. This information, in turn, should help to define the primary goals, memberships, and operational priorities of a coalition.
The interpretation of these results is subject to several limitations. Because the CDC required that the CAACP projects operate through coalitions, those communities that applied most likely represented a subset of groups with either a history of collaboration or a commitment to the coalition approach—possibly not characteristic of all communities. Moreover, CAACP coalitions had substantial financial support for administrative functions and for interventions not generally available to other coalitions. Much of the funding for the CAACP was subcontracted to partners, possibly promoting cooperation and collaboration. Although the continuation of the coalitions and their interventions a year beyond project funding suggests they were able to achieve sustained change in the short term, measuring the durability of that change is an important future project.
The outcomes listed were self-reported by the site representatives and were not independently validated. CDC leads did, however, have knowledge of the level of accomplishment through routine site visits and semi-annual written reports from each site. While the methodology captured specific and concrete outcomes, because of variations in the nature and potential impact of different outcomes, it did not determine the relationship between the number of added-value outcomes and coalition size, structure, governance, membership, length of existence, or other factors as reported in other publications.21
It also did not determine which types of projects or activities were best implemented through coalitions as opposed to small, project-oriented partnerships. An additional limitation is the non-systematic collection of information regarding the mechanisms through which added value was achieved. Future applications of this methodology should consider the literature on social capital which is concerned with understanding how social–structural relationships within collaborating groups effect what can be accomplished by the group.22
The work group used inclusion and exclusion criteria to standardize the language and apply consistent terms to the process of attributing outcomes to coalitions. These criteria and the definition of 7 added-value components may have limited thinking about components of added value falling outside the framework. The criteria themselves are subject to interpretation. They should be considered an imperfect, initial step in a process that needs to be validated and further refined through appropriate research strategies similar to those used in the development of inclusion and exclusion criteria for psychiatric diagnoses.23
The methodology presented here is capable of further development to advance the knowledge base about coalitions. Defining and tracking costs would, for example, be an important next step. Project costs include staff time for developing policies and procedures, organizing coalition and committee meetings, and communicating with members, as well as administrative overhead and material costs associated with meetings. The costs can be substantial to participants preparing for, travelling to, and attending meetings. Opportunity costs also confront individuals and organizations in a coalition, in that activities that might have been undertaken independently are delayed or deferred.
Ideally, added-value outcome data and coalition costs should be collected prospectively. This would require funders and program managers to agree on the procedures by which to collect data, including a system for tracking costs, during a project’s planning phase. Funders and program managers would also have to reach consensus regarding the definition, components, terms, and criteria for documenting added value. In addition to creating a credible database, such advance planning would articulate and communicate the coalition’s goals at the institutional, community, and policy levels. Although the nature of the inputs, outputs, and outcomes may preclude a formal cost-effectiveness analysis, a “real time” tracking of costs and added value outcomes would permit funders and other stakeholders to assess whether they have struck a reasonable balance between costs incurred through the coalition approach and the outcomes uniquely attributable to the coalition. Further, the routine identification and documentation of the mechanisms through which added value is achieved—or perceived to be achieved—can provide helpful feedback to the coalition members and useful guidance for future efforts.