|Home | About | Journals | Submit | Contact Us | Français|
To be effective and sustain themselves over time, public-private partnerships must make evaluation a priority. Specifically, partnerships should evaluate 1) their infrastructure, function, and processes; 2) programs designed to achieve their mission, goals, and objectives; and 3) changes in health and social status, organizations, systems, and the broader community. This article describes how to 1) develop a comprehensive evaluation strategy based on partnership theory; 2) select short-term, intermediate, and long-term indicators to measure outcomes; 3) choose appropriate methods and tools; and 4) use evaluation results to provide accountability to stakeholders and improve partnership function and program implementation.
Public-private partnerships fundamentally bring together the expertise of the private and public sectors and allow each to do what it does best, so that products and services can be delivered efficiently and effectively. These partnerships also can help overcome organizational boundaries and allow parties to work together on a shared goal. For example, even though collaborating with private-sector businesses may cause tension among nonprofit public-sector partners, the businesses may bring new skills and funding to the partnership and enhance the partnership's scope of influence. An illustration: 3 partnerships were sponsored by a national association of health plans to improve the quality of diabetes care in New Mexico, Missouri, and New York. The main evaluation finding was that the competing health plans and local organizations established and sustained viable partnerships around a shared goal, despite significant challenges (1). The process of evaluation discovered the effects of these vital partnerships by answering the following questions: 1) What attributes and events resulted in competing organizations coming together? 2) What is necessary to sustain these partnerships? 3) What recommendations can help replicate this approach for other chronic conditions in other communities? and 4) To what extent do local market characteristics and structures set partnership direction and influence success?
Evaluation is a critical task for any partnership and determines whether the organization and its activities are sustained over time. Effective evaluation provides ongoing, systematic information that strengthens the partnership during implementation and provides outcome data to assess the extent of change among participants or within systems (2).
Both the for-profit and nonprofit sectors place a high value on evaluation and regard it as a necessity rather than an enhancement. However, their underlying value systems and motivating factors may differ. For the nonprofit sector, partnership evaluation fulfills underlying process goals, such as identifying new approaches, increasing community awareness and support, informing policy decisions, and contributing to the understanding of what works (3). The public health issues have developed over time, and the public health community's expectations are that desired evaluation outcomes may take years or even decades to accomplish. Conversely, the for-profit sector is driven by increasing its market share, improving its performance through continuous innovation, having good financial management and stability, increasing efficiency, increasing customer accountability and satisfaction, improving its access to government officials, and improving public relations. The time horizon for outcomes in the business sector is often a few months to 2 years. As the value systems of the for-profit and nonprofit sectors converge, evaluation also will become more of a shared value. Partnering certainly can enable this process.
This article describes how evaluation is viewed by nonprofit and for-profit sectors, levels of partnership evaluation, and a step-by-step model for evaluating partnerships. I conclude by presenting the challenges to evaluating partnerships and recommending solutions.
Many partnership evaluations are based on collaborative or participatory approaches according to who controls the process, who participates, and how much (4). These approaches are well suited to partnerships, although they have disadvantages (Appendix A) (5).
Participatory approaches to evaluation are generally comfortable and customary for the public sector. The private sector may not be as familiar with these types of evaluations nor as patient with the extra time and effort that it takes to be democratic and attentive to the needs of all partners, priority populations, and communities served. Discussing the extra benefits that result from such evaluations, such as better understanding and acceptance of findings that may improve performance, may enable for-profit partners to be more open to these approaches and learn by participating in the evaluation process (6).
When diverse partners work together, evaluation approaches and terms must be clarified. The private sector and the public sector may differ in their approaches to the evaluation or assessment process, the standards and methods they use to gather data, how they define terms, and the kinds of indicators they plan to measure.
For example, the VERB campaign used the best practices of private-sector marketing to children. The VERB brand created an emotional affinity between the product (physical activity) and the user (tween), and engaged tweens at key places and times when they might be both inactive and receptive to the brand (7). The public sector redefined its terms — education had to include persuasion, VERB was not a program but a brand, and pooling resources with the private sector allowed the public agency, the Centers for Disease Control and Prevention (CDC), to buy media time and the talents of marketing experts. The agency also adjusted its concepts of performance measures and outcomes to meet both private investors' needs and public health's goals.
Coming to consensus on definition of terms, methods, and measures is a crucial step in building trust when beginning a partnership evaluation. Although public health professionals may feel comfortable using the terms process, impact, and outcome measures, using the terms short-term, intermediate, and long-term indicators is more descriptive and avoids confusion when working with partners from diverse professional backgrounds. Different evaluation terms that the for-profit and nonprofit sectors may use in a public-private partnership are listed in Table 1.
In public-private partnerships, evaluation may measure 1) processes that sustain and renew partnership infrastructure and function; 2) programs intended to accomplish targeted activities or those that work directly toward the partnership's goals; and 3) changes in health status or the community. Appendix B details these 3 levels and sample measures for each. The aim of every partnership should be to evaluate something in each level. Conducting a member survey to assess satisfaction with how the organization functions (level 1), evaluating 1 program or activity that the partnership conducts (level 2), and collecting extant data on key health indicators (level 3) are reasonable expectations for an annual evaluation plan.
Many practical frameworks and models exist that can help partnerships develop evaluation plans, but the focus here is on the Framework for Program Evaluation in Public Health (8,9). The framework guides its users in selecting evaluation strategies that are useful, feasible, ethical, and accurate — its 6 steps help increase understanding of a partnership's context as well as its outcomes. The steps have been refined and described below to apply to partnership evaluation.
Engaging stakeholders means fostering participation and power sharing among people invested in the evaluation and its findings. Stakeholders include 1) those involved in program operations (eg, partners, public relations professionals, lawyers, sponsors, funders, collaborators, administrators, managers, business owners, staff), 2) those served or affected by the program (eg, clients, customers, families, neighborhood organizations, academic institutions, elected officials, advocacy groups, professional associations, opponents), and 3) primary users of the evaluation. Stakeholders must understand the organizational structure, history, and goals of the partnership and how politics affect program implementation and impact. This understanding can be attained by creating an environment where stakeholders discuss their values, philosophies and assumptions, and capabilities. Stakeholders may 1) provide resources for evaluation such as staff and in-kind supplies, 2) clarify partnership goals and objectives, 3) identify and prioritize evaluation questions, 4) develop and pilot evaluation methods and tools, 5) collect data, and 6) interpret and report results (10).
Stakeholders' needs, concerns, and demands for specific outcomes differ widely, even though they may agree with the partnership's goals and objectives. To motivate the stakeholders to participate in the partnership and its evaluation, data could be gathered from them about what they need the evaluation to measure. A sample set of criteria is included in Table 2 (11).
This description should focus on the purpose, goals, objectives, resources, current and planned activities, expected outcomes, stage of development, and environmental context of the partnership. In a public-private partnership, objectives are based on compromise among partners with different political, social, and economic aims. Divergent interests concerning actions and expected impacts must be taken into account (11). Developing a logic model is 1 way to help stakeholders clarify the partnership's rationale, strategies, and conditions. It serves as a road map of the program, prioritizes the sequence of activities, summarizes expected change by linking processes to eventual outcomes, shows how partnership programs are linked to other ongoing efforts, and displays the infrastructure needed to support the partnership (9). An effective logic model will be refined and changed many times, as the partners learn about how and why the partnership works. A sample logic model for a state partnership is in CDC's National Heart Disease and Stroke Prevention Program's Evaluation Guide (10).
Partnerships develop in stages: 1) formation — initial building of the organization, 2) implementation — strategic planning and conducting of activities to address goals, 3) maintenance — sustaining activities until goals are accomplished, and 4) institutionalization — collaborative attainment of goals in permanent structures within the community (12,13). The current stage of the partnership should be assessed to determine the proper focus for evaluation (8). For example, evaluation activities should focus on identifying and recruiting partners if the partnership is in the formation stage; communication and decision making in the maintenance stage; and community changes in the institutionalization stage. A comprehensive evaluation of a mature partnership includes measures at each stage.
The evaluation should focus on issues of greatest concern to stakeholders, while efficiently using time and resources. A written plan that summarizes evaluation goals and procedures and outlines the stakeholders' roles and responsibilities is essential. The plan should include evaluation questions and practical methods for sampling, data collection, data analysis, and interpretation. Stakeholders can help prioritize the questions to determine which are critical, are likely to improve the partnership, and can be answered with available resources. Questions may include the following:
The evaluation design is linked to the priority questions, and the choice of design has implications for what data will be collected and how. A pretest-posttest design uses a comparison group, measures the partnership on given parameters before and after it implements planned improvement strategies, or both. A case study design is used to study the partnership's context, history, structure, and function. Case studies usually rely on multiple sources of information such as observations, interviews, audiovisual material, documents, and reports. Appendix C provides a sample evaluation plan for partnerships (14).
After deciding on the evaluation questions and design, the partnership must decide what data it needs to answer the questions, where and how the data can be obtained, and how the data should be analyzed and used. Adequate data may be available and easily accessed, or new data may have to be collected. Evaluation data should provide a well-rounded picture of the partnership and its programs so stakeholders can perceive the results as believable and relevant. Integrating qualitative and quantitative data increases the likelihood that data will be balanced and accepted by all stakeholders (15).
For each evaluation question, at least 1 indicator or data point must be defined and tracked. Examples of indicators for partnerships might include measures of 1) partnership effectiveness (eg, participation in meetings and activities, usefulness of partnership structures), 2) partnership activities (eg, participation rate, completion of state plan objectives), and 3) partnership effects (eg, number of policies or practices that were amended or adopted, health status changes). Practitioners and researchers have summarized measures that document changes in partnership knowledge, attitudes, practices, community environment, policies, and health status (2,16,17).
For each evaluation question and indicator, sources of data must be identified. Data from documents, key informant interviews, meeting observations, member surveys, and focus groups provide different perspectives of the partnership and enhance the comprehensiveness and credibility of the evaluation. Census data (including economic data and demographics), health survey data (eg, Behavioral Risk Factor Surveillance System survey results), or behavioral outcome data (eg, emergency medical transports, hospital admissions) represent likely data sets that partnerships may use to assess health and quality of life status. A rule of thumb is to collect only data that will be used and to use all data collected.
In deciding what instruments to use, partnerships may develop their own questionnaires or interview frameworks, use validated and reliable tools, or modify an existing tool to fit their priority population(s), community culture, and issues. Coalitions and Partnerships in Community Health (2) lists tools and resources for these evaluations. Stakeholders should develop clear procedures for gathering, analyzing, and interpreting data, and training staff, partnership members, and volunteers to collect quality data.
After designing an evaluation, data must be collected, described, analyzed, and synthesized to summarize the findings, then interpreted to decide what it means in the context of the partnership. Investing enough time and resources in analysis and interpretation is critical because this is when decisions are made and actions are taken. Once data are collected, they are returned to stakeholders for reflection and verification. Stakeholders should look beyond the raw data to ask what the results mean, what led to the findings, and whether they are significant. Each partner has different criteria for judging success and weighs them differently; using multirater analysis may help (11). Conclusions are justified and will be used with confidence when they are supported by data, consistent with the agreed-on values of the stakeholders, and linked to recommended actions.
The partnership should provide continuous feedback to stakeholders regarding interim findings to ensure that evaluation conclusions lead to appropriate decisions or actions. Stakeholders are more likely to use evaluation results if they feel they own the evaluation process and if they function cohesively as a team. During each planning and implementation step, stakeholders should discuss the best ways to communicate evaluation findings and use them. Frequent communication will increase the commitment to act on the results and refine the evaluation design, questions, methods, and interpretations. Having a positive experience with evaluation changes participants' attitudes; they begin to base decisions on judgments instead of assumptions (6).
Researchers agree that partnerships are difficult to evaluate. Measuring system-level changes is more difficult than evaluating program outcomes because multiple levels and community readiness must be considered (18). Other practical and methodologic issues include the following:
Partnerships may be the best vehicles available to address the chronic diseases of our time, so evaluation methods continuously need to be refined. The following are proposed solutions to partnership evaluation issues (2,3,20,23-27):
Public-private partnerships can be powerful agents for preventing and managing chronic disease. However, such partnerships become more complex as the public sector works more closely with private-sector partners. The following issues must be considered in developing evaluations that lead to improvements in partnerships and their programs and services:
How well these issues are addressed will help determine the effectiveness of partnership evaluations. Evaluations that meet stakeholder needs and focus on mutually acceptable and measurable systems-level outcomes will make partnership support and sustainability more likely in the end.
Level 1 — Partnership infrastructure, function, and processes
In the early stages, partnerships create mission statements, set up work groups, conduct assessments, and develop action plans. To sustain momentum, a partnership has to recruit and orient new members, train leaders, prepare members to assume leadership when turnover occurs, address and resolve conflict, engage in public relations, raise funds, and celebrate its accomplishments. Process evaluation will document what was done, how people were recruited and engaged in partnership efforts, and whether the partnership is functioning optimally or as originally intended. This type of evaluation essentially assesses the short-term outcomes of the partnership's development as an organization. Collecting and analyzing annual reports, attendance records, contribution records, meeting minutes, activity logs, and surveys that measure members' levels of satisfaction, commitment, and participation are methods and measures that may be used to document partnership structure and function.
Level 1 measures are short-term outcome measures used to document changes in the number and type of partners, the perceptions or skills of staff and members, or the mission or direction of the partnership.
Level 2 — Partnership programs and interventions
To achieve program outcomes, partnership activities (eg, training, advocacy, education programs) must be fully implemented and reach priority populations. For example, a partnership to prevent asthma might develop curricula, public awareness campaigns, or advocate changes in clean indoor air policies. Successful implementation depends on available resources, a time-phased action plan, and a supportive environment. This level of evaluation is conducted not only to prove that programs work but also to improve them. Adaptations of interventions that have been previously evaluated or accepted as promising practices increase the likelihood that interventions will result in systems-level change, and ultimately, desired health and social outcomes. Intermediate outcomes that are associated with changes in the priority population, the partnership, or the state's capacity to achieve long-term outcomes should be measured. For example, observations of clinic functioning, home visit logs, media reach reports, and event attendance forms provide short-term evidence of whether programs are implemented effectively and with fidelity. Patient record reviews help determine whether medical personnel adopt practice guidelines, and legislative records provide evidence of when and how proposed policies were introduced or passed.
Level 2 measures are short-term and intermediate outcomes that focus on activities and programs that the partnership accomplishes; the people, organizations, and groups it serves and affects; and the scope of the efforts it initiates. These measures include accomplishment of program outcomes such as changes in knowledge, attitudes, and behaviors.
Level 3 — Health and systems change outcomes
For partners and funders, the bottom line is whether the partnership achieves its ultimate, long-term goals. Systems change does not happen quickly, and many outcomes are difficult to measure using traditional quantitative methods. Participatory and qualitative evaluation methods increase understanding about how and why community-based initiatives work. Epidemiologic data will indicate whether health status indicators have changed, but key informants can identify the partnership programs that are institutionalized within their organizations.
Level 3 measures focus on ultimate partnership outcomes beyond programmatic activities — long-term outcomes such as changes in health status, quality of care, effectiveness of community institutions, and overall changes in the community's capacity and competence to deal with emerging problems.
|Questions (Evaluation Measure)||Type of Data Collection||Type of Design|
|Survey/ Scale||Structured Interview||Self-Report/ Log||Direct Obser-vation||Archival Records||Case Study||Pretest-Posttest Control Group||Time Series|
|Planning and implementation issues (descriptive and process measures)|
|Who participates? (demographic data)||X||X||X|
|Why do partners drop out? (partners' reasons for dropping out)||X||X||X|
|Are different activities generated? (type and frequency of activities)||X||X||X|
|Assessing attainment of objectives (outcome measures)|
|How many participate? (no. of partners)||X||X||X||X||X|
|How many hours are partners involved? (no. of hours by activity)||X||X||X||X||X|
|How many people are trained? (no. of partners per workshop/retreat)||X||X||X||X||X|
|Impact on participants|
|How do attitudes and behavior change by participating in program? (changes in attitude and behavior)||X||X||X||X||X||X||X|
|Does participation affect incidence, prevalence, or management of disease? (incidence/prevalence of asthma, diabetes, heart disease, and stroke)||X||X||X||X|
|Are participants satisfied with experience? (satisfaction ratings)||X||X|
|Impact on community|
|What resulted from program? (changes in programs, policies, and practices of partner organizations)||X||X||X||X||X||X|
|Do partnership benefits outweigh costs? (cost-benefit data)||X||X||X||X|
|Are community members satisfied with partnership and services they provide? (beneficiaries and community members/ satisfaction ratings)||X||X|
The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the US Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only and does not imply endorsement by any of the groups named above. URLs for nonfederal organizations are provided solely as a service to our users. URLs do not constitute an endorsement of any organization by CDC or the federal government, and none should be inferred. CDC is not responsible for the content of Web pages found at these URLs.
Suggested citation for this article: Butterfoss FD. Evaluating partnerships to prevent and manage chronic disease. Prev Chronic Dis 2009;6(2). http://www.cdc.gov/pcd/issues/2009/apr/08_0200.htm. Accessed [date].
Frances Dunn Butterfoss, Coalitions Work. 1109 Moore House Rd, Yorktown, VA 23690, Phone: 757-898-7454, Email: email@example.com.