Center TRT’s approach to evaluating policy interventions is based on the use of emergent logic models, in other words, logic models that evolve as a project progresses (11
). Logic models are a systematic and visual way to present the relationships among an initiative’s inputs, activities, outputs, and outcomes (12
). The traditional approach to logic modeling involves the creation of a single model depicting a linear causal pathway from inputs to outcomes. This approach is difficult to apply to policy interventions because they involve a continual and often cyclical interplay among inputs, activities, and outputs and because the path to successful outcomes is neither linear nor constant (11
). Practitioners can respond to this challenge by iteratively revising their logic model to fit progressive stages in the policy-making process to identify the emergence of new inputs, activities, and outputs over time (11
The cornerstone of Center TRT’s approach to evaluation is its evaluation framework (). The framework is intended to provide a comprehensive overview of the entire process with the expectation that practitioners will select and adapt those components that are relevant to the stage of their initiatives and use them to develop their own progressive logic models (11
). Center TRT’s framework draws on a range of theories and frameworks related to policy making and evaluation in public health (). Foundational to the framework is CDC’s guidance on the 6 steps essential to all evaluations (): continuously engaging stakeholders and intended users, describing the program, focusing the evaluation design, gathering credible evidence, justifying conclusions, and disseminating and using findings (15
). The framework is organized according to the 4 standard sections of a logic model: inputs, activities, outputs, and outcomes (12
). We describe these 4 sections in more detail and provide an illustration of how practitioners might apply them to evaluate a state-level farm-to-school policy to create infrastructure and allocate funding to coordinate the purchase and distribution of locally grown foods to schools.
Figure 1 Center TRT’s evaluation framework incorporates elements from multiple policy-making and evaluation frameworks (9,15-19). The framework is intended to support practitioners as they develop logic models to describe and evaluate policy making initiatives. (more ...)
Center TRT’s evaluation framework includes the 4 components of a standard logic model: inputs, activities, outputs, and outcomes. These components are embedded within the Centers for Disease Control and Prevention’s evaluation framework, which includes continuously engaging stakeholders and intended users, describing the program, focusing the evaluation design, gathering credible evidence, justifying conclusions, and disseminating and using findings. The activities section of the framework includes each of the stages of the policy-making process: formulation, enactment, implementation, and maintenance/modification. The outputs section of the framework addresses the reach, adoption, implementation, and maintenance components of the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework. The outcomes section of the framework includes short-term changes to the environment, intermediate changes to social norms and health behaviors, and long-term changes in population-level health outcomes that are equitably distributed and cost-effective.
Overview of Frameworks and Theories Used in Center TRT Evaluation Framework
are the resources and contextual factors that support and influence each step in the policy-making process. The selection of inputs to include in the framework was informed by Kingdon’s theory that 3 elements need to align to create a “window of opportunity” for new policy: problems, solutions to those problems, and politics (16
). Using farm-to-school policy as an example, inputs related to the problem include assessment data on the prevalence and distribution of obesity and unhealthy eating behaviors among the state’s school-aged children. Inputs related to solutions include content experts, evidence-based approaches, existing state policies, and model policies such as exemplar farm-to-school policies enacted in other states. Politics are critical to whether a problem rises to the top of the political agenda and consist of the attitudes and activities of policy makers, stakeholder groups, advocates/champions, and the general public. Political inputs may include the extensiveness of farm-to-school activities at local levels and the attitudes of politicians, school staff, teachers, farmers, and health advocates.
After a policy is enacted, additional inputs include the administrative structures, staffing, and material resources necessary to implement the policy and the systems that will be employed for policy monitoring and enforcement. The systems in place to purchase foods and distribute them to schools or to monitor school districts’ implementation of existing policies are examples of these inputs.
are the actions or events that engage and transform inputs to produce outputs and outcomes. Engaging stakeholders, raising awareness, and advocating for change are essential activities throughout the policy-making process (9
). Stakeholders are defined broadly to include special interest groups, the general public, policy makers, representatives from the sectors or settings that will implement the policy, and others with an investment in the policy and its outcomes. Additional activities in the framework are organized according to commonly recognized steps in the policy-making process: formulation, enactment, implementation, and maintenance/modification (9
. Activities involved in formulating policy include reviewing evidence on the problem and potential solutions, gaining stakeholder agreement on priority problems and preferred approaches, and drafting policies in the form of laws, rules, and funding priorities (20
Enactment. To enact a policy, activities focus on looking for windows of opportunity, identifying the policy makers who will sponsor or promote the policy, and formally enacting the drafted policy in the form of a law, regulation, or budget. Enacting farm-to-school policy, for example, might involve passing new legislation and allocating funding.
Implementation. Implementing the policy may include activity at multiple levels of the executive branch of government or by involved settings and sectors. For example, in implementing new state-level farm-to-school policy, the Department of Education may reallocate resources and draft new rules governing how the policy will be implemented across school districts. School districts and schools would then develop and implement plans and allocate the resources necessary to implement the new policy.
Maintenance/modification. At the final stage in the process, activities are directed toward maintaining policy implementation and ensuring sustained effect through monitoring, enforcement, and further policy modification. The framework includes a feedback loop from maintenance to formulation to indicate that policy making is cyclical; modifications strengthen policies over time.
Outputs are the direct, tangible results of activities. Outputs include media coverage and other communication (eg, policy briefs). They also may include evidence of increased stakeholders’ awareness of the problem, engagement in the policy-making process, and political will to take action.
The central outputs of the first 2 steps of policy making (formulation and enactment) are the actual policies proposed and enacted. Practitioners may assess the extent to which policies employ evidence-based approaches, in other words, approaches that have been found in prior research to be effective at improving health environments (2
). Evaluation may also assess the extent to which policies followed model policy guidance from organizations with expertise in obesity prevention policy (eg, National Policy and Legal Analysis Network to Prevent Childhood Obesity). For example, does the farm-to-school policy include language that is strong enough to ensure the resources necessary to support implementation (5
Outputs of the second 2 steps of policy making (implementation and maintenance/modification) were drawn from elements of the RE-AIM framework. The RE-AIM framework is widely used to assess an intervention’s potential impact and includes the following criteria: whether the intervention reaches the priority population, is effective in achieving intended outcomes, is adopted by providers and settings, and is implemented with fidelity and maintained over time (17
). The outputs of the implementation and maintenance steps of the framework address 4 of the 5 RE-AIM criteria: reach, adoption, implementation, and maintenance. The fifth criterion, effectiveness, is addressed in outcomes.
Reach is the absolute number, proportion, and representativeness of people exposed to the policy. For farm-to-school policy, people reached may include children, family, community members, school employees, and farmers. The policy could be evaluated on the basis of both the absolute number and the proportion of the state’s school-aged children affected. Reach also accounts for the extent to which affected children represent the state’s overall population of school-aged children, particularly those at disproportionate risk for obesity.
Adoption, in the case of voluntary policies, is the absolute number, proportion, and representativeness of settings/sectors that decide to participate or to implement the policy. For example, farm-to-school policies require that local farmers, food distribution systems, and schools agree to participate (ie, adopt). Adoption is less applicable to mandatory policies. Adoption can also apply to multiple levels of government, such as a local governing body’s decision to enact policy to operationalize state-level policy. For example, a state may enact enabling legislation in support of complete streets. County-level governments may then “adopt” the legislation by enacting local regulations.
Implementation is the extent to which a policy is applied as intended among levels, settings, and sectors. For example, implementing farm-to-school policy can occur at the state, school district, and local school levels. Additional outputs include acceptability and affordability. Evaluation of farm-to-school implementation might address whether locally grown foods were delivered to schools and served in cafeterias as intended; cafeteria staff, parent, and student response (ie, acceptability); and the effects on the cost of school lunches (ie, affordability).
Maintenance/modification involves assessing continuation of funding, partnerships, implementation, and reach over time. For farm-to-school policies, outputs include whether schools continue to order and serve the same or greater amounts of locally grown foods as when the program started. Policies may also need to be modified. For example, municipalities may have restrictions on foods served in school cafeterias; policies may need to be modified to allow schools to serve foods grown in school gardens.
are the desired and unanticipated results of a policy. Short-term outcomes are changes to the environment that promote healthier foods and increase physical activity. Changes can occur in the following types of environments (18
- physical (eg, proximity to healthier food and spaces for physical activity)
- economic (eg, changes to prices, taxes)
- social (eg, changes to social networks)
- communication (eg, advertisements, point-of-decision prompts).
An increase in the amount of fruits and vegetables available in school cafeterias is an example of a short-term outcome of farm-to-school policy.
Intermediate outcomes refer to behavior changes that occur as a result of a policy’s effects on environments and include changes in dietary intake, physical activity, screen time, and breastfeeding. Intermediate outcomes also refer to changes in social norms related to obesity and the behaviors that prevent it (18
). Thus, intermediate outcomes of a farm-to-school policy may include changes to school children’s attitudes toward and consumption of fruits and vegetables.
Practitioners cannot anticipate all possible consequences and, therefore, should assess both positive and negative unintended outcomes (22
). For example, incorporating local foods into unhealthy recipes is a potential unintended consequence of farm-to-school policies.
The long-term goal of obesity prevention policy is to be effective, equitable, and cost-effective at the population level (19
). The policy should have the potential to contribute effectively to improvements that are distributed equitably across subgroups to reduce disparities in obesity and obesity-related health outcomes (2
). Policies also should be cost-effective; in other words, they should use resources in ways that contribute to improvements that are equal to or greater than alternative policies or programs.
Although the framework outlines short-term, intermediate, and long-term outcomes, many initiatives will capture change only in short-term outcomes (ie, changes to the environment). CDC guidance on community-level obesity prevention identifies a range of measures that can be used to assess environmental change (23
). Changing behaviors and health outcomes at the population level will require multiple initiatives and an extended period and will, therefore, often exceed the scope of any single initiative (2