PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jurbhealthspringer.comThis journalToc AlertsSubmit OnlineOpen ChoiceThis journal
 
J Urban Health. 2009 September; 86(5): 668–671.
Published online 2009 August 7. doi:  10.1007/s11524-009-9395-y
PMCID: PMC2729875

Evaluating Community-based Prevention Programs

On June 2, 2009, Trust for America’s Health, The New York Academy of Medicine, the Robert Wood Johnson Foundation, and The California Endowment convened policymakers, researchers, and evaluators to discuss strategies for assessing the impact of the $650 million investment in community-based prevention in the American Recovery and Reinvestment Act of 2009 (ARRA). The ARRA investment was discussed as an opportunity to demonstrate the value of community-based prevention. Meeting participants generated six recommendations to address the areas of concern policymakers typically have regarding investment in community-based prevention programs.

Community-based prevention has the support of academics and practitioners in public health and preventive medicine. As a result of philanthropic investments and the growing success of local policies and programs, there has been a paradigm shift away from focusing primarily on individual behavior change to promote health, toward environmental and policy change for health improvement. However, outside the fields of public health and preventive medicine, the effectiveness of this strategy is less well known. There was consensus at the meeting that ideally, the ARRA program and evaluation of its supported interventions will be designed to help address the concerns and questions of policymakers in particular.

The following were identified as some of the areas of concern policymakers typically have regarding investment in community-based prevention programs:

  • Outcomes

Identified Concern While randomized controlled trials of clinical treatments can often show direct influence on health outcomes within a relatively short period, community interventions typically demonstrate influence on intermediate variables, and may not be able to report direct links to health outcomes within a short time frame. With only intermediate markers of success available, how can policymakers be confident they are investing in health improvement?

Further Discussion Project time frames, insufficient data, and lack of infrastructure and resources to carry out adequate evaluation are some of the reasons community-based projects are not always able to demonstrate improvements in health status. It was noted, however, that even within a limited time frame such as 2 years, projects with sufficient resources and data-collection infrastructure can capture biological or health outcomes such as physical fitness, reduced healthcare visits, and behavior changes. It was also discussed that health-related outcomes are not always essential to arguments for investment in community-based prevention. Reduced absenteeism, improved test scores, and economic benefits are examples of other outcomes that can be persuasive to policymakers. The Institute of Medicine’s Evaluation Framework for Obesity Prevention Policies and Interventions was reviewed. It illustrates that community interventions typically yield outcomes in five categories: environmental; cognitive and social; behavioral; structural/institutional/systems; and health. Projects should be framed so that results are expected in the appropriate outcome category and are not unfairly deemed unsuccessful. It was also discussed that macro-level, aggregate comparisons may be useful for explaining the value of community interventions. For example, comparison of communities who participated in the CDC REACH program versus those that did not shows that the REACH interventions are closing gaps in health disparities.

  • Replicability

Identified Concern Community-based interventions often have multiple components and are tailored to address the needs and assets of individual communities. They are also typically undertaken by communities particularly motivated to make changes. The often idiosyncratic nature of the interventions does not serve well the strong preference among policymakers to isolate one single intervention that can be replicated in a standardized manner many times.

Further Discussion The CDC’s Guide to Community Preventive Services was discussed as a resource for identifying effective, potentially replicable program and policy interventions. It is a work-in-progress, however, and due to inadequate funding, has subject area gaps. It also lacks information about barriers to implementation. Communities need technical assistance to adapt evidence-based interventions for local conditions, and program and evaluation schemes need to provide leeway for local adaptations. It was discussed that cancer treatment may provide a useful analogy: it is well-accepted that a mix of interventions (radiation, surgery, chemotherapy, etc.) is needed to treat cancer, and that this mix varies by patient. Similarly, there are core practices that are vital for successful community interventions, but they should be applied according to local needs. School policies were discussed as a category of interventions that lend themselves well to replication. However, political will is important to their implementation. Policymakers need to understand that community “readiness for change” is an integral part of the potential success of an intervention, particularly an intervention that seeks to transform the physical or policy environment. Accordingly, efforts to cultivate leadership and coalitions should be funded and evaluated.

  • Cost

Identified Concern Studies of community-based efforts rarely capture cost information, and when cost information is available, it can be hard to separate program costs from research costs. This information is needed to understand adequate levels of funding and potential returns on investment.

Further Discussion It was agreed that intervention costs are not captured in most studies, and that when they are reported, the information is not usually sufficient to enable a calculation of return on investment. There was discussion of whether the economic benefits of prevention are adequately appreciated. For example, improved productivity and tax revenues generated by a healthy workforce are not generally incorporated into evaluations of the benefit of investment in health. Also, it is often argued that prevention is not an effective investment because in prolonging life, it increases opportunities for additional health care expenditures across a lifetime. It was noted that this is also true of but not raised against investment in clinical treatment. It was suggested that cost questions be reframed to ask “How can we get the most benefit per dollar invested in health?” The issue of where prevention savings are realized was discussed. Federal prevention programs are funded through discretionary funding streams. Any resulting savings in health care expenditures are realized in mandatory spending accounts (Medicare/Medicaid). Current program-scoring policies do not allow the Congressional Budget Office to acknowledge savings that occur in an account different from the account where the expenditure occurs. This is a policy decision that could be changed. It is also possible that if Medicare or Medicaid pay for community-based prevention, then savings could be accrued within the mandatory spending account and the prevention program could be more adequately scored.It was noted that the ARRA program designers have certain statutory constraints and political questions to consider. First, the program needs to demonstrate results within 2 years. It is understood that chronic disease rates can only begin to be affected within this timeframe. Therefore, the program must identify interim markers that can show progress toward eventually reducing chronic disease and its associated costs. Secondly, it is important that any measures of success be easily understood by a lay public. A baseline is needed, as well as a means to make comparisons against communities that did not receive funding. Lastly, it is not clear which populations or communities should be funded: those in most need, or those that have stronger community infrastructure and greater readiness, and so can quickly leverage the new funding.

To address the concerns regarding community-based interventions, and some of the questions raised specifically around the ARRA program, meeting participants generated the following recommendations for the ARRA program design and evaluation:

  1. Identify evidence-based milestones communities should achieve. The program should identify evidence-based measures of progress toward the desired health outcomes. For example, if the program is focusing on obesity, an evidence-based pathway should be described that progresses from coalition-building to policy change to increased physical activity (and/or food consumption changes) to reductions in BMI. Grantees would have to demonstrate where they are starting and how they move along that pathway.
  2. Support data collection. The program should facilitate identification or collection of baseline data for each community as well as ongoing measures of progress. Establishing the baseline may require enhancing existing data sets (e.g., the Behavioral Risk Factor Surveillance System) or creating new measures, or both. Where possible, nationally-comparable data sets should be used to allow comparison across sites that are receiving funds or to show comparisons with those not participating in the program.
  3. Treat the program as a study of community-based intervention. To help answer questions of replicability, it would be useful to stratify funded communities according to certain characteristics, and to document processes in ways that permit identification of common core versus locally variable aspects. Generating case studies concerning certain subpopulations may also be helpful.
  4. Support communities that are ready to implement change. Coalition building and community development are critical to effective change, but they take time. Public and private investment has yielded many communities across the country that are ready to act, including some communities with high levels of need. Such targeted funding will support a demonstration of proof of concept, while allowing other communities to move toward readiness. This approach is essential to the long-term sustainability of funding for community-based prevention.
  5. Define a limited number of evidence-based interventions from which communities may choose. The interventions may need to be adapted to local conditions, but having multiple communities select from a list of preferred, evidence-based interventions will help answer questions of replicability and scalability. They will also be more likely to produce positive outcomes within the program’s timeframe.
  6. Require the collection of cost data associated with the intervention. An understanding of the costs of programs is one of the weakest links in the current community-based prevention literature. Good cost accounting that separates research costs from program costs will enable more accurate cost-benefit analyses.

Articles from Journal of Urban Health : Bulletin of the New York Academy of Medicine are provided here courtesy of New York Academy of Medicine