|Home | About | Journals | Submit | Contact Us | Français|
In 2003, the Monterey County Health Department, serving Salinas, California, was awarded one of 12 grants from the Steps to a HealthierUS Program to implement a 5-year, multiple-intervention community approach to reduce diabetes, asthma, and obesity. National adult and youth surveys to assess long-term outcomes are required by all Steps sites; however, site-specific surveys to assess intermediate outcomes are not required.
Salinas is a medically underserved community of primarily Mexican American residents with high obesity rates and other poor health outcomes. The health department's Steps program has partnered with traditional organizations such as schools, senior centers, clinics, and faith-based organizations as well as novel organizations such as employers of agricultural workers and owners of taquerias.
The health department and the Stanford Prevention Research Center developed new site-specific, community-focused partner surveys to assess intermediate outcomes to augment the nationally mandated surveys. These site-specific surveys will evaluate changes in organizational practices, policies, or both following the socioecological model, specifically the Spectrum of Prevention.
Our site-specific partner surveys helped to 1) identify promising new partners, select initial partners from neighborhoods with the greatest financial need, and identify potentially successful community approaches; and 2) provide data for evaluating intermediate outcomes matched to national long-term outcomes so that policy and organizational level changes could be assessed. These quantitative surveys also provide important context-specific qualitative data, identifying opportunities for strengthening community partnerships.
Developing site-specific partner surveys in multisite intervention studies can provide important data to guide local program efforts and assess progress toward intermediate outcomes matched to long-term outcomes from nationally mandated surveys.
Public health in recent decades has focused on a socioecological approach that emphasizes social and cultural modifications to change health behaviors (1,2). This approach addresses multiple societal influences on individual behavior with coordinated, multicomponent programs. These programs often combine more traditional health promotion methods with widespread health education and environmental and policy-level change to influence behavior (3). An important aspect of the socioecological approach includes developing novel partnerships to access or focus on smaller, at-risk populations often overlooked by existing approaches (4).
The socioecological approach helps health professionals develop improved models that encompass the complexity of multiple program components and interconnected intervention strategies that encourage sustainable systems change (5). The Spectrum of Prevention (the Spectrum) (6)The socioecological approach helps health professionals develop improved models that encompass the complexity of multiple program components and interconnected intervention strategies that encourage sustainable systems change (5). The Spectrum of Prevention (the Spectrum) (6) is one socioecological model used by many local and federal public health programs applying community-based approaches. The Spectrum is composed of six interrelated action levels: 1) strengthening individual knowledge and skills, 2) promoting community education, 3) educating providers, 4) fostering coalitions and networks, 5) changing organizational practices, and 6) influencing policy and legislation (6). This framework is especially useful for evaluating outcomes of community-based approaches that involve many interrelated and complex programs and partners at multiple levels of implementation.
In 2003, the Monterey County Health Department in California was one of 12 initial sites selected to receive funding from the Steps to a HealthierUS (Steps) cooperative agreement program. This funding is being used to implement a 5-year, multiple-intervention community approach to reduce the burden of diabetes, asthma, and obesity. Evaluation is a substantial component of the Steps program's national and community-level investment. All Steps program sites annually implement mandated Behavioral Risk Factor Surveillance System (BRFSS) surveys and Youth Risk Behavior Surveillance System (YRBSS) surveys. In Salinas, these two surveys have been modified to delete questions unrelated to the interventions and to add more detailed sociodemographic questions, so evaluators can determine whether outcomes differ among population subgroups. All Steps sites use a national logic model and guidance documents to assess community-level progress toward a set of long-term outcome measures (6). These surveys assess long-term outcomes; they do not focus on more intermediate outcomes related to site-specific intervention efforts.
At the start of the program, the health department used the Spectrum framework to plan the Steps to a Healthier Salinas program interventions. This plan included developing partnerships with traditional partners such as schools, senior centers, clinics, and faith-based organizations, as well as with novel partners such as employers of agricultural workers and owners of taquerias. The aims are to 1) build on and expand a set of past and current programs related to preventing obesity, diabetes, and asthma by increasing physical activity and good nutrition, and by decreasing smoking; and 2) emphasize disease management through more effective referral and treatment systems in secondary prevention programs. Plans are guided by evidence-based and cost-effective strategies and are tailored to the health literacy, linguistic, and cultural needs of Hispanics and other groups in the community.
To assess change at all levels of the Spectrum and to specifically assess progress toward intermediate outcomes linked to long-term outcomes in a local logic model, the Salinas program developed site-specific partner surveys to gauge organizational practices, policies, or both as implemented by community partners. This paper describes 1) the development and implementation of these surveys, 2) how the surveys were used to prioritize recruitment of community partners and to identify potentially successful community approaches, and 3) how these surveys will be used to evaluate intermediate outcomes matched to mandated national long-term outcomes.
Salinas, California, with 157,000 residents in 2005 (7), is a medically underserved community known for its international agribusiness industry. Hispanics make up 64% of the population, among whom 88% are of Mexican descent and 15% are agricultural workers (8). Random surveys of community and agricultural labor camp samples, conducted in 1990 and 2000, showed that most residents were young, had low education levels, spoke a language other than English at home, and had lived in the United States 10 years or more (9). The surveys also showed that from 1990 to 2000, many residents were at continuing or increased risk for chronic disease; for example, the prevalence of obesity increased by 47% (among community samples) and by 91% (among labor camp samples) (8). In 2000, residents also reported a low consumption of fruits and vegetables and a high consumption of fried foods.
The Spectrum was used as a unifying organizational framework for linking Steps interventions with traditional and novel community partners. Interventions are numerous and specific to partner organizations. The interventions include activities such as developing walking clubs; providing training on obesity, asthma, and diabetes in partnership with community members; and facilitating the adoption of exercise and nutrition policies. The Spectrum was also used to identify gaps in the proposed national evaluation framework, leading the evaluation team to develop site-specific partner surveys to capture intermediate and long-term outcomes related to organizational change that may result from specific intervention efforts in Salinas.
We in the health department collaborated with the Stanford Prevention Research Center to develop a local program logic model that incorporated resources and activities or strategies. The model linked proposed interventions with expected local intermediate and national long-term outcomes. We then used telephone directories and county agency lists to create a comprehensive list of potential partners that included all schools, senior centers, clinics, faith-based organizations, employers of agricultural workers, and owners of taquerias with a Salinas address. We developed five site-specific partner surveys to ensure that intermediate outcomes were assessed. Other tracking mechanisms not reported here were developed for assessing change in community clinics. The five surveys complemented the mandated national annual BRFSS and biennial YRBSS. The surveys were administered at baseline in year 1 (2004) before the start of the interventions and will be repeated in year 3 at the midpoint of the interventions and in year 5 at the conclusion of the interventions.
When developing the site-specific surveys, we reviewed the literature to identify existing survey tools that assess similar types of programs and interventions, and when they were available and appropriate, we modified standardized survey questions to better represent the Salinas community approaches and outcomes. Most questions on the surveys were new and focused on organizational and policy areas: 1) healthy food policies, 2) healthy exercise policies, 3) smoke-free policies, 4) healthy food marketing strategies, and 5) wellness programs. The surveys also contained questions about the size of the organizations and profiles of their members.
A senior epidemiologist oversaw all survey development. Trained health department survey workers conducted the in-person and telephone administration of surveys. For surveys that were mailed and not returned promptly, survey workers made telephone calls until a completed survey was returned, participation was declined, or repeated telephone calls were not returned. Surveys were assessed for face validity by the senior epidemiologist, several chronic disease intervention specialists, and a biostatistician, and were revised on the basis of their suggestions. Surveys took 10 to 20 minutes to complete and were administered in English, except for a few surveys that were completed in Spanish during telephone call-backs to faith-based organizations. Other details of the five site-specific surveys follow:
Although we hope that all partners will eventually adopt measures related to improving intermediate and longer-term outcomes, limited staff and resources required that we prioritize which organizations and businesses to partner with during the initial years of the interventions. To assist with selecting initial partners and ensure that partners were selected from neighborhoods with the greatest financial need, we divided the 25 Salinas census tracts into three groups, based on annual family income: low income (the bottom 25% of family income distribution; range of average incomes, $25,145–$34,112), moderate income (the middle 50%; range, $34,162–$53,500), and high income (the top 25%; range, $54,571–$83,123). The distribution of potential community partners among the three income categories was assessed with analysis of variance and chi-square analysis. A matrix of partner groups, intervention areas, and outcomes was developed to ensure that the surveys encompassed all intervention areas and outcomes.
Significant geographic stratification of sociodemographic characteristics existed within Salinas by level of census tract income (Table 1). Enumeration of organizations indicated a large base of potential partners with which to develop our interventions on policy and organization change. When we reviewed the geographic distribution of our potential community partners, we found that taquerias, faith-based organizations, and employers of agricultural workers (but not senior centers) were concentrated in the lower-income census tracts where individuals in greatest need reside. School districts were not included in this analysis because they overlapped census tracts.
Response rates to the five site-specific partner surveys varied from 100% for school districts, senior centers, and taquerias to 70% for agricultural worker employers and 69% for faith-based organizations. For agricultural worker employers, five companies declined to participate because of seasonal time constraints, and no response was obtained from six companies despite repeated callbacks. For faith-based organizations, 23 refused to participate and 9 were never reached at the listed phone number, despite repeated attempts.
Table 2 presents a summary of the content of the five site-specific surveys and how the surveys will track local intermediate outcomes matched to national long-term outcomes. Survey questions match the five intervention areas and corresponding local intermediate outcomes that can be linked to national long-term outcomes. This linkage will enable the local evaluation team to produce analytic reports for program use. For example, outcome data can be presented by type of organization and intervention area (e.g., changes in healthy food policies in schools and faith-based organizations) and posted on the program Web site for community stakeholders.
The site-specific surveys, administered before the start of the interventions, also contributed to targeting program planning. The administration of the surveys served as the first contact between the health department staff and potential partners, when survey workers introduced the program and asked for permission to conduct the survey. Information about the potential partner helped health department staff decide which organizations should be contacted next by the intervention staff. For example, for employers of agricultural workers and for faith-based organizations, data on the size of the organizations as well as their geographic distribution were useful in allocating staff resources for developing partnerships and interventions. In addition, the baseline surveys helped identify organization-specific concerns about proposed community approaches. For example, we found that taqueria owners initially mistrusted the health department because taquerias' past involvement with the health department mainly related to obtaining operating permits. This mistrust was partially overcome when survey workers developed a personal rapport with the owners and provided them with health information about diseases common to families and customers, in particular diabetes and obesity. As a result of these conversations, community approaches for taquerias were modified to include developing and promoting healthy food choices for diabetic patrons of the taquerias. We also identified organization-specific concerns from agricultural worker employers related to the timing of our planned interventions. We learned that because of the seasonality of their work, agricultural employers preferred to begin their participation during the off-season (November through March). We also learned that faith-based organizations were often reluctant to complete surveys because of concerns about the involvement of the government, as represented by the health department, in their operations. As a result, program staff modified their plans to focus on developing relationships with faith-based organizations before introducing intervention programs.
Evaluation of the Steps to a Healthier Salinas Program's success toward creating sustainable community and environmental change within a 5-year period is a seemingly difficult goal. This is true for evaluating long-term outcomes only (e.g., changes in overweight and obesity) that are the focus of the national surveys, and for evaluating the intermediate outcomes that are the focus of our site-specific partner surveys (e.g., changes in food policies adopted by faith-based organizations).
Developing site-specific surveys has been important for several reasons. First, the surveys of all potential community partners helped us to identify promising new partners, select initial partners who were from neighborhoods with the greatest financial need, and learn about potentially successful community approaches. Second, the surveys will provide data for gauging progress toward intermediate, local outcomes that are more likely than long-term outcomes to occur in the short time frame of the intervention. Data on intermediate outcomes, linked to interventions, may also help in obtaining continued funding after the completion of the grant. Third, the surveys served to assess how the health department and organizations can work together to identify areas of need and potential challenges at the organization level. The benefits of these site-specific surveys relate to the importance of the social environment in defining issues to be addressed and to how a community defines the boundaries of possible action and change when implementing comprehensive community-based approaches (10). Thus, although the surveys focused on outcomes, they also helped guide the intervention staff in areas of qualitative project management. This process included developing new community partnerships and providing insight into the merit of various strategies during program development and potentially reducing staff time needed to establish program direction. The surveys also provided opportunities for program staff to identify areas for bridging social capital within and between traditional and novel partners. Identification of community forces, such as areas of resistance that may affect desired outcomes, has also been proposed as another important next step in understanding change in the context of multi-intervention programs (11).
We developed our site-specific partner surveys to augment the mandated national surveys and to help assess changing organizational practices or policies using the Spectrum framework. Questions on these surveys match intervention areas and corresponding local intermediate outcomes that are matched to national long-term outcomes. Such careful planning provides important quantitative evaluation data for the Spectrum levels of organizational and policy change. Interestingly, the site-specific surveys also provide valuable qualitative data that have informed areas of need for community education and have identified opportunities for developing or strengthening community partnerships, two other important levels of the Spectrum.
The community partner surveys for the Steps to a Healthier Salinas serve as a case study for developing a methodology to evaluate a multiple-intervention program in a community. These surveys will gauge progress toward intermediate outcomes that are matched to long-term outcomes. The documentation of site-specific intermediate outcomes are valuable to our community partners, health professionals, and community members as they work together to enhance the health of the Salinas population.
Funding for the Steps to a Healthier Salinas Program is provided by the U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, as part of its Steps to a HealthierUS program. John Snider and Marilyn Winkleby are coprincipal investigators. Special thanks to colleagues involved with the Salinas Steps program, especially Christina Pitts for survey implementation in 2004.
The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, Centers for Disease Control and Prevention, or the authors' affiliated institutions. Use of trade names is for identification only and does not imply endorsement by any of the groups named above.
Suggested citation for this article: Hanni KD, Mendoza E, Snider J, Winkleby MA. A methodology for evaluating organizational change in community-based chronic disease interventions. Prev Chronic Dis 2007;4(4). http://www.cdc.gov/pcd/issues/2007/oct/07_0015.htm. Accessed [date].
Marilyn A Winkleby, Professor of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine. 211 Quarry Rd, Hoover Pavilion, Room N 229, Stanford, CA 94305-5705, Phone: 650-723-7055. Fax: 650-725-6247, Email: winkleby/at/stanford.edu.
Krista D Hanni, Monterey County Health Department, Salinas, California.
Elsa Mendoza, Monterey County Health Department, Salinas, California.
John Snider, Monterey County Health Department, Salinas, California.