The EA checklist had face validity as an instrument to assess the environment in and around the worksite. It is hard to do environmental assessment in these intervention studies, but it is important, since the PACE intervention targets the environment as part of its strategies. As an instrument, the EA checklist discriminated between detected changes in environmental information and lack of differences in the physical environment at follow-up. From a program evaluation perspective, in spite of some limitations, the EA checklist was able to detect changes in the information environment, as a result of the PACE intervention, for both physical activity and healthy eating behaviors.
Our work adds to the literature on developing environmental assessments. As with any measurement tool, the instrument used in our study must have face validity, reliability, inter-rater reliability, and external validity. Our study was not able to assess all of these, but pointed up difficulties with inter-rater reliability that called for adjustment in our analyses. We discovered that small to medium sized worksites have very variable facilities, and often share buildings with other companies. This posed challenges in making the counts of bike provisions etc. comparable with other worksites.
Several groups have adopted and adapted the CHEW 9
for use in their worksite studies, and have carefully described the construction of scales and subscales within the different conceptual domains, using different methods to effect this. For example DeJoy et al 13
developed a prototype for use in very large worksites. They expanded the domains beyond those in the CHEW, and allocated points to different components of those domains. Some parts of the resulting Environmental Assessment Tool (EAT) were completed by company staff, and led to multiple measures of the same worksite (location). Inter-rater reliability was high (above 80%)13
. The same group used the EAT to inform the intervention developed for selected large companies and demonstrated considerable variability across worksites in access to physical activity opportunities and access to healthy food choices.14
Few studies have evaluated changes in the worksite environment in response to an intervention, before the results of the current studies that form the consortium of worksite obesity prevention trials become available. A report from the World Health Organization on monitoring and evaluating worksite heath promotion programs 15
underscores the need for such evaluations. Regrettably, work developing such an environmental checklist has proceeded differently in different studies, and the lack of consensus on the development methodology is a current limitation of this field.
Our results using the EA checklist show that our PACE intervention program significantly improved the information environment in the worksite at two year follow-up. The fact that it did not have a measurable effect on physical fitness facilities or bike provisions or stairway enhancements is disappointing but perhaps not surprising. The intervention did not emphasize making physical changes but placed emphasis on working with the available resources of the worksite and optimizing their use. Changes in the physical environment require resources on a scale not provided by this intervention, and not easily incorporated into the company budget of small to medium sized businesses. The cost of physical improvements and the labor involved are orders of magnitude greater than the provision of signage, for example. For worksites that share buildings with other companies, the shared building further limits a worksite’s ability to significantly alter their physical environment, including altering the appearance of stairwells or building outdoor walking paths.
Nonetheless, it is possible, that given more elapsed time since the introduction of the PACE intervention, we will see an effect of the intervention on the EA checklist variables. As employees become more active, make more use of bicycles and use the stairs more, we could expect that they will demand more improvements in provision of bike racks, and easy access to stairways. The Importance of changes in the physical environment in relation to both physical activity and healthy eating may increase relative to individual behavior changes, as time elapses since initiation of behavior change. Sustaining long term changes in behavior requires support from the physical as well as the social environment, leading to changes in social norms.
Our study has limitations. We did not include an independent rating of the worksite environments in the same worksite in the same period, by either the same rater or different raters. We are therefore unable to estimate the reproducibility of the scales, or precisely to estimate the inter-rater reliability of the scales. No attempt has been made to evaluate the interventions according to dose of intervention delivered, or strength of the EAB, since this would violate the intent to treat principle of analysis of a randomized controlled trial. Such evaluations would be better done as a secondary analysis of individual level behavior or obesity risk variables, in a linear mixed model where there is greater statistical power.
The strengths of the study include its group randomized design, the relatively large number of independent worksites participating, the length of follow-up (two years), the partnerships between study staff and EAB members, the multilevel, phased intervention, and the multiple outcome measures used, of which the EA checklist is only one.
In summary, we discovered that the checklist is not easy to complete, and may require significant training and supervision of raters to achieve standardization of ratings. A consensus should be actively developed regarding a common methodology for environmental assessment that is appropriate for a wide rang of company sizes and environmental circumstances.