This study found no evidence to support the hypotheses that improved access to supermarkets, or that less exposure to fast-food restaurants or convenience stores within walking distance improve diet quality or reduce BMI among Californian youth. There are isolated significant coefficients, but the number of significant coefficients is about what would be expected due to chance.
No single study resolves a major research question. Establishing reliable empirical relationships (even without establishing causality) requires the accumulation of evidence through many studies. Every study will have its own set of limitations and our analysis certainly has many. The response rate of CHIS (29.5% in 2005 and 21.1% in 2007) remains low, and our study sample has a large proportion of missing values (30.6% for children and 30.9% for adolescents). The data are not complete dietary recalls but single item questions without probing or guidance on serving size. Similarly, self-reported height and weight (and even more so for parent-reported height and weight) is likely to have substantial measurement errors. Relatively small sample sizes and noisy measures of dependent variables lower the statistical power to detect small but true associations.
Possibly even more of a limitation is the quality of the InfoUSA business listings, although this is a criticism that applies to all similar studies, including those reporting significant findings. Powell et al. (2011)21
advises caution when using commercial listings, reporting only fair agreement between commercial data and field observations for supermarkets, grocery stores, convenience stores and full-service restaurants, and poor agreement for fast-food restaurants. Field studies by Bader et al. (2010)22
and Liese et al. (2010)23
find reasonably good predictive values, although there are substantial discrepancies. The precision of coding on a very small scale (i.e., less than 100 meters) is unreliable. That is not surprising, however, as 100 meters is a distance smaller than a shopping center (or even a strip mall), and street-address geocoding will not match to the exact location within a shopping center. More generally, categorizing food outlets by type tends to be insufficient to reflect the heterogeneity of outlets and it is possible that more detailed measures, such as store inventories, ratings of food quality, and measuring shelf space, would be more predictive for health outcomes.24–26
Unfortunately, such data are very costly and time consuming to collect and may never exist on a national scale. Simple measures will remain important for surveillance and tracking on a large scale where feasibility is paramount. This is reflected in the recommendations by the CDC to use the number of full-service grocery stores and supermarkets as one community measures in efforts to prevent obesity.6
But unless such measures have predictive value for what are the ultimate desired outcomes (e.g., to improve diet or lower obesity rates), they are not useful to inform policy.
Our findings seem to be in conflict with a recent study that reports positive association between proximity of fast-food restaurants surrounding schools, and soda consumption and obesity among adolescents using the California Healthy Kids Survey.27
That study focuses on BMI and the BMI results clearly differ from ours. The inconsistency could result from statistical power as that study has much larger sample size (over half a million survey respondents). Even so, there a few issues remain unexplained. For instance, the regression coefficient for counts of soda consumption in that study is statistically nonsignificant just as in our analysis (in both cases, the point estimate is positive). Nor didis that study find a significant effect on fried potato consumption, the diet measure with a direct plausible causal relationship to nearby fast-food outlets. No relationships between other type of food outlets and consumption patterns are reported.. The study by Powell et al. (2007)10
is cited as support for the hypothesis that supermarkets have a protective effect, but that study reports no results on fast-food outlets, although that variable was analyzed as well.
While our null findings may be due to technical limitations (e.g., data quality, sample size), there are substantive reasons why the association between local food outlets and consumption or BMI may be much weaker than commonly believed. Today’s society is very mobile and the role of transportation has altered the definition of the shopping environment - both across areas and individuals.28
Access to transportation could be a more essential determinant of dietary behaviors than immediate availability, an issue highlighted in the USDA report on “food deserts.”28
In a Los Angeles study, Inagami et al. (2006) found that less than 20 percent of respondents shop in their census tract.29
Only 3% of households in the 2007 CHIS data report not having access to a car.
Research on how environmental factors affect obesity and related health behaviors is rapidly growing. One particular problem in new fields of investigation is that early results often do not hold up, or require some qualification that is only detectable through replication, a central principle of scientific method. The rate of false-positive results is particularly high for new and competitive research topics, which has led some methodologists to claim that “most published research results are false.”30–31
Research on environmental impacts on obesity is probably not dissimilar to other emerging research areas where there is an initial explosion of findings, but successful replication rates are low.32
Our study can only provide one data point, but reporting a full range of results is important as a selective focus on significant results (and especially those that appear to confirm - rather than contradict - a hypothesis) may lead policy astray. In contrast to basic research, publications on associations between obesity and the environment have an immediate and sizeable impact on policy. Accelerating this “shake down” period through systematic replication is thus potentially beneficial. At least equally important is the research design. Existing studies examining the environmental impact on body weight are mostly correlational. To infer causality from the mechanisms through which community retail food outlets might (or might not) influence youth’s diet and obesity, new studies should focus on improvement in research design by examining the critical intervening variables (such as shopping and purchasing practices), through experimentation, or through the rigorously-founded and carefully-implemented quasi-experimental methods.33