Given epidemiologists’ general concern about measurement accuracy, surprisingly little information on sources of measurement error exists in this field (8, 9). To our knowledge, only 1 study has examined the validity of a commercial database and an Internet listing of food outlets, by conducting an independent ground-truthing effort in 12 Montreal, Canada, census tracts (10). Indirect evidence comes from a study on the agreement of a local government listing of food outlets with field observation in Glasgow, United Kingdom (11) and the agreement between InfoUSA and field observation in Chicago, Illinois (12). These studies were conducted in large urban areas and suggest that the secondary data sources were quite complete. For instance, Paquet et al. (10) reported sensitivities and positive predictive values (PPVs) of 84% and 90%, respectively, for food stores in Montreal. The level of agreement between listing and actuality was reported at 88% in Glasgow (11) and above 80% in Chicago (12). However, no published validity studies have been conducted in rural environments or larger urban areas in the United States. Moreover, previous experience by our group in 1 rural county suggests that some secondary data sources may not be complete (13).

Another concern with secondary data is its geospatial or positional accuracy (14), a function of the completeness of the address information and the road network database which feed into the geocoding process. The accuracy of several commercial geocoding vendors has been described in detail (15). Many secondary data sources used to characterize the food environment provide geocoded addresses, but, to our knowledge, no study has specifically focused on geospatial error related to geocoded food outlets.

The purpose of this study was to assess the validity of 3 readily available, secondary data sources on food outlets within an 8-county region of South Carolina by conducting a comprehensive field census, including ground-truthing and ascertainment of geographic coordinates. We assessed validity by considering both count accuracy and geospatial accuracy and the influence of level of urbanization.

Table 1 shows the results of the validation effort relative to the secondary data sources. The DHEC database had the largest number of listed outlets (n = 1,694), including 417 stores and 1,277 restaurants. InfoUSA listed 1,657 outlets (672 stores and 985 restaurants), and D&B listed 1,573 (751 and 822, respectively). Of the outlets listed by the DHEC, only about 11% could not be confirmed by the field census, because they were either not found (6.6%) or closed (4.2%). The InfoUSA database was similar: Approximately 14% of listed outlets were not confirmed, largely because they were not found (9%) or closed (3.5 %). In contrast, D&B had the highest proportion of outlets that could not be confirmed (22%), with 12.3% not being found, 2.3% not being found because of a post office box address, and 7.6% being closed.

We found 183 food outlets during the validation effort that were not listed in any of the 3 data sources. The number of outlets newly found ranged from 696 for the DHEC to 774 for InfoUSA to 985 for D&B. The majority of outlets discovered relative to the DHEC were stores, which is not surprising since many stores do not fall under the licensing regulations enforced by the DHEC. However, 183 restaurants were discovered that were not listed by the DHEC. The InfoUSA database was missing 349 stores and 425 restaurants. The D&B database was missing 330 stores and a very large number of restaurants (n = 655). Combining D&B and InfoUSA data sources would have yielded 2,170 unique, listed outlets (944 stores and 1,226 restaurants), of which 1,727 were found and open during the field survey (data not shown). Relative to this combined listing, 481 new outlets were discovered during fieldwork.

Validity statistics are shown in Table 2. For all outlets combined, the sensitivity (i.e., the ability to capture food outlets that truly existed in the area) was moderate to fair (68% for the DHEC, 65% for InfoUSA, and 55% for D&B). Both D&B and InfoUSA exhibited moderate sensitivities for food stores (63% for D&B and 61% for InfoUSA) and ranked significantly better than the DHEC (43%). This implies notable undercounting of existing stores for both commercial databases—approximately 37% for D&B and 39% for InfoUSA. Combining the 2 commercial databases would have resulted in a significant improvement in sensitivity for food store identification (81%). Likewise, for supermarkets and grocery stores, all 3 databases had similar levels of sensitivity, ranging from 71% to 76%. The combination of InfoUSA and D&B data would have resulted in a marked improvement in supermarket sensitivity (90%). Combination of DHEC and InfoUSA data or DHEC and D&B data or data from all 3 sources would have resulted in very good to excellent sensitivity for the identification of supermarkets: 86%, 91%, and 97%, respectively (data not shown). No data are shown for DHEC with respect to dollar/variety stores and drug stores/pharmacies because of the DHEC's focus on licensing prepared food outlets. Had we excluded dollar/variety stores and drug stores/pharmacies entirely from the DHEC analysis, the sensitivity of the overall food store category would have improved to 54% (95% confidence interval: 51, 58). Of the other types of food stores, D&B and InfoUSA demonstrated very good sensitivities at 80% or above for drug stores and pharmacies.

With respect to restaurants (lower portion of Table 2), the DHEC database had very good sensitivity of 86% (i.e., an undercount (discovery rate) of only 14%). InfoUSA (67%) and D&B (50%) performed significantly worse. With respect to ranking of the 3 databases across type of restaurants, the findings were consistent. Combining the 2 commercial databases would have resulted in a significant increase in sensitivity compared with using either one alone, but sensitivity would not have reached the level of the DHEC database. However, the combination of DHEC data with InfoUSA or D&B data or the combination of all 3 databases would have resulted in excellent sensitivity values for restaurants (91%, 92%, and 94%, respectively; data not shown).

Table 2 also shows PPVs, which can also be interpreted as verification rates—that is, the likelihood that a listed food outlet actually existed and was open. PPVs ranged from good to very good: 78% for D&B, 86% for InfoUSA, and 89% for DHEC, for all food outlets combined. For stores, the PPV was highest (92%) for the DHEC database, significantly better than for any other database (D&B, 76%; InfoUSA, 82%). For restaurants, DHEC and InfoUSA data performed equally well with respect to PPV (88% and 90%, respectively), with D&B performing significantly worse (79%). This ranking between the databases remained consistent for all 3 restaurant types.

We subsequently evaluated potential differences in the validity of the 3 secondary data sources across levels of urbanization (Table 3). For stores, there were no marked differences between levels of urbanization in any of the 3 databases or the combined D&B and InfoUSA databases. A similar picture emerged for restaurants, the exception being significantly higher sensitivity in urban areas in the D&B data. We additionally evaluated the potential influence of tract racial composition or poverty on the validity estimates but found no evidence for any systematic differences (data not shown).

Geospatial accuracy statistics are shown in Table 4 and are limited to located and open outlets because of the need to have both geocodes from the database and GPS coordinates from the field census. The geospatial accuracy varied widely, with a median Euclidian difference of 76 m (n = 1,507) for DHEC, 92 m for D&B (n = 1,213), and 92 m for InfoUSA (n = 1,434). The percentage of outlets for which the geocoded position was less than 100 m from the GPS location ranged from 53% for both D&B and InfoUSA to 56% for DHEC. The correct allocation of outlets to census tracts was high (83%, 85%, and 84% for DHEC, D&B, and InfoUSA, respectively). No notable differences in the median distances were observed by type of food outlet; hence, all outlet types were combined. As expected, the Euclidian distance differences were lowest for the urban areas for all 3 data sources, intermediate for the suburban and large-town areas, and highest for small-town and rural areas (P < 0.0001 for all contrasts).

Finally, to combine our evaluation of count accuracy with the geospatial accuracy, we calculated the proportion of open outlets that had been both listed in the respective database and geocoded to a position less than 100 m from the actual GPS-recorded location (Table 5). Overall, between 29% (D&B) and 39% (DHEC) of open food outlets were listed with geocodes that would place them less than 100 m from their actual location. DHEC performed best for restaurants (49%) and worst for stores (24%). D&B and InfoUSA did not differ significantly for stores (31% vs. 29%), but InfoUSA performed significantly better than D&B for restaurants (38% vs. 28%).

To our knowledge, the only other comprehensive validity study on the built food environment published to date was conducted in 12 census tracts in the Montreal metropolitan region (10). Paquet et al. (10) reported undercounts of 16% for a commercial listing of 171 food stores and 34% for an Internet-based listing of 123 food stores. In our study of 8 rural and urban counties and 2,208 food outlets, undercounts of food stores were much more common, with 37%–39% of food stores and 33%–50% of restaurants not being listed in the D&B and InfoUSA databases. Furthermore, we found the lowest undercount error in the DHEC database, from which only 14% of restaurants were missing (but 57% of stores, which largely are not subject to DHEC regulatory permitting). Similar to our study, Sharkey et al. (7) reported that 21% of the 213 food stores found during ground-truthing in a rural environment were not listed; however, they did not present any other validity statistics in the published article. In our study, combining both commercial databases would have decreased the undercount to 19% of stores and 24% of restaurants, and combining all 3 sources would have decreased the undercounts to 12% and 6%, respectively. Thus, combining multiple data sources and including data from state regulatory agencies is a strategy future investigators should consider in order to reduce undercount error.

Overcount error—that is, listed outlets’ either not being present or being found to be closed upon ground-truthing—occurred less frequently (22% in D&B, 14% in InfoUSA, and 11% in DHEC). Had we not conducted substantial data cleaning efforts to remove ineligible outlets prior to the field census, the overcount error would have been much higher. In the Glasgow study, Cummins and Macintyre (11) aimed at characterizing food availability and prices in stores, reporting that approximately 88% of listed outlets were confirmed as open (i.e., an overcount error of only 12%). Unfortunately, combining multiple data sources would have a detrimental effect on this type of error, in that it increases the overcount.

Similar to a previous study of commercial listings of physical activity facilities by Boone et al. (21), our study suggests that, given the dominating nature of undercount error, it is unlikely that the effects of under- and overcounting would balance each other out. This implies that epidemiologic studies using commercial secondary data to characterize the food environments of participants would most likely have underestimated participants’ true exposure status. Furthermore, our data suggest that the ratio of undercount error to overcount error may differ somewhat by type of food outlet. In future studies, investigators should consider carefully the amount and type of error specific to the food-outlet type under study in selecting secondary data sources.

We also evaluated the geospatial accuracy of the geocodes contained in each of the 3 databases. Geospatial accuracy factors into any analysis involving distances or precise geospatial location relative to predefined boundaries. For instance, in research on the built food environment, exposures typically include distance measures such as proximity to the nearest supermarket (6) or count or density measures such as number of supermarkets per square mile in a 1-mile (1.6-km) buffer around a residence (3) or within a census tract (1). We found varying levels of geospatial accuracy for the 3 data sources, using the conservative Euclidian (straight-line) distance estimate. Correct allocation to a census tract was high (above 80%), recognizing that in the rural parts of our area census tracts tended to be very expansive. However, on a smaller spatial scale, marked inaccuracies became apparent, with only about 50% of outlets being allocated correctly within 100 m. Furthermore, we attempted to estimate the overall amount of error by combining geospatial accuracy with an estimate of undercount. The results were sobering: Only 29%–39% of outlets listed had geocodes that placed them within 100 m of their actual location. Therefore, studies relying on very small definitions of neighborhoods may incur larger amounts of error than those utilizing larger buffers or census tracts.

There were several limitations to our study. Given the large geographic area and the number of food outlets, data collection spanned 10 months. Therefore, our data represent more of a period prevalence, while other efforts represent a period as short as 1 month (10). While we attempted to be comprehensive in our field efforts, we cannot exclude the possibility that some outlets were overlooked. In addition, some food outlets discovered may have been listed in a secondary data source but under an NAICS code that we did not request (e.g., code 446191—food/health supplement stores). For the geospatial accuracy analyses, GPS data were used as the gold standard, although this method is subject to some error which can arise from issues including satellite-related errors, signal propagation errors, and receiver errors (22). Lastly, it is important to keep in mind that commercial databases are developed for purposes other than scientific research.

Our study is likely the largest of its kind to date, in terms of both geographic area and the number of food outlets. We included food stores and restaurants, since both have been of interest in research on the built food environment (3, 23), and rural areas, which have not been included in previous studies. We have presented our key results according to specific types of food outlets within the larger groups of stores or restaurants, to assist in future efforts that may be tailored to specific outlets.