PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Clin Epidemiol. Author manuscript; available in PMC 2010 August 1.
Published in final edited form as:
PMCID: PMC2706913
NIHMSID: NIHMS97156

Twenty-Four Hour Dietary Recalls by Fourth-Grade Children Were Not Influenced by Observations of School Meals

Abstract

Objective

To investigate whether school-meal observations influenced children’s 24-hour dietary recalls.

Study Design and Setting

Over three school years, 555 randomly selected fourth-grade children were interviewed to obtain a 24-hour dietary recall; before being interviewed, 374 children were observed eating two school meals (breakfast, lunch) and 181 children were not observed. Within observation-status groups (observed; unobserved), children were randomized within sex to one of six combinations from two target periods (prior-24-hours; previous-day) crossed with three interview times (morning; afternoon; evening).

Results

For each of five variables (interview length, meals/snacks, meal components, items, kilocalories), naïve and adjusted equivalence tests rejected that observation-status groups were different, indicating that school-meal observations did not influence children’s 24-hour dietary recalls. There was a target-period effect on length (P<0.0001) (longer for prior-24-hour recalls), a school year effect on length (P=0.0002) (longer for third year), and a target-period-by-interview-time interaction on items (P=0.0110) and kilocalories (P=0.0047) (both smaller for previous-day recalls in the afternoon than prior-24-hour recalls in the afternoon and previous-day recalls in the evening), indicating that variables were sufficiently sensitive and psychometrically reliable.

Conclusion

Conclusions about 24-hour dietary recalls by fourth-grade children observed eating school meals in validation studies are generalizable to 24-hour dietary recalls by comparable but unobserved children in non-validation studies.

Keywords: 24-hour dietary recalls, children, validation, observation, school breakfast, school lunch

What is new?

Key findings

  • For each of five response variables, naïve and adjusted equivalence tests rejected differences between observation-status groups, indicating that our implementation of school breakfast and school lunch observations did not influence fourth-grade children’s 24-hour dietary recalls. Furthermore, results concerning a target-period effect and a target-period-x-interview-time interaction indicated that the response variables were sufficiently sensitive and psychometrically reliable to detect differences.

What this adds to what is known

  • Two previously conducted studies that investigated this found conflicting results.

What is the implication and what should change now

  • Observation of school meals is an appropriate method to use in validation studies to investigate the accuracy of reports of these meals in children’s 24-hour dietary recalls.
  • Conclusions about 24-hour dietary recalls by fourth-grade children observed eating school meals in validation studies can be generalized to 24-hour dietary recalls by comparable but unobserved children in non-validation studies (such as national surveys, epidemiologic studies, and nutrition interventions).
  • Assessing the impact of measurement in validation studies is essential to generalizing the results of validation studies to research subjects for whom reference information has not been collected.

Introduction

Dietary-reporting validation studies compare reported information to reference information that, ideally, is from a method independent of the subject’s memory. Observation is the best method for validating dietary reports, and observations should occur in cafeteria-type settings familiar to subjects [1]. Because children eat meals in locations where parents are not present (e.g., at school), it is necessary to obtain dietary information from children. School-meal observations provide an excellent method to validate reports of these meals in children’s 24-hour dietary recalls [2]. Observations of children eating meals in private homes may cause substantial reactivity [3]. Reactivity is less problematic when observations occur at school [2] where children are accustomed to being watched while eating in groups [2, 4]. In most validation studies for which children provided dietary recalls without parental assistance, reference information was obtained by observing one or two school meals [5].

A goal of validation studies is to generalize conclusions to subjects in non-validation studies for whom reference information is not collected [3]. Ideally, validation methods have no effect on subjects’ reports [3, 6]. However, collecting reference information in validation studies may influence subjects’ responses or behaviors. For example, observing children eating school (or other) meals may influence what or how much children eat, children’s memories about what they ate, and/or children’s subsequent dietary recalls [5, 7]. Such influence may restrict generalizability of validation-study conclusions.

To our knowledge, only two studies have investigated the effects of school-meal observations on children’s 24-hour dietary recalls [5, 7]. In a study by Baranowski et al. [7], fourth-grade children were or were not observed eating school lunch, and provided two recalls collected back-to-back (one with child-operated software; one with a dietitian-administered interview). Baranowski et al., who defined “intrusions” as items reported in child-operated software interviews but not in dietitian-administered interviews, found that observed children averaged significantly lower on intrusions than unobserved children and concluded that school-lunch observations decrease reports of uneaten foods. However, children’s observation status was partially confounded with a bogus pipeline manipulation intended to enhance recall accuracy. (Children provided hair samples and were told “We can tell some of what you eat from a chemical analysis of your hair.”) Our assessment of the unconfounded observation-status effect showed a non-significant difference in intrusions between observed and unobserved children [5]. Furthermore, completing back-to-back recalls is similar to completing multiple interview passes in two prominent 24-hour dietary recall protocols. (One is the Nutrition Data System for Research [http://www.ncc.umn.edu/about/missionandhistory.html]; the other is the U.S. Department of Agriculture’s automated multiple-pass method [http://www.ars.usda.gov/Services/docs.htm?docid=7710].) Multiple passes provide respondents with numerous opportunities to report additional foods, so obtaining back-to-back recalls from children may have had implications for items Baranowski et al. defined as “intrusions”.

In a study with fourth-grade children who were or were not observed eating two school meals (breakfast, lunch), Smith et al. [5] used a different approach to investigate whether school-meal observations influenced children’s 24-hour dietary recalls. Each of 120 children was randomized to one of 12 conditions created by crossing two observation-status groups (observed; unobserved), two target periods (prior-24-hours; previous-day), and three interview times (morning; afternoon; evening). The Figure illustrates intake covered in recalls in these 12 conditions. For each child, five response variables were measured — interview length, meals/snacks for the target period, and, for two school meals, meal components, weighted items, and kilocalories. None of these response variables depended on observation status. However, target period affected reported intake, with more meals/snacks and longer interviews for prior-24-hour recalls than previous-day recalls, which indicated that response variables were sufficiently reliable to detect manipulations. Together, results suggested that school-meal observations did not affect fourth-grade children’s 24-hour dietary recalls, which in turn suggested, but did not guarantee, that conclusions about 24-hour dietary recalls by fourth-grade children observed eating school meals may be generalized to 24-hour dietary recalls by comparable children not observed [5].

FigureFigure
Intake Covered in 24-Hour Dietary Recalls in 12 Conditions Created by Crossing Six Target-Period-by-Interview-Time Combinations with Two Observation-Status Groups

For this article, we investigated whether school-meal observations influenced 24-hour dietary recalls in a large sample of fourth-grade children. This was the secondary aim of a study for which the primary aim was to investigate target-period and interview-time effects on children’s dietary recall accuracy. For the primary aim, children were observed eating two school meals and interviewed to obtain a 24-hour dietary recall; recall accuracy is described elsewhere [8]. For the secondary aim, additional children were interviewed to obtain a 24-hour dietary recall without having been observed eating either school meal for the target period. For this article, we investigated the statistical equivalence [911] of characteristics of 24-hour dietary recalls by observed and unobserved children.

Material and methods

The study was approved by the appropriate institutional review board. Written child assent and parental consent were obtained. Table 1 provides details concerning data collection and quality control methods.

TABLE 1
Details Concerning Key Aspects of Data Collection Methods and Quality Control

Data collection occurred during the 2004–05, 2005–06, and 2006–07 school years in 17, 17, and 8 schools, respectively. From participating children within the center 95% of the age distribution, children were randomly selected with the constraints that half were girls and all were usual school-meal eaters.

In the final sample of 555 children (half girls; 96% Black; mean ± standard deviation 10.1±0.79 years of age), 374 children were observed eating two school meals (breakfast, lunch) and interviewed to obtain a 24-hour dietary recall, and 181 children were interviewed to obtain a 24-hour dietary recall without having been observed eating either school meal in the target period. Assignment to observation-status group was random; within observation-status groups, children were randomized to one of six target-period-by-interview-time combinations (see Figure) with the constraint that each combination had 62 or 64 children (half girls) in the observed group, and 30 or 31 children (half girls) in the unobserved group. Also, insofar as possible, within school year and school, randomization to target-period-by-interview-time combinations within observation-status groups by sex was stratified across classes.

Observers followed a written protocol based on previously used procedures [5, 12]. Interobserver reliability was assessed using established procedures [13].

Interviewers followed written multiple-pass protocols based on previously used procedures [5, 12]. Quality control for interviews was assessed for each interview using established procedures [14].

Observation-status groups were compared on the five response variables used by Smith et al. [5] — interview length, meals/snacks for the target period, and, for two school meals, meal components, weighted items, and kilocalories. Interview length was determined by subtracting each interview’s beginning time from its ending time. Specific criteria were used to treat meals reported in 24-hour dietary recalls as school meals (Table 2, footnote §). Items reported eaten in non-zero amounts at school meals were classified by meal component (Table 2, footnote ¶) and weighted (Table 2, footnote ||). Amounts eaten were reported in servings of standard school-meal portions, quantified, and converted to kilocalories (Table 2, footnote **).

TABLE 2
Means ± Standard Deviations for Five Response Variables from Fourth-Grade Children’s 24-Hour Dietary Recalls *

To investigate statistical equivalence [911] between 24-hour dietary recalls by observed and unobserved children, for each response variable, we conducted a naïve (unadjusted) equivalence test to investigate the effect of observation-status group. Each equivalence test was a pair of one-sided two-sample t-tests based on a tolerance δ>0 denoting the smallest nonnegligible difference:

H0:μunobservedμobserved+δversusH1:μunobserved<μobserved+δH0:μunobservedμobservedδversusH1:μunobserved>μobservedδ

Rejecting both null hypotheses leads one to accept the alternative hypotheses

μobservedδ<μunobserved<μobserved+δ

which implies equivalence (on average) of observation-status groups. We used Cohen’s definition of a small effect δ=(0.20)sobserved [15]. By convention, results of equivalence tests are summarized using the larger P value from the pair of two-sample t-tests.

Subsequently, each response variable was modeled using analysis of variance (for interview length, weighted items, and kilocalories) or Poisson regression (for meals/snacks and meal components) with observation-status group, target period, interview time, target-period-by-interview-time interaction, observation-status-group-by-target-period interaction, observation-status-group-by-interview-time interaction, observation-status-group-by-target-period-by-interview-time interaction, sex, interviewer, and school year in the model. For each response variable, after fitting a full model, non-significant (P>0.05) terms were removed, and an equivalence test on adjusted means was conducted to assess differences for observation-status groups.

Analyses used Stata 10.0 (Stata, Inc., College Station, TX) and SAS 9.0 (SAS Institute, Inc., Cary, NC). A significance criterion of 0.05 was used; post hoc pairwise comparisons were interpreted using Bonferroni adjustments.

Results

Thirty-nine of 374 observed children (10%) and 22 of 181 unobserved children (12%) failed to meet criteria for reporting both school meals (Table 2, footnote §). Because in practice one would not know whether interviewed children had eaten particular meals, children who failed to meet criteria for reporting both school meals were included in analyses by using zeroes for their three response variables concerning two school meals — meal components, weighted items, and kilocalories [5]. Analyses with and without these 61 children essentially agreed; we indicate the four discrepant results, none of which involved observation-status group.

Table 2 shows means and standard deviations for response variables for each target period (by observation-status group and averaged across observation-status group), each target-period-by-interview-time combination (by observation-status group and averaged across observation-status group), each observation-status group, and averaged across all interviews.

For each response variable, naïve equivalence tests rejected hypotheses that observation-status groups differed (P values <0.0083). Thus, we concluded that means of observation-status groups were equivalent, indicating that school-meal observations did not influence children’s 24-hour dietary recalls.

For interview length, there was a target-period effect (P<0.0001), with interviews 3 minutes longer on average for prior-24-hour recalls than previous-day recalls (Table 2). There was an interviewer effect (P=0.0159); mean interview length for the four interviewers ranged from 17.5 to 18.8 minutes, but no post hoc pairwise comparison was significant. There was a school-year effect (P=0.0002); means for the first, second, and third school years were 17.7, 17.6, and 20.1 minutes, respectively, and post hoc pairwise comparisons indicated that interviews were longer for the third than first (P<0.0001) and second school years (P<0.0001).

For weighted items for two school meals, only the target-period-by-interview-time interaction was significant (P=0.0110). Post hoc pairwise comparisons indicated that 1.3 fewer weighted items on average were reported for two school meals for previous-day recalls in the afternoon than prior-24-hour recalls in the afternoon (P=0.0031) and previous-day recalls in the evening (P=0.0022) [Table 2]. When children who failed to meet criteria for reporting both school meals were excluded from analyses, two results were discrepant from those in this paragraph. First, there was a school-year effect (P=0.0402), with weighted items for two school meals ranging on average from 6.3 to 6.9 for the three school years, but no pairwise comparison was significant. Second, although there was a significant target-period-by-interview-time interaction (P=0.0155), post hoc pairwise comparisons were discrepant, indicating that average numbers of weighted items for two school meals differed between prior-24-hour recalls in the morning (5.8) and both prior-24-hour recalls in the evening (6.9; P=0.0019) and previous-day recalls in the morning (7.0; P=0.0015).

For kilocalories for two school meals, only the target-period-by-interview-time interaction was significant (P=0.0047). Post hoc pairwise comparisons indicated that 156 fewer kilocalories on average were reported for two school meals for previous-day recalls in the afternoon than both prior-24-hour recalls in the afternoon (P=0.0020) and previous-day recalls in the evening (P=0.0019) [Table 2]. When children who failed to meet criteria for reporting both school meals were excluded from analyses, two results were discrepant from those in this paragraph. First, there was a sex effect (P=0.0096); on average, more kilocalories were reported by boys (681) than girls (615). Second, although there was a significant target-period-by-interview-time interaction (P=0.0332), with average kilocalories reported for the two school meals ranging from 573 to 687 across the six target-period-by-interview-time combinations, no post hoc pairwise comparison was significant.

For meals and snacks for the target period, and for meal components for two school meals, no significant effects or interactions were found.

For no response variable was there a significant effect of observation-status group, or a significant interaction of observation-status group with any other effect.

For adjusted equivalence tests, all P values were <0.0016. Thus, results of adjusted and naïve equivalence tests were consistent; for each response variable, we rejected the pair of null hypotheses and concluded that observation-status group means were equivalent.

Discussion

Results are compelling that our implementation of school-meal observations did not influence fourth-grade children’s 24-hour dietary recalls. Specifically, for each of five response variables, in naïve and adjusted equivalence tests, we rejected hypotheses that observation-status groups differed, which allowed us to conclude that means for the observation-status groups were statistically equivalent. Furthermore, results concerning a target-period effect on interview length and a target-period-by-interview-time interaction on weighted items and kilocalories indicated that response variables were sufficiently sensitive and psychometrically reliable to detect differences, providing added support for the conclusion that school-meal observations did not affect fourth-grade children’s 24-hour dietary recalls [3, 6]. These results are similar to those from the Smith et al. study [5] for which the design was comparable but the sample was too small to permit equivalence testing.

Our conclusion differs from that of Baranowski et al. [7], but our approach differed. Smith [3] noted that when investigating the effect on subjects’ reports of collecting reference information, such information is not available for one group; thus, groups cannot be compared on accuracy measures, and other measures must be used. This methodology is appropriate for assessing the impact of school-meal observations on children’s 24-hour dietary recalls.

Investigation of the effect of collecting reference information, and establishing statistical equivalence to “no-reference-method” groups (if possible), would be appropriate in other endpoint validation studies [3, 5, 6]. For example, studies to validate self-report questionnaires about taking medication may use electronic monitors that record every opening of medication containers [e.g., 16], and studies to validate self-report questionnaires about physical activity may use accelerometers that record motion [e.g., 17]. To determine whether validation methods of electronic monitors and accelerometers affect subjects’ questionnaire responses, such studies could include a no-reference-method group of subjects who do not use electronic monitors or accelerometers but do complete questionnaires.

Limitations included homogeneity in children’s race/ethnicity, grade level, and geographical location. We did not investigate the effect of being observed eating meals at home due to reactivity.

Strengths included the large sample, which allowed us to conduct conservative equivalence tests (using Cohen’s definition of a small effect [15]) to compare observation-status groups. The effect of being observed eating two school meals was investigated. Interviewers were blinded to children’s observation status. Children were usual school-meal eaters. Quality control methods were rigorous.

These results have important implications. Children will likely continue to provide thousands of 24-hour dietary recalls each year for national surveys [e.g., 18] and interventions [e.g., 19], and to assess the relative validity of food frequency questionnaires [e.g., 20]. There is a crucial need for validation studies to improve children’s dietary recall accuracy to enhance dietary-data quality in non-validation studies that utilize children’s dietary recalls. Our findings justify using our implementation of school-meal observations in validation studies to investigate children’s accuracy for these meals in 24-hour dietary recalls. Conclusions about 24-hour dietary recalls by fourth-grade children observed eating school meals in validation studies can be generalized to 24-hour dietary recalls by comparable but unobserved children in non-validation studies.

Acknowledgments

This research was supported by grant R01 HL074358 from the National Heart, Lung, and Blood Institute of the National Institutes of Health; Suzanne D. Baxter was Principal Investigator. The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or the National Institutes of Health.

The authors appreciate the cooperation of children, faculty, and staff of elementary schools, and staff of Student Nutrition Services, of the Richland One School District (Columbia, SC).

The authors thank Elizabeth J. Herron for her comments on earlier versions of this manuscript.

A brief podium presentation with some results in this article was given at the American Dietetic Association’s Food and Nutrition Conference and Expo held in Chicago, Illinois in October, 2008.

The authors appreciate the contribution to this work of Amy F. Joye, MS, RD, who was Project Director for this grant until she suffered severe brain damage due to a medical tragedy. The Amy Joye Research Fund is being established through the American Dietetic Association Foundation to award nutrition research grants on an annual basis in Amy’s honor.

References

1. Mertz W. Food intake measurements: Is there a “gold standard”? Journal of the American Dietetic Association. 1992;92:1463–5. [PubMed]
2. Simons-Morton BG, Baranowski T. Observation in assessment of children’s dietary practices. Journal of School Health. 1991;61(5):204–7. [PubMed]
3. Smith AF. Concerning the suitability of recordkeeping for validating and generalizing about reports of health-related information. Review of General Psychology. 1999;3(2):133–50.
4. Baxter SD, Thompson WO, Litaker MS, Guinn CH, Frye FHA, Baglio ML, et al. Accuracy of fourth-graders’ dietary recalls of school breakfast and school lunch validated with observations: In-person versus telephone interviews. Journal of Nutrition Education and Behavior. 2003;35(3):124–34. [PMC free article] [PubMed]
5. Smith AF, Baxter SD, Hardin JW, Guinn CH, Royer JA, Litaker MS. Validation-study conclusions from dietary reports by fourth-grade children observed eating school meals are generalisable to dietary reports by comparable children not observed. Public Health Nutrition. 2007;10:1057–66. [PMC free article] [PubMed]
6. Webb EJ, Campbell DT, Schwartz RD, Sechrest L. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally; 1966.
7. Baranowski T, Islam N, Baranowski J, Cullen KW, Myres D, Marsh T, et al. The Food Intake Recording Software System is valid among fourth-grade children. Journal of the American Dietetic Association. 2002;102:380–5. [PubMed]
8. Baxter SD, Hardin JW, Guinn CH, Royer JA, Mackelprang AJ, Smith AF. Journal of the American Dietetic Association. Fourth-grade children’s dietary recall accuracy is influenced by retention interval (target period and interview time) (in press) [PMC free article] [PubMed]
9. Rogers JL, Howard KI, Vessey JT. Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin. 1993;113(3):553–65. [PubMed]
10. Seaman MA, Serlin RC. Equivalence confidence intervals for two-group comparisons of means. Psychological Methods. 1998;3(4):403–11.
11. Tyron WW. Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods. 2001;6(4):371–86. [PubMed]
12. Baxter SD, Smith AF, Litaker MS, Guinn CH, Shaffer NM, Baglio ML, et al. Recency affects reporting accuracy of children’s dietary recalls. Annals of Epidemiology. 2004;14:385–90. [PubMed]
13. Baglio ML, Baxter SD, Guinn CH, Thompson WO, Shaffer NM, Frye FHA. Assessment of interobserver reliability in nutrition studies that use direct observation of school meals. Journal of the American Dietetic Association. 2004;104:1385–93. [PMC free article] [PubMed]
14. Shaffer NM, Baxter SD, Thompson WO, Baglio ML, Guinn CH, Frye FHA. Quality control for interviews to obtain dietary recalls from children for research studies. Journal of the American Dietetic Association. 2004;104:1577–85. [PMC free article] [PubMed]
15. Cohen J. A power primer. Psychological Bulletin. 1992;112(1):155–9. [PubMed]
16. Zeller A, Schroeder K, Peters TJ. An adherence self-report questionnaire facilitated the differentiation between nonadherence and nonresponse to antihypertensive treatment. Journal of Clinical Epidemiology. 2008;61:282–8. [PubMed]
17. Wendel-Vos GC, Schuit AJ, Saris WHM, Kromhout D. Reproducibility and relative validity of the short questionnaire to assess health-enhancing physical activity. Journal of Clinical Epidemiology. 2003;56:1163–9. [PubMed]
18. Gordon A, Fox MK. School Nutrition Dietary Assessment Study-III, Summary of Findings. Alexandria, VA: US Department of Agriculture, Food and Nutrition Service, Office of Research, Nutrition, and Analysis; 2007. [accessed October 11, 2008]. Available from: http://www.fns.usda.gov/oane/MENU/Published/CNP/FILES/SNDAIII-SummaryofFindings.pdf.
19. Luepker RV, Perry CL, McKinlay SM, Nader PR, Parcel GS, Stone EJ, et al. Outcomes of a field trial to improve children’s dietary patterns and physical activity: The Child and Adolescent Trial for Cardiovascular Health (CATCH) JAMA. 1996;275(10):768–76. [PubMed]
20. Field AE, Peterson KE, Gortmaker SL, Cheung L, Rockett H, Fox MK, et al. Reproducibility and validity of a food frequency questionnaire among fourth to seventh grade inner-city school children: Implications of age and day-to-day variation in dietary intake. Public Health Nutrition. 1999;2(3):293–300. [PubMed]