This is not clear. The authors searched electronic databases such as MEDLINE, HealthStar, Psych Info and Embase for trials conducted during the period from 1966 to February 2000 for a previous review.1
Additional electronic database searches for studies published between February 2000 and October 2004 were conducted to add to data obtained previously. The authors further reviewed the Cochrane Library and seminal work by ‘prominent authors in the field of obesity’ to locate additional relevant citations. The authors, however, report no effort to identify unpublished literature or attempt to assess the effects of publication bias on results.
Approximately half of the initially identified trials were excluded due to ‘poor methodological quality’, but we are not told the nature of these methodological problems, whether standardized exclusion criteria existed, and if so, what these exclusion criteria were. Of 19 studies identified as appropriate for potential inclusion in the meta-analysis that were not excluded because of methodological problems, 2 were subsequently excluded because they lacked control groups. Lack of a control group, to us, represents a substantial methodological flaw.
Moreover, 5 more of the 19 studies were not included in the meta-analysis because Katz et al
. were unable to obtain or calculate an effect size from the published report. There is no description of an attempt to contact the authors for the requisite information, to place plausible boundaries on the sample effect size from incomplete data or to back-calculate effect sizes from test statistics as can typically be done in such situations (cf., Allison et al
). Several of the excluded studies were relatively large and strong studies, so their inclusion may make a difference. One excluded study was that of Sallis et al
and yet they wrote ‘Effect sizes were large for boys’ …BMI (d
= 0.83).’ So, at least for boys, the exclusion was not necessary. No calculation was needed, the original authors had provided the effect size. Similarly, Table 2 of Caballero et al
a study listed in Katz et al
.’s Table 1 as one for which an effect size could not be calculated, seems to have all the requisite information for calculating an effect size.
Thus, we are unable to place full confidence in the process that led to studies being included in the meta-analysis as being one that fully represented the available data.