In this study we found that in a sample of top-tier general medical journals, recently published nutrition and obesity trials are reported at a similarly high quality level between funding categories. Additionally, reporting quality in the areas of study protocol, statistical analysis, and presentation of results were similar. One limitation of our study is that we categorized funding source based on what was reported in the manuscript acknowledgments section. Therefore, it is possible that some studies’ funding category assignments were incorrect due to incompletely reported information. Also, our assignment system did not have a third category for mixed types of reported funding, but this applied to only four of the 38 studies we examined (10%).
Another possible limitation is that we used a single method for assessing reporting quality. There are many methods available in the literature to assess the quality of a research reporting but there is no consensus as to which method is best and none have risen to the top as the clear, preferred method. Indeed, the Agency for Healthcare Research and Quality (AHRQ) evaluated many scoring systems and reduced the number of those generic systems down to nineteen that they concluded fully address their key domains of quality, these being labeled as “generally informative” (13
). The Chalmers method(10
) we used numbers among those cited by the AHRQ, although, as noted above, we made minor modifications based on the specific types of studies we examined. The paired samples approach is commonly used and has been supported when comparing groups of studies (14
). Additionally, we compared the papers for each Chalmers Index subcategory (10
) and found no significant difference between funding categories in any subcategory.
In the earlier analysis that examined obesity trial studies published prior to 2004 (9
), the method used was the CONSORT reporting criteria as described by Thabane, et al. (15
) This could be one explanation for why our present results are different from the results reported by Thomas, et al., which indicated that reports of industry-funded studies were of a statistically significant, higher quality of reporting (9
). Another possible explanation is that the more prevalent use of the CONSORT guidelines (1
) by researchers and journal editors of late is having the desired effect in improving the reporting quality of all types of studies. Since our present sample of papers were published in what are generally considered as “top tier” general medical journals, diligence by authors and editorial staff to assure that the level of reporting meets high standards could have been a key factor in our present results showing uniform quality across funding classes.
Future studies of this type might benefit from comparing studies using two or more methods for rating quality most appropriate to the type of study being evaluated. While there is no currently accepted superior method for assessing reporting quality, cross comparisons between methods may illuminate strengths and weaknesses of the methods that may lead to the development of better rating systems. Since our question pertained to potential issues of bias based on funding sources, we opted for a system that allowed for assessing potential key omissions within the publication limits of the published record. Other future studies that would help to resolve perceptions about the influence of funding sources and publication bias might examine the distribution of funding sources across a wide range of journals to assess whether the “top-tier” journal lean towards publishing studies of one type of funding versus another as compared to some less widely read journals. Our focus on selected, matched, “top-tier” journal articles did not address funding type frequencies within and across these journals in general.
The importance of this topic of assessing the quality of reported science extends beyond the obesity literature to important issues of public safety and policy making, such as the case with bisphenol A (16
). One source of bias that will likely always be in the scientific dialogue is that of confirmation bias – the tendency to seek or selectively attend to information that supports current attitudes or beliefs (17
). Recent understanding concerning the mechanisms of this common bias reveal that the “congeniality” of the new information (how much this information supports existing beliefs) impacts the preferences of the receiver of the information, but this bias is reduced when the information is of higher quality (19
). Efforts to increase the overall quality of the scientific literature can be achieved through activities that support high quality in reporting and create transparency (4
). Research funding is beneficial and necessary, and scientists should remain vigilant against bias of all types, including that which attempts to discredit information based on funding source.