|Home | About | Journals | Submit | Contact Us | Français|
There are several examples in nutrition of discordance between the results of observational studies and randomized controlled trials (RCTs). We hypothesized that this discordance is attributable to differences in the translational paths of nutrient–disease associations. Translational paths can be assessed using citation analysis.
We compared the characteristics of citation networks using examples, where RCTs and observational studies agreed (long-chain n-3 polyunsaturated fatty acids [n-3 PUFA]) or disagreed (vitamin E). We performed systematic reviews in each example, constructed citation networks, and compared them with respect to the number of articles and citation relationships between them, as well as the distribution of articles’ hub and authority scores.
For n-3 PUFA, meta-analyses of 14 RCTs and 10 observational studies both suggested that higher intake was associated with lower cardiovascular mortality. For vitamin E, the meta-analysis of 14 RCTs excluded a clinically significant effect, whereas 14 observational studies reported a significant inverse association. The respective citation networks consisted of 392 (n-3 PUFA) and 351 (vitamin E) articles. No differences between the characteristics of the two networks were identified. There was no evidence that the observational studies predated RCTs in the translational process in either example.
In the two examples, citation network characteristics do not predict concordance in the results of observational studies and RCTs.
Observational data have suggested a strong association between a number of dietary factors and chronic disease risk [1-5]. However, randomized controlled trials (RCTs) designed to assess the efficacy of these dietary factors with respect to health outcomes have yielded, for the most part, negative results (fiber and colon cancer , vitamin E and cardiovascular disease , vitamin E and lung cancer , β-carotene and cancer , β-carotene and coronary events , vitamin C and cardiovascular disease , and folate and cardiovascular disease ).
The outcomes of these trials were both disappointing to the healthcare community and confusing to the general public. The trials were expensive to conduct, and in some cases, identified adverse effects of nutrient supplementation . The discrepancies raised serious questions about the currently used approach for determining whether the evidence base is adequate to justify launching a large-scale RCT with hard endpoints as the outcome measure.
Deciding which specific nutrient–disease association to further evaluate in human intervention trials is challenging. Apart from the expected impact of a nutritional intervention on public health, and the feasibility and logistics of conducting a trial, additional critical components need to be factored into the decision. These pertain to the maturity and reliability of the relevant evidence base (that is, the strength of the data supporting a potential nutrient–disease association), the biological plausibility of the association, the reliability of existing data, and the likelihood of bias and systematic errors affecting the interpretation of the available data. The evidence base is formed by the interplay of various translational paths, in which an initial hypothesis-forming observation supports subsequent research and is eventually “translated” to interventions for preventing or treating human disease. It is possible that nutrient associations where RCT and observational data are concordant have a more extensive and mature evidence base, compared with associations where the data are discordant. Therefore, further understanding of the translational paths that shape the evidence base in each topic is of interest. Fig. 1 describes alternative scenarios for the possible cascade of translational events. The simplistic model in Fig. 1a purports a linear progression from an initial experiment in the laboratory to a succession of research studies that eventually lead to observational studies and then to RCTs in humans. If anything, anecdotal observations suggest a much more circuitous path, such as that in Fig. 1b, in which there is no clear succession of study types. To some extent, such patterns of information flow can be assessed with citation analysis, which is a qualitative and quantitative representation of citation relationships among publications.
We hypothesize that differences in the observed flow of information (as captured by citations that are received or made among publications) through the various translational paths and the content of the propagated information are associated with concordance or discordance in the results of observational studies and RCTs. For example, a limited evidence base and information flow may indicate inconsistency of study results and thus may be associated with topics where RCTs and observational studies disagree. Reciprocally, a large evidence base with higher information flow may indicate consistency of findings and general agreement between RCTs and observational studies. Of course, these are not one-to-one relationships; it is conceivable that profound inconsistencies and disagreements between studies could lead to considerable discourse among investigators, which in turn would increase information flow. We set out to empirically explore our hypothesis by analyzing and comparing characteristics of the citation networks in two nutritional associations with disease: one where the two research designs generally agree, and one where they disagree.
For proof of concept, we decided to study the evidence base for two nutrient associations of disease outcomes. After feedback from a technical expert panel composing of four nutritional epidemiology and methodology experts, we selected two nutritional associations with cardiovascular mortality: one where the results of RCTs and prospective cohorts (“observational studies”) were statistically significant and had the same direction (“concordant” example); and one where the summary of observational studies showed a statistically significant effect, but that of the RCTs excluded any clinically important effect (“discordant” example). We chose a hard clinical outcome because such outcomes have high potential impact on public health and strongly influence clinical practice guidelines.
For the “concordant” example, we selected the relationship between the very long-chain n-3 polyunsaturated fatty acids (n-3 PUFA), eicosapentaenoic acid (EPA) and docosahexaenoic acid (DHA), and cardiovascular mortality. Based on systematic reviews and meta-analyses from our team [12,13] and others [14,15], both observational studies and RCTs associated higher EPA and DHA intake with lower cardiovascular mortality. For the “discordant” example, we selected the relationship between vitamin E intake and cardiovascular disease mortality. A meta-analysis of observational studies suggested a statistically significant association with decreased risk of coronary heart disease , whereas a meta-analysis of large RCTs excluded any clinically important effect .
For each topic, we performed de novo systematic reviews and meta-analyses of RCTs and observational studies to become familiar with the literature and verify characterizations of “concordance” or “discordance.” We searched PubMed to identify eligible reports of RCTs and observational studies, screened them for eligibility according to predefined criteria, extracted data, and performed meta-analyses. Details of the systematic review methodology, the characteristics of the eligible studies, and the results of the quantitative synthesis are provided in the online Appendix on the journal’s Web site at www.jclinepi.com.
The search strategies included keywords related to the nutrients of interest (n-3 PUFA or vitamin E), cardiovascular disease, and study design terms related to observational studies or RCTs (see online Appendix for the search strategies on the journal’s Web site at www.jclinepi.com). Searches were conducted during June and July of 2010 and were limited to English-language publications. We considered that the set of publications identified in PubMed contain an adequate representation of the corresponding clinical evidence base. We also assumed that this evidence base would include influential articles along major translational paths.
We then constructed the citation networks of the RCTs and observational studies included in our systematic reviews (“index” articles), and represented them as citation graphs. The online Appendix (on the journal’s Web site at www.jclinepi.com) provides details for how we formed the citation graphs and how we verified that they possess and fulfill theoretically anticipated properties and constraints. Briefly, among the set of articles returned by the PubMed searches, we identified those cited by the index studies directly or following citation links through one or more intermediary papers using the Thompson ISI database.
A citation graph describes citation relationships between articles in the evidence base (Fig. 2). Articles are represented by vertices and citation relationships by arcs that connect pairs of vertices. The direction of each arc is from the cited toward the citing article (following the flow of information). Representing citation relationships as graphs allows us to use tools from graph theory and network analysis [18-21] to characterize the evidence base and the apparent information flow. Without loss of clarity, we use the terms “citation network” and “citation graph” interchangeably.
The citation networks generated are treated as operational representations of the clinical evidence base for n-3 PUFA and vitamin E. A limited evidence base (in terms of number of articles or number of citation connections between them) may indicate inconsistencies in the findings of several studies, and thus may be encountered in topics where RCTs and observational studies disagree. Reciprocally, fields where there is general congruence in the study findings may have many articles and numerous citation connections.
In the past, we have encountered topics where there are more reviews and commentaries than papers with primary data [22-25]. It is therefore important to limit the analysis of citation relationships to the subset of articles that report primary data in humans. To this end, we reviewed the abstracts of the articles in the citation networks and recorded whether the article included primary data in humans (yes, no). We also recorded whether there was a clear statement describing the “intervention” or “exposure” as the nutrient of interest, and whether any clinical cardiovascular outcomes were reported. Articles containing primary data in humans and describing the exposure and the outcomes of interest were deemed pertinent to the association. Examples of nonpertinent articles are methodology articles, and those referring to different nutrients (e.g., alpha-linolenic acid instead of EPA or DHA for n-3 PUFA or antioxidants in general rather than vitamin E), or endpoints other than clinical cardiovascular outcomes (e.g., depression, diabetes, hypertension, lipid profiles).
Both for the initial citation networks and those limited to pertinent studies, we counted the total number of articles and the total number of citation relationships between articles in the network. These were only a subset of the total counts of citations received by an article. For example, the study on vitamin E and coronary heart disease in men by Rimm et al. , received 101 citations within the network but has received over 1,582 citations in total. We also recorded the number of citations made and number of citations received by each article (calculating the density of the citation relationships as per Fig. 2), and the articles’ hub and authority scores. These quantify an article’s importance in the citation network. The hub score is higher for articles that cite a lot of other articles (“integrate more information”), which are in turn connected to other articles that make many citations. The authority score is higher for articles that receive a lot of citations and are also connected to other articles that receive a lot of citations. These metrics convey information on the connectivity of the graph and the relative importance of articles in a network. We compared them across the n-3 PUFA and vitamin E examples using quantile–quantile (QQ) plots and Mann–Whitney tests. We used chi-square tests for discrete characteristics.
We collected the enrollment periods of the RCTs included in the systematic reviews and examined whether they preceded the year of publication of the observational studies. It is likely that most RCTs are conceived and designed at least a year before randomizing the first patient. Therefore, the start of the enrollment period is a conservative proxy of the time when the RCT was conceived. When enrollment periods were not reported, we examined other publications of the same RCT (if available). If no data were found, we imputed the start of the enrollment period as (year of publication—follow-up duration—1 year) and its end as (year of publication—1 year).
Main path articles integrate maximal information from previous articles and propagate information to other articles [18,19,27,28]. They represent the most important stream of propagation of information in a citation network [18,19,29]. By representing “major” citation relationships between key articles, the main path can provide an abstraction of the type of information that flows in a citation network. We calculated the main path of each citation network using an iterative algorithm [27,28], and reviewed the full text of its articles. Examples of articles that are likely to be selected in the main path are major RCTs or major observational studies (including seminal articles) because they are often cited as key articles; and systematic reviews and meta-analyses, because they make references to relevant articles, and because they are often cited by subsequently published articles as state-of-the-science summaries.
We performed separate meta-analyses of RCTs and observational studies (prospective cohorts) for the n-3 PUFA and vitamin E examples. The online Appendix (on the journal’s Web site at www.jclinepi.com) describes the characteristics of the eligible studies for each systematic review, and the results of the quantitative analyses for cardiovascular mortality outcomes.
Briefly, in the n-3 PUFA example, we identified 14 RCTs (reported in 15 publications) [30-44] and 10 prospective cohorts [45-54] (see Tables 1 and 2 in the online Appendix on the journal’s Web site at www.jclinepi.com). Summary results from meta-analyses of RCTs and observational studies were statistically significant and suggested lower risk of cardiovascular mortality with higher n-3 PUFA intake or supplementation. In RCTs, the random-effects meta-analysis relative risk was 0.88 (95% confidence interval [CI]: 0.82, 0.95—in 10 out of 14 trials with analyzable data [45-54], that is, 10 of the 14 trials of the systematic review that contributed to the meta-analysis). In 6 out of 10 prospective cohorts with analyzable data on cardiovascular mortality [45,46,49-51,54], mixed-effects meta-regressions suggested a dose–response association between higher EPA and DHA intakes up to 0.20 g per day and decreased risk of cardiovascular mortality (OR = 0.62, 95% CI: 0.45, 0.86 per 0.20 g of daily intake), with no statistically significant change in risk for higher average intakes.
In the vitamin E example, we identified 14 RCTs [9,10,55-66] and 14 prospective cohorts [26,67-79] (see Tables 3 and 4 in the online Appendix on the journal’s Web site at www.jclinepi.com). The summary relative risk of the 14 RCTs was statistically nonsignificant and excluded any clinically important effect (relative risk = 0.97, 95% CI: 0.91, 1.03), with no evidence of heterogeneity. In contrast, a random-effects meta-analysis of eight observational studies with analyzable data [67-74] suggested an association between higher daily vitamin E intake and lower risk of cardiovascular mortality (summary hazard ratio 0.85, 95% CI: 0.78, 0.93), with little evidence of heterogeneity (I2 = 26%, P-value = 0.13).
Our searches returned 2,741 and 2,825 articles for n-3 PUFA and vitamin E, respectively. Both citation networks were sparsely connected; citation graph densities (ratio of observed to possible citation relationships) were in the order of 10−3. As shown in Table 1, although the two citation networks had similar number of articles, the n-3 PUFA network had more citation relationships (n = 2,193) than the vitamin E network (n = 1,519).
For n-3 PUFA, 392 (out of 2,741) articles were connected to at least one index publication via a citation relationship vs. 351 (out of 2,825) articles for vitamin E (P = 0.04, by chi-square test; Table 1, Fig. 3).
The QQ plots in Fig. 4 compare distributions of the number of citations received (panel a) or made (panel b) by each article from other publications. The plots suggest that these numbers were larger for n-3 PUFA than vitamin E (Mann–Whitney test P < 0.001 and P = 0.013 for panels 4a and 4b, respectively).
In both topics, the top three most cited articles were published in general medical journals with high impact factors (Journal of the American Medical Association, The Lancet, and the New England Journal of Medicine). The three most cited articles in the n-3 PUFA citation network were an observational study in the Netherlands (Zutphen study) that associated higher fish intake with lower cardiovascular mortality (108 citations received) ; a prospective nested case–control study that found a strong association between serum levels of n-3 PUFA and lower risk of sudden death in patients with history of heart disease (an index study that received 71 citations) ; and an observational study (Chicago Western Electric Study) reporting an association between higher fish intake and lower risk of sudden and other cardiovascular death (55 citations received) .
All three most cited papers in the vitamin E example were index articles: a report from the Health Professionals Follow-up Study suggesting a negative association between the use of vitamin E supplements and coronary heart disease in men (101 citations received) ; an analysis from the Nurses’ Health Study prospective cohort suggesting that among middle-aged women, the use of vitamin E supplements was associated with a reduced risk of coronary heart disease (85 citations received) ; and the Cambridge Heart Anti-oxidant Study RCT that found a beneficial effect of vitamin E supplementation on the rate on nonfatal myocardial infarctions, but not on cardiac mortality (CHAOS trial, 70 citations received) .
As shown in Fig. 5, citation networks limited to the subset of pertinent articles were comparable across the two examples. For n-3 PUFA, 71 out of the 392 articles (18%) had primary data in humans and were pertinent to the association of EPA or DHA with clinical cardiovascular outcomes. The corresponding number for vitamin E was 69 out of 351 articles (20%; P = 0.64 by chi-square test for the comparison between the two topics).
For both n-3 PUFA and vitamin E, the enrollment periods of at least two RCTs started before or in the same year as the earliest-published observational study in our systematic review (Fig. 6). For n-3 PUFA, only one  of the three largest RCTs (corresponding to four publications in the systematic review) [32,33,37,41] started enrolling participants after several (three) observational studies were published. In vitamin E, no index observational study was published before the start of the enrollment period of two [55,61] of the three largest RCTs [10,55,61].
We were not able to identify qualitative differences in the main path articles in the two examples. Main path articles in the n-3 PUFA example pertained to the relationship of fish oil or EPA/DHA with cardiovascular risk factors such as lipid profiles, blood pressure/hypertension, hemostasis, intermediate outcomes such as arrhythmia, and hard cardiovascular outcomes including myocardial infarction, stroke, and cardiovascular mortality (Table 2). They included several large prospective cohorts and RCTs, and three systematic reviews or meta-analyses. The articles on the main path of the vitamin E example included seven RCTs and six observational studies on the relationship between vitamin E or other antioxidants with cardiovascular morbidity and cardiovascular or all-cause mortality, and three systematic reviews with meta-analyses (Table 3).
We analyzed the citation networks of publications on the associations of n-3 PUFA and vitamin E with cardiovascular mortality. We observed that, in both examples, citation networks were grossly similar with respect to their quantitative characteristics (such as the number of articles and citation relationships among articles), and the connectivity of the articles. This was also true for citation networks limited to the subset of articles that described primary data in humans and were pertinent to the association between the nutrients and clinical cardiovascular outcomes. In both examples, there was no indication that the research that led to the index RCTs and observational studies proceeded in a sequential fashion, with observational studies preceding RCTs. For both examples, at least two RCTs enrolled their first patient before or on the same calendar year that the earliest-published index observational study appeared in the literature.
Research builds on previous findings. Biomedical knowledge advances as scientific data and its interpretation are communicated toward informing further research. Scientific publications are the generally accepted vehicle for such communications, and the citation networks they form are a representation of the flow of biomedical knowledge through various translational paths . From this point of view, analysis of citation networks may provide insights on the size and maturity of the relevant evidence base. Further in some extreme cases, it may demonstrate the propagation of erroneous conclusions, and help understand the formation and establishment of unfounded belief systems in the medical community .
We hypothesized that nutrient associations of disease where RCTs and observational studies agree in the direction and significance of their findings may differ in the size of their evidence base, and the patterns of information flow in it. Although we found small differences in the quantitative characteristics of the citation networks of the n-3 PUFA and the vitamin E examples, these were not dramatic. Thus, there is no suggestion that quantitative analyses of citation relationships can distinguish topics where RCTs and observational studies agree or disagree. Our negative finding is easy to explain. Only the most dramatic aberrations in a translational path would have been identified in an analysis of citation relationships. An example would be a whole body of literature generated from an unwarranted extrapolation of previous findings. There is no evidence that any such aberrations occurred in the two examples we explored.
Notwithstanding the aforementioned observations, this article introduced methodologies for the quantitative analysis of citation networks as proxies for understanding translational paths. In analyzing the two examples, we found no empirical support of the notion that research translation progresses in a linear fashion, at least in the rightmost end of the translational spectrum, where hypotheses are being evaluated in humans. If anything, the index RCTs and observational studies (the reports that represent the state of knowledge in the two examples) were published over the same time period. If anything, Fig. 6 questions whether the design and launching of most index RCTs could had been informed by the majority of index observational studies.
There are limitations to using citation analysis to understand translational paths in a given topic. Citing previous research is a complex process. As much as we would like citation practices to be impartial and scientific, they are influenced by personal beliefs, biases, and preferences, and are subject to citation distortion. The latter includes citation bias, that is, when one systematically ignores articles that contain content at odds with ones claims; citation amplification where a lot of citations propagate a belief without any evidence support; and citation invention, that includes citing content but ascribing different meaning to it and converting an hypothesis to fact through citation alone . Our searches may have missed early publications that potentially influenced subsequent work, either observational or experimental, and most of the publications in the citation network (with the exception of those included in the systematic reviews) were not assessed in detail for content, pertinence to the association, or methodological quality. The two examples chosen both had cardiovascular mortality as an outcome and results might have been different for non-mortality outcomes. Publication bias, which occurs as a result of researchers publishing only studies with significant positive results and ignoring studies that report nonsignificant results, also affects the citation network. Lastly, much of the work cited for the two examples was published before a registry of clinical trial was established (www.clinicaltrials.gov). Hence, we cannot rule out the possibility that reporting bias confounded our conclusions.
Despite the above limitations, we believe that citation analysis is one of the few representations of the translational process that can be objectively quantified. It merits attention as it can provide a framework to analyze the series of research steps that led to RCTs in humans, and perhaps identify unwarranted extrapolations in the translational process, if any exist . We believe that an empirical exploration of citation relationships for a larger number of nutrient/disease associations is necessary to adjudicate our conclusion that most index RCTs could not had been informed by the mass of corresponding observational studies in either example examined in the present study.
Funding: This article is based on work performed with funding from the Office for Dietary Supplements commissioned through the Agency for Healthcare Research and Quality (Contract number HHSA 290 2007 100551, task order 5). None other than the authors had any role in study design, collection, analysis, or interpretation of data, in the writing of the report, or the decision to submit the paper for publication. The authors are independent from the funders.
Contributors: T.A.T. and J.Lau had the initial idea. T.A.T. designed the study and drafted the protocol with input from J.Lau. T.A.T., D.M., W.W.Y., J.Lee, and M.C. acquired the data. T.A.T. developed software for citation graph manipulation and analyses and wrote the first draft of the paper. All authors critically revised the paper. No other person including medical editors has assisted in any way in the writing or the preparation of the article. T.A.T. had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.jclinepi.2011.07.006.
Competing interests: We declare that none of the authors has a conflict of interest in this submission. All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that no author has support from companies for the submitted work. No other relationships or activities that could appear to have influenced the submitted work.