Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Crit Care Med. Author manuscript; available in PMC 2017 April 1.
Published in final edited form as:
PMCID: PMC4793280

Measuring Impact: Citation Rates and the Quest to Identify Meaningful Clinical Research in Critical Care Medicine

“If you think you’re too small to have an impact, try going to bed with a mosquito.”

Annita Roddick

In times of increasing costs and limited funding for clinical trials, identifying trial designs that generate meaningful data to improve the care of patients in the intensive care units is crucial. In this issue of Critical Care Medicine, Marshall and colleagues present an analysis of 391 clinical trials each enrolling more than 100 critically ill patients, that seeks to identify which design factors (such as source of funding, type of study, and the research model) affect the overall impact of a given study, as measured by how often it is cited in the literature.1 Studies conducted by investigator-led clinical trials groups or consortia, as well as those addressing a comparative effectiveness research question, were independently associated with significantly higher annual citation rates. In contrast, the funding model (industry versus peer review funding) and the specific study focus did not correlate with higher citation rates. The investigators’ efforts are ambitious and should be applauded; their analysis provides interesting insight into the relationship between specific trial characteristics and the likelihood of producing widespread discourse within the scientific community. At the same time, this study raises questions about the meaning of citation rates and whether this metric is an adequate tool for identifying clinical research that has the greatest positive impact on patient care.

While citation rates have long been used to judge the efficiency of grant funding and as criterion for academic advancement, there are limitations to using this approach. Notably, citation rates often favor older studies compared with more recent trials (a lead time advantage). Furthermore, one needs to look below the surface to understand why a study has a high citation rate. For example, examining the 16 most cited papers identified by Marshall et al.—9 of which were led by a trials group and cited more than 100 times annually—reveals the potential pitfall of assuming that studies with high citations rates have had a positive and lasting impact on improving patient care. Four of the sixteen studies that showed a benefit of either tight glucose control,2,3 steroids and the utility of ACTH stimulation testing,4 or activated protein C5 were later refuted by four other studies on the same list.69 Furthermore, the trial of early goal-directed therapy—the study with the second most median citations per year—has recently been refuted by three large multicenter, clinical trials.1012 Ironically, in most cases (3/4) the paper that disproved the original trial results and limited the use of potentially harmful interventions—arguably more meaningful and durable work—was cited less often. As well, some highly cited studies have design flaws that make the interpretation of the results challenging and thus difficult to apply at the bedside. For example, the study designs for both the ARDSnet trial13 and the TRICC trial14 have been questioned for discounting the titrated nature of standard practice.15 The lack of such a usual titrated care control arm limited the ability to assess the clinical impact in the ARDSnet and TRICC trials of the two treatments tested.16,17 In total, these findings would suggest that citation rate does not necessarily identify clinical trials that advance and improve patient care. In some cases, high citation rates may even be indicative of controversial, non-reproducible science that is “too good to be true.”

Since the creation of The Canadian Critical Care Trials Group in 1989,18 a number of other investigator-led consortia have formed and are dedicated to designing and conducting clinical trials in the field of critical care.19,20 These groups have led numerous multicenter, randomized controlled trials in the past two decades, generating valuable clinical research. The role that these groups have had in advancing the understanding of critical care medicine therapies is unquestioned and their willingness to design and conduct clinical trials in critically ill patients, a population that has often been neglected by industry, is commendable. However, trial groups are only one source of valuable data: multiple streams of clinical research are most likely to advance knowledge and improve patient care. While no one would argue that smaller trials are inherently superior to larger trials,21 it is important to recognize the ingenuity and the contribution of studies that were designed and carried out by investigators not involved in such groups, sometimes performed at single centers. Examples of small studies that have had a lasting impact on patient care include the first studies showing the benefit of bedside ultrasound for critical care procedures,22 the use of steroids to treat severe Pneumocystis jiroveci pneumonia in AIDS patients,23 or the use of transluminal balloon dilatation for coronary artery stenosis.24

Unfortunately there is no perfect instrument for identifying quality, high impact clinical research and it is important that we recognize the inherent limitations of citation rates and the ongoing need for peer review. Trial groups, small investigator-led teams and industry bring talent, insight and perspective to many of the clinical problems faced by intensivists and the patients they care for. Whether their work proves meaningful to patient care depends mostly on the relevancy of the question, the quality of the trial design and the reproducibility of the findings.


Copyright form disclosures: Dr. Cortés-Puch received support for article research from the National Institutes of Health (NIH). Dr. Natanson disclosed other support (US government employee work done as part of official US Government work and part of the public domain there are no patent rights).


The opinions expressed in this article are the authors’ own and do not represent any position or policy of the National Institutes of Health, the Department of Health and Human Services, or the United States government.

Dr. Sweeney disclosed that he does not have any potential conflicts of interest.


1. Marshall JC, Kwong WK, Kommaraju K, Burns KEA. Determinants of Citation Impact in Large Clinical Trials in Critical Care: The Role of Investigator-led Clinical Trials Groups. Crit Care Med. 2016 in press. [PubMed]
2. van den Berghe G, Wouters P, Weekers F, et al. N Engl J Med. 2001;345(19):1359–1367. [PubMed]
3. Van den Berghe G, Wilmer A, Hermans G, et al. Intensive insulin therapy in the medical ICU. N Engl J Med. 2006;354(5):449–461. [PubMed]
4. Annane D, Sebille V, Charpentier C, et al. Efludrocortisone on mortality in patients with septic shock. JAMA. 2002;288(7):862–871. [PubMed]
5. Bernard GR, Vincent JL, Laterre PF, et al. Efficacy and safety of recombinant human activated protein C for severe sepsis. N Engl J Med. 2001;344(10):699–709. [PubMed]
6. Investigators N-SS. Finfer S, Chittock DR, et al. N Engl J Med. 2009;360(13):1283–1297. [PubMed]
7. Brunkhorst FM, Engel C, Bloos F, et al. Intensive insulin therapy and pentastarch resuscitation in severe sepsis. N Engl J Med. 2008;358(2):125–139. [PubMed]
8. Sprung CL, Annane D, Keh D, et al. N Engl J Med. 2008;358(2):111–124. [PubMed]
9. Ranieri VM, Thompson BT, Barie PS, et al. N Engl J Med. 2012;366(22):2055–2064. [PubMed]
10. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693. [PMC free article] [PubMed]
11. Investigators A, Group ACT. Peake SL, et al. N Engl J Med. 2014;371(16):1496–1506. [PubMed]
12. Mouncey PR, Osborn TM, Power GS, et al. Trial of early, goal-directed resuscitation for septic shock. N Engl J Med. 2015;372(14):1301–1311. [PubMed]
13. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. The Acute Respiratory Distress Syndrome Network. N Engl J Med. 2000;342(18):1301–1308. [PubMed]
14. Hebert PC, Wells G, Blajchman MA, et al. A multicenter, randomized, controlled clinical trial of transfusion requirements in critical care. Transfusion Requirements in Critical Care Investigators, Canadian Critical Care Trials Group. N Engl J Med. 1999;340(6):409–417. [PubMed]
15. Deans KJ, Minneci PC, Suffredini AF, et al. Crit Care Med. 2007;35(6):1509–1516. [PubMed]
16. Klein HG, Flegel WA, Natanson C. Red Blood Cell Transfusion: Precision vs Imprecision Medicine. JAMA. 2015:1–2. [PubMed]
17. Klein HG, Cortes-Puch I, Natanson C. Clinical practice: Blood-transfusion decisions not simple. Nature. 2015;521(7552):289. [PubMed]
18. Marshall JC, Cook DJ, Canadian Critical Care Trials G Investigator-led clinical research consortia: the Canadian Critical Care Trials Group. Crit Care Med. 2009;37(1 Suppl):S165–172. [PubMed]
19. Jones DA, Cooper DJ, Finfer SR, et al. Advancing intensive care research in Australia and New Zealand: development of the binational ANZIC Research Centre. Crit Care Resusc. 2007;9(2):198–204. [PubMed]
20. In FGHNC. InFACT: a global critical care research response to H1N1. Lancet. 2010;375(9708):11–13. [PubMed]
21. Wiedermann CJ, Wiedermann W. Beautiful small: Misleading large randomized controlled trials? The example of colloids for volume resuscitation. J Anesthesiol Clin Pharmacol. 2015;31(3):394–400. [PMC free article] [PubMed]
22. Legler D, Nugent M. Doppler localization of the internal jugular vein facilitates central venous cannulation. Anesthesiol. 1984;60(5):481–482. [PubMed]
23. Nielsen TL, Eeftinck Schattenkerk JK, Jensen BN, et al. Adjunctive corticosteroid therapy for Pneumocystis carinii pneumonia in AIDS: a randomized European multicenter open label study. Journal of acquired immune deficiency syndromes. 1992;5(7):726–731. [PubMed]
24. Gruntzig A. Transluminal dilatation of coronary-artery stenosis. Lancet. 1978;1(8058):263. [PubMed]