|Home | About | Journals | Submit | Contact Us | Français|
The effects of clinical-trial funding on the interpretation of trial results are poorly understood. We examined how such support affects physicians’ reactions to trials with a high, medium, or low level of methodologic rigor.
We presented 503 board-certified internists with abstracts that we designed describing clinical trials of three hypothetical drugs. The trials had high, medium, or low methodologic rigor, and each report included one of three support disclosures: funding from a pharmaceutical company, NIH funding, or none. For both factors studied (rigor and funding), one of the three possible variations was randomly selected for inclusion in the abstracts. Follow-up questions assessed the physicians’ impressions of the trials’ rigor, their confidence in the results, and their willingness to prescribe the drugs.
The 269 respondents (53.5% response rate) perceived the level of study rigor accurately. Physicians reported that they would be less willing to prescribe drugs tested in low-rigor trials than those tested in medium-rigor trials (odds ratio, 0.64; 95% confidence interval [CI], 0.46 to 0.89; P = 0.008) and would be more willing to prescribe drugs tested in high-rigor trials than those tested in medium-rigor trials (odds ratio, 3.07; 95% CI, 2.18 to 4.32; P<0.001). Disclosure of industry funding, as compared with no disclosure of funding, led physicians to downgrade the rigor of a trial (odds ratio, 0.63; 95% CI, 0.46 to 0.87; P = 0.006), their confidence in the results (odds ratio, 0.71; 95% CI, 0.51 to 0.98; P = 0.04), and their willingness to prescribe the hypothetical drugs (odds ratio, 0.68; 95% CI, 0.49 to 0.94; P = 0.02). Physicians were half as willing to prescribe drugs studied in industry-funded trials as they were to prescribe drugs studied in NIH-funded trials (odds ratio, 0.52; 95% CI, 0.37 to 0.71; P<0.001). These effects were consistent across all levels of methodologic rigor.
Physicians discriminate among trials of varying degrees of rigor, but industry sponsorship negatively influences their perception of methodologic quality and reduces their willingness to believe and act on trial findings, independently of the trial’s quality. These effects may influence the translation of clinical research into practice.
A large proportion of clinical trials of new treatments is funded by the pharmaceutical industry.1,2 There is increasing concern over how such support can affect the design of these trials3 and the reporting of the results.4–9 Some practitioners may be swayed by potential bias in industry-supported publications to use certain medications more widely than would be appropriate, whereas other practitioners may be skeptical about the validity of company-funded research.10 In the past two decades, to ensure full transparency in the reporting of clinical trials, disclosure of financial support has become the norm in medical research publications.11–13 Some biomedical journals include disclosures in the abstracts themselves, where their effect on readers might be greatest.14
Financial disclosure may not fully address the problem of potential bias in published research results.15,16 One study showed that physicians ignored disclosure statements,17 whereas other studies have shown that disclosure of substantial conflicts of interest can lead clinicians to discount the reported results of a trial.18–20 Some reports suggest that the disclosure of conflicts of interest can have a paradoxical effect,21 such as enhancing the trustworthiness of the discloser.22
The methodologic rigor of a trial, not its funding disclosure, should be a primary determinant of its credibility. Skepticism about results that is based on a trial’s funding sources may be less appropriate for well-controlled, double-blind, randomized trials than for poorly controlled or unblinded trials, in which conflicts of interest may have a stronger effect on interpretation of the data. However, little is known about how information concerning study design interacts with information concerning funding sources to influence physicians’ interpretation of research. We therefore conducted a randomized study involving simulated research abstracts to assess the role that disclosure plays in physicians’ interpretation of the results of medical research.
We developed hypothetical scenarios in which three new drugs were being evaluated for the treatment of disorders commonly encountered in primary care: hyperlipidemia, diabetes, and angina. “Lampytinib” would be used for dyslipidemia in patients who had unacceptable side effects from statins, “bondaglutaraz” would be used for diabetes and low levels of high-density lipoprotein cholesterol in patients who were taking metformin and a sulfonylurea and who were unwilling or unable to add insulin, and “provasinab” would be used for angina in patients with untreatable multivessel coronary disease who were taking maximal doses of beta-blockers. Since internists report that they frequently read only the abstracts when reviewing the medical literature,23 we created abstracts describing trials of these drugs in which we varied the drug being tested, the trial’s methodologic rigor, and the funding source. For each drug, one trial had a high level of rigor, one had a medium level of rigor, and one had a low level of rigor. The features defining these levels were based on guidelines24,25 and on our experience in conducting randomized trials26,27 and in studying evidence-based drug-evaluation and prescribing practices28,29 (Table 1). Differences in methodologic rigor among the trials were consistent across drugs. All the trials had similar effect sizes, and statistically significant results.
We then added a variable describing the trial’s funding status. Each abstract included one of three variations: no funding source mentioned, funding by the National Institutes of Health (NIH), or funding by a pharmaceutical company, with financial involvement in that company on the part of the lead author. (Companies named in the disclosure statements were randomly selected from the top 12 global pharmaceutical enterprises.30)
Assigning one of the three conditions to each of the three variables (drug, study design, and funding source), we created 27 different abstracts describing a hypothetical new-drug trial. The abstracts, along with the study protocol, are provided in the Supplementary Appendix, available with the full text of this article at NEJM.org. The survey was pretested among 12 physicians certified in internal medicine.
The study was approved by the institutional review board at Brigham and Women’s Hospital, with written informed consent implied by the participant’s completion of the survey. There were no agreements concerning confidentiality of the data between the authors and the institutions providing financial support for the study.
The American Board of Internal Medicine (ABIM) maintains a list of diplomates with active certification and maintenance of certification. The data set included demographic characteristics and information about medical training and responses to the ABIM Practice Characteristics Survey.31,32 From a potential sample of 45,398 physicians, we randomly identified 514 who reported spending 40% or more of their time and 21 hours or more per month in activities related to patient care and spending 50% or less of their time in the intensive care unit, the emergency department, or the cardiac catheterization laboratory. Of these possible participants, 11 had noncurrent contact information, resulting in a potential sample of 503 physicians.
Between July 2011 and October 2011, physicians in the sample received two postcards and three e-mail messages from ABIM indicating that they had been randomly selected to participate in a study investigating how physicians make prescribing decisions. These communications included the names of the sponsoring institutions and the lead investigators, a link to the online survey, an opportunity to opt out, and an offer of a $50 honorarium on completion of the survey (see the Supplementary Appendix). Physicians who did not respond were mailed a printed version of the invitation along with a $5 bill and an offer of $45 on completion of the survey. Those who still did not respond received a telephone reminder.
Participants were presented with three abstracts, each of which described a trial of a different hypothetical new drug. Participants were told to assume that the hypothetical drug had recently been approved by the Food and Drug Administration and was covered by insurance and that the abstract was from a “high-impact” biomedical journal and written by academic physicians at established U.S. universities. We randomly varied the level of methodologic rigor and the disclosure so that the three abstracts that the physicians received described trials at each level of methodologic rigor and with each disclosure variation.
Each abstract was followed by the same set of questions, with the choice of answers based on a 7-point Likert scale. Physicians were asked about their likelihood of prescribing the new drug (ranging from a score of 1 for “very unlikely to prescribe” to a score of 7 for “very likely to prescribe”), the level of methodologic rigor of the trial (ranging from a score of 1 for “not at all rigorous” [low rigor] to a score of 7 for “very rigorous” [high rigor]), and their confidence in the study investigators’ conclusions (ranging from a score of 1 for “not confident at all” to a score of 7 for “very confident”). Secondary outcomes were participants’ rating of the importance of the trial20 and their level of interest in reading the full article. (These questionnaires are provided in the Supplementary Appendix.)
After reading the third abstract, participants were asked to estimate how many abstracts describing trials of medications they had read in biomedical journals in the past month and to respond to the following question: “Do you think that pharmaceutical company funding is likely to influence the outcome of scientific studies about the efficacy and safety of pharmaceuticals in favor of the drug in question?” Finally, participants were asked about any financial support they had received from drug, device, or other medically related companies in the previous year.33
For each question, we estimated a hierarchical proportional-odds regression model, using the appropriate Likert-scale response as the outcome. This model included a random intercept for each physician to account for within-physician correlation of responses across abstracts, as well as fixed effects for methodologic rigor (low, medium, or high), disclosure statement (industry funding, NIH funding, or no statement), and drug. A second model included interactions between the indicators for funding source and methodologic rigor.34 Because we developed the three hypothetical drugs solely to obtain data about the effects of the level of methodologic rigor and variation of disclosure, we did not separately analyze differences among the drugs.
To investigate the association between characteristics of the physicians and survey responses, we created proportional-odds models that included random intercepts for physicians; indicators for the variables describing methodologic rigor, type of disclosure statement, and drug; and terms for physician characteristics (i.e., age, sex, location of medical school [U.S. vs. non-U.S.], type of practice [general internal medicine vs. subspecialty], proportion of time spent in clinical care [≥80% vs. <80%], hours per month spent in clinical care [≥80 vs. <80], acceptance of gifts from industry [any vs. none], and opinion about whether industry funding influences the outcome of trials in favor of the drug being tested [a score of 6 or higher vs. a score of 5 or lower on the 7-point scale, with higher numbers indicating a perception of a stronger influence]).
Of the 503 physicians who were invited to participate in the survey, 269 responded (a 53.5% response rate); 192 (71.4%) responded online (spending a median of 14 minutes), and 77 (28.6%) responded by returning a paper copy of the survey. We evaluated the responses of the 263 physicians who answered all the questions for one or more abstracts.
The demographic and practice characteristics of the respondents did not differ significantly from those of the nonrespondents (Table 2). Most of the respondents (75.5% [188 of 249]) reported accepting at least one type of industry support. Respondents generally agreed with the statement that industry funding could influence the outcome of clinical trials of pharmaceutical agents in favor of the drug in question (mean score, 5.4 [out of a possible 7.0]; 95% confidence interval [CI], 4.0 to 6.9). Responses to other questions are provided in the Supplementary Appendix.
We found strong associations between the methodologic rigor of the hypothetical trials and physicians’ perceptions of rigor. In comparisons with trials having a medium level of rigor, physicians were less likely to identify low-rigor trials as rigorous (odds ratio, 0.55; 95% CI, 0.40 to 0.76; P<0.001) (Fig. 1A) and were more likely to report high-rigor trials as rigorous (odds ratio, 3.95; 95% CI, 2.81 to 5.55; P<0.001). Similarly, in comparisons with medium-rigor trials, physicians had less confidence in the results of low-rigor trials (odds ratio, 0.48; 95% CI, 0.35 to 0.66; P<0.001) and more confidence in the results of high-rigor trials (odds ratio, 2.73; 95% CI, 1.95 to 3.82; P<0.001) (Fig. 1B) and would be less likely to prescribe drugs described in low-rigor trials (odds ratio, 0.64; 95% CI, 0.46 to 0.89; P=0.008) and more willing to prescribe drugs described in high-rigor trials (odds ratio, 3.07; 95% CI, 2.18 to 4.32; P<0.001) (Fig. 1C).
We found clear associations between the funding disclosure variations and physicians’ perceptions of a trial’s rigor and results. Regardless of the actual study design, physicians were less likely to view a trial as having a high level of rigor if funding by a pharmaceutical company was disclosed than if no disclosure statement was included (odds ratio, 0.63; 95% CI, 0.46 to 0.87; P = 0.006) (Fig. 2A). Similarly, in comparisons with trials for which no funding was listed and regardless of the study design, physicians were less likely to have confidence in the results of trials funded by industry (odds ratio, 0.71; 95% CI, 0.51 to 0.98; P = 0.04) (Fig. 2B) and were less willing to prescribe drugs described in such trials (odds ratio, 0.68; 95% CI, 0.49 to 0.94; P = 0.02) (Fig. 2C). These effects were even greater when industry-funded trials were compared with trials described as having NIH support.
In comparisons with NIH-funded trials, respondents were also less likely to describe industry-funded trials as “important” (odds ratio, 0.59; 95% CI, 0.42 to 0.82; P = 0.002), and were less likely to want to read the full article (odds ratio, 0.67; 95% CI, 0.47 to 0.95; P = 0.03). In comparisons between industry-funded trials and trials with no funding statement, the differences were smaller with respect to respondents’ ratings of the importance of the trial (odds ratio, 0.88; 95% CI, 0.63 to 1.22; P = 0.43) and their desire to read the full article (odds ratio, 0.88; 95% CI, 0.63 to 1.23; P = 0.46).
The reduction in physicians’ willingness to prescribe drugs studied in industry-funded trials as compared with drugs studied in trials without funding disclosures was consistent across levels of methodologic rigor (P = 0.87 for high-rigor trials vs. medium-rigor trials and P = 0.83 for low-rigor trials vs. medium-rigor trials). The greater willingness to prescribe drugs described in NIH-funded trials was also constant across rigor levels (P = 0.81 for high-rigor trials vs. medium-rigor trials and P = 0.56 for low-rigor trials vs. medium-rigor trials). We found no interaction between methodologic rigor and disclosure with respect to variations in physicians’ perceptions of the rigor of a trial or their confidence in the results (data not shown).
Irrespective of the disclosure and level of methodologic rigor, physicians who strongly agreed that funding by pharmaceutical companies can influence the results of a trial (score of 6 or 7 on the Likert scale) were less likely to prescribe the hypothetical drugs than were physicians with lower scores (odds ratio, 0.58; 95% CI, 0.37 to 0.91; P = 0.02). For all the abstracts presented, U.S.-trained physicians were substantially less likely to say they would be willing to prescribe the hypothetical drugs than were physicians who were trained elsewhere (odds ratio, 0.44; 95% CI, 0.27 to 0.70; P = 0.001). Physicians who were older than the median age of 48 years for the sample were more likely than the younger physicians to report that they would prescribe the simulated drugs (odds ratio, 1.65; 95% CI, 1.04 to 2.61; P = 0.03). Adjustment for physicians’ characteristics did not alter the effect estimates of methodologic rigor or disclosure statement, indicating that the randomization effectively balanced these characteristics.
In this randomized study, we found that practicing internists understood and appreciated methodologic differences when they read abstracts describing hypothetical studies of new drugs. They discounted small, poorly designed trials and assigned greater validity to large trials that tested clinical end points. We also found that respondents downgraded the credibility of industry-funded trials, as compared with the same trials randomly characterized as having NIH funding or having no source of support listed. The magnitude of this reduction in perceived methodologic rigor was about the same as that for low-rigor trials as compared with medium-rigor trials. Physicians’ skepticism of industry-funded research affected their responses to high-rigor and low-rigor trials similarly.
Well-publicized controversies related to industry-funded research may help explain these findings. Reports have emerged of trials that with-held critical data35–37 or that presented positive results while withholding negative results.38,39 Other concerns stem from reports of industry-financed articles that were ghostwritten40 or published primarily as instruments of marketing.41 Physicians’ skepticism of industry-funded research may be a response to such trends.
These findings have important implications. Despite the occasional scientific and ethical lapses in trials funded by pharmaceutical companies, it is also true that the pharmaceutical industry has supported many major drug trials that have been of particular clinical importance.42 Excessive skepticism concerning trials supported by industry could hinder the appropriate translation of the results into practice. For example, after publishing the results of a large, well-designed trial describing a new use for a widely prescribed class of drugs,43 a leading biomedical journal noted that many of its readers believed that the results of the trial did not justify a change in clinical management, citing industry funding as a key reason for this conclusion.44
The results of our study were based on physicians’ responses to descriptions of hypothetical trials of three new drugs, each of which was described in a single trial with varying attributes, 45 whereas actual prescribing behavior integrates drug information from many sources. Nevertheless, prescribing decisions made when a drug is first approved may rely principally on a single published study, as presented in these scenarios. Our response rate was similar to the mean rate in published surveys of physicians,46 and our respondents were similar to other internists in terms of the characteristics we measured.47 However, unmeasured variables may have differed between respondents and nonrespondents, contributing to bias in our sample. Finally, the findings from this survey of board-certified internists may not be generalizable to other specialties.48
Pharmaceutical companies seeking to enhance the appropriate use of important new products or to expand the appropriate uses of existing products must address the attitudes that our survey revealed,49,50 so that the credibility of the results of industry-supported trials is more likely to be based on methodologic rigor than on funding sources. Exactly how to change such attitudes was not the subject of this research. Currently, journal reviewers and editors, those who conduct systematic reviews, or even interested physicians can refer to the ClinicalTrials.gov database to see whether trial data as reported reflect the planned study design.51 This retrospective check could alleviate concerns about the possibility that trial outcomes were changed after the data were gathered and analyzed. However, the information provided to this database may have missing values or may be of poor quality.51,52 We do not have empirical data that address whether concordance between the study design and the reporting of results influences physicians’ perceptions of methodologic rigor.
We found that physicians assigned the highest level of credibility to NIH-funded trials. Thus, an increase in the number of clinical trials funded by the NIH or by the new Patient-Centered Outcomes Research Institute might reduce clinicians’ skepticism and lead to more data-driven changes in practice.53 Despite the initial financial outlay, such publicly funded trials are likely to save more than they cost.54 Partnerships between the NIH and industry55 may also serve this purpose if their jointly funded trials feature characteristics that are a routine part of NIH-funded trials, including data and safety monitoring boards and public reporting of protocols.
It is reassuring that the physicians in our study were attentive to the level of methodologic rigor. They also clearly took notice of funding sources for trials, according greater credibility and import to NIH-funded research than to industry-funded research. Although attention to potential sources of bias is necessary, such skepticism apparently can also reduce the credibility and acceptance of even high-quality research that is industry-supported. Financial disclosure is important, but more fundamental strategies, such as avoiding selective reporting of results in reports of industry-sponsored trials, ensuring protocol and data transparency, and providing an independent review of end points, will be needed to more effectively promote the translation of high-quality clinical trials — whatever their funding source — into practice.
Supported by a grant from the Edmond J. Safra Center for Ethics at Harvard University, a career development award from the Agency for Healthcare Research and Quality (K08HS18465-01) and a Robert Wood Johnson Foundation Investigator Award in Health Policy Research (both to Dr. Kesselheim), a fellowship at the Petrie–Flom Center for Health Law Policy, Biotechnology, and Bioethics at Harvard Law School (to Dr. Robertson), and a grant from the National Cancer Institute (2R25CA092203-08, to Dr. Rose).
We thank the American Board of Internal Medicine Foundation for help with administrative support; Adam Licurse, Maureen Garmon, and David Yokum for their help in the research process; Lawrence Lessig and Christine Cassel for their comments on earlier drafts of the manuscript; and members of the Research Ethics Program of the Harvard Catalyst.
Disclosure forms provided by the authors are available with the full text of this article at NEJM.org.