|Home | About | Journals | Submit | Contact Us | Français|
To determine if there is evidence of a time-lag bias in the publication of pediatric antidepressant trials.
We conducted a meta-analysis of published and unpublished randomized, placebo-controlled trials of serotonin reuptake inhibitors (SRIs) in subjects less than 18 years old with major depressive disorder. Our main outcomes were (1) time to publication of positive versus negative trials, and (2) proportion of treatment responders in trials with standard (< 3 years after study completion) versus delayed publication.
We identified 15 randomized, placebo-controlled trials of SRIs for pediatric depression. Trials with negative findings had a significantly longer time to publication (median years ± standard deviation = 4.2 ±1.9) than trials with positive findings (2.2 ±0.9; log-rank χ2 = 4.35, p = 0.037). The estimated efficacy in trials with standard publication time (number needed to treat = 7, 95% CI: 5 – 11) was significantly greater than those with delayed publication (17, 95% CI: 9 – ∞; χ2 = 4.98, p = 0.025). The inflation-adjusted impact factor of journals for published trials with positive (15.33 ±11.01) and negative results (7.54 ±7.90) did not statistically differ (t = 1.4, df = 10, p = 0.17).
Despite a small number of trials of SRIs for pediatric antidepressants we found a significant evidence of time-lag bias in the publication of findings. This time-lag bias altered the perceived efficacy of pediatric antidepressants in the medical literature. Time-lag bias is not unique to child psychiatry and reflects a larger problem in scientific publishing.
Publication bias is a well-described form of bias that can affect the estimated efficacy of interventions.1, 2 Publication bias occurs when studies with positive results are published more frequently than those with unfavorable findings, thus creating an overrepresentation of efficacious findings in the medical literature.3 For example, previous research has demonstrated that only 51% of the antidepressant trials registered with the FDA had been positive.4 By contrast, as many as 94% of trials published in the peer-reviewed literature evaluating antidepressant agents were positive.4 A meta-analysis published in 2004suggested that this type of bias may have affected our estimates of the risk/benefit profile of antidepressant use in children.5 A larger meta-analysis conducted after the publication of several trials subsequent to the FDA Black Box warning indicated that this effect of publication bias may have dissipated.6 The problem of publication bias is by no means unique to psychiatry. A recent systematic review in the field of internal medicine suggests that trials with positive results were nearly twice as likely to be published compared to trials with negative results.7
Time-lag bias is another form of bias that can also affect perceived efficacy of interventions, although it has been much less well studied in the scientific literature. Time-lag bias occurs when the results of negative trials take substantially longer to publish than positive trials.8 For example, one study assessing efficacy trials of HIV treatments concluded that the time from study enrollment to publication was significantly longer for negative trials than that for positive trials.9 Likewise, of the phase 3, randomized controlled trials presented at the annual American Society of Clinical Oncology meetings, as many as 81% of those with positive findings were published within 5 years of presentation, whereas only 68% of negative trials were published within this time period.10 Individual medical practices are potentially vulnerable to research results that are readily disseminated and accessible to clinicians. Time-lag bias is a particularly important form of bias that creates an environment in which treatments may be inaccurately portrayed as efficacious in a shorter period of time, amidst the existence of negative, though not yet published data.
To our knowledge there have been no studies investigating time-lag bias in the child psychiatry literature. The purpose of this study was to examine time-lag bias in pediatric antidepressant trials. We examined published and unpublished randomized controlled trials of serotonin reuptake inhibitors (SRIs) for the treatment of major depressive disorder in children to determine (1) if time-lag bias existed (i.e. whether negative antidepressant trials took significantly longer to be published than positive trials), (2) if time-lag bias affected estimates of the efficacy of SRIs in the treatment of pediatric depression and (3) if trials with positive results were published in higher impact journals than those trials with negative findings.
Two reviewers (MMR and KEP) searched PubMed (between June 2009 and July 2009) for relevant studies using the search strategy: (serotonin uptake inhibitors (MeSH) or SSRI or citalopram or duloxetine or escitalopram or fluoxetine or fluvoxamine or paroxetine or sertraline or venlafaxine) AND (depressive disorders (MeSH) or depress* or dysthymi*). The search was further limited to randomized, placebo-controlled clinical trials, meta-analyses and reviews involving children (0–18 years). Randomized clinical trials were examined for eligibility for inclusion in this meta-analysis. The references of included articles, as well as review articles and meta-analyses in this area, were searched for citations of further relevant published and unpublished research. We further searched the U.S. Food and Drug Administration (FDA) Center for Drug Evaluation and Research online database on approved drug products (http://www.accessdata.fda.gov/Scripts/cder/DrugsatFDA) for additional trials in “Approval History, Letters, Reviews, and Related Documents” related to medications included in this review. We additionally searched the U.S. National Institutes of Health (NIH) clinical trials database (http://www.clinicaltrials.gov) searching with the following search terms: serotonin uptake inhibitors or SSRI or citalopram or duloxetine or escitalopram or fluoxetine or fluvoxamine or paroxetine or sertraline or venlafaxine combined with interventional studies, age group: child, and depression or dysthymia to identify additional eligible trials.
The titles and abstracts of studies obtained by this search strategy were scrutinized by two reviewers (MMR and KEP) to determine if they were potentially eligible for inclusion in this review. Eligibility for the study was based upon scrutiny of the full articles for the following inclusion criteria (1) they were randomized, clinical trials comparing an SSRI or serotonin-norepinephrine reuptake inhibitors (SNRI: duloxetine or venlafaxine) to placebo in the treatment of depression symptoms and (2) participants included were children and adolescents less than 18 years of age. The studies were excluded if they were unpublished, compared our antidepressants of interest with another active medication (i.e. another antidepressant agent, mood stabilizer, antipsychotic agent or stimulant) or if they were discontinuation studies. We also excluded placebo-controlled trials of other antidepressant agents such as 5-HT2 antagonists (mirtazapine, trazadone or nefazodone), tricyclic antidepressants, monoamine oxidase inhibitors and buprioprion in order to eliminate as much noise from heterogeneity of effect between agents as possible.
Our first a priori analysis was whether time to publication differed between trials with positive versus negative outcomes. A trial was considered positive when any of the primary outcomes defined in the study manuscript demonstrated a statistically significant benefit of a medication compared to placebo. We used the Kaplan-Meier life-table method of survival analysis to analyze these results using the LIFETEST procedure command in SAS 9.2. Time to publication was computed in months by subtracting publication date of a manuscript from the date of study completion. The date of study completion was identified from date reported (in order of preference) from (1) published manuscript, (2) FDA report,11 (3) NIH clinical trials online database study completion date (http://www.clinicaltrials.gov), (4) FDA Center for Drug Evaluation and Research online database (http://www.accessdata.fda.gov/Scripts/cder/DrugsatFDA) (5) company website or (6) correspondence with the primary author or principal investigator of a study.
Our second a priori hypotheses for this meta-analysis was to determine if the estimation of antidepressant efficacy between pediatric antidepressant trials which were published after a standard amount of time (less than 3 years after study completion) was different when compared to trials with delayed publication (published greater than 3 years after study completion). The threshold of 3 years to differentiate standard versus delayed publication was chosen based on a median split of time to publication in included trials. The decision about the point of stratification for this outcome was made prior to conducting any analyses or examining individual study results. Our primary outcome measure was proportion of treatment responders using the criteria chosen by the study authors. When a primary criterion of response was not chosen by the study author we used those most commonly selected in other manuscripts such as: clinical global improvement score of less than 3 or a reported % reduction in the Children’s Depression Rating Scale (CDRS), Montgomery-Ashberg Depression Rating Scale (MADRS) or Hamilton Depression Rating Scale (HAM-D). We used absolute risk of response as our summary statistic for this analysis. We chose absolute risk difference (ARD: proportion of treatment responders in the medication group – proportion of treatment responders in the placebo group) and number needed to treat (NNT) as our summary outcomes. NNT is defined as 1/ARD. The confidence interval for NNT is defined by the reciprocals of the confidence interval for absolute risk difference. We chose ARD and NNT as our summary outcome measures (rather than that of a continuous measure such as effect size) because we wanted to use the most clinically salient measures of outcome. We used a random effects model in Review Manager (RevMan) version 5 to analyze these outcomes.12 For our main comparisons we stratified by whether they had a standard or delayed time to publication. We used the chi-square test for differences between subgroups to investigate if the difference between these two subgroups was significant.13 We computed the improvement in the chi-squared statistic of heterogeneity of studies when studies were stratified into subgroups based on time to publication compared to if all trials were categorized together (Qimprovement = Qnosubgroups −(Qstandard + Qdelayed)). The improvement in the chi-squared statistic of heterogeneity (Qimprovement) was then examined using a chi-squared distribution (df = 1).
Results from all the published trials were entered into a funnel plot (trial effect size plotted against sample size) to detect any evidence of additional publication bias.14 Heterogeneity of treatment response was assessed from the forest plot of absolute risk of response for individual studies. Statistical estimates of heterogeneity were performed using the I-square heterogeneity statistic in RevMan.12 Since the I-square test has low power to detect heterogeneity in a meta-analysis that has few trials with small sample sizes, the threshold for statistical significance was set at p < 0.1. This threshold for significance using the I-squared test is conventional in a meta-analysis. When heterogeneity was present between trials, differences in duration of trial length, patient population and antidepressant agent used were examined.
We conducted additional stratified sensitivity analyses to examine the effects of study quality as rated by the Quality Rating Scale, number of study sites and duration of study recruitment on response rates to pediatric antidepressants.15 Since these analyses were conducted post-hoc, we divided the studies based on a median split of eligible studies for each of these analyses. We conducted an additional sensitivity analysis to examine whether publication of trials before or after the black box warning was associated with response rates to pediatric antidepressants. We used the chi-square test for differences between subgroups to investigate if the difference between subgroups was significant for all these analyses.13
In order to determine whether trials with significant results (as opposed to those with non-significant findings) and trials with standard publication (as opposed to articles with delayed publication) were published in higher impact medical journals we examined journal impact factor. In order to account for impact factor inflation that occurs in medical journals, we utilized an equation from economics used to determine the time value of money adjusting for inflation. Impact factor values were adjusted for inflation based on the following equation:
Where x equals the inflation adjusted impact factor in 2009, y is the impact factor of the journal in year of publication at the time of publication, and z is the year of publication. The value 1.039 was derived from the estimated rate of inflation for psychiatry journals according to previous studies in the area (3.9%). 16 An unpaired 2-sided t-test was used to evaluate the difference in inflation-adjusted impact factors for significant versus non-significant studies and trials with standard versus delayed publication times. When two trials were published within the same article the article was counted only once.
The results from a small, pilot trial of fluoxetine were discussed in a longitudinal follow-up of study participants and several review articles but were never formally published. 29 We, therefore, did not include this trial in the meta-analysis. Thirteen of the studies examined the efficacy of SSRIs for the treatment of pediatric depression (2- citalopram, 2-escitalopram, 4-fluoxetine, 3-paroxetine, and 2-sertraline). The other 2 studies examined the efficacy of venlafaxine ER. Overall, 4 of the 15 trials reported statistically significant positive results (27%) and the remaining 11 trials were non-significant (73%).30 Table 1 describes the characteristics of the included studies. In one manuscript two negative trials were combined to demonstrate a positive result post-hoc.19, 30, 31
Time to publication was significantly longer in trials with negative results compared to those trials with positive findings. The median time to publication of trials with negative results was 4.2 ±1.9 years compared to 2.2 ±0.9 years in trials with positive results (log-rank test χ2 = 4.35, p = 0.037; Figure. 2). A similar disparity existed in the mean publication time between trials with negative and positive findings (4.4 ±0.6 years and 2.3 ±0.5 years, respectively).
The impact factor for trials with positive results (mean +/− standard deviation: 15.33 +/− 11.01) and negative results (7.54 +/− 7.90) did not statistically differ (t = 1.4, df = 10, p = 0.17). When we compared trials that were published within 3 years of trial completion (standard publication time) compared to those that were not (delayed publication), trials with standard publication time tended to be published in higher impact journals although this trend also did not reach statistical significance (t = 2.3, df = 6.2, p = 0.06). The trials with standard publication time had a mean impact factor of 14.07 +/− 10.79 compared to 4.64 +/− 1.05 in trials with delayed publication. Non-parametric tests of the impact factor analysis were also not statistically significant.
Figure 3 depicts the forest plot of this meta-analysis comparing response rates in trials with a standard publication time compared to those with delayed publication time. Trials with a standard publication time had a significantly higher estimated efficacy of pediatric antidepressants than those with a delayed publication time (χ2 = 4.98, df = 1, p = 0.025). The number needed to treat (NNT) in trials with standard publication was 7 (95% CI: 5 –11) compared to 17 (95% CI: 9 – ∞) in trials with delayed publication. Overall, the NNT for response of depression was 10 (95% CI: 8 – 17).
Trials which employed less than 25 study sites (NNT=7, 95% CI: 5–11) showed a greater effect of pediatric antidepressants than those trials which used more than 25 study sites (NNT=17, 95% CI: 7–∞, χ2 = 4.9, df = 1, p <=0.025). Studies judged to be of higher quality (QRS score≥34: NNT=8, 95% CI: 6–14) tended to show a larger effect of pediatric antidepressants than lower quality studies (NNT=17: 95% CI: 8–100) but this stratified analysis failed to reach our threshold for statistical significance (χ2 = 2.93, df = 1, p = 0.09). We found no association between study recruitment time and response rates of pediatric antidepressants (χ2 = 0.60, df = 1, p = NS). Trials published before the FDA Black Box Warning (NNT=8: 95% CI: 5–13) was issued in October 2004 tended to show a greater effect of pediatric antideprssants compared to trials published after the Black Box Warning (NNT=14: 95% CI: 8–50) but this difference failed to reach statistical significance (χ2 = 3.05, df = 1, p = 0.08)..
In this meta-analysis we demonstrate a significant time-lag bias in the publication of trials of SRIs for the treatment of pediatric depression. The median time to publication of trials with negative results was nearly double that of trials with positive results. Furthermore, there was evidence that time-lag bias distorted the estimated efficacy of study medications. The NNT to induce treatment response using SRIs for pediatric depression was more than double that in trials with delayed publication compared to those with standard publication time.
The problems of time-lag bias and publication bias are by no means unique to child psychiatry. It is well established in the general medical literature that trials with negative findings are published less frequently than their positive counterparts.4, 7 Time-lag bias and publication bias might occur at the level of the investigator (i.e. individuals or companies that may be less likely to submit trials with undesirable findings),32, 33 or during the publication process (i.e. journal editors may be more inclined to publish trials with positive results that are more likely to be cited in the future, or reviewers may be more interested in positive trials that may change clinical practice).
There was no statistically significant difference between journal impact factors of positive versus negative studies. Likewise, no statistical difference was detected between studies published in a standard and delayed time period. Although we did not determine a statistically significant difference in our analysis, there is a substantial absolute difference in impact factor values that should be acknowledged. In addition, one manuscript with positive results published in a high impact journal had evaluated data from two negative trials.19 These trials were represented separately in our analysis and the high impact factors associated with each likely inflated the variance in the negative trial group. The observation that many of the negative trials were published in journals with a substantially lower impact factor than positive trials highlights the potential importance of the publication process in contributing to time-lag bias. We, however, do not believe it is plausible that the peer review process alone explains the greater than 3 year difference in the publication time of positive compared to negative SRI trials for pediatric depression.
Several recent steps taken by Congress to increase the authority of the US Food and Drug Administration (FDA) should make the issues of bias less of a problem for clinical trials in the future. The FDA Amendments Act of 2007 included a mandate requiring all clinical trials be registered and accessible through www.clinicatrials.gov.34, 35 Furthermore, the results of trials must be updated every 12 months. As it is unclear to what extent clinicians will utilize www.clinicaltrials.gov, much of daily medical practice may continue to rely on peer-reviewed medical publication. Therefore, investigators, sponsors and journals have a shared responsibility to ensuring that submitted articles provide data on recently performed trials to prevent a systematic time-lag bias in the literature.35
Despite advances in the regulation of the reporting of biomedical research, bias continues to be an issue and the current regulatory policies are still with limitations. A recent study found that less than half of studies reported in high impact factor medical journals were adequately registered.36 Furthermore, of those adequately registered, one third had discrepancies between the outcome measures initially registered and those actually reported in the peer-reviewed literature.36 Thus, selective reporting is yet another form of bias than can manifest despite newer regulations. It should also be noted that the FDA Amendment Act only covers the reporting of future compounds and not those that have already received approval.34, 37 Therefore, the established perceived efficacy of several other classes of medications, widely prescribed in the practice of child psychiatry (such as antipsychotics and psychostimulants), may be influenced by the fact that the availability of trial results to clinicians and researchers is limited. In light of the important findings of our meta-analysis, it is important to recognize its limitations. Only 14 double-blind, placebo-controlled trials examining the efficacy of SRIs for pediatric depression were included in this meta-analysis. This small number of trials limited our power to detect significant differences. When examining whether time to publication influenced the perceived efficacy of SRIs for pediatric depression we used proportion of treatment responders as well as absolute risk difference and NNT as our primary outcome measure. We chose treatment response as our primary outcome because it is an important outcome measure to clinicians and almost universally reported in antidepressant studies. However, in many cases, this dichotomous outcome measure differed from the primary outcome reported in the peer-reviewed publication. Trial investigators were also not surveyed to determine the factors that influenced each trial’s time to publication and therefore we cannot ascertain the exact causes responsible for the delay in publication of negative articles.
Additionally the small number of trials included in this study limited our ability to examine the effects of potential confounders on the relationship between study outcome (or response rate) and time to publication. Potential confounders include funding source and study quality. Funding source represents a particularly important potential confounder in the case of pediatric antidepressants. In 1997, Congress passed the Food and Drug Administration Modernization Act (FDAMA) 38. Section 505A of FDAMA, known as the Pediatric Exclusivity Provision, provided an additional 6 months of patent protection, or marketing exclusivity, in return for performing studies specified by the FDA. Obtaining additional patent exclusivity thus did not depend on showing significant benefit of the medication, but rather, simply on having conducted the study according to the terms of a written request from the FDA. Thus the quality of many of the industry funded studies of pediatric antidepressants has been called into question because they may have been hastily done or included many sites without appropriate standardization and training. Of note, the only 2 NIH funded trials in this meta-analysis both demonstrated a significant benefit of antidepressants and were published within a standard period of time. By contrast only 2 of 13 industry funded studies showed a significant benefit of medications and only 5 of 13 industry-funded studies were published in less than 3 years. Furthermore, sensitivity analysis suggested that trials with lower quality and with greater than 25 sites showed a smaller effect of pediatric antidepressants. Although, the former analysis did not reach our threshold for statistical significance. A previous meta-analysis demonstrated that a greater number of study sites was associated with a higher placebo response rate in pediatric antidepressant trials.39
Despite this potential confounding effect by funding source, the problem of time-lag bias is likely not a uniquely industry asspociated problem. Although private companies clearly have a profit motive to suppress or delay publication of trials with negative results, non-industry funded investigators may have similar incentives.. These incentives may be caused by a perceived decreased ability to get funding for future trials (either industry or non-industry based) or for related research grants Additionally, investigators may be slower in publishing negative studies b cuase of decreased enthusiasm or a decreased ability to publish the results in high impact journals. Further research should examine the interaction between funding source and time-lag bias in psychiatric clinical trials.
Our study demonstrates the presence of a significant time-lag bias in the publication of SRI trials for pediatric depression. Trials with positive results were published significantly faster than trials with negative ones. Data also suggest that time-lag bias altered the perceived efficacy of these medications for pediatric depression. The situation for pediatric antidepressant trials is rather unique in psychiatry. Concerns related to the potential association between pediatric antidepressant use and suicidality caused pharmaceutical companies to be under tremendous pressure from the FDA, Congress and the State of New York to publish the results of previously unreported trials. Furthermore, the subsequently issued FDA Black Box Warning highlighted the existence of many negative pediatric antidepressant even before they were published. These factors undoubtedly caused some pediatric antidepressant trials that otherwise would have remained suppressed to get published. Unfortunately, there may still exist vast areas in psychiatry and medicine where the results from negative trials remain unpublished or are delayed. Such bias has the potential to effect expert opinion, meta-analyses, practice guidelines and clinical practice and should be examined for presence in other psychiatry and medical fields.
The authors acknowledge the support of the Klingenstein Third Generation Foundation Donald J. Cohen Medical Student Fellowship Program (MMR, MHB and AM), the National Institute of Mental Health support of the Yale Child Study Center Research Training Program (MHB).
Disclosures: Dr. Bloch receives research support from the National Institutes of Health Loan Repayment Program, the APIRE/Eli Lilly Psychiatric Research Fellowship, the American Academy of Child and Adolescent Psychiatry / Eli Lilly Pilot Research Award, the Trichotillomania Learning Center, and the National Alliance for Research on Schizophrenia and Depression. Dr. Martin receives royalities from Lippincott Williams and Wilkins. Ms. Reyes and Ms. Panza report no biomedical financial interests or potential conflicts of interest.