We detected a modest but statistically and clinically significant advantage for the two treatment conditions relative to the control group on self-reported depression, but not on functioning. To the best of our knowledge, this study is the first to find significant effects for a pure self-help or “unattended” Internet program, where the intervention was delivered without any adjunct person-to-person contact.
This study is also the first to find Internet intervention effects in the context of a TAU control condition. TAU was essentially another potentially active treatment, with 93% of participants receiving at least some traditional mental health care in the year following randomization (84% through the week 16 follow-up), the majority of which was antidepressant medication. This high background level of depression treatment and other health care had the potential to obscure differences between conditions. Nonetheless, we still observed an advantage for the ODIN intervention.
While the magnitude of this outcome was relatively modest, it compares favorably with other traditional, stand alone bibliotherapy interventions such as self-help books [3
]. More importantly, the potential public health implications of these findings are considerable. The low incremental costs of delivering this Internet program makes it feasible to offer this or similar programs to very large populations (health plans, large employer groups, universities) where Internet access is widespread. Interventions with a small average effect may have substantial public health impact when applied to a large number of people, as a modest but meaningful number of patients will not develop the target disorder as a result of this small, average improvement [30
Is the observed effect size of 0.277 standard deviation units (0.537 in cases with higher baseline depression) of sufficient magnitude to merit much enthusiasm? In meta-analyses of depression evidence-based psychotherapy efficacy
randomized controlled trials (where the control condition is typically an easily surmounted no treatment or waitlist control), the difference in effect size is typically much higher, averaging around 1.56 standard deviation units [31
]. However, when (as in this randomized controlled trial) the evidence-based psychotherapy is provided in the context of TAU [32
], this effect size advantage typically shrinks substantially. Gaffan [31
] and Gloaguen [33
] find only small to medium mean effect sizes favoring CBT when it is compared to behavioral therapy (0.27), “other” psychotherapy (0.23), or pharmacotherapy (0.27). In this context, our TAU control condition is best thought of as a blend of evidence-based and non-evidence-based psychosocial and pharmacotherapy treatments [34
]. Therefore, the observed effect size of 0.277 standard deviation units is roughly consistent with the effect sizes of this meta-analysis when traditional, face-to-face CBT is compared to these other treatments.
The mail and telephone reminders similarly increased the frequency of visits to the ODIN site, relative to our first study with no reminders [11
]. We are therefore inclined to use postcard reminders in the future because they are much less costly than telephone reminders.
Our failure to detect effects on health care utilization was not unexpected. A follow-up period of two years or more is typically needed to detect impacts of an intervention on health care utilization [35
]. Further, because health care utilization typically has very high variance (a small number of patients use an extreme amount of health care), very large samples are typically needed for adequate power [36
This study had several limitations. First, despite providing gift certificates for completed assessments, follow-up rates averaged around 66%—although 82% of participants completed at least one follow-up assessment. These rates are comparable to the follow-up rates obtained in our earlier study [11
] and are similar to, if not better than, rates seen in other Internet intervention trials (reviewed by Eysenbach [37
Second, subjects lost to follow-up were slightly more depressed, slightly older, and less likely to be in the control group. All these factors, but particularly the interaction between experimental condition and attrition, limit our confidence in our results, although post-hoc analyses suggest that confounding effects were unlikely to have accounted for the observed results.
Our enrollment rates were also quite low, with 3.3% of the “depressed” recruitment sample and 0.9% of the “nondepressed” recruitment sample enrolling in the study, respectively. We have no information on why so many declined to enroll. Because the majority of the “depressed” recruitment sample was receiving traditional depression care (all had depression diagnoses in their medical charts), perhaps they felt no need to augment their traditional care with our self-help program. Among the nominally “nondepressed” recruitment sample, we had hoped to enroll previously unrecognized cases of depression [38
]. However, the 1% “nondepressed” enrollment rate suggests that only a small minority of these undetected cases found our study of interest. Perhaps some of these individuals did not recognize their own depression and thus would not have seen the program as applicable. Still others may have been receiving other depression care outside of this HMO, which we could not know about from the HMO records. Regardless of the reasons for the low enrollment, these rates are not an indication of the acceptability
of this intervention or any Internet program offered outside of a research trial. The unique features of randomized trials (a chance of being assigned to the no-access control group, repeated reminders to complete assessments over time, burdensome questionnaires) create barriers to participants that likely contribute to lower research enrollment rates, but which have no counterparts in usual clinical care implementation of these types of programs.
This study was also limited by its reliance on a single, self-reported measure of depression. We decided against using research diagnostic interviews because the accompanying in-person or telephone interview contacts had the potential to impart quasi-therapeutic benefits that, in turn, might have swamped the small benefit expected from the ODIN intervention. Further, the target population for the ODIN website includes persons who may have low level or subdiagnostic depression symptoms, as well as individuals who meet full diagnostic criteria for major depression or other DSM mood diagnoses. Relying on DSM mood diagnosis as a primary outcome might have missed the effects of the ODIN intervention on depression symptoms below the level of a full diagnosis.
Finally, our follow-up period of 16 weeks was extremely brief. We must examine this intervention's longer term impacts on depression, health care utilization, and quality of life. Future studies should include a much longer follow-up and a broader range of assessment domains.
The lessons we have learned from this investigation are guiding our development of a completely new Internet intervention for depressed young adults. This new program emphasizes behavioral activation, or increasing pleasant activities, as the main therapeutic technique [39
We are encouraged by the results of this study, while acknowledging the positive effects are modest in magnitude. Nonetheless, we view low intensity, widely available interventions as an important piece of an overall, population-based strategy for reducing depression disorder and symptomatology. The marginal costs of delivering this pure self-help Internet program to each additional individual are very minimal, given that there is no staff time associated with the delivery of the intervention content. Therefore, it is feasible to offer this type of program to entire populations where Internet access is widespread, such as universities and large employers.