PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jmirEditorial BoardMembershipSubmitCurrent IssueArchiveReviewSearchAboutJMIR PublicationsJMIR
 
J Med Internet Res. 2004 Oct-Dec; 6(4): e46.
Published online 2004 December 22. doi:  10.2196/jmir.6.4.e46
PMCID: PMC1550629

A Comparison of Changes in Anxiety and Depression Symptoms of Spontaneous Users and Trial Participants of a Cognitive Behavior Therapy Website

Helen Christensen, PhD,corresponding author1 Kathleen M Griffiths, PhD,1 Ailsa E Korten, BSc,1 Kylie Brittliffe, BA(Hons),1 and Chloe Groves, BA(Hons)1
Helen Christensen, Centre for Mental Health Research, The Australian National University, Canberra ACT, 0200, Australia, Phone: +61 2 6125 8409, Fax: +61 2 6125 0733, Helen.Christensen/at/anu.edu.au.
Reviewed by John Powell

Abstract

Background

In randomized controlled trials Internet sites have been shown to be effective in the treatment of depression and anxiety. However, it is unclear if the positive effects demonstrated in these trials transfer to community users of such sites.

Objective

To compare anxiety and depression outcomes for spontaneous visitors to a publicly accessible cognitive behavior therapy website (MoodGYM) (http://moodgym.anu.edu.au) with outcomes achieved through a randomized controlled efficacy trial of the same site.

Methods

All community visitors to the MoodGYM site between April 2001 and September 2003 were sampled: 182 participants in the BlueMood Trial who had been randomly assigned to the MoodGYM site as part of a large trial and 19607 visitors (public registrants) to the site. Symptom assessments (quizzes) were repeated within the website intervention to allow the examination of change in symptoms across modules. Outcome variables were (1) age, gender, initial depression severity scores, and number of assessments attempted, and (2) symptom change measures based on Goldberg anxiety and depression scores recorded on a least two occasions.

Results

Public registrants did not differ from trial participants in gender, age, or initial level of depression, which was high for both groups relative to previously published epidemiological data sets. Trial participants completed more assessments. No significant differences in anxiety or depression change scores were observed, with both public registrants and trial participants improving through the training program.

Conclusions

Public registrants to a cognitive behavior therapy website show significant change in anxiety and depression symptoms. The extent of change does not differ from that exhibited by participants enrolled on the website for a randomized controlled trial.

Keywords: Internet, mental health, depression

Introduction

The Internet is increasingly seen as a resource to disseminate self-help and clinician-based mental health interventions. To date, there have been 6 published trials evaluating the delivery of cognitive behavior therapy (CBT) using the Internet for anxiety and depression [1-6], of which 4 reported positive outcomes [1,2,4,5]. As yet, however, there has been no assessment of whether these evaluated websites are as useful for public users as for trial participants. The experience of a user of an open website is less structured than the experience of a participant in a randomized controlled trial (RCT). For example, RCT participants are encouraged by trial managers, either in person or by phone. They are tracked and monitored over the trial period, and may be called to complete questionnaires over the phone or on the Internet. These aspects of the intervention may be responsible for the efficacy of these sites, either directly through personal support provided to the participant, or indirectly through the greater adherence that results from this support.

In this paper, we compare the mental health outcomes of public registrants and trial participants using the MoodGYM website, a cognitive behaviour therapy website [1, 7]. Public registrants directed themselves to the site using search engines and links from relevant Web pages. Trial participants were Internet users in the community in Canberra, Australia, who had elevated depression symptoms. Previous research on the site has demonstrated the effectiveness of the site in an RCT (subsequently referred to as the "BlueMood Trial") [1] and change in symptoms by site users [7]. However, no previous investigation has directly compared treatment outcomes from the two samples, or evaluated factors of age or initial depression severity in treatment outcome.

Methods

Sample

All participants, except 71 students (who were recruited via tutorial) were recruited from among MoodGym registrants between April 2001 and September 2003 [7]. Registrants are individuals who enroll on the site (by providing a name and password) and who create a record in our database. The BlueMood trial participants were recruited by survey and subsequently randomized to the MoodGYM website as part of a three arm trial (the other arms were a psycho-education intervention and an attention placebo condition; see [1]). Trial participants scored more than 12 on the Kessler Psychological Distress Scale [8] and were not receiving clinical care from either a psychologist or a psychiatrist at the time of recruitment. The latter requirement was imposed so that the trial included individuals who were the target audience for self-help interventions and excluded individuals who might already be receiving CBT. Because in this paper our central goal was to evaluate change in symptom scores over the course of the MoodGYM training program, only those individuals from both samples who completed more than one assessment on the website were included.

There were 19789 people registered between April 2001 and September 2003, of whom 182 were participants in the BlueMood RCT trial, and 19607 were members of the public who had registered online. Among the public registrants, 12141 (61.9%) completed at least one quiz, but only 3055 (15.6%) completed at least 2 of the Goldberg Depression Scales. Among the BlueMood RCT trial participants, 157 (86.3%) completed at least one quiz, while 121 (66.5%) completed 2 or more of the Goldberg depression assessments.

That is, 3176 participants provided sufficient data to allow change in depression scores to be assessed (see Table 1). Change in Goldberg anxiety score could be assessed for 1668 people, all of whom had completed 2 or more anxiety scales. Appendix A provides exact participant numbers for the major analyses.

Table 1
Mean (SD) and n of the Goldberg Depression and Goldberg Anxiety Scales for the first module

Site Description

The site consists of a set of 5 cognitive behavioral training modules, a personal workbook (containing 29 exercises and assessments) that records and updates each user's responses), an interactive game, and a feedback evaluation form. Module 1 introduces the site “characters” (who model patterns of dysfunctional thinking) and demonstrates the way mood is influenced by thinking, using animated diagrams and interactive exercises. Module 2 describes types of dysfunctional thinking and the methods to overcome them, and provides scales for self-assessment of “warpy” (dysfunctional) thoughts. Module 3 provides behavioral methods to overcome dysfunctional thinking, and includes sections on assertiveness and self-esteem training. Module 4 assesses life-event stress, pleasant events, and activities, and provides three downloadable relaxation tapes. Module 5 covers simple problem-solving and typical responses to relationship break-up. Workbook exercises are integrated into each of the modules.

Each module was designed to take between 30 and 45 minutes to complete, but users can opt to skip sections. Module 1 has approximately 30 pages but many of these contain browser-supported interactive features (creating additional pages) and supplementary pop-up windows. Module 3 has more than 60 pages, but users are directed to specific sections depending on their scores on earlier tests and thus may not access all pages (see multimedia presentation [7]). Although users were encouraged to proceed through the assessments and modules in order, they were free to move about within the site at will. Thus, some registrants started with later modules and did not necessarily work through them in order; however, research previously conducted that matched log files to user profiles indicated that most individuals do approach the modules sequentially [7]. No assessments were compulsory in the original MoodGYM site. Since September 2003, however, a new version of the software has been installed (Mark II) that consists of compulsory core assessments and a requirement that modules be completed in order.

The MoodGYM site provides CBT for depression over 5 modules. Users complete online anxiety and depression scales in each of the 5 modules. This allows us to assess whether anxiety and depression scores change over the training program.

Assessments

Online assessments included the anxiety and depression items from the Goldberg Depression and Anxiety Scales [9]. Each of the Goldberg scales comprises 9 items. These assessments were introduced at the beginning of each of the modules, so it was possible for individuals to complete between 1 and 5 assessments. Other online assessments included the Warpy Thoughts Quiz, a 42-item scale that measured dysfunctional thinking [10]; the Life Whacks Questionnaire (adapted with permission from the Tennant & Andrews scale to measure the stress of life events) [11]; the Measure of Parenting Style (the MOPS, used with permission from G. Parker) [12]; and the Pleasant Events Schedule [13]. These additional assessments are not reported on in this paper. Age and gender variables were collected at registration, and initial level of depression was measured by Goldberg scales at the beginning of module 1.

Analysis Strategy

Differences in response rates were analyzed using logistic regression. Differences in symptom change scores were analyzed using analysis of variance (ANOVA) with repeated measures using SPSS software. Models predicting final depression score accounting for initial score were derived from linear regression models. SPSS 11.5 software was used [14].

Results

Demographic Variables

Two thirds of all registrants were female (66.1%). Among those registered, females were more likely than males to complete at least 2 Goldberg Depression Scales (odds ratio [OR] controlling for sample [public or trial] 1.24; 95% confidence interval [CI] 1.14-1.34), while there was no difference between the sexes for completion of at least two anxiety scales (OR 0.97; 95% CI 0.87-1.1). Appendix A provides full participant numbers.

Initial Depression and Anxiety Scores

Initial levels of depression, as measured by the Goldberg Depression Scale in module 1, are shown in Table 1. Differences between the samples were not significant (F[1, 10004]=1.25; P=.26), nor were any interactions between gender and number of modules completed (F[1, 10004]=2.661; P=.10) Those who completed more than one module had significantly higher scores (F[1, 10004]=15.7; P<.001). Appendix B provides full details of analysis.

Initial levels of anxiety are shown in Table 1. Differences between the samples were not significant, nor were any interactions between sample, gender, or number of modules completed (see Appendix B).

Change in Depression and Anxiety Scores

For 3176 people, it was possible to compare the initial and final completed depression scales, although there were only 138 for whom the profile across all 5 modules could be traced. The change in depression score between the first and last modules attempted was analyzed using repeated measures ANOVA controlling for sample, gender, number of modules completed (1 to 5), and all interaction terms. The sample numbers are presented in Appendix C. Sample (public or trial) was not significant, nor were any of the interaction terms. A final model included gender and number of modules completed; the parameter estimates are shown in Table 2.

Table 2
Parameter estimates for prediction of first and last Goldberg depression and Goldberg anxiety scores completed as derived from repeated measures ANOVA with factors being gender and number of modules completed

Improvement increased as the number of modules completed increased from 2 to 4, but there was no significant difference in estimated improvement between 4 and 5 modules completed. Females had higher depression scores than males for the first module, but there were no differences between the sexes for the final depression score. Figure 1 shows the estimated marginal means for the first and final modules completed, by gender and number of modules completed.

figure jmir_v6i4e46_fig1
Estimated means with standard error bars for initial and final scores on Goldberg Depression Scale, by gender and number of modules completed, as derived from repeated measures ANOVA with fixed factors being gender and number of modules completed

For 1668 people, it was possible to compare the first and last completed anxiety scales, although there were only 76 for whom the profile across all 5 modules could be traced. The change in anxiety score between the initial and final modules attempted were analyzed using repeated measures ANOVA controlling for sample (public or trial), gender, number of modules completed (1 to 5) and all interaction terms. Neither sample type nor any of the interaction terms was significant.

A final model included gender and number of modules completed; parameter estimates are shown in Table 2. Improvement increased as the number of modules completed increased from 2 to 4, but there was no significant difference in estimated improvement between 4 or 5 modules completed. Females had higher scores than males. Figure 2 shows the estimated marginal means for the first and final modules completed, by gender and number of modules completed.

figure jmir_v6i4e46_fig2
Estimated means with standard error bars for initial and final scores on Goldberg Anxiety Scale, by gender and number of modules completed, as derived from repeated measures ANOVA with fixed factors being gender and number of modules completed

Predicting Depression Scores at End of the Intervention

We also examined the expected magnitude of the final score for different starting levels of depression. In a linear regression with dependent variable equal to the final score and independent variables gender, number of modules (treated as 3 dummy variables), initial score, and a quadratic term in the initial score, all independent variables and the interaction between initial score and number of modules were significant. Males are expected to be 0.193 (SE = .095) higher than females, controlling for the initial level and number of modules. For initial scores above 2, it is expected that the final score will indicate improvement, the improvement increasing with the number of modules attempted. Figure 3 illustrates these relationships for the Goldberg Depression Scale.

figure jmir_v6i4e46_fig3
Estimated final Goldberg depression score by initial score, number of modules completed, and gender. Derived from general linear model with final score as dependent variables, gender, and number of modules completed as fixed factors, and linear and quadratic ...

Discussion

The present study evaluated the outcomes of public registrants using MoodGYM delivered openly on a website by comparing them with trial participants in a RCT using the same site. We found that there were no differences in the initial level of depression or anxiety in the samples, that the gender balance did not differ significantly, and that the rate of change in symptoms was not different for the two samples. Both public registrants and trial participants exhibited significant improvement in symptoms.

Our findings suggest that MoodGYM can be used effectively in the community. If we had found stronger effects in the BlueMood Trial participants, we might have concluded that structured weekly human contact was a necessary condition for the site to deliver effective outcomes. However, we conclude that community users of the site are similar in gender distribution and severity of depression, and that site exposure results in similar significant symptom improvement.

Although there were no differences in outcome among participants who stayed in the program and completed at least 2 modules, we did find that community users were less likely than the trial participants to adhere to the full treatment program. Thus, the community users infrequently progressed beyond the first module of the program. Among the public, only 15.6% completed 2 or more of the modules, while over 66% of the trial participants completed 2 or more of the modules. This finding suggests that the formal structure of the trial may be important for compliance. Merrill et al reported a similar finding for clinic-based CBT: community clients attended substantially fewer sessions than clients in RCTs, yet “still showed similar levels of improvement” [15]. Only 15% completed 2 or more modules, a finding that may reflect the usability of the site, the acceptability of CBT type interventions, commitment to change, symptom level, user preference, or other factors. Factors such as these are likely to influence the uptake of non-Internet-based services as well, although comparative published data on potential users of standard health care services are not readily available. The low completion rate is not a major problem for free Internet services, which do not have specific costs based on user numbers (few additional costs are incurred for large numbers of non-completers).

There are limitations to the present study that need to be acknowledged. As is the case for studies using a benchmarking strategy, the samples involved are likely to be quite different given the operation of selection biases. The trial participants were Canberra-based, and from a population sample with high education and occupation levels. In contrast, the user sample is international, with participants from more than 62 countries. There was selective attrition, with many participants from the public sample dropping out before completion, and greater retention for the trial sample (although there were many opportunities for trial participants to be excluded or to drop out before randomization). These sample differences are difficult to characterize. However, they should be acknowledged as having the potential to mask differences in outcomes. Other sample characteristics, such as the concurrent use of evidence-based treatments other than psychotherapy (antidepressants, other medications, physical activity), were not measured but did not preclude participation in either sample.

The possibility cannot be ruled out that community users were assisted in the program by clinicians or other counselors. If this occurred for the majority of community users, our claim that the trial participants were the subject of greater assistance/support would be invalid. However, we consider it highly unlikely that the majority of public participants were assisted by a counselor or other person.

Clinicians and researchers have argued that although treatment efficacy needs to be established, it is crucial to demonstration of the effectiveness of treatments in real world settings [15]. Demonstrations of real-world effectiveness often employ benchmarking strategies where RCTs are chosen to compare results to community settings [16]. We have employed this strategy in the present study using outcomes from our own RCT. Unlike benchmarking in clinical settings where there can be considerable flexibility in application of the clinical therapy, Internet sites have the advantage of transferring the treatment with fidelity, so that differences that may exist between the trial and the real-world site can be more reliably attributed to external factors such as degree of human contact, capacity to maintain compliance, sample characteristics, and intensity of monitoring.

The findings of the present study demonstrate the effectiveness of Internet-based interventions in the early treatment and prevention of depression. Health systems in developed countries are expected to change radically over the next 10 years, with self-help and self-responsibility for health forming a new tier of the health system [17]. Sites such as MoodGYM are likely to provide both tools for the self-delivery of evidence-based prevention/treatment and resources to be used as adjuncts to professionally managed primary care.

Acknowledgments

This research was supported by a Program grant from the National Health and Medical Research Council of Australia. We wish to thank Mr David Berriman and Mr Richard Pass for their contribution to this research.

Abbreviations

CBT
Cognitive behavior therapy
RCT
Randomized controlled trial

Appendix A

Table 3

Participant numbers for all analyses §

PublicBlueMood trialTotal
Total number registeredMales6640 (33.9%)52 (28.6%)6692 (33.8%)
Females12967 (66.1%)130 (71.4%)13097 (66.2%)
Total19607 (100%)182 (100%)19789 (100%)
Completed at least one quizMales3823 (31.5%)40 (25.5%)3863 (31.4%)
Females8318 (68.5%)117 (74.5%)8435 (68.6%)
Total12141 (100%)157 (100%)12298 (100%)
Completed 2 or more Goldberg Depression Scales*Males918 (30.0%)30 (24.8%)948 (29.8%)
Females2137 (69.9%)91 (75.2%)2228 (70.1%)
Total3055 (100%)121 (100%)3176 (100%)
Completed 2 or more Goldberg Anxiety ScalesMales537 (34.6%)30 (25.6%)567 (34.0%)
Females1014 (65.4%)87 (74.4%)1101 (66.0%)
Total1551 (100%)117 (100%)1668 (100%)
Completed both Warpy Thoughts quizzesMales34 (34.7%)10 (25.0%)44 (31.9%)
Females64 (65.3%)30 (75.0%)94 (68.1%)
Total98 (100%)40 (100%)138 (100%)
Completed Life Whacks quizMales268 (36.2%)20 (20.6%)288 (34.4%)
Females473 (63.8%)77 (79.4%)550 (65.6%)
Total741 (100%)97 (100%)838 (100%)
Completed Mum and Dad quizMales107 (39.2%)27 (26.5%)134 (35.7%)
Females166 (60.8%)75 (73.5%)241 (64.3%)
Total273 (100%)102 (100%)375 (100%)
Completed Pleasant Events quizMales103 (37.2%)26 (24.8%)129 (33.8%)
Females174 (62.8%)79 (75.2%)253 (66.2%)
Total277 (100%)105 (100%)382 (100%)

* Among participants who completed at least 1 quiz, all but 125 completed at least 1 Goldberg Depression Scale.

§ 1370 people completed the quizzes for module 2 and 566 completed them for module 5.

Appendix B

Linear regression model with initial levels of depression and anxiety as independent variables for participants who completed at least 2 modules

Table 4

Tests of between-subject effects (depression)

SourceType III Sum of SquaresdfMean SquareFP
Corrected model255.616551.1238.817<.001
Intercept11555.687111555.6871993.061<.001
Gender6.40216.4021.104.29
Module number*91.030191.03015.700<.001
Sample7.26517.2651.253.26
Gender xmodule number15.431115.4312.661.10
Gender x sample22.536122.5363.887.049
Error57968.00699985.798

* Module number refers to whether participants had at least 2 completed tests for either the depression or anxiety tests.

Table 5

Tests of between-subject effects (anxiety)

SourceType III Sum of SquaresdfMean SquareFP
Corrected model582.699783.24313.435<.001
Intercept6817.01716817.0171100.211<.001
Gender5.64015.640.910.34
Sample5.084E-0215.084E-02.008.93
Module number1.25911.259.203.65
Gender x sample2.24612.246.362.55
Gender x module number1.62911.629.263.61
Sample x module number9.467E-0219.467E-02.015.90
Error55851.61890146.196
Total339306.0009022
Corrected Total56434.3179021

Appendix C

Table 6

Number of people who completed at least 2 modules by the total number of depression modules completed

Number of Modules CompletedPublic Registrants
n
BlueMood Trial Participants
n
Total

n
Males26775682
31526158
4601272
529736
at least 291830948
Females21611181629
334723370
411116127
56834102
at least 22137912228
Total22288232311
349929528
417128199
59741138
at least 230551213176

Footnotes

Conflicts of Interest:

None declared.

References

1. Christensen Helen, Griffiths Kathleen M, Jorm Anthony F. Delivering interventions for depression by using the internet: randomised controlled trial. BMJ. 2004 Jan 31;328(7434):265. doi: 10.1136/bmj.37945.566632.EE. http://bmj.bmjjournals.com/cgi/content/full/328/7434/265. [PMC free article] [PubMed] [Cross Ref]bmj.37945.566632.EE
2. Klein B, Richards JC. A brief internet-based treatment for panic disorder. Behav Cog Psychother. 2001;29(1):113–117. doi: 10.1017/S1352465801001138. [Cross Ref]
3. Clarke Greg, Reid Ed, Eubanks Donna, O'connor Elizabeth, Debar Lynn L, Kelleher Chris, Lynch Frances, Nunley Sonia. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res. 2002 Dec 17;4(3):e14. http://www.jmir.org/2002/3/e14/ [PMC free article] [PubMed]
4. Richards J, Alvarenga M. Extension and replication of an internet-based treatment program for panic disorder. Cogn Behav Ther. 2002;31(5):41–47. doi: 10.1080/16506070252823652. [Cross Ref]
5. Kenardy J, Mccafferty K, Rosa V. Internet-delivered indicated prevention for anxiety disorders: a randomized controlled trial. Behav Cog Psychother. 2003;31(3):279–289. doi: 10.1017/S1352465803003047. [Cross Ref]
6. Patten Scott B. Prevention of depressive symptoms through the use of distance technologies. Psychiatr Serv. 2003 Mar;54(3):396–8. doi: 10.1176/appi.ps.54.3.396. http://ps.psychiatryonline.org/cgi/content/full/54/3/396. [PubMed] [Cross Ref]
7. Christensen Helen, Griffiths Kathleen M, Korten Ailsa. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res. 2002 Feb 15;4(1):e3. doi: 10.2196/jmir.4.1.e3. http://www.jmir.org/2002/1/e3/ [PMC free article] [PubMed] [Cross Ref]
8. Andrews G, Slade T. Interpreting scores on the Kessler Psychological Distress Scale (K10) Aust N Z J Public Health. 2001 Dec;25(6):494–7. [PubMed]
9. Goldberg D, Bridges K, Duncan-jones P, Grayson D. Detecting anxiety and depression in general medical settings. BMJ. 1988 Oct 8;297(6653):897–9. [PMC free article] [PubMed]
10. Christensen H, Griffiths K, Groves C. MoodGYM Training Program: Clinician's Manual. Canberra: Centre for Mental Health Research; 2004.
11. Tennant C, Andrews G. A scale to measure the stress of life events. Aust N Z J Psychiatry. 1976 Mar;10(1):27–32. [PubMed]
12. Parker G, Roussos J, Hadzi-pavlovic D, Mitchell P, Wilhelm K, Austin M P. The development of a refined measure of dysfunctional parenting and assessment of its relevance in patients with affective disorders. Psychol Med. 1997 Sep;27(5):1193–203. doi: 10.1017/S003329179700545X. [PubMed] [Cross Ref]
13. MacPhillamy D, Lewinsohn PM. The Pleasant Events Schedule: studies on reliability, validity and scale intercorrelation. J Consult Clin Psychol. 1982;3:363–380.
14. SPSS Inc, authors. SPSS(11.5) for Windows. Chicago, Illinois: SPSS; http://www.spss.com/spss.
15. Merrill Kari A, Tolbert Valerie E, Wade Wendy A. Effectiveness of cognitive therapy for depression in a community mental health center: a benchmarking study. J Consult Clin Psychol. 2003 Apr;71(2):404–9. doi: 10.1037/0022-006X.71.2.404. [PubMed] [Cross Ref]
16. Weersing V Robin, Weisz John R. Community clinic treatment of depressed youth: benchmarking usual care against CBT clinical trials. J Consult Clin Psychol. 2002 Apr;70(2):299–310. doi: 10.1037//0022-006X.70.2.299. [PubMed] [Cross Ref]
17. Coiera Enrico. Four rules for the reinvention of health care. BMJ. 2004 May 15;328(7449):1197–9. doi: 10.1136/bmj.328.7449.1197. http://bmj.bmjjournals.com/cgi/content/full/328/7449/1197.328/7449/1197 [PMC free article] [PubMed] [Cross Ref]

Articles from Journal of Medical Internet Research are provided here courtesy of Gunther Eysenbach