|Home | About | Journals | Submit | Contact Us | Français|
ObjectivesThis study evaluates the usability and feasibility of a Web-based intervention (Web-MAP) to deliver cognitive behavioral therapy (CBT) to adolescents with chronic pain and their parents.MethodsThe Web site was evaluated in two stages. In stage one, recovered adolescents and parents (n = 5 dyads), who had completed office-based CBT through a pediatric pain management clinic, completed ratings of Web site content, usability, appearance, and theme. In stage two, treatment-seeking adolescents and their parents (n = 6 dyads) completed the full-length Web program. Program usage data were obtained to assess interaction with the Web site. ResultsParticipants rated moderate to strong acceptability of the program. Usage data indicated that participants interacted with the site and used communication features.ConclusionsFeedback from usability testing provided important information in the process of designing a feasible Web-based treatment for adolescents with chronic pain for use in a randomized controlled trial.
There is strong empirical support for the use of cognitive behavioral therapy (CBT) in the treatment of pediatric chronic pain (Eccleston, Morley, Williams, Yorke, & Mastroyannopoulou, 2002). Despite support for CBT interventions, many youth do not have access to outpatient psychological services for chronic pain due to a number of barriers, including lack of trained clinicians, expense, competing school and work schedules, and travel distances. To date, the largest treatment advances in making CBT more accessible have been minimal therapist contact and self-administered treatment delivery strategies. These treatments use written materials as a supplement to or substitute for outpatient appointments with a therapist. Self-administered treatments for headache management have produced results that are equal or better than the equivalent therapist-led treatments (Larsson, Daleflod, Hakansson, & Melin, 1987; Richardson & McGrath, 1989).
Recently, two randomized controlled trials have tested computer-based technologies to deliver minimal therapist contact CBT to children with headache and abdominal pain. Headstrong, a CD-ROM intervention for 7–12-year-old children with recurrent headache successfully reduced headache frequency, severity, and duration (Connelly, Rapoff, Thompson, & Connelly, 2006). In the second study, an online treatment program was developed for 9–16 year olds with headache and/or recurrent abdominal pain, demonstrating clinically significant pain reduction (Hicks, von Baeyer, & McGrath, 2006). This 7-week program included online content, written materials for children and parents, and therapist telephone and email contact. These studies provide support for the feasibility of delivering CBT for chronic pain via computer-based technology.
While the existing computer-based interventions for pediatric recurrent pain are quite promising, they do not utilize the interactivity, multimedia formats, personalization, and communication possible with the Internet. These features may increase engagement and positively impact treatment outcomes (Wade, 2004). Initial research on Internet interventions for delivering behavioral treatments has demonstrated feasibility for a variety of pediatric conditions (Ritterband et al., 2003; Wade, 2004). Web-based interventions also provide researchers with unobtrusive measurement of Web site usage and access patterns, which can be used to assess compliance and engagement (Danaher, Boles, Akers, Gordon, & Severson, 2006). Evaluating the feasibility and usability of Web-based interventions early in Web site development is important, as these factors may impact frequency of use, understanding, and the likelihood that a user will implement recommendations (Ritterband et al., 2003). Evaluation of usability can also help determine the appropriateness of Web site interface and content for a target population (Gustafson & Wyatt, 2004). Lack of compliance with accessing Web-based information is a significant barrier to this type of intervention, and improvement of Web content and usability has been suggested as one means of improving compliance (Ritterband et al., 2005).
The purpose of this report is to describe usability testing of an online family CBT program for adolescents with chronic pain (Web-based Management of Adolescent Pain; Web-MAP). This formative evaluation model may be applied in the development of other online treatments. We hypothesized that acceptance of content and usability of Web-MAP would be high, and that participants would appropriately interact and engage with the Web site.
This study was approved by the Institutional Review Board at the academic medical center where the study was conducted. Written informed consent was obtained from parents and written assent was obtained from adolescents for participation in this study.
We utilized a three-phase model based on existing program evaluation models (Rand, 2004) and previous research on Web-based interventions. In Phase 1, program development, program content is developed based on existing evidence-based treatments, and is adapted to fit within the interface and functional features of a Web site. The adaptation must be appropriate to the intended audience. We utilized independent expert feedback to evaluate the program adaptation in Phase 1. Phase 2, formative evaluation, may include monitoring of program usage, as well as evaluation of the acceptability of program content, usability, and other features by users. This phase is designed to test the adaptation completed in Phase 1 and identify problems that may need to be addressed. Phase 3, outcome evaluation, measures the impact that the fully developed Web-based treatment program has on predetermined outcome measures. This phase may also include evaluation of user's satisfaction with the program, as well as usage patterns, which may be investigated as predictors of outcomes (Gustafson & Wyatt, 2004). We focus this article on Phase 2, formative evaluation of the Web-MAP program.
Program content was developed from existing evidence-based, outpatient CBT treatments for chronic pain (Kashikar-Zuck, 2006; Kashikar-Zuck, Swain, Jones, & Graham, 2005; Robins, Smith, Glutting, & Bishop, 2005), including child pain management skills and parent interventions (e.g., reinforce positive coping and activity participation). The adolescent program includes topics considered core components of CBT for chronic pain, including education, relaxation training, and other cognitive and behavioral strategies. The parent program includes training on reinforcement of positive coping, reward systems for activity participation, modeling positive coping skills, and communicating with teenagers (see additional description in Table I). The Web-based program aims to reduce pain and associated functional limitations. Content was reviewed by an independent expert in CBT for adolescents with chronic pain, and by pediatric pain experts from the fields of nursing, psychology, pediatrics, and anesthesiology.
The online program is designed to be used by teens and their parents and includes multimedia elements to enhance information. Peer modeling is included through video interviews of teens who have experienced chronic pain and their parents. Audio files of relaxation instructions are available for practice. Web-MAP is designed to be interactive and personalized. Content includes over 200 page views with graphics. Each user creates a login, chooses treatment goals, and interacts with fields in the Web site. The program is travel-themed, and users visit eight destinations (e.g., China, Costa Rica; one per week) each designed to take about 30 min to complete. Users are instructed to spend additional time practicing skills, and are required to complete assignments at the end of six modules. At some destinations, users receive postcards from previous places they have visited reminding them to practice skills. A message center allows users to ask questions as they progress through the program. A study therapist reviews assignments and messages from participants and provides individualized feedback through the Web-MAP message center. Prospective diary measures of pain and activity limitations are incorporated into the Web site to assess pain and activity limitations before and after program completion.
Web site usability ratings were obtained during two stages of formative evaluation from two groups of patients. In stage one, recovered patients rated program content in each module and basic Web site function using an HTML version of the Web system. Module content was improved based on participant feedback from stage one. In stage two, treatment-seeking patients were given access to the fully functional Web system to rate overall program content and to obtain information on their interaction with the Web site.
A purposeful sample of adolescent patients with a history of chronic pain and their parents (n = 5 dyads) was recruited from a multidisciplinary pediatric chronic pain clinic. Adolescents had previously received 4–14 (M = 7.40, SD = 4.16) sessions of office-based CBT over the course of 3–8 months (M = 5.4, SD = 1.95), and were recovered such that they were no longer experiencing significant pain or disability. Adolescents were 13–17 years of age (M = 15.80, SD = 1.23), 60% (n = 3) were female, and had a history of headache (n = 2), abdominal (n = 1), or musculoskeletal (n = 2) pain. Parents were four mothers and one father.
Patients who were currently experiencing chronic pain and associated functional disability and were seeking treatment were recruited from a multidisciplinary pediatric chronic pain clinic. Six consecutive adolescents and their parents (n = 6 dyads) completed the full 8-week Web-MAP program. Adolescents were 12–16 years of age (M = 14.58, SD = 1.59), four were female, and had chronic headache (n = 3), abdominal (n = 2), or musculoskeletal (n = 1) pain. Parents were four mothers and two fathers.
In stage one, six recovered adolescent–parent dyads were contacted by telephone and five dyads (83%) agreed to participate in the study. Participants were given access to a preliminary HTML version of Web-MAP on their home computers for 2 weeks. This version of the Web-MAP program included all text, basic navigation features, and some images, but no other functional features such as interactive fields, assignment submission, or the message system. A survey designed to measure perceptions of the content and usability of Web-MAP was mailed to participants for completion while viewing the eight treatment modules. Recovered participants rated the overall program, as well as the content of each individual module. Module content was improved based on recovered participant feedback prior to stage two.
In stage two, consecutive adolescents seeking treatment for chronic pain and their parents were recruited during pediatric pain management clinic visits. Participants were instructed to complete the fully functional Web-MAP program over 8 weeks. After program completion, surveys designed to measure perceptions of overall Web site content and usability were mailed to participants. Treatment-seeking participants rated the Web site content and features overall, but did not rate the content of each module of the program.
This self-report measure was designed to assess a user's perception of the Web-MAP content and Web site usability. Users rated content (“How useful or relevant was the Web site/module content?”) and ease of Web site use (“How easy or difficult was it to use and navigate the Web site/module?”) on Likert scale items (Table I). Likeability of the Web site's appearance and theme were assessed using Likert scale items ranging from 0 (Did not like at all) to 5 (Liked very much). Two open-ended items asked users to “List 2–3 things that you liked about this module/the Web site” and “List 2–3 things that you did not like about this module/the Web site,” and open-ended items asked users “What would you suggest we remove from/add to the Web site to make it more fun or useful?” and solicited other comments or suggestions for improvement. Recovered participants rated the overall Web site and each module, while treatment-seeking participants rated only the overall Web site.
In stage two, information about program usage was obtained for treatment-seeking participants who accessed the program. Program completion time, number of electronic messages exchanged with study staff via the study Web site, number of fill-in-the-blank fields completed while progressing through modules, number of completed assignments, and word count of completed assignments was collected from Web-MAP administrative databases.
Ratings of perceived usefulness and ease of use were moderate to high, with some variation across modules (Range = 3.20–4.80; Table I). Ratings for the Web-MAP appearance were high, with a mean of 4.60 (SD = 0.52) on a 0–5 scale, as were ratings for the travel theme (M = 4.65; SD = 0.67). In responses to open-ended questions, parents reported that site content was helpful, particularly content about pain and stress, instructions for reinforcing positive coping, recommendations regarding modeling, and information about lifestyle factors. Adolescents reported that content was helpful, including information about the pain pathway, content about stress and worry, relaxation instructions, and instructions for using distraction strategies. Users liked a number of images, including a diagram of the pain signal pathway. Users also provided suggestions for improvement, including removal of particular images, and areas of content that were difficult to understand, lengthy, or repetitive. These comments were addressed and full Web site functionality was implemented before the site was used by treatment-seeking patients in stage two.
Ratings of perceived usefulness and ease of use of the overall program were moderate to high among treatment-seeking patients and their parents. Mean ratings for overall Web-MAP appearance were moderate (M = 3.67, SD = 1.30). Ratings for ease of use were high (M = 4.50, SD = 0.64). Usefulness of Web-MAP content was rated as moderate (M = 3.58, SD = 1.62), as was the travel theme (M = 3.42, SD = 1.56). While treatment-seeking patients’ ratings were slightly lower than those of recovered patients, no differences were statistically significant. Treatment-seeking patients liked that the program was easy to navigate, video and audio features, personalization, and content, particularly parent behavioral recommendations. Users disliked some images, long Web site load times, and portions of the content that were repetitive.
Treatment-seeking participants took 7–15 weeks to complete the program, with a mean completion time of 10.26 weeks (SD = 2.33). Teen users (n = 6) completed on average 30.17 (SD = 4.40) of 35 interactive fields while progressing through the modules. All teen users entered at least three personal treatment goals, such as “To not allow pain to bring my mood down,” “To swim for longer periods,” and “To be able to see friends more.” All teens completed the six assignments. Total word count of assignments ranged from 121–280 (M = 187.33, SD = 53.14). Teens used the message system to communicate with staff (range = 1–7 messages; M = 3.00; SD = 2.10).
Parent users (n = 6) completed on average 26.17 (SD = 5.88) of 44 interactive fields while progressing through the modules. Five of the six parent users entered three or more personal treatment goals. Examples of parent treatment goals include “To be more active,” “To participate more in activities she loves,” and “To have a low level of pain that he can manage.” All parents completed the seven assignments. Total word count of assignments ranged from 204–879 (M = 497.50, SD = 241.93). All parents utilized the Web-MAP message system to communicate with study staff (range = 1–11 messages; M = 4.00; SD = 3.90).
This study was designed to assess the usefulness and ease of comprehension of program content, as well as the usability, appearance, and theme of the Web-MAP program before testing the intervention in a randomized trial. It also provides a model for formative evaluation of other online interventions. User responses were positive, with users providing moderate to high ratings for the content, usability, appearance, and theme. Program usage patterns showed that users completed the program as instructed, utilized communication features, and were engaged through goal setting and completing personalized information.
Recovered patients rated the Web-MAP program more positively than treatment-seeking patients, although these differences did not reach significance likely due to the small sample size. Because families had previously participated in CBT, ratings of content may have been confounded by previous knowledge, and high acceptability ratings may have been influenced by prior success with CBT. The two stage model of evaluation used in this study allowed feedback from recovered patients about content while also incorporating usage data from patients who were currently seeking treatment. This model may prove useful in future studies evaluating content and usability of such interventions.
There are a number of limitations to the current study. The sample was small and limited to treatment-seeking and recovered adolescents. Thus, it is unknown whether the treatment would be feasible or preferred by adolescents with chronic pain in the community. While this study allowed for comparison of recovered and treatment-seeking patient ratings, interpretation of the precise meaning of ratings is difficult. For instance, we cannot infer that a 4/5 rating is significantly less favorable than a 5/5 rating. Without comparison ratings of a similar treatment program we are limited in our ability to draw conclusions in this area. However, we were able to identify Web site modules that users rated less highly than others and will use this information to target future program improvement efforts. Word count of assignments was used as a proxy for engagement, which presents similar problems as no comparison is available to assist with interpretation. We examined user acceptability ratings, while other usability studies have examined real time user performance data such as navigational errors and number of pages viewed. Future research will investigate the clinical efficacy of Web-MAP in a randomized controlled trial. In the context of this trial, additional usability information will be available such as number of visits to the site. The ability to track patterns of user activity and Web site usage is a research advantage provided by such programs (Marks, Cavanaugh & Gega, 2007). Future studies might investigate predictors of compliance and Web site access (Ritterband et al., 2005), and associations between program usage and treatment outcomes.
This study provides strong initial support for the usability and feasibility of Web-MAP, a Web-based family CBT intervention for adolescents with chronic pain. Results also support the importance of incorporating feedback from potential treatment-seeking consumers into the design of Internet interventions. The Internet may be critical to reaching the large population of children with chronic pain who are not able to receive behavioral treatment in their communities. Research during this formative phase of Web-based treatment design may be an important step in developing Internet interventions that are effective and acceptable to specific populations.
This work was supported by the National Institutes of Health (HD050674 to T.P.). The authors would also like to thank Anne Manees, Psy.D. for her assistance with this research.
Conflicts of interest: None declared.