Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 2012 Jul-Aug; 19(4): 514–522.
Published online 2012 January 3. doi:  10.1136/amiajnl-2011-000307
PMCID: PMC3384097

Systematic review and evaluation of web-accessible tools for management of diabetes and related cardiovascular risk factors by patients and healthcare providers



To identify and evaluate the effectiveness, clinical usefulness, sustainability, and usability of web-compatible diabetes-related tools.

Data sources

Medline, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, world wide web.

Study selection

Studies were included if they described an electronic audiovisual tool used as a means to educate patients, care givers, or clinicians about diabetes management and assessed a psychological, behavioral, or clinical outcome.

Data extraction

Study abstraction and evaluation for clinical usefulness, sustainability, and usability were performed by two independent reviewers.


Of 12616 citations and 1541 full-text articles reviewed, 57 studies met inclusion criteria. Forty studies used experimental designs (25 randomized controlled trials, one controlled clinical trial, 14 before–after studies), and 17 used observational designs. Methodological quality and ratings for clinical usefulness and sustainability were variable, and there was a high prevalence of usability errors. Tools showed moderate but inconsistent effects on a variety of psychological and clinical outcomes including HbA1c and weight. Meta-regression of adequately reported studies (12 studies, 2731 participants) demonstrated that, although the interventions studied resulted in positive outcomes, this was not moderated by clinical usefulness nor usability.


This review is limited by the number of accessible tools, exclusion of tools for mobile devices, study quality, and the use of non-validated scales.


Few tools were identified that met our criteria for effectiveness, usefulness, sustainability, and usability. Priority areas include identifying strategies to minimize website attrition and enabling patients and clinicians to make informed decisions about website choice by encouraging reporting of website quality indicators.

Keywords: Diabetes mellitus, medical informatics, medical education, patient education, endocrinology, knowledge translation, quality improvement, process redesign, CPOE, information, EMR

Diabetes mellitus affects 285 million people worldwide and is a leading cause of death in most high-income countries.1 Clinical care gaps are common in diabetes care. For example, in an American population-based survey, only 62% of patients with diabetes had low-density lipoprotein cholesterol measured annually.2 Reviews of interventions targeting patients and healthcare providers to optimize diabetes care have shown small effects on provider performance and patient outcomes.3–5

Given that consumers are increasingly using the world wide web as a source of health information,6 web-based tools offer potential for optimizing quality of diabetes care. Use of web-based media may improve knowledge, social support, behavior change, and clinical outcomes.7 8 However, existing diabetes websites have wide variations in the quality of evidence provided9 and offer didactic information at high reading levels with little interactive technology, social support, or problem-solving assistance.10 Similarly, although healthcare providers increasingly use online resources for patient care, the volume, breadth, editorial quality, and evidence-based methodology upon which they were developed are highly variable.11

The effectiveness of these tools in changing clinical outcomes has been the subject of reviews in other topic areas; for example, a systematic review of consumer health informatics applications in diverse topic areas, including breast cancer, found that these applications improved clinical outcomes.12 Their effectiveness in a research setting may not translate to effectiveness in clinical practice; factors that affect their adoption into clinical practice include clinical usefulness, usability, and sustainability.13 Specifically, a clinically useful tool, defined as a tool that provides clinically useful answers and is easy to use, access, and read,14 may differ in a research context; for example, while a website on carbohydrate counting may be useful in a research setting with a research dietician, it may be less useful to the consumer trying to use it alone in a real-life setting. Similarly, usability of the tool (defined as the extent to which a product can be used by specified users to complete tasks successfully, in time, and with satisfaction in a specified context15) may be underemphasized in research studies,16 where participants are routinely oriented to and trained on the use of the tool. Finally, sustainability, defined as the degree to which an innovation continues to be used after initial effort to secure adoption is completed,17 is a critical component in addressing the gap between research and practice,18 yet is often not addressed or assessed.19 Critical appraisal of web tools should thus consider their effectiveness and their clinical usefulness, usability, and sustainability. Previous studies have not evaluated the validity, clinical usefulness, usability, and sustainability of web-compatible, diabetes-related tools for patients and providers, which was the objective of this study. Because of the importance of multifactorial vascular risk reduction as well as comprehensive lifestyle modification in the care of patients with diabetes,20 we were interested in diabetes-specific tools and tools for blood pressure, lipid, smoking, obesity, nutrition, physical activity and weight management.20


Data sources and searches

Published literature search strategy

In consultation with an information scientist, Medline, EMBASE, CINAHL, and the Cochrane Central Register of Controlled Trials were searched from their earliest date to June 1, 2011. The following search terms were used: diabetes, hypertension, smoking cessation, weight reduction, online, computer-based, and internet. The complete search strategy is provided in online appendices. Additional articles were identified through review of reference lists of identified studies and discussions with experts.

Grey literature search strategy

The world wide web was searched using the Google search engine on June 14, 2009 with preselected phrases (online appendices). We used these phrases to search websites of interest that had been identified on the basis of expert knowledge.21 Sixty web ‘hits’ were captured for every phrase. The first 30 hits that met our definition of a web-compatible diabetes care tool were retained.

Study inclusion and exclusion criteria

Studies were included if they evaluated a web-compatible diabetes-related care tool, defined as an audiovisual tool that is provided in electronic form to be used as a means to educate, support, or advise patients, care givers, or healthcare providers, and that addressed one of the following aspects of diabetes and pre-diabetes management: glycemic control; cardiovascular risk factor assessment and modification (including hypertension, dyslipidaemia, and smoking cessation); nutrition, physical activity, and weight management; self-management and psychological issues; and complication screening and management. Tools were included if they addressed a relevant topic area, regardless of whether patients with diabetes were the target population. Studies could have observational or experimental designs and had to include at least one psychological, behavioral, or clinical outcome.

Studies were excluded if they (1) did not include an evaluation of the tool, (2) were in a language other than English or French, or (3) evaluated a tool that (a) consisted of an electronic database with no exportable stand-alone tool (such as an electronic medical record), (b) had supplementary hardware or software requirements that were not readily available to the average user, or (c) required a real-time operator (such as a healthcare provider).

Study selection

Titles and abstracts were screened for relevance by two independent reviewers (CHY, SES; figure 1). Potentially eligible full articles were then retrieved and reviewed independently by two reviewers (CHY, SES) to determine whether they met inclusion criteria. A third reviewer was available in cases of disagreement.

Figure 1
Published literature search algorithm. PRISMA flow diagram outlining results of published literature search and tool identification process.

Determination of tool accessibility

Tools were evaluated for accessibility by attempting to access the web-connected tool on a personal computer with standard software. If the tools were not readily accessible online, two attempts were made to contact the authors by email for additional information or access.

Data extraction and quality assessment

For each study that identified an accessible tool, two reviewers independently extracted study characteristics using electronic data extraction forms. We used a modification of the Cochrane Effective Practice and Organization of Care Group data abstraction form.22 These forms characterized study design, participants, tool description, study outcomes, and results. Differences in data extraction were reconciled by consensus.

Tools identified from the grey literature search were categorized by content and educational focus (online appendices). We randomly selected five websites from each of these categories and reviewed them to determine if there was evidence of clinical effectiveness. To assess these sites, we developed an instrument based on a framework by Straus and Haynes14 and tested its face validity with relevant experts. This instrument contained 18 items and characterized the evidence base for the content and effectiveness of the tool.

Tool evaluation

Clinical usefulness and sustainability

Each tool was independently reviewed for clinical usefulness and sustainability by two members of a team of clinical experts (CHY F Kim, H Halapy, and C West; see Acknowledgments). Differences were reconciled by consensus. As there were no validated instruments to assess clinical usefulness or sustainability, we developed instruments using a framework from the literature and input from experts14 23; these instruments were determined to have face validity by experienced clinicians and experts in knowledge translation. The clinical usefulness instrument contained five items and assessed clinical relevance and ease of access using a Likert scale with scores ranging from 0 to 5. A score of 5 denoted ‘clinically useful answers are available most of the time, and are easily accessible and readable within a few minutes’, and a score of 0 denoted ‘not useful clinically.’ The sustainability instrument contained six items and assessed continued relevance of the topic, potential barriers to sustainability, and engagement of a group to keep the tool up to date. The instrument was designed to exclude major barriers to sustainability.


Each tool was independently reviewed for usability by two members of a team of human factors engineers (S Jovicic, A Xu, H Takeshita, and F Wan; see Acknowledgments). The instrument incorporated questions from three industry-standard usability instruments (ISO 9241-110 Usability Heuristics,24 Gerhardt-Powals Research-based Heuristic,25 and Site Assessment Tool26) and contained 27 items characterizing suitability to user's skill, ease of navigation, reduction in cognitive load, and appearance. For each desirable usability characteristic, raters scored ‘yes’, ‘no’, or ‘not applicable.’

Data synthesis and analysis

Inter-rater reliability for data abstraction for clinical effectiveness, clinical usefulness, sustainability, and usability were calculated.

Owing to heterogeneity in study design, population, interventions, and outcomes, meta-analyses by intervention type or outcome were not possible. However, we described study quality and performed a descriptive analysis of studies with evidence of impact on outcomes. In addition, we performed a meta-regression of all eligible studies (irrespective of intervention type or outcome) to assess whether clinical usefulness ratings and usability ratings were moderators of the effectiveness of these interventions. The meta-regression was performed in R27 version 2.12.0 using the contributed package metafor28 version 1.4-0. The Mad29 version 0.8 package was used to convert the treatment effects into a standardized treatment effect. Hedges G30 Studies were excluded if there were insufficient data to determine a treatment effect or its variance. Studies without a ‘true’ control group were also excluded, as there is no way to incorporate clinical usefulness or usability for both groups.


Published literature

Results of the literature search, study, and tool selection are detailed in figure 1. While 393 studies and 219 unique tools were identified, 127 tools were not accessible, and thus we were not able to evaluate them. We evaluated the remaining 92 tools and corresponding 57 studies.

Inter-rater reliability

Inter-rater reliability was moderate to good: κ for data abstraction items for clinical effectiveness ranged from 0.66 to 0.72. Weighted κ values31 for assessment of clinical usefulness and sustainability were 0.50 (95% CI 0.36 to 0.64) and 0.77 (95% CI 0.63 to 0.87), respectively. Cohen's κ32 ranged from 0.50 to 0.65 for each component of the scale (ISO, 0.65; Gerhardt-Powals, 0.50; Site Assessment Tool, 0.55).

Description of studies

Study quality and type

Of 57 studies, 40 studies used experimental designs (25 randomized controlled trials (RCTs),33–56 one controlled clinical trial,57 14 uncontrolled before–after studies39 47 58–68), and 17 studies used observational designs (one case–control trial,69 seven cross-sectional studies,70–79 nine cohort studies76–78 80–85). One article consisted of four studies including two RCTs and two uncontrolled before–after studies.47 One article consisted of one RCT and one uncontrolled before–after study.39

Risk of bias

The methodological quality of all studies is described in online appendix tables 1 and 2. Characteristics of the RCTs are summarized in appendix table 1. Methodological quality was variable; intention-to-treat analysis and description of loss to follow-up occurred in approximately half of the studies, and calculation of statistical power, randomization, concealment of allocation, and follow-up of more than 6 months were described in the minority of studies.

Description of tools

Tool formats

Formats included static websites, decision aids,34 49 52 56 63 64 interactive websites,34–36 38–41 44–46 48 51 54–62 65–76 78 79 81 82 85 CD-ROM games or DVD,33 42 43 and email feedback programs.34 37 47 52 53 77

Target audience

Five tools targeted patients with type 1 or 2 diabetes.33 42 55 74 75 79 The remaining tools did not specifically target patients with diabetes, but did address an aspect of comprehensive diabetes care in overweight adults,40 53–55 57 smoking adults,35 47 49 51 56 59 60 70 71 76 77 80 depressed adults,41 58 69 children,43 62 smoking adolescents,48 50 and adolescents at risk of type 2 diabetes.68 With respect to informal care givers and healthcare professionals, 10 studies targeted healthcare providers, with six targeting physicians,54 65 72 73 82–84 three studies targeting nurses,38 39 and one study targeting public health professionals61; there were no studies that targeted care givers.

Clinical usefulness, sustainability, and usability

Clinical usefulness, sustainability, and usability ratings are summarized in table 1. The most common usability error (found in 50% of tools) was not utilizing images to facilitate learning, a feature that has been demonstrated to aid data interpretation and improve recognition and recall.25

Table 1
Summary of clinical usefulness, sustainability, and usability ratings

Clinical effectiveness of tools

Patient outcomes, including knowledge, skill development, behavior change, and psychological and clinical outcomes, were examined in 17 studies.37 40–53 55 Clinician knowledge and skill were evaluated in three studies 38 39 54 (online appendix table 3).

Patient diabetes education tools

One study examined the effect of a multimedia general diabetes education computer application for low-literacy patients to use in clinic waiting rooms.42 At 1 year, the intervention group had a greater awareness of diabetes complications and a greater reduction in HbA1c than the control group (online appendix table 3).

Patient self-management and coping tools

Two studies examined the effect of a self-management website or DVD in patients with poorly controlled diabetes.33 55 In the first study,55 the intervention group had a greater reduction in HbA1c and systolic blood pressure, an increase in high-density lipoprotein cholesterol, and reduction in triglycerides compared with the control group at 12 months (online appendix table 3). In addition, greater website use correlated with greater clinical improvements: persistent website users had greater reduction in HbA1c from baseline compared with intermittent users (−1.9% vs −1.2%, p=0.051). Similarly, larger numbers of website data uploads were associated with a larger decline in HbA1c (highest tertile −2.1% vs lowest tertile −1.0%, p<0.02). The second study, which compared in-person class-based programs with DVD-based self-management programs, showed no differences in self-management behavior or psychological or clinical outcomes.33

One study examined the effect of an interactive website providing tailored advice on lifestyle modification and risk factor screening, based on a questionnaire on family history and lifestyle habits.36 Compared with a control group who received standardized messages, the intervention group improved their diet and physical activity, although there were no changes in smoking rates and assessment of blood glucose or blood pressure, and a reduction in cholesterol screening.

Patient nutrition and physical activity tools

Seven studies examined the effect of nutrition and physical activity websites37 40 44–46 52 53 on waist circumference,40 weight,53 body mass index (BMI),44 percentage body fat,44 blood pressure,44 quality of life,37 and coronary RR.40

Measures of obesity (waist circumference, weight, BMI, and percentage body fat) were significantly improved,40 44 53 as was coronary RR 44 and self-assessed health status37 (online appendix table 3). In a study of a web-based physical activity program,40 waist circumference decreased significantly in the intervention group compared with the controls. Similarly, an interactive personalized health promotion website was found to result in a greater reduction in percentage body fat compared with the control, although there was no significant change in BMI or blood pressure.44 This study also found a significant reduction in coronary RR in the intervention group, compared with the control group. A third study compared the effect of a weight loss website in combination with human-email counseling, computer-automated email counseling, or no counseling53 and found that, at 3 months, there was a significantly greater weight loss in the human-email group and computer-automated email group than in the no counseling group. However, at 6 months, only the human-email group retained significantly greater weight loss compared with the no counseling group. Finally, self-assessed health status was significantly greater in the group using an email physical activity and diet program than the control group.37

One study examined the effect of an educational CD-ROM game about nutrition and physical activity for children.43 Whereas there was a greater reduction in BMI in girls (p=0.04), a greater increase in BMI was noted in boys (p=0.04) 5 weeks after the intervention.

Patient smoking cessation tools

Eight studies examined the effect of five online smoking prevention and cessation websites on clinical outcomes (smoking initiation,50 cigarette use,48 1-day smoking abstinence,35 7-day smoking abstinence,34 47 49 30-day point prevalence51 56). Study quality was variable. There was inconsistent reporting of loss to follow-up, similarity of groups at baseline, or adequacy of randomization (online appendix table 3). It was unclear whether analysis was conducted by intention-to-treat in the studies that reported positive outcomes.47 48

One study of an interactive, multimedia smoking prevention and cessation curriculum demonstrated lower rate of smoking initiation compared with use of a self-help booklet.50 With respect to smoking cessation, one study showed no change in cigarette use at 6 months,48 three studies demonstrated no difference in quit rate (as measured by 1-day reported abstinence,35 7-day reported abstinence,34 47 48 30-day point prevalence51 56), and one study showed an improvement in quit rate.47 However, in an exploratory analysis by website utilization, Rabius found that higher smoking-quit rates were associated with the two more highly utilized websites compared with the three less frequently utilized sites (12.5% vs 10.6%, p=0.03).51 Similarly, participants who visited a site more than five times were twice as likely to quit than participants who visited a site less than five times (20.0% vs 9.8%, p<0.001). In addition, higher quit rates were found with more interactive, tailored sites compared with the static control site (13% vs 10%, p=0.04).51

Clinician education tool

One study examined the effect of an online continuing education seminar on physician knowledge of diabetes management.54 Physicians' recommendation of appropriate quality-of-care measures was assessed immediately after the intervention using a non-validated clinical vignette score and did not change, with the exception of one process-of-care measure (‘ordering an eye exam’).

Clinician behavior change counseling tool

Two studies examined the effect of an interactive web-based motivational interviewing educational program on teaching effectiveness immediately after the intervention.38 39 In both studies, teaching effectiveness, as measured by qualitative analysis and coding of written responses to counseling vignettes and a multiple choice questionnaire, was higher in the intervention group.

Role of interactivity

More interactive tools resulted in greater clinical improvement; for example, Tate et al found that interaction in the form of human- or computer-email counseling resulted in greater weight loss than no counseling.53 Similarly, Goran and Reynolds found that an interactive multimedia CD-ROM game resulted in greater reduction in BMI than a static educational CD-ROM.43 This observation is seen also in patient smoking cessation websites; Rabius et al found that interactive tailored smoking cessation sites resulted in greater quit rates than a static site,51 and Munoz et al found that individually timed educational messages resulted in greater quit rates than the static smoking guide alone.47 A similar finding was seen in tools for healthcare providers; Carpenter et al found that an interactive tutorial was more effective in teaching motivational interviewing techniques than reading material.38 39 The role of tool interactivity on continued website use is highlighted in the study of the interactive personalized health promotion website: Hurling et al found that the interactive site resulted in a significantly greater percentage of participants logging in throughout the study period, with less attrition than the static, control site.45

Grey literature

Tool selection and evaluation are described in figure 2. Of the 360 websites reviewed, two86 87 had been evaluated for clinical effectiveness. Both evaluations had been identified in the published literature search47 48 and the tools described previously.

Figure 2
Grey literature search algorithm. Modified PRISMA flow diagram outlining results of grey literature search and tool identification process.

Clinical usefulness and usability as potential moderators of tool effectiveness

Figure 3A,B shows the results of the meta-regression, with clinical usefulness ratings and usability ratings as potential moderators of tool effectiveness. Twelve studies were included, comprising a total sample size of 2731 participants. There was significant heterogeneity, with τ of 0.87 and 0.85, respectively. While the standard meta-analysis demonstrated a significant positive effect on outcomes (standardized treatment effect, Hedges G 0.64, 95% CI 0.15 to 1.13, p=0.01), neither clinical usefulness nor usability had a moderating effect on tool effectiveness (regression coefficient 0.26 (95% CI −1.4 to 1.9, p=0.76) and −1.5 (95% CI −6.4 to 3.4, p=0.55), respectively).

Figure 3
(A) Modified forest plot demonstrating lack of moderating effect of clinical usefulness ratings on tool effectiveness. Squares with lines are the observed treatment effects and CI. Grey diamonds show the predicted treatment effects based on the model. ...


Although a large number of studies and tools were identified, many tools were not accessible, and thus we were not able to evaluate them. These tools would also not be accessible to patients or healthcare providers; thus their exclusion does not affect the applicability or relevance of our findings. The 57 studies and tools identified were very different in terms of participants, settings, and outcomes, which meant we could not perform a meta-analysis by intervention or outcome. Although there were a number of studies with positive results, these results must be viewed with caution because of concerns about the reported study designs. Half of the studies were pre–post designs or included a comparative group that was non-randomized or not adequately randomized. Many studies would have been strengthened through use of validated outcome measures and longer-term follow-up of 1 year or more. With respect to the tools, although the evidence base of the tool's content was high, only 25% had easily accessible, clinically useful answers most of the time. Six percent of tools were free of usability errors, but 60% had three or more errors in usability. Common usability errors included limited use of images, icons, and other visual elements to facilitate learning, and lack of intuitiveness in navigation and expected next steps. These and other usability errors can negatively affect users' experience with a tool and may lead them to stop using the tool. In one study, ease of usability was one of the main determinants of an individual's satisfaction and willingness to engage with a website.88 While a correlation between usability and tool effectiveness was not demonstrated in this study, our meta-regression was limited by the number of studies that adequately reported data, as well as the heterogeneity in interventions, populations, and outcomes. This high prevalence of usability errors is mirrored in other reviews of usability of healthcare websites89–91 and highlights the need to ensure that websites provide useful and usable formats and undergo usability testing before they are launched.

Our review of the literature has identified areas for further exploration. First, greater improvements in patient outcomes were seen with greater use of the tool.42 49 51 55 For example, persistent website users had greater improvement in HbA1c than intermittent users, and a larger number of website data uploads was associated with a larger decline in HbA1c.55 Caution should be used when interpreting this association, given possible confounders that can result in reduced use such as depression or lack of progress with respect to goals. A recent systematic review found mixed results in the association between adherence and outcomes; analysis was limited because of heterogeneity of adherence and outcome measures, although logins appeared to be associated with outcomes of physical health interventions, while module completion appeared to be associated with outcomes of psychological health interventions.92 In conjunction with the literature on website usage attrition,93–95 these findings have implications for website development and website evaluation. Second, this review suggests a mechanism by which to minimize attrition and thus maximize clinical improvement, through the use of interactivity and feedback.44 51 53 For example, Hurling et al found that an interactive health promotion site resulted in a significantly greater percentage of participants logging in throughout the study period, with less attrition, than the control static site.44 Although greater interactivity may result in better outcomes, it may require higher levels of health literacy, navigation skills, and computer experience.96 97 However, with careful user testing, highly interactive applications can be designed to be user friendly98 and can have positive effects on user satisfaction, effectiveness, efficiency, and overall attitude toward the tool.99 Other intervention characteristics that enhance use include peer or counselor support, email or phone contact, and updates regarding the intervention website.100

These findings have implications for website developers, researchers, patients, and clinicians. Web-based tool developers must incorporate strategies—such as optimization of website usability and interactivity—to maximize frequency and persistence of website use, and researchers must evaluate these strategies and their impact on website usage and clinical outcomes, as well as characteristics of users who are predisposed to persistent website use. Given the degree of variability in website quality, patients and clinicians should critically appraise these resources for effectiveness, relevance, and usability before selecting them for use. Given the burden of time and expertise required to make these assessments, development of a transparent recognized peer-review system to assess clinical effectiveness, usefulness, sustainability, and usability of web-based tools, as well as a requirement for standard reporting of these characteristics by website developers, would enable both patients and clinicians to make informed decisions in a timely manner. Although website certification systems do exist, these primarily address the evidence base of the website content rather than website use, and do not address clinical usefulness or usability. For example, Health On the Net Foundation (HON, is a non-profit organization that established HONcode certification, an ethical standard aimed at offering quality online health information. A review of HONcode-accredited sites found that 87% were too difficult to read for the average adult population.101 In addition, this system has not been universally adopted, with only 28% of diabetes patient education sites being HonCODE-accredited.102

This review is limited by the number of accessible tools, exclusion of tools for mobile devices, the quality of the studies identified, use of non-validated scales, and publication bias. In addition, the broad interventions included, as well as the number of study outcomes (clinical effectiveness, clinical usefulness, sustainability, and usability), limited the ability to synthesize data with a standard meta-analytical approach. Mobile devices represent a highly accessible portal to health information resources and thus have the potential to transform healthcare delivery; however, assessing tools for mobile devices was beyond the scope of this review. Although the rating scales used were not formally validated, the items were derived from the literature14 23–26 and were assessed for face validity by content experts. We chose to be inclusive when selecting interventions, given the multi-system involvement of the diabetic disease process and the importance of comprehensive management (including vascular risk modification) in the care of the individual with diabetes. We also chose to assess the non-traditional outcomes of clinical usefulness, sustainability, and usability, as these are important predictors of knowledge use and transfer.14 88 We strove to reduce publication bias by including a comprehensive search of the grey literature.

The strengths of this review include: an extensive literature search that included the grey literature; the comprehensive review of each tool for clinical effectiveness, usefulness, sustainability, and usability; and the generalizability of findings regarding website use in health promotion and chronic disease management. To our knowledge, although other reviews of health informatics tools have addressed clinical outcomes12 103 and usability16 individually, no other systematic review of any informatics intervention has considered all of these issues.

Web-based tools have the potential to improve health outcomes and complement healthcare delivery, but their full potential is hindered by limited knowledge about their effectiveness, high prevalence of usability errors, and high attrition rates. A development and research agenda should include: developing strategies to reduce website attrition in order to maximize clinical outcomes; standardizing website quality indicators; and transparent reporting of these indicators in order to allow patients and clinicians to make informed decisions about website choice.

Supplementary Material

Supporting Statement:
Supporting Statement:
Supplementary Data:


We thank: Laure Perrier for carrying out the published and grey literature searches; David Newton for information technology support; Julie Hukui, Daniel Chong, Patrick Ong, and Kristin Tokarsky for reference management; Mahmood Beheshti, Nathan McKibbon, Alex Rogovic, and Violetta Sochka for website review; our clinician experts (Florence Kim, Henry Halapy, and Christine West) for rating clinical usefulness and sustainability; and our human factors engineers (Sasha Jovicic, Annie Xu, Harumi Takeshita, and Flora Wan) for rating usability.


Funding: This research was supported by the Ontario Ministry of Health and Long Term Care, who had no role in study design, in the collection, analysis and interpretation of data, in the writing of the report, or in the decision to submit the article for publication. SES is supported by a Tier 1 Canada Research Chair.

Competing interests: None.

Contributed by

Contributors: CHY conceived the study, participated in its design, and drafted the manuscript. RB participated in the coordination, performed data abstraction, and drafted portions of the manuscript. EL and MSO performed data abstraction and drafted portions of the manuscript. SES and AL participated in its design and revised the manuscript critically for important intellectual content. All authors had access to the data and read and approved the final manuscript.

Provenance and peer review: Not commissioned; externally peer reviewed.


1. Unwin N, Whiting D, Gan D, et al. IDF Diabetes Atlas. 4th edn International Diabetes Federation, 2009
2. Saadine JB, Cadwell MS, Gregg EW, et al. Improvements in diabetes processes of care and intermediate outcomes: United States, 1988–2002. Ann Intern Med 2006;144: 465–74 [PubMed]
3. Ferlie EB, Shortell SM. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q 2001;79:281–315 [PubMed]
4. Vermeire E, Wens J, Van Royen P, et al. Interventions for improving adherence to treatment recommendations in people with type 2 diabetes mellitus. Cochrane Database Sys Rev 2005;(5):CD003638 [PubMed]
5. Renders CM, Valk GD, Griffin SJ, et al. Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings. Cochrane Database Sys Rev 2001;(1):CD001481 [PubMed]
6. Statistics Canada: Canadian Internet Use Survey 2008. (accessed 6 May 2009).
7. Jeste DV, Dunn LB, Folsom DP, et al. Multimedia educational aids for improving consumer knowledge about illness management and treatment decisions: a review of randomized controlled trials. J Psychiatr Res 2008;42:1–21 [PubMed]
8. Murray E, Burns J, See TS, et al. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev 2004;(4):CD004274. [PubMed]
9. Seidman JJ, Steinwachs D, Rubin H. Design and testing of a tool for evaluating the quality of diabetes consumer-information Web sites. J Med Internet Res 2003;5:e30. [PMC free article] [PubMed]
10. Bull SS, Gaglio B, McKay HG, et al. Harnessing the potential of the internet to promote chronic illness self-management diabetes as an example of how well we are doing. Chronic Illn 2005;1:143–55 [PubMed]
11. Banzi R, Liberati A, Moschetti I, et al. A review of online evidence-based practice point-of-care information summary providers. J Med Internet Res 2010;12:e26. [PMC free article] [PubMed]
12. Gibbons MC, Wilson RF, Samal L, et al. Impact of consumer health informatics applications. Evid Rep Technol Assess (Full Rep) 2009:1–546 [PubMed]
13. Mackert M, Kahlor L, Tyler D, et al. Designing e-health interventions for low-health-literate culturally diverse parents: addressing the obesity epidemic. Telemed J E Health 2009;15:672–7 [PMC free article] [PubMed]
14. Straus S, Haynes R. Managing evidence-based knowledge: the need for reliable, relevant and readable resources. CMAJ 2009;180:942–5 [PMC free article] [PubMed]
15. ISO/IEC Ergonomic Requirements for Office Work with Visual Display Terminals (VDT)s - Part 11 Guidance on Usability. ISO/IEC 9241-11 1998:(E).
16. Bock B, Graham AL, Sciamanna C, et al. Smoking cessation treatment on the Internet: content, quality, and usability. Nicotine Tob Res 2004;6:207–19 [PubMed]
17. Rogers E. Diffusion of Innovations. 5th edn New York: Free Press, 2005:429
18. Mendel P, Meredith LS, Schoenbaum M, et al. Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health 2008;35:21–37 [PMC free article] [PubMed]
19. Perry KJ, Hickson M, Thomas J. Factors enabling success in weight management programmes: systematic review and phenomenological approach. J Hum Nutr Diet 2011;24:301
20. Gaede P, Lund-Andersen H, Parving H, et al. Effect of a multifactorial intervention on mortality in type 2 diabetes. N Engl J Med 2008;358:580–91 [PubMed]
21. Canadian Agency for Drugs and Technologies in Health Grey Matters: A Practical Tool for Evidence-Based Searching. (accessed 6 May 2009).
22. Cochrane Effective Practice and Organisation of Care Group EPOC Resources for Review Authors.
23. Davies B, Edwards N. The action cycle: sustaining knowledge use. In: Sharon S, Jacqueline T, Ian DG, editors. , eds. Knowledge Translation in Health Care: Moving from Evidence to Practice. West Sussex, UK: Blackwell Publishing Ltd, 2009
24. Ergonomics of Human-System Interaction Part 110: Dialogue Principles. ISO 2006:9241:9110.
25. Gerhardt-Powals J. Cognitive engineering principles for enhancing human computer performance. Int J Hum Comput Interact 1996;8:189–21
26. U.S. Department of Health & Human Services: Site Assessment Tool Adapted from Research-Based Web Design & Usability Guidelines. 2009. (accessed 13 Aug 2009).
27. R Development Core Team A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation Statistical Computing, 2010, ISBN 3-900051-07-0.
28. Wolfgang V. Conducting meta-analyses in R with metafor package. J Stat Softw 2010;36:1–48
29. Del Re AC, Hoyt WT. Mad: Meta-Analysis with Mean Differences R package version 08. 2010
30. Cooper H, Hedges LV, Valentine JC. The Handbook of Research Synthesis and Meta Analysis. New York, US: Russell Sage Foundation, 2009
31. Fleiss JL, Cohen J, Everitt BS. Large-sample standard errors of kappa and weighted kappa. Psychol Bull 1969;72:323–7
32. Cohen J. A coefficient for agreement for nominal scales. Educ Psychol Meas 1960;20:37–46
33. Glasgow RE, Edwards LL, Whitesides H, et al. Reach and effectiveness of DVD and in-person diabetes self-management education. Chronic Illn 2009;5:243–9 [PubMed]
34. Muñoz RF, Barrera AZ, Delucchi K, et al. International Spanish/English Internet smoking cessation trial yields 20% abstinence rates at 1 year. Nicotine Tob Res 2009;11:1025–34 [PMC free article] [PubMed]
35. Etter J. Comparing computer-tailored, internet-based smoking cessation counseling reports with generic, untailored reports: a randomized trial. J Health Commun 2009;14:646–57 [PubMed]
36. Ruffin MT, 4th, Nease DE, Jr, Sen A, et al. Effect of preventive messages tailored to family history on health behaviors: The Family Healthware Impact Trial. Ann Fam Med 2011;9:3–11 [PubMed]
37. Block G, Sternfeld B, Block CH, et al. Development of Alive! (A lifestyle intervention via email), and its effects on health-related quality of life, presenteeism, and other behavioral outcomes: Randomized controlled trial. J Med Internet Res 2008;10:e43. [PMC free article] [PubMed]
38. Carpenter KM, Cohn LG, Glynn LH, et al. Brief interventions for smoking cessation: Using the internet to train healthcare providers. Int Electron J Health Educ 2008;11:1–12 [PMC free article] [PubMed]
39. Carpenter KM, Watson JM, Raffety B, et al. Teaching brief interventions for smoking cessation via an interactive computer-based tutorial. J Health Psychol 2003;8:149–60 [PubMed]
40. Carr LJ, Bartee RT, Dorozynski C, et al. Internet-delivered behavior change program increases physical activity and improves cardiometabolic disease risk factors in sedentary adults: Results of a randomized controlled trial. Prev Med 2008;46:431–8 [PubMed]
41. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: Randomized controlled trial. BMJ 2004;328:265–9 [PMC free article] [PubMed]
42. Gerber BS, Brodsky IG, Lawless KA, et al. Implementation and evaluation of a low-literacy diabetes education computer multimedia application. Diabetes Care 2005;28:1574–80 [PubMed]
43. Goran MI, Reynolds K. Interactive multimedia for promoting physical activity (IMPACT) in children. Obes Res 2005;13:762–71 [PubMed]
44. Hurling R, Catt M, Boni MD, et al. Using internet and mobile phone technology to deliver an automated physical activity program: Randomized controlled trial. J Med Internet Res 2007;9:e7. [PMC free article] [PubMed]
45. Hurling R, Fairley BW, Dias MB. Internet-based exercise intervention systems: Are more interactive designs better? Psychol Health 2006;21:757–72
46. Martinson BC, Crain AL, Sherwood NE, et al. Maintaining physical activity among older adults: Six-month outcomes of the keep active Minnesota randomized controlled trial. Prev Med 2008;46:111–19 [PMC free article] [PubMed]
47. Munoz RF, Lenert LL, Delucchi K, et al. Toward evidence-based internet interventions: A Spanish/English web site for international smoking cessation trials. Nicotine Tob Res 2006;8:77–87 [PubMed]
48. Norman CD, Maley O, Li X, et al. Using the internet to assist smoking prevention and cessation in schools: a randomized, controlled trial. Health Psychol 2008;27:799–810 [PubMed]
49. Pike KJ, Rabius V, McAlister A, et al. American cancer society's QuitLink: randomized trial of internet assistance. Nicotine Tob Res 2007;9:415–20 [PubMed]
50. Prokhorov AV, Kelder SH, Shegog R, et al. Impact of a smoking prevention interactive experience (ASPIRE), an interactive, multimedia smoking prevention and cessation curriculum for culturally diverse high school students. Nicotine Tob Res 2008;10:1477–85 [PubMed]
51. Rabius V, Pike KJ, Wiatrek D, et al. Comparing internet assistance for smoking cessation: 13-month follow-up of a six-arm randomized controlled trial. J Med Internet Res 2008;10:e45. [PMC free article] [PubMed]
52. Sternfeld B, Block C, Quesenberry CP, et al. Improving diet and physical activity with alive: a worksite randomized trial. Am J Prev Med 2009;36:475–83 [PubMed]
53. Tate DF, Jackvony EH, Wing RR. A randomized trial comparing human e-mail counseling, computer-automated tailored counseling, and no counseling in an internet weight loss program. Arch Intern Med 2006;166:1620–5 [PubMed]
54. Weston CM, Sciamanna CN, Nash DB. Evaluating online continuing medical education seminars: Evidence for improving clinical practices. Am J Med Qual 2008;23:475–83 [PubMed]
55. McMahon GT, Gomes HE, Hohne SH, et al. Web-based care management in patients with poorly controlled diabetes. Diabetes Care 2005;28:1624–9 [PMC free article] [PubMed]
56. Graham AL, Cobb NK, Papandonatos GD, et al. A randomized trial of Internet and telephone treatment for smoking cessation. Arch Intern Med 2011;171:46–53 [PMC free article] [PubMed]
57. Woolf SH, Krist AH, Johnson RE, et al. A practice-sponsored web site to help patients pursue healthy behaviors: an ACORN study. Ann Fam Med 2006;4:148–52 [PubMed]
58. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002;4:e3. [PMC free article] [PubMed]
59. Graham AL, Cobb NK, Raymond L, et al. Effectiveness of an internet-based worksite smoking cessation intervention at 12 months. J Occup Environ Med 2007;49:821–8 [PubMed]
60. Thieleke J, McMahon J, Meyer G, et al. An evaluation of the freedom from smoking online cessation program among Wisconsin residents. WMJ 2005;104:41–4 [PubMed]
61. Sears KE, Cohen JE, Drope J. Comprehensive evaluation of an online tobacco control continuing education course in Canada. J Contin Educ Health Prof 2008;28:41–4 [PubMed]
62. Bell JA, Patel B, Malasanos T. Knowledge improvement with web-based diabetes education program: Brainfood. Diabetes Technol Ther 2006;8:444–8 [PubMed]
63. VanWormer JJ, Pronk NP, Boucher JL. Experience analysis of a practice-based, online pedometer program. Diabetes Spectrum 2006;19:197–200
64. Lalonde L, O'Conner AM, Drake E, et al. Development and preliminary testing of a patient decision aid to assist in pharmaceutical care in the prevention of cardiovascular disease. Pharmacotherapy 2004;24:909–22 [PubMed]
65. Wells S, Furness S, Rafter N, et al. Integrated electronic decision support increases cardiovascular disease risk assessment four fold in routine primary care practice. Eur J Cardiovasc Prev Rehabil 2007;15:173–8 [PubMed]
66. Smith KE, Levine BA, Clement SC, et al. Impact of MyCareTeam for poorly controlled diabetes mellitus. Diabetes Technol Ther 2004;6:828–35 [PubMed]
67. Speck RM, Hill RK, Pronk NP, et al. Assessment and outcomes of HealthPartners 10,000 Steps program in an academic work site. Health Promot Pract 2010;11:741–50 [PubMed]
68. Long JD, Armstrong ML, Amos E, et al. Pilot using world wide web to prevent diabetes in adolescents. Clin Nurs Res 2006;15:67–79 [PubMed]
69. Christensen H, Griffiths KM, Korten A, et al. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 2004;6:e46. [PMC free article] [PubMed]
70. Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. J Med Internet Res 2008;10:e54. [PMC free article] [PubMed]
71. Cobb NK, Graham AL, Bock BC, et al. Initial evaluation of a real-world internet smoking cessation system. Nicotine Tob Res 2005;7:207–16 [PMC free article] [PubMed]
72. Riddell T, Lindsay G, Kenealy T, et al. The accuracy of ethnicity data in primary care and its impact on cardiovascular risk assessment and management: Predict CVD-8. N Z Med J 2008;121:40–8 [PubMed]
73. Riddell T, Kenealy T, Wells S, et al. Audit of health data captured routinely in primary healthcare for the clinical decision support system predict (PREDICT CVD-4). HCIRO 2008;12:15–21
74. Zrebiec J. Internet communities: do they improve coping with diabetes? Diabetes Educ 2005;31:825–8, 830–2, 834, 836. [PubMed]
75. Zrebiec JF, Jacobson A. What attracts patients with diabetes to an internet support group? A 21 month longitudinal website study. Diabet Med 2001;18:154–8 [PubMed]
76. An LC, Schillo BA, Saul JE, et al. Utilization of smoking cessation information, integrative and online community resources as predictors of abstinence: Cohort study. J Med Internet Res 2008;10:e55. [PMC free article] [PubMed]
77. Lenert L, Munoz R, Perez JE, et al. Automated e-mail messaging as a tool for improving quit rates in an internet smoking cessation intervention. J Am Med Inform Assoc 2004;11:235–40 [PMC free article] [PubMed]
78. Petersen R, Sill S, Lu C, et al. Effectiveness of employee internet-based weight management program. J Occup Environ Med 2008;50:163–71 [PubMed]
79. Herrejon K, Hartke JL, Scherer J, et al. The creation and impact evaluation of “Your guide to diet and diabetes,” an interactive web-based diabetes tutorial. Diabetes Technol Ther 2009;11:171–9 [PubMed]
80. Lenert L, Munoz RF, Stoddard J, et al. Design and pilot evaluation of an internet smoking cessation program. J Am Med Inform Assoc 2003;10:16–20 [PMC free article] [PubMed]
81. Ware LJ, Hurling R, Bataveljic O, et al. Rates and determinants of uptake and use of an internet physical activity and weight management program in office and manufacturing work sites in England: Cohort study. J Med Internet Res 2008;10:e56. [PMC free article] [PubMed]
82. Marshall RJ, Zhang Z, Broad JB, et al. Agreement between ethnicity recorded in two New Zealand health databases: Effects of discordance on cardiovascular outcome measures (PREDICT CVD3). Aust N Z J Public Health 2007;31:211–16 [PubMed]
83. Selak V, Wells S, Whittaker R, et al. Smoking status recording in GP electronic records: The unrealised potential. Inform Prim Care 2006;14:235–41; discussion 242–5. [PubMed]
84. Whittaker R, Bramley D, Wells S, et al. Will a web-based cardiovascular disease (CVD) risk assessment programme increase the assessment of CVD risk factors for maori? N Z Med J 2006;119:U2077. [PubMed]
85. Sarna L, Bialous S, Wewers ME, et al. Nurses trying to quit smoking using the Internet. Nurs Outlook 2009;57:246–56 [PubMed]
86. The Stop Smoking Center Welcome to the Stop Smoking Center 6.1!. http://wwwstopsmokingcenternet/ (accessed 1 Mar 2010).
87. The American Cancer Society Do You Need Help to Quit? 2008. http://wwwcancerorg/docroot/PED/content/PED_10_13x_Smoking_Habits_Quizasp?sitearea=&level= (accessed 1 Mar 2010).
88. Fu L, Salvendy G. The contribution of apparent and inherent usability to a user's satisfaction in a searching and browsing task on the Web. Ergonomics 2002;45:415–24 [PubMed]
89. Seidman JJ, Steinwachs D, Rubin H. Design and testing of a tool for evaluating the quality of diabetes consumer-information web sites. J Med Internet Res 2003;5:136–56 [PMC free article] [PubMed]
90. Sutherland LA, Wildemuth B, Campbell MK, et al. Unraveling the web: an evaluation of the content quality, usability, and readability of nutrition web sites. J Nutr Educ Behav 2005;37:300–5 [PubMed]
91. Bock BC, Graham AL, Whiteley JA, et al. A review of web-assisted tobacco interventions (WATIs). J Med Internet Res 2008;10:e39. [PMC free article] [PubMed]
92. Donkin L, Christensen H, Naismith SL, et al. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011;13:e52. [PMC free article] [PubMed]
93. Gunther E. The law of attrition. J Med Internet Res 2005;7:e11 http://wwwjmirorg/2005/1/e11/v7e11 [PMC free article] [PubMed]
94. Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006;8:e21 http://wwwjmirorg/2006/3/e20/v8i3e20 [PMC free article] [PubMed]
95. Gunther E. The law of attrition revisited: Author's Reply. J Med Internet Res 2006;8:e21 http://wwwjmirorg/2006/3/e20/v8i3e21 [PMC free article] [PubMed]
96. Morrel R. Older Adults, Health Information and the World-Wide Web. New Jersey, US: Lawrence Erlbaum Associates, 2001
97. Milne SAD, Carmichael A, Sloan D, et al. . Are guidelines enough? An introduction to designing web sites accessible to older people. IBM Syst J 2005;43
98. Boquete L, Rodríguez-Ascariz JM, Amo-Usanos C, et al. User-Friendly Cognitive Training for the Elderly: A Technical Report. Telemed J E Health 2011;17:456–60 [PubMed]
99. Teo HH, Oh LB, Liu C, et al. An empirical study of the effects of interactivity on web user attitude. Int J Hum Comput Stud 2003;58:281–305
100. Brouwer W, Kroeze W, Crutzen dNJ R, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011;13:e2. [PMC free article] [PubMed]
101. Kusec S, Brborovic O, Schillinger D. Diabetes websites accredited by the Health On the Net Foundation Code of Conduct: Readable or Not? Stud Health Technol Inform 2003;95:655–60 [PubMed]
102. Thakurdesai PA, Kole PL, Pareek RP. Evaluation of the quality and contents of diabetes mellitus patient education on Internet. Patient Educ Couns 2004;53:309–13 [PubMed]
103. Austin Boren S, Gunlock TL, Krishna S, et al. Computer-aided diabetes education: a synthesis of randomized controlled trials. AMIA Annu Symp Proc 2006:51–5 [PMC free article] [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association