|Home | About | Journals | Submit | Contact Us | Français|
Warfarin is a high-risk medication where patient information may be critical to help ensure safe and effective treatment. Considering the time constraints of healthcare providers, the internet can be an important supplementary information resource for patients prescribed warfarin. The usefulness of internet-based patient information is often limited by challenges associated with finding valid and reliable health information. Given patients' increasing access of the internet for information, this study investigated the quality, suitability and readability of patient information about warfarin presented on the internet.
Previously validated tools were used to evaluate the quality, suitability and readability of patient information about warfarin on selected websites.
The initial search yielded 200 websites, of which 11 fit selection criteria, comprising seven non-commercial and four commercial websites. Regarding quality, most of the non-commercial sites (six out of seven) scored at least an ‘adequate’ score. With regard to suitability, 6 of the 11 websites (including two of the four commercial sites) attained an ‘adequate’ score. It was determined that information on 7 of the 11 sites (including two commercial sites) was written at reading grade levels beyond that considered representative of the adult patient population with poor literacy skills (e.g. school grade 8 or less).
Despite the overall ‘adequate’ quality and suitability of the internet derived patient information about warfarin, the actual usability of such websites may be limited due to their poor readability grades, particularly in patients with low literacy skills.
The World Wide Web (WWW), or simply the ‘web’ or the ‘internet', has become a significant source of health information that is increasing in popularity.1,2 Data from the USA and Europe shows that as many as 61% of the general adult population, including older people (aged 65 years and over), seek information on the internet about their health and related medical issues.1,2 Evidence suggests that the use of internet-based health information has encouraged some patients to be more proactive in the management of their own health/medical conditions.3 It is important to note,however, that this cost-effective and easily accessible resource4,5 for health information is largely unregulated.6 The internet may potentially contain poor quality and unsuitable information,7, 8 which is difficult to read and understand.4,5,9,10
Despite its potential as a significant patient information resource, the internet's usefulness is often limited by the challenges associated with finding good quality health information that comes from authentic and reliable sources.8,9 Previous studies9,11 have reported that more than half of websites provide poor quality health information. Currently available quality evaluation tools, e.g., Health-Related Website Evaluation Form (HRWEF)12 and Quality Component Scoring System (QCSS)13,14 can be used to evaluate the quality of internet-based health information using criteria such as: purpose of the content; disclosure of authors/sponsors; currency of information; accuracy and reliability of information; accessibility and interactivity (e.g. allows patients to make comments or post questions online); readability of information; and graphics/layout of information.6,9,11,15 However, since none of these quality evaluation tools individually addresses each of these criteria,6,16 a comprehensive evaluation of the quality of information available on the internet requires the application of multiple tools.
Suitability is an important aspect of written health information that helps to predict how well the information can be read and understood by general patient populations, and in particular, those with limited literacy skills. Inadequate attention may be paid to the suitability of internet-based health information despite recognition that the internet readership includes an adult population with more than 25% having low literacy skills.1,17 The Suitability Assessment of Materials (SAM)18 is an available and commonly used10,19,20 rating scale, which measures suitability in terms of content, literacy demand, graphics and layout, learning stimulation/motivation and cultural specificity.
Readability formulae, such as the Flesch-Kincaid (F-K) grade formula21 and the SMOG (Simple Measure of Gobbledygook) grade formula,22 are commonly used to assess the readability of health information.23,24 Previous studies9,23,24 using such formulae have shown that in most cases (e.g., between 60-96%) health information on the internet is written at high grade levels (e.g. school grade 12). This is particularly concerning for older patients with poor literacy skills, estimated to be approximately 16% amongst those aged between 60-65 years, and 58% among those aged 85 years and above.25 Further, this older group of patients are more likely to be cognitively challenged, often taking several medications for co-morbid conditions.26 It is recommended therefore that health information should be written at a 6 to 8 school grade level9,27 to ensure that it can be read and understood by the general adult patient population, including those older patients with poor literacy skills.
Increasingly, patients and carers are turning to the internet for information pertaining to complex health problems and/or complicated therapies. A case in point is warfarin therapy, which is one of the 10 most prescribed medications used worldwide and its use has increased by approximately 8-10% per year, mostly because of the increased prescribing of warfarin for older patients (at risk of chronic thromboembolic complications) who have been diagnosed with atrial fibrillation (AF).28-30 Evidence suggests that 55-60% of older patients (aged 65 years or more) with AF are currently treated with warfarin.31-33 Although a potentially life-saving medication, warfarin therapy carries a risk of excessive and potentially life-threatening bleeding complications owing to its complex pharmacology and very narrow therapeutic range of dosage.34 For example, the rate of major bleeding events associated with oral anticoagulation therapy is 7.2 per 100 patient-year as shown in a meta-analysis.35 Further, warfarin is attributed to about 10% of all preventable adverse drug events in high-risk patient groups such as elderly patients.36 Providing patient education and information about warfarin is therefore an essential component for safe and effective warfarin management along with other measures that include regular blood testing and dosage adjustment.34-37 However, health practitioners short of time could fail to effectively convey important warfarin information to their patients.38 The internet may therefore be seen as a very useful supplementary information resource for many patients receiving warfarin therapy. The quality, suitability and readability of patient information about warfarin on the available websites we evaluated two years ago, are unknown. Therefore, our aim in this study was to evaluate the quality, suitability and readability of patient information on the internet about warfarin. The specific objectives were to inform health professionals about the weaknesses and strengths of the available information, as well as to demonstrate a process for the evaluation of the quality, suitability and readability of internet-based health information.
A quantitative study, comprising the evaluation of quality, suitability and readability of health information about warfarin for patients extracted from systematically selected websites, was conducted during August-September, 2009.
Websites providing information about warfarin for adult patients were identified via the key internet search engines: Google, Yahoo, Bing and AltaVista, using search terms such as ‘warfarin', ‘oral anticoagulation', and ‘website'. The first 200 websites (first 20 search pages containing 10 entries per page) yielded by each of the search engines were screened to identify potential websites providing patient information about warfarin, and then accessed to review the content. Inclusion criteria for selecting websites for assessment were: written in the English language, dedicated to patients only, and specific to warfarin alone. Additionally, websites that could not be accessed due to a broken/dead link were excluded.
Validated tools were used to assess the quality, suitability and readability of web-based patient information about warfarin. A brief description of selected evaluation tools is provided below and in Table 1.
The Health-Related Website Evaluation Form (HRWEF)12 and the Quality Component Scoring System (QCSS)13,14 were used to evaluate the quality of the selected websites (Table 1). The principal researcher and three other independent researchers assessed the quality of the information using both tools.
The validated and reliable SAM instrument18 was used to evaluate the suitability of information on the selected websites (Table 1). Flesch-Kincaid reading grades for each of the websites were used by the researchers to determine the ‘reading grade level’ criterion of the SAM instrument. The evaluation of suitability was conducted by the principal researcher and three other independent researchers.
It is generally accepted that, in evaluating the readability grades/scores of written information, the use of more than one readability formula improves the reliability of readability scores,39 hence we used two readability formulae in this study (F-K grade level formula21 and the SMOG formula.22 For F-K calculations, written information from each selected website was copied and pasted into a blank Microsoft Office Word (Professional Edition 2003) document which was then evaluated for readability. The final grade level (i.e., the average school grade level of reading ability required to comprehend the information) for each website was reported as the average of the combined individual grade levels calculated for each webpage. SMOG readability grades were measured by using both the manual SMOG formula22 and the online SMOG calculator.40 Manual SMOG calculations involved copying and pasting the relevant patient information from the websites into a separate blank document, and then evaluated for readability by the principal researcher as well as two independent researchers using the same 30 lines from the beginning, middle, and end of the document. Online SMOG calculations, however, involved cutting and pasting the relevant patient information from each website into the online tool to generate an automatic SMOG readability grade. In doing so, the online SMOG calculator served to confirm the manual SMOG calculations.
To verify the reliability of the findings, the quality and suitability scores were cross-checked against the evaluations undertaken by all the eight assessors (in some cases an individual researcher was involved in more than one evaluation). The quality and suitability coding of all websites were assessed for inter-rater reliability via intra-class correlation coefficients (ICCs), with a high ICC value (maximum 1.0) indicating no variance in the scoring between different assessors. The ICC values calculated for HRWEF, QCSS, and the SAM were 0.8, 0.8 and 0.7 respectively, indicated a fair to good level of consistency for the quality and suitability rating measurements. Since the F-K grades were calculated using computerised software, inter-rater consistency was not measured. The ICC value for manually calculated SMOG grade levels was 1.0 which indicated perfect agreement between the different assessors. The Statistical Package for the Social Sciences (SPSS)41 was used to conduct descriptive statistics (e.g., mean, standard deviation, proportion, range) and to calculate ICC values.
The selection of the potential websites is clearly outlined in Figure 1. Based on the stated inclusion criteria, 11 websites were finally evaluated for the quality, suitability and readability of information. Four of these websites were identified as commercial sites (Table 2) (e.g. published by the pharmaceutical industry or for-profit organisations) and the remaining seven were non-commercial sites (e.g. published by government/education/non-profit organisations or patient support groups).
Table 2 highlights that the quality of the internet-based information about warfarin was at least ‘adequate', ‘good’ or ‘moderate’ for the majority of sites based on the overall scores from the HRWEF and QCSS instruments. The commercial sites were found to have overall poorer quality scores/ratings compared to the non-commercial sites.
Using the HRWEF instrument, none of the websites scored an ‘excellent’ (>90%) rating for quality (Table 2). Whilst seven of the sites achieved ‘adequate’ scores for quality, the remaining four sites (three of which were commercial sites: www.ptinr.com; www.coumadin.com; and www.warfarininfo.com) attained ‘poor’ scores.
Using the overall QCSS scores, two websites, www.anticoagulation.com.au and www.stoptheclot.org, were found to provide information of ‘excellent’ quality, while six other sites provided information of at least ‘good’ quality (Table 2). Similar to HRWEF findings, the commercial sites www.ptinr.com and www.coumadin.com achieved overall ‘poor’ quality scores.
In summary, the non-commercial website www.anticoagulation.com.au and the commercial site www.ptinr.com consistently attained the highest and lowest quality scores/ratings, respectively. Overall, fairly consistent results relating to the quality scores/ratings were yielded using the HRWEF and QCSS evaluation tools (Table 2), except for the www.warfarinfo.com site, which achieved a ‘poor’ quality rating using the HRWEF tool and a ‘very good’ rating using the QCSS.
Based on overall SAM scores (Figure 2), none of the websites achieved ‘superior’ ratings for suitability. Of the six sites attaining ‘adequate’ suitability score, two were commercial sites (www.coumadin.com and www.ptinr.com) (Figure 2). Regarding individual SAM criteria, less than half of the sites adequately addressed issues relating to layout and graphics, and learning motivation (Table 3). For example, relevant graphics/illustrations or subheadings were presented on only three of the non-commercial websites (www.anticoagulation.com.au; www.clotcare.com; and www.mybloodthinner.org). None of the sites addressed the cultural specificity of information relating to language, experience or provision of examples to patients from diverse socio-demographic backgrounds based on the SAM tool. In summary, those websites achieving the highest and lowest suitability scores/ratings were www.anticoagulation.com.au and www.warfarinfo.com, respectively.
Readability grades for all evaluated websites are shown in Table 4. Whilst there was some variability in the actual readability grades attained, the ranking order of the sites (lowest versus highest grades) was consistent across each of the tools used. Brief descriptions of the readability grades determined by each of the readability tools are as follows:
The mean F-K readability grade level was measured as 9.6 (SD 2.1; 95% CI 8.2-11.0). The F-K formula found that four of the websites (including two non-commercial sites; www.anticoagulation.com.au, www.tigc.org; and two commercial sites: www.ptinr.com, www.warfarinfo.com) were written at an approximately grade 8 school level or below (Table 4), in line with what is the recommended level for written health information. The www.ptinr.com site (a commercial site) provided information that was written at the lowest readability grade (grade 6) based on the F-K grades, whereas www.clotcare.com, www.ismaap.org (noncommercial sites) and www.coaguchek.com (commercial site) provided information that was written at the highest readability level (approximately grade 12).
The mean SMOG readability grade levels were measured as 11.0 (SD 1.6; 95% CI 10.1-12.3) and 13.4 (SD 1.7; 95% CI 12.3-14.5) for the manual and online SMOG formulae, respectively. Table 4 highlights that the SMOG readability grades measured by the manual and online calculator ranged between grades 913 and grades 10.4-15.3, respectively (i.e., varying by 1-3 grade levels).
This study is the first, to our knowledge, to have systematically evaluated websites providing information for patients about warfarin therapy. Specifically, this study has audited the quality, suitability and readability of the content of these websites to help gauge their utility for the general adult patient population including those with low literacy skills.34 The results of this study provide some important insights regarding medicines information on the internet, specifically information about warfarin therapy. Overall, the aspects of quality and suitability are adequate; the readability is generally poor and targeted toward patients with high skills.
This study found that the quality of internet-based information about warfarin on most of the evaluated websites was generally adequate. These findings are consistent with the findings from previous studies,14,42,43 which have evaluated health information available on the internet for a range of different chronic diseases. This study also highlights that the quality of information about warfarin on the evaluated commercial websites is poor, which is also consistent with the findings of other studies.9,26,42 This is an important finding given the increasing reliance of patients on the internet as an information resource,1,2 as well as the increasing referral of patients by healthcare professionals to such websites. The relative advantages and disadvantages of non-commercial and commercial sites need to be carefully identified and communicated to patients, given that some commercial sites may not always be reliable sources of good quality information about warfarin.
Similar to the findings of a US-based study10 evaluating the suitability of health information available on the internet about osteoporosis using the SAM instrument, the present study found that information about warfarin on most of the selected websites (including two commercial sites) was generally adequate (i.e. satisfactory) for the general adult population with limited literacy skills. Despite the overall adequate suitability ratings of information on these selected websites, specific deficiencies were identified regarding specific SAM criteria, such as graphics, layout and cultural appropriateness. This study's finding relating to the limited use of graphics/illustrations is consistent with those of other studies27,44 evaluating health information available on the internet. This is unfortunate given that these features help to effectively convey and define complex medical words and terminologies, and/or findings from clinical studies (e.g., risk versus benefit), thus having the potential to improve patient understanding of health information.45,46
In ethnically diverse countries, it is important to consider the cultural appropriateness of the information presented, given the ubiquitous nature of the internet making such information accessible to patients from a range of social, ethnic and cultural backgrounds.47,48 The present study highlights the issue that internet-based health information about warfarin does not always consider issues relevant to patients from non-mainstream ethnic groups, and/or how people from different ethnicities may interpret or apply the information. This reflects previous studies23,48 that have evaluated health information available on the internet about cancer therapy and which reported similar findings. Whilst it is difficult to cater to the needs of all existing socio-ethno-cultural groups, several key health websites have implemented simple measures to help meet the needs of their target populations; for example, the Canadian Breast Cancer Network (www.cbcn.ca) provides links to culturally relevant breast cancer information for aboriginal people, ethnic minorities and those for whom English is a second language. In regard to warfarin therapy, where complex information about lifestyle issues must be clearly communicated to patients (e.g., drug interactions with food/diet, risks of bleeding with normal activities of daily living), it is important to consider and address relevant socio-ethno-cultural ‘habits’ (e.g., diets, religious practices, health beliefs) within internet-based health information.
In regard to the readability, this study highlights that the information presented on most websites is written at readability levels well beyond (e.g. grade 12) that of the average adult population. This result is consistent with Estrada et al (2000)25 and is important given that many patients receiving warfarin therapy are older patients with poor literacy skills. 25 For these patients, as well as others with poor literacy skills (e.g. poorly educated, culturally and linguistically diverse backgrounds), patient information about warfarin should be written at approximately school grade 8 or less to facilitate better understanding.6,25 Importantly, although a difference by approximately 2-4 grades was observed between the readability grades measured by the SMOG and F-K readability formulae, such a difference is not uncommon and is considered the result of variation between different measurement scales.24 Similarly, even though there is a disparity between the calculated reading grade levels for the manual and online SMOG formulae, they are all consistent with regard to the trends in increased reading grade levels required for the different websites. However, the comparatively higher readability grades generated by the online SMOG calculator compared to that of the manual SMOG formula warrant that care should be taken when using the online tools to measure the readability levels of health information available on the internet.
In summary, a wide variability in the quality, suitability and readability scores of internet-based health information about warfarin has been identified in this study. The overall scores indicate that whilst a website may score highly regarding quality parameters it may also achieve poor scores for other evaluated criteria, such as suitability and readability. In the current study, only www.anticoagulation.com.au consistently attained higher scores/ratings in terms of the quality, suitability and readability of information abut warfarin.
Collectively, the study highlights that there are key areas for improvement to help increase the utility of the health and medicines information in relation to warfarin therapy. As a first measure, healthcare professionals might actively be aware of the information presented on websites, as well as purposefully identifying websites that patients may be accessing. By doing so, they will be able to not only identify misinformation but better direct their patients to more effective websites. Secondly, developers of internet-based health information could carefully consider each of these criteria and ensure that the information presented on their sites is relevant and suitable for their target audience (patient population) across each of the three criteria.
In interpreting the findings of this study, it is important to consider some of its potential limitations. Only English language sites were evaluated, and therefore the findings may not be generalisable to those websites written in other languages. The subjective nature of some quality and suitability criteria may potentially introduce variability in scoring, although a fair to good level of inter-rater consistency across the ratings was demonstrated. Furthermore, the SAM instrument principally evaluates the suitability of health information for the general adult population with limited literacy and it is not known to what extent this caters to other patient groups (e.g. older patients). The readability tools may have overestimated the required readability levels because they do not discriminate between commonly and infrequently used terms/words. For example, the analysis would not include commonly used, albeit polysyllabic, clinical and medical terms such as ‘warfarin’ and ‘anticoagulation'. Finally, a conflicting finding regarding the quality score/rating was measured by the HRWEF and the QCSS evaluation tools for the site www.warfarinfo.com. However, such a finding may not be entirely unexpected given the different scoring/rating systems used and characteristics of evaluation criteria included in the above quality evaluation tools.
Whilst the quality and suitability of internet-based health information about warfarin is generally adequate, the actual usability of the sites examined in this study may be limited due to poor readability levels, which could be problematic in patients with poor literacy skills. Since the internet can be readily accessed as a valuable patient information resource, healthcare professionals have an opportunity to direct patients to websites that provide readable information of good quality and suitability.
The authors gratefully acknowledge Renske Hebing, Maya von Moos, Isabella Renman, Shirley Sparla, Judith van Dalem, Ashraf Eissa and Viki Yaputra for their assistance in testing the tools used in the study, and Dr Kylie Williams and Dr Warren Rich for their feedback. The preliminary results of this study were presented (as poster presentation) at the Australasian Pharmaceutical Sciences Association annual scientific meeting, Hobart (Tasmania, Australia) 9- 11 December, 2009.
Not commissioned. Externally peer reviewed
CONFLICTS OF INTEREST
The authors declare that they have no competing interests
No financial support was received for this study
ETHICS COMMITTEE APPROVAL
No ethics approval was required for this study
Please cite this paper as: Nasser S, Mullan J, Bajorek B. Assessing the quality, suitability and readability of web-based information about warfarin for patients. AMJ 2012, 5, 3, 194-203. http//dx.doi.org/10.4066/AMJ.2012.862