PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Med Care. Author manuscript; available in PMC Apr 1, 2013.
Published in final edited form as:
PMCID: PMC3318988
NIHMSID: NIHMS355343
5-Step Methodology for Evaluation and Adaptation of Print Patient Health Information to Meet the <5th Grade Readability Criterion
Felicia Hill-Briggs, PhD,1,2,3 Kristina P. Schumann, M.A.,2,4 and Ogechi Dike, B.A.5
1Welch Center for Prevention, Epidemiology, and Clinical Research, Johns Hopkins Medical Institutions
2Department of Medicine, Johns Hopkins University School of Medicine
3Department of Health, Behavior, and Society, Johns Hopkins Bloomberg School of Public Health
4Department of Psychology, University of Maryland Baltimore County
5UMDNJ-Robert Wood Johnson Medical School
Corresponding Author: Felicia Hill-Briggs, PhD, ABPP, Welch Center for Prevention, Epidemiology, and Clinical Research, 2024 E. Monument Street, Suite 2-600, Baltimore MD, 21287, Phone (410) 502-2794. Fax (410) 955-0476, fbriggs3/at/jhmi.edu
Background
In the setting of declining U.S. literacy, new policies include use of clear communication and low literacy accessibility practices with all patients. Reliable methods for adapting health information to meet such criteria remain a pressing need.
Objectives
To report method validation (Study 1) and method replication (Study 2) procedures and outcomes for a 5-step method for evaluating and adapting print health information to meet the current low literacy criterion of <5th grade readability.
Materials
Sets of 18 and 11 publicly-disseminated patient education documents developed by a university-affiliated medical center.
Measures
Three low-literacy criteria were strategically targeted for efficient, systematic evaluation and text modification to meet a <5th grade reading level: sentence length <15 words, writing in active voice, and use of common words with multisyllabic words (>2–3 syllables) minimized or avoided. Inter-rater reliability for the document evaluations was determined.
Results
Training in the methodology resulted in inter-rater reliability of 0.99–1.00 in Study 1 and 0.98–1.00 in Study 2. Original documents met none of the targeted low literacy criteria. In Study 1, following low-literacy adaptation, mean reading grade level decreased from 10.4±1.8 to 3.8±0.6 (p<0.0001), with consistent achievement of criteria for words per sentence, passive voice, and syllables per word. Study 2 demonstrated similar achievement of all target criteria, with a resulting decrease in mean reading grade level from 11.0±1.8 to 4.6±0.3 (p < 0.0001).
Conclusions
The 5-step methodology proved teachable and efficient. Targeting a limited set of modifiable criteria was effective and reliable in achieving <5th grade readability.
Low health literacy, a persisting public health challenge in the U.S., is linked to poorer health status, increased morbidity and mortality, and higher healthcare costs.1,2 The most recent National Assessment of Adult Literacy (NAAL) revealed that only 12% of American adults demonstrate proficient literacy with written materials.3,4 Although lower literacy among racial and ethnic minorities has been found to contribute to health disparities,5,6 the NAAL revealed a population decline in literacy over the past ten years, with greatest decline among White adults and persons with a high school diploma, GED, or some vocational education.3,4
These findings suggest that lower literacy impacts the U.S. adult population broadly, warranting population-level intervention. A universal precautions approach has been recommended;7 recent federal legislative action mandates use of clear, plain language in written documents for the public, across government sectors.8
The current, low-literacy criterion for readability of health information is <5th grade,911 yet most patient education materials and websites remain at 12th grade or college level readability.12,13 Resources are available to assist with awareness of health literacy recommendations and materials adaptation (See Appendix A). However, an efficient, systematic approach that consistently and reliably results in <5th grade readability, especially while preserving essential medical terms, remains a need for healthcare organizations and professionals.14,15
Appendix A
Appendix A
Agency Health Literacy Resources
This paper describes a 5-step approach for essential evaluation and efficient adaption of print patient education text to meet <5th grade readability. We report: a) a method validation study (Study 1) to test effectiveness of the 5-step approach in achieving targeted low-literacy criteria in a set of 18 publicly-disseminated patient education documents, and b) a method replication study (Study 2) to test reliability of the personnel training and consistency of the 5-step approach in achieving targeted literacy criteria in an additional set of 11 patient education documents.
Development of the Literacy Evaluation and Adaptation Methodology
Low-Literacy Criteria
Criteria for low-literacy patient education materials were derived from the Centers for Disease Control and Prevention (CDC),9 National Institutes of Health/National Cancer Institute (NIH/NCI),10 and Doak and colleagues.11 Of 32 consensus criteria available on literacy demand, formatting, and behavioral activation characteristics14 (See Appendix B), the following literacy-demand criteria were selected for efficiency and effectiveness in adapting text to meet a <5th grade reading level: 1) sentence length <15 words; 2) writing in active voice (<5% passive voice); and 3) use of common words, with multisyllabic words (>2–3 syllables) minimized or avoided. These criteria were deemed modifiable, and reading grade level, sentence length and passive voice can be readily evaluated using available electronic resources. (See Hill-Briggs and Smith14 for descriptions of modifiable low-literacy criteria, operational definitions, and procedures.)
Appendix B
Appendix B
Consensus Criteria for Evaluation of Literacy-Demand (n = 23) and Behavioral Activation (n = 9) Characteristics of Print Patient Education Materials
5-Step Methodology
The 5-steps in the approach are shown in Table 1. Importantly, the approach begins with comprehensive evaluation to determine adaptation needs at the sentence level of analysis. For the two studies reported herein, readability statistics available in Microsoft Word, 2003/2007 were used to evaluate and adapt the documents. The Flesch-Kincaid Grade Level test rates text on a U.S. school grade level, with a score of 5.0 indicating that an average fifth grader can understand the document.16
Table 1
Table 1
5-Step Methodology
Process of Adapting Documents for Low Literacy
Following document evaluation, low-literacy adaptation was approached strategically, targeting those sentences that were at or above a 5th grade reading level, and the specific elements of the evaluated sentence that were not within the low-literacy targets. Specific adaptation methods included use of common, familiar words; use of resources (e.g. thesaurus) for word and phrasing alternatives; use of a simple, direct, and straightforward manner for sentence wording; expressing one main thought at a time; refraining from adding multiple phrases, clauses, or modifiers within a sentence; use of formatting changes such as bullets or tables when appropriate; avoidance of scientific jargon or medical terminology that was above the target readability; and consistency in use of the chosen words or phrases used to substitute for frequently occurring medical language. When medical terminology that exceeded the readability level was necessary, those technical, concept, category, or value judgment terms were introduced with an understandable explanation or example, prior to introduction of the scientific terminology. The scientific term, rather than the explanation, was placed in parentheses. This allows the reader to encounter the understandable explanation in the course of the text, with the scientific term treated as parenthetical to the explanation. Following low-literacy adaptation, the adapted documents underwent evaluation of readability criteria, and the readability statistics of the original and literacy-adapted documents were compared.
Study 1: Methodology Validation
The 5-step methodology was tested using a set of 18 publicly disseminated patient education documents covering a variety of diabetes and lifestyle topics, developed by a university-affiliated medical center. Three research assistants (high school, bachelor’s and master’s level education, respectively, with no prior experience in low literacy evaluation, writing, or document adaptation) were hired and trained as literacy adaptation specialists for the study. Assistants received an initial 2-hour in-person training in the 5-step methodology prior to working with the documents. They then received a 1-hour supplemental training, during which instruction and materials were provided to address specific challenges in the treatment of medical and legal terminology, and supplemental material in documents (e.g. resource lists). A combined literacy adaptation manual was developed for training new personnel, with the following content: Performance of the 5-Step Method, Additional Exclude/Include Criteria for Readability Evaluations, Treatment of Medical Terminology, Legal Content, Tip Sheet, and Demonstration of Method(s) and Practice. Each assistant received supervision to ensure quality of the adaptation and consistency of content with the original document. Each document was evaluated by two assistants for determination of inter-rater reliability.
Study 2: Methodology Replication
Study 2 was conducted using a set of 11 publicly disseminated patient education documents from the same original source. Two different literacy evaluators/adapters (bachelor’s level education, with prior editorial experience, but without formal training in low literacy evaluation and adaptation) participated in this study. They completed the comprehensive 2-hour total training, using the manual created in Study 1.
Data Analysis
Inter-rater reliability for document-level literacy statistics was assessed using Pearson product-moment correlation coefficients. Inter-rater reliability was determined for original and adapted documents. Document-level literacy statistics of the original and adapted documents were compared using paired-samples t-tests to determine statistical significance of mean improvements in literacy variables. Analyses were conducted using PASW Statistics 18, Release Version 18.0.0 (© SPSS, Inc., 2008, Chicago, IL).
Study 1
Following the basic training, inter-rater reliability for document-level readability evaluation ranged from 0.81 – 0.99 for the set of original documents. After adding the 1-hour supplemental training, inter-rater reliability of the literacy evaluations increased to 0.99 – 1.00 for each pair of evaluators, and for each original document. Inter-rater reliability remained at 0.99 – 1.00 for adapted documents.
Readability statistics for the documents are shown in Table 2. Mean reading grade level of the original documents was 10.4 ± 1.8, with no documents meeting low-literacy criteria investigated. Following adaptation, mean reading grade level of the documents was 3.8 ± 0.6, with each document meeting the < 5th grade criterion. All adapted documents met targets for passive voice and number of words per sentence. Table 3 illustrates an original document excerpt and the literacy-adapted version of that excerpt. Rephrasing of text, shortening of sentences to simple sentence constructions, and word substitutions were the most commonly used text adaptations in the set of 18 documents, while the most commonly used format changes were bulleting and tables, particularly for content requiring medical terminology (i.e. medication lists and descriptors).
Table 2
Table 2
Comparison of Readability Statistics of Original and Literacy-Adapted Documents in Method Validation Study (Study 1)
Table 3
Table 3
Sample Original and Literacy-Adapted Document Excerpts with Low-Literacy Readability Evaluation
Syllables per word was compared with the number of characters per word readability statistic (Table 4). Both statistics improved with document literacy adaptation. The use of common, familiar words in the adaptation process resulted in a decrease in mean percentage of words with >3 syllables from 8.3% ± 4.9 in the original documents to 2.5% ± 2.2 in the adapted documents.
Table 4
Table 4
Average Characters per Word and Average Syllables per Word in Paragraph Exceptsa
Study 2
Readability statistics are shown in Table 5. After 2-hour comprehensive training, inter-rater reliability for literacy evaluation of original documents was 0.99 – 1.00, and for adapted documents, 0.98 – 1.00. Mean reading grade level of the original documents was 11.0 ± 1.8., and no original documents met target low-literacy criteria. Study 2 yielded equally successful adaptations as Study 1 (p-values <0.0001).
Table 5
Table 5
Comparison of Readability Statistics of Original and Literacy-Adapted Documents in Method Replication Study (Study 2)
The 5-step methodology, which relies on commonly available electronic readability tools, demonstrated effectiveness as an evaluation and adaptation approach to meet the <5th grade readability criterion for public health information. Effectiveness of the approach was demonstrated in both the method validation and replication studies. Personnel with no prior experience were able to be trained to perform the method successfully.
Multiple techniques, used in combination, were needed to meet targeted readability statistics. Mere word substitution was not effective as an overall technique for document adaptation; it was prone to inaccurate or awkward sentence constructions, and it often did not reduce sentence-level readability sufficiently. Rephrasing was necessary to preserve meaning and to introduce essential medical terms, while still reducing literacy demand. All instances of passive voice required rephrasing. Changing passive voice phrasings (e.g. Eating foods that have fewer calories is used to help with losing weight) to declarative statements (e.g. Eat foods with fewer calories to help lose weight) not only reduced reading grade level but made the messages more actionable for the reader. Further, we found that meeting the <5th grade criterion was realistic, and reading levels as low as 2nd and 3rd grade were achievable without using presentation styles or language that might be perceived as more appropriate for children than adults.
Our method utilized the Flesch-Kincaid formula to assess reading grade level. It is important to note additional available readability formulas that are widely used, including the FOG Index, the Simple Measure of Gobbledygook (SMOG), and the Fry readability graph. Although these formulas are each calculated differently, they have been found to correlate highly with each other, and there has been no evidence to support increased accuracy of one over another in determining reading grade level.17,18 All four formulas can be calculated manually; the Flesch-Kincaid formula has an accessible electronic calculation, which provides the benefit of time efficiency and practicality in routine use and with large numbers of documents. A limitation of using the Microsoft Word automated readability statistics is that the syllables per word statistic is not provided; characters per word is calculated. This study evaluated number of syllables per word using the Doak11 method, and a comparison table, with number of characters per word, is provided. During the adaptation process, multisyllabic words were readily identified without a formal calculation. The process of using common, more familiar words automatically resulted in fewer syllables at the sentence and document levels of analysis.
With regard to literacy adaptation content evaluation, the articles did not undergo a process of pre-/post- evaluation for preservation of meaning as part of research. However, after completing literacy adaptation, the adapted articles were reviewed alongside the original articles, by the institutional business office responsible for materials development and copyright. The business office approved the literacy-adapted articles as consistent in content and meaning. The articles then underwent review by representatives of a national healthcare organization which, after reviewing the articles, requested use of the full sets of literacy-adapted (but not the original) articles for dissemination as routine patient education materials to their members/clients. Although data were not collected during these reviews, both professional entities determined the literacy-adapted materials to be of high quality and value, consistent with or exceeding that of their non-literacy-adapted counterparts.
Testing acceptability and effectiveness of the materials in educating the public is ultimately needed to evaluate utility of the 5-step method. The method has been used to adapt patient education materials in previously reported clinical outcome studies, with findings of high satisfaction and ease of understanding ratings by adults with literacy as low as ≤3rd grade as well as those with average literacy,19 and by persons with mild cognitive and/or visual impairment,20 another subgroup for whom clear communication is particularly necessary. Intervention studies using this literacy-adaptation method have resulted in increased patient knowledge19,21 and improved behaviors and disease control21 in lower-literate samples.
There are directions deserving of further research and development. First, the 2-hour training, which was efficient and reliable, warrants testing in alternative instruction modes (e.g. web-based, electronic media such as CD-Rom). Second, in developing and testing the 5-step method, consistent “rules” for reducing literacy demand emerged for several elements of language (e.g. treatment of numbers, contractions, sentence structure for active voice) that may be amenable to software programming. Third, there is a need to evaluate the efficacy of the 5-step process in other languages. The overall framework for a systematic approach to low literacy adaptataion is transferrable, as are some elements of formatting. However, we do not expect that the specific literacy demand criteria will hold across languages and cultures. Effective low literacy text modification is based upon understanding of the language construction, parts of speech, characters utilized (length, letter pairings, etc.), and meaning/messaging, all of which determine how information is processed, stored, and learned. The neuropsychology literature reveals significant differences in the structure, processing and comprehension of Spanish language, for example, such that the English language low-literacy criteria may not be directly transferrable. Low-literacy adaptation in Spanish, and other languages, does require examination of those features of the respective language itself.
Finally, the problem of declining literacy in the U.S. over the past decade is one of great importance that likely requires both educational and political reform to reverse. The methodology described herein allows health information to be accessible to the growing proportion of the population with basic to below basic literacy; this is essential for improving patient self-management and health outcomes.2225 However, meeting this current demand should not obfuscate the concurrent need for actions directed toward stopping and reversing the low literacy trends in the U.S.
Acknowledgments
This research was supported by NIDDK Diabetes Research and Training Center grant P60 DK079637 (O.D.) and NHLBI grant R01HL08975 (F.H-B., K.S.). Data were presented in part at the Annual Health Literacy Conference, Washington, DC, October 2009.
1. Berkman ND, DeWalt DA, Pignone M, Sheridan SL, Lohr KN, Lux L, Sutton SF, Swinson T, Bonito AJ. Summary, Evidence Report/Technology Assessment No. 87. AHRQ Publication No. 04-E007-1. Rockville, MD: Agency for Healthcare Research and Quality; 2004. Literacy and health outcomes.
2. Institute of Medicine. Health literacy: A prescription to end confusion. Washington, D.C: National Academies Press; 2004.
3. Kutner M, Greenberg E, Jin Y, Boyle B, Hsu Y, Dunleavy E. Literacy in Everyday Life: Results from the 2003 National Assessment of Adult Literacy NCES 2007-480. Washington, DC: National Center for Education Statistics, U.S. Dept. of Education; 2007.
4. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: Results from the 2003 National Survey of Adult Literacy NCES 2006-483. Washington, DC: National Center for Education Statistics, U.S. Dept. of Education; 2006.
5. Sentell TL, Halpin HA. Importance of adult literacy in understanding health disparities. J Gen Intern Med. 2006;21:862–866. [PMC free article] [PubMed]
6. Osborn CY, Paasche-Orlow MK, Davis TC, Wolf MS. Health literacy: An overlooked factor in understanding HIV health disparities. Am J Prev Med. 2007;33(5):374–378. [PubMed]
7. Agency for Healthcare Research and Quality. AHRQ Publication No. 10-0046-EF. Rockville, MD: AHRQ; 2010. [September 8, 2011]. Health Literacy Universal Precautions Toolkit. http://www.ahrq.gov/qual/literacy/index.html.
8. Plain Writing Act of 2010. [Accessed April 16, 2011];Public Law 111-274, 111th Congress [Plain Language web site] Available at: http://www.plainlanguage.gov/plLaw/index.cfm.
9. Centers for Disease Control and Prevention. Scientific and Technical Information Simply Put. 2. Atlanta: Office of Communication, Centers for Disease Control and Prevention; 1999.
10. National Cancer Institute. Clear and Simple: Developing Effective Print Materials for Low-Literate Readers. Bethesda, MD: U.S. Department of Health and Human Services; 1994.
11. Doak CC, Doak LG, Root JH. Teaching Patients With Low Literacy Skills. 2. Philadelphia: JB Lippincott Company; 1996.
12. Aldridge MD. Writing and designing readable patient education materials. Nephrol Nurs J. 2004;31:373–377. [PubMed]
13. McInnes N, Haglund BJ. Readability of online health information: implications for health literacy. Inform Health Soc Care. 2011 EPub Ahead of Print. [PubMed]
14. Hill-Briggs F, Smith AS. Evaluation of diabetes and cardiovascular disease print patient education materials for use with low health literate populations. Diabetes Care. 2008;31(4):667–671. [PubMed]
15. Kellerman R, Weiss BD. Health literacy and the JAMA Patient Page. JAMA. 1999;282(6):525–527. [PubMed]
16. Microsoft Office Support. [Accessed April 16, 2011];Test your document’s readability: Understand readability scores [Microsoft Office Support web site] Available at: http://office.microsoft.com/en-us/word-help/test-your-document-s-readability-HP010148506.aspx#BM2.
17. Mead CD, Smith CF. Readability formulas: cautions and criteria. Patient Educ Couns. 1991;17(2):153–158.
18. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006 Jun;33(3):352–73. [PubMed]
19. Hill-Briggs F, Renosky R, Lazo M, Bone L, Hill M, Levine D, Brancati FL, Peyrot F. Development and Pilot Evaluation of Literacy-Adapted Diabetes and CVD Education in Urban, Diabetic African Americans. J Gen Intern Med. 2008;23(9):1491–1494. [PMC free article] [PubMed]
20. Hill-Briggs F, Lazo M, Renosky R, Ewing C. Usability of a diabetes and cardiovascular disease education module in an African American, diabetic sample with physical, visual, and cognitive impairment. Rehabil Psychol. 2008;53(1):1–8.
21. Hill-Briggs F, Lazo M, Peyrot M, Chang Y, Doswell A, Hill M, Levine D, Wang N, Brancati F. Effect of problem-solving-based diabetes self-management training on diabetes control in a low income patient sample. J Gen Intern Med. 2011;26(9):972–8. [PMC free article] [PubMed]
22. Williams MV, Baker DW, Parker RM, Nurss JR. Relationship of functional health literacy to patients’ knowledge of their chronic disease: a study of patients with hypertension and diabetes. Arch Intern Med. 1998;158:166–172. [PubMed]
23. Schillinger D, Grumbach K, Piette J, et al. Association of health literacy with diabetes outcomes. JAMA. 2002;288:475–482. [PubMed]
24. Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997;87:1027–1030. [PubMed]
25. Yin HS, Johnson M, Mendelsohn AL, Abrams MA, Sanders LM, Dreyer BP. The health literacy of parents in the United States: a nationally representative study. Pediatrics. 2009;124 (Suppl 3):S289–298. [PubMed]