|Home | About | Journals | Submit | Contact Us | Français|
In 2005 the Spoken Knowledge in Low Literacy in Diabetes scale (SKILLD) was introduced as a diabetes knowledge test. The SKILLD has not been validated since its introduction.
To perform a validation analysis on the SKILLD.
Cross-sectional observational study of 240 patients with diabetes at an academic family practice center.
SKILLD’s correlation with an oral form of the Diabetes Knowledge Test (DKT) was used to assess criterion validity. A regression model tested construct validity, hypothesizing that SKILLD score was independently related to health literacy and education level. Content validity was tested using Cronbach’s Alpha for inter-item relatedness and by comparing SKILLD items with the content of a National Institutes of Health (NIH) diabetes education website. We assessed inter-rater reliability and bias using Spearman correlation coefficients and sign-rank tests between interviewers scoring the same interview.
The SKILLD demonstrated fair correlation with the DKT (Pearson’s coefficient 0.54, 95% CI=0.49 to 0.66, p<0.001). Health literacy, education level, male gender, household income, and years with diabetes were independent predictors of SKILLD score in the regression model. Cronbach’s Alpha for inter-item relatedness was 0.54. There were some topics on the NIH website not addressed by the SKILLD. The inter-rater correlation coefficient was 0.79 (95% CI 0.56 to 0.91, p<0.001).
The SKILLD is an adequate diabetes knowledge test and is appropriate for people of all literacy levels. However, it should be expanded to more completely evaluate diabetes knowledge.
Diabetes mellitus is a leading cause of morbidity, diminished quality of life, and mortality in the United States.1,2 Although guidelines exist for diabetes care3, less than half of those with diabetes have good control of their glucose.4 The reasons for this lack of good control includes physician, patient, and system factors.5
Self-management support, one of the principles of the Chronic Care Model, involves a collaborative effort of clinicians helping patients acquire the necessary knowledge, skills and confidence to manage their diabetes.6 Patients with diabetes must acquire a significant degree of new knowledge after diagnosis. For example, they must learn to recognize symptoms, risks and adverse consequences of hyperglycemia and hypoglycemia, engage in proper foot care, calculate carbohydrates, and perhaps administer insulin.
Studies have shown that patients who score well on a diabetes knowledge test, with or without an educational intervention, generally have better clinical outcomes than those who score poorly.7–18 Higher diabetes-related knowledge (DRK) has been associated with improved glucose control11, lower blood pressure11, enhanced quality of life10,14,19,20, and improved self-care behaviors such as dietary adherence11,12,21–23, home glucose monitoring11,12,20,21, increased exercise11,21, and foot inspection11,21. Assessment of DRK by the validated Michigan Diabetes Knowledge Test (DKT)24 may help determine what diabetes education is required, and to monitor knowledge educational progress over time.
Unfortunately, limited health literacy may pose a barrier to delivery of diabetes care.25–28 Health literacy is “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”29 Limited health literacy is prevalent30–33 and associated with poorer glycemic control.26,27,34–36
This disparity is addressed with interventions designed to improve diabetes-related knowledge for a range of literacy levels38–40, and with those designed exclusively for patients with low literacy.37,41–43 However, there is only one published DRK test specifically designed for individuals with low literacy: the Spoken Knowledge in Low Literacy in Diabetes scale (SKILLD).44 Since its introduction in 2005, the SKILLD has not been validated in a population of patients with diabetes and wide range of health literacy levels.
The purpose of this study is to further validate the use of the SKILLD for application in clinical and research practice. To do so, we sought to answer the following questions: Does the SKILLD relate to other validated measures of DRK (criterion validity)? Does it relate to other variables the way a satisfactory DRK test should (construct validity)? Are its test items a reasonable sample of what a diabetic patient should know (content validity)? Are the scores meaningfully reproducible between interviewers (inter-rater reliability) and populations (external validity)? We hypothesized that the SKILLD would adequately perform these tasks.
This study is part of a larger research protocol designed to study the association of health literacy with DRK, patient behaviors, and health outcomes. The Ohio State University Biomedical Institutional Review Board approved this protocol.
As part of the larger study protocol, participants were invited to sit for a one-hour interview conducted by trained research assistants. After consenting to participate, subjects provided demographic information and answered a verbal questionnaire. Data were collected from their medical record. Subjects took in random order the following battery of tests: the Michigan Diabetes Knowledge Test (DKT), the Rapid Estimate of Adult Literacy in Medicine (REALM), and the SKILLD, and two other literacy tests related to the larger protocol. Participants were asked at the end of each test to rate the difficulty of that test.
The DKT is a multiple-choice exam meant to test a patient’s knowledge about topics relevant to diabetes self-care.24 The first 16 questions are relevant to any patient with diabetes, whereas the last seven apply only to those who are using insulin. This test was administered orally, which has been shown to correlate extremely well with the written DKT and is recommended for persons with unknown health literacy or poor visual acuity (Miser WF, Jeppesen KM, Coyle J, unpublished data, 2009).
The REALM, perhaps the most widely used test of health literacy that can be administered in one to two minutes, consists of three lists of 22 words each, to be read aloud by the test taker.45 A score is given based on how many words the subject pronounces correctly. Although this test is meant primarily as a screening tool, its results correlate well to other previously developed literacy tests.
The SKILLD was developed in 2005 by Rothman et al. to screen patients with diabetes and low literacy for deficits in self-care knowledge.44 Administered orally, it is a 10-item, open-answer test. Questions ask about behaviors patients should have to best manage their diabetes. As the questions are open-ended, alternative wording is provided for each of the 10 questions. Question #3 of the SKILLD asks how to treat low blood sugar and requires an answer regarding some form of oral glucose plus rechecking blood sugar. Although this is correct according to the medical standards established by the American Diabetes Association3, there is some concern that checking the glucose may not be considered by some to be “treatment.” As such, we separately recorded whether the subjects mentioned to check their blood sugar. A subsample of SKILLD tests was audio recorded and graded by three different graders to determine inter-rater reliability.
The population consisted of all patients diagnosed with diabetes mellitus (ICD code 250.xx) seen at least twice at the Ohio State University Rardin Family Practice Center in Columbus, Ohio from August 25, 2006 to December 5, 2007. Patients were contacted either in person at the office, via telephone shortly prior to an appointment, or randomly if no appointment was scheduled. Potential participants were excluded if they were under 18 years of age, pregnant, did not speak English, or had a known cognitive defect that would interfere with their ability to complete the interview.
Time to complete the test, difficulty ratings, and relationships of SKILLD with demographic variables were summarized and face validity was assessed. Formal analyses were designed to test criterion validity, construct validity, content validity and reliability of the SKILLD. All analyses were performed with STATA SE version 9.2 (StataCorp. 2005. College Station, TX). P-values were considered significant at alpha<0.05 unless otherwise mentioned.
To assess criterion validity, the extent to which a test relates to another previously validated metric, we compared SKILLD scores to oral DKT scores using Pearson’s correlation coefficient, with an acceptable value being>0.6. We calculated confidence intervals using Fisher’s Z transformation. To establish construct validity, the extent to which a test behaves as we would expect it to given our understanding of the concept to be measured, we hypothesized that DRK would be independently related to both education level attained and health literacy level. We created a regression model predicting SKILLD using the REALM and highest education level attained as independent variables. The model also controlled for age, gender, race, household income, use of insulin, and years with diabetes. We considered the construct valid if both education level and health literacy were independent, significant predictors of SKILLD score in this model, after adjusting for covariates.
We tested content validity, the extent to which a test’s material matches the material we wish to test, using two methods. First, we calculated Cronbach’s Alpha for inter-item relatedness, considering 0.6<alpha <0.9 an ideal range. Second, we compared the concepts tested in the SKILLD to the domain of topics found in seven educational pamphlets, a video, and an informational web page produced by the National Institute of Health’s National Diabetes Education Program (NDEP).46 To assess inter-rater reliability, agreement between various test administrators, we calculated the Fleiss Kappa statistic for each item using scores obtained from the three different graders previously described. We tested overall reliability and accuracy by calculating Spearman correlation coefficients and sign-rank test p-values, comparing the original grader to the other two graders.
We contacted 384 patients with diabetes to participate in this study; 40 were excluded (6 were blind, 25 spoke little or no English, nine had a cognitive deficit). Of the 344 remaining, 243 consented to participate (71% response rate). During test administration, one subject was excluded due to newly discovered dementia, one subject refused to finish the battery of tests, and results from one subject’s SKILLD test was incomplete, leaving complete data available for analysis for 240 subjects. Subjects who participated were not significantly different from non-participating patients with respect to age, gender, race, and years with diabetes. Table Table11.
The SKILLD took a median of 225 (inter-quartile range=182, 273) seconds to complete. Its median difficulty rating was “easy,” compared to “okay” for the oral DKT. Crude associations with SKILLD score are given in Table 2 (“Unadjusted” columns). A higher SKILLD score was obtained more frequently in subjects with non-black race, higher income, use of insulin, higher education level, higher REALM score, shorter test time, and easier difficulty rating.
The research assistants indicated that the SKILLD was able to discriminate a person’s understanding of diabetes. However, they observed that for question #1 and #2, regarding hyper- and hypoglycemia, subjects often stated, “I know how it feels” to have high or low blood sugar, but were unable to indicate specific symptoms. Questions were clearly written, with the exception of #3. “Check blood sugar” is a required part of the answer, but the question appears to focus more on treatment than monitoring. Only 4% of subjects mentioned checking blood sugar in their response.
The open-ended style of the questions, though helpful in an oral test, may lead to ambiguities in grading. The following illustrate responses that may rightly be graded as correct or incorrect. Question #6 asks about frequency of eye checks. “Every six months” is more frequent than the expected answer of “at least yearly”, but may be what a subject’s ophthalmologist has recommended. Question #7 is regarding normal blood sugar values. “90-100” is not the given answer range, but it falls within the accepted range of “70 (or 80) to 120”. Question #9 asks about frequency and duration of exercise. “Every day for 30 minutes” or “three times a week for an hour” could be acceptable exercise regimens but fall outside of the limits of a correct response (“3-5 times per week for a total of 30–45 minutes each”).
The correlation coefficient between the SKILLD and the oral DKT was 0.58 (95% CI 0.49 to 0.66, p<0.001). The regression model assessing construct validity is found in Table 2 (“Adjusted” columns). Both REALM score and education level were independently associated with SKILLD score after correcting for confounders. Male gender, household income, and years with diabetes were also significantly related to SKILLD in this model. The inter-item Cronbach’s Alpha for the SKILLD scale was 0.54.
Educational materials from the NDEP provide enough information to answer all questions on the SKILLD, but the program also includes information not addressed by the SKILLD, such as frequency of HbA1c testing, blood pressure, triglyceride and cholesterol goals, nutrition, mental health implications, smoking cessation, oral care, and causes of high or low blood glucose.
We recorded 23 interviews for inter-reliability testing. Kappas for individual items are given in the appendix. All calculated Kappa p-values were <0.001. Grading of SKILLD scores were significantly correlated between graders: Spearman’s rho was 0.79 (95% CI 0.56 to 0.91, p<0.001).
Rothman et al. developed and validated the SKILLD in 217 low income, poorly educated, low literacy patients with poorly controlled diabetes (average A1c was 10.8%).44 They found that the SKILLD had good reliability and validity in measuring diabetes knowledge in those with type 2 diabetes and low literacy. They also demonstrated that individuals with low literacy or lower economic status had poorer knowledge of diabetes, and that the results of the SKILLD was associated with metabolic control (that is, the higher the score of the SKILLD, the better the A1c).
Our construct validity analysis leads us to believe that health literacy and education level correlate well with SKILLD score independent of one another. Rothman et al. reported similar findings. This is indicative of a good knowledge test. Other portions of our analysis raise some noteworthy questions.
We analyzed the criterion validity using Pearson’s coefficient in which we evaluated how well the SKILLD predicted knowledge compared to the oral DKT. Previously two of the authors (WFM and KMJ) observed that the oral DKT evaluates knowledge as well as the written DKT (unpublished, 2009). The results of the current analysis demonstrate an only moderate correlation between the knowledge evaluated by the SKILLD and that evaluated by the oral DKT. This shows that, while they are similar, the SKILLD and the oral DKT are not the same test. It does not, however, indicate that one test is superior to the other, a point brought up in the original paper, which had no criterion validity measure. The SKILLD may be preferred because of its lower perceived difficulty, the practical nature of the questions, and its coverage of general concepts about diabetes management.
Our content validity analysis revealed a separate shortcoming that should be addressed. The domain of knowledge provided by the NDEP website was not sufficiently tested in the SKILLD, which likely led to test items that had little to do with one another resulting in a lower inter-item relatedness (Cronbach’s Alpha=0.54). Rothman et al. attempted to create a valid short version of the SKILLD due to time constraints of administering a knowledge test in a busy clinical practice.44 Our analysis suggests that the SKILLD may require more questions to fully evaluate the full set of knowledge available on the NDEP website. Attempting to shorten the test may miss useful information. Rothman et al. make the same point, noting that the SKILLD is more skill-related and lacks questions related to nutrition or medication.
Our conclusion about the deficits observed in criterion and content validity is that 1) the format and style of questions in the SKILLD are more practical than those in the DKT and may be an excellent way to test for DRK, but 2) the SKILLD does not have enough questions to adequately assess all domains of DRK. Consideration should be given to expanding the SKILLD into these other domains. Modifying question # 3 could be done as part of this process, since patients may omit “checking blood sugar” as they may not realize this to be a “treatment”.
Our regression analysis indicates that certain variables appear to predict DRK as measured by the SKILLD score. The results support previous findings that patients with a higher education level, health literacy, and income, and use insulin are more likely to have a higher SKILLD score.44 Previous findings found that duration of disease, younger age, and lower HbA1c were also related to higher SKILLD scores.44 We did not investigate a relationship between the SKILLD score and clinical outcomes as our aim was to determine if the SKILLD is a good knowledge test, independent of whether it is associated with metabolic control. Of note, our crude analysis identified a relationship between “non-black race” and DRK, but our regression analysis did not identify race as a predictor of DRK. We, therefore, conclude that race may have an association with another variable that independently correlates with DRK.
Most previous studies have shown a relationship between diabetes education and improved metabolic control.8–16 Our current investigation used schooling level as a measure of education, as did Rothman et al.44 A relationship between education in a disease management program and highest level of education attained may exist, but our literature search yielded no studies that drew direct comparisons. While further investigation is necessary to elucidate such a relationship, our analysis suggests that individuals with a higher education level have more DRK than those with a lower education level. Disease management programs may be used as a tool to increase DRK among those with lower health literacy. However, it is important to note that educational interventions increase knowledge of disease for all participants, and those with lower health literacy are unlikely to experience increases in disease knowledge sufficient to “catch up” to those with higher levels of health literacy enrolled in the same program.37
Current literature suggests that a higher level of health literacy27,37 and education7–16 are related to improved knowledge of disease as well as better disease outcomes.26,27,34,35 There is also an association between higher levels of DRK and many measurements of disease outcomes and self-care behaviors,4,6–8,16–20,36 however this association does not appear to be as strong,11,14,19,20,23,47 suggesting that the relationship between DRK and behaviors that lead to better self-care is not always predictable. “While adequate diabetes knowledge is a prerequisite to good self-care…proper knowledge does not always correlate with patient behaviors”.44
Our analysis demonstrated that higher levels of health literacy and education level independently correlate with higher levels of DRK. We did not attempt to demonstrate the link between DRK (i.e. SKILLD score) and HbA1c (a disease outcome) but we feel confident that the majority of the current literature (as well as our analysis) indicates that the model proposed in Figure 1 is accurate (with the understanding that the relationship between DRK and behaviors may be affected by other variables as well).36
Our patient population was different than that of Rothman et al. who first introduced the SKILLD.40 Our population appears to have a higher level of income, education, and health literacy. While cross verifying a study in a different population is helpful in establishing test validity, neither of these populations was sampled to be nationally representative. Different conclusions may be drawn relating to the use of the SKILLD in private or rural practice, for example. Secondly, although our sample size was adequate to draw our conclusions, we recognize that a more powered study may have led to different conclusions, particularly regarding criterion validity of the SKILLD or other variables that may have predicted SKILLD score.
In summary, we consider the SKILLD an adequate diabetes knowledge test with a very useful question format, but should be refined to more fully evaluate DRK. We maintain that any modality to improve the quality of care for patients, enhance the delivery of care by providers, and decrease the cost of treating expensive chronic diseases should be evaluated.
The authors would like to thank Mark Stevens, Charlie Pizanis, Stephen Wilkes, Scott Gspandl, Laura Holzer, Patrick Ellsworth, Joan Allen, and the very helpful staff at The Ohio State University Rardin Family Practice Center.
This research was funded by The Ohio State University Crisafi -Monte Primary Care Cardiopulmonary Endowment, and by grant T32RR023260 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH).
Conflict of Interest KMJ, BPH, and MR report no conflicts of interest. WFM has received research funding from Abbott Laboratories, Centocor Ortho Biotech, Inc., Bristol-Myers Squibb, Daiichi Sankyo Co., LTD, Eli Lilly and Company, Forest Research Institute, GlaxoSmithKline, Merck & Co., Inc., Novartis, Novo Nordisk, Pfizer Inc., Sanofi Pasteur, and Wyeth-Ayerst.