PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of amjepidLink to Publisher's site
 
Am J Epidemiol. 2012 October 15; 176(8): 738–743.
Published online 2012 September 25. doi:  10.1093/aje/kws156
PMCID: PMC3571247

Validity and Reliability of Self-reported Diabetes in the Atherosclerosis Risk in Communities Study

Abstract

The objective of this study was to assess the validity of prevalent and incident self-reported diabetes compared with multiple reference definitions and to assess the reliability (repeatability) of a self-reported diagnosis of diabetes. Data from 10,321 participants in the Atherosclerosis Risk in Communities (ARIC) Study who attended visit 4 (1996–1998) were analyzed. Prevalent self-reported diabetes was compared with reference definitions defined by fasting glucose and medication use obtained at visit 4. Incident self-reported diabetes was assessed during annual follow-up telephone calls and was compared with reference definitions defined by fasting glucose, hemoglobin A1c, and medication use obtained during an in-person visit attended by a subsample of participants (n = 1,738) in 2004–2005. The sensitivity of prevalent self-reported diabetes ranged from 58.5% to 70.8%, and specificity ranged from 95.6% to 96.8%, depending on the reference definition. Similarly, the sensitivity of incident self-reported diabetes ranged from 55.9% to 80.4%, and specificity ranged from 84.5% to 90.6%. Percent positive agreement of self-reported diabetes during 9 years of repeat assessments ranged from 92.7% to 95.4%. Both prevalent self-reported diabetes and incident self-reported diabetes were 84%–97% specific and 55%–80% sensitive as compared with reference definitions using glucose and medication criteria. Self-reported diabetes was >92% reliable over time.

Keywords: diabetes, validation study

Diabetes is often defined exclusively by using a self-reported diagnosis in cross-sectional and prospective epidemiologic studies (13). Previous studies have found that self-reported diabetes is highly specific when medical chart reviews or physician surveys are used to confirm cases (47). However, the degree of underascertainment of diabetes cases by self-report is less well described, and previous studies have not quantified the performance of self-reported diabetes by using different reference definitions.

In the United States, diabetes is estimated to affect approximately 26 million people; about one third of these cases are undiagnosed (8). The degree of underascertainment of diabetes based on self-reported information depends on the reference (“gold”) standard used for comparison. Typical reference definitions incorporate fasting glucose, hemoglobin A1c (HbA1c), and/or use of diabetes medications. With the recent recommendations by the American Diabetes Association for the use of HbA1c for the diagnosis of diabetes (9), assessing the concordance of self-reported diabetes with HbA1c is important. Our objective was to assess the validity of both prevalent and incident self-reported diabetes by using different reference definitions and to assess the reliability (repeatability) of self-reported diabetes in a community-based population.

MATERIALS AND METHODS

Study population

The Atherosclerosis Risk in Communities (ARIC) Study is a community-based prospective cohort of 15,792 persons from 4 US communities, which included in-person visits (10). All visits included an interview, physical examination, and blood collection. Participants were asked to bring current medication bottles to the visits, and medications were recorded. Visit 4 (1996–1998) was attended by 11,656 participants and is the baseline for this analysis. Information on fasting glucose, glucose-lowering medication use, and self-reported diabetes was collected at visit 4. For validation of prevalent self-reported diabetes cases, we restricted the population to visit 4 participants who were not missing data on self-reported diabetes, fasting glucose, or medication use, leaving a sample size of 10,321 participants (including 1,041 with self-reported diabetes at visit 4). Beginning after this visit, new cases of diabetes have been (and continue to be) identified during annual telephone phone calls to all participants.

Between 2004 and 2005, a subsample of ARIC Study participants (n = 2,066) took part in the carotid magnetic resonance imaging (MRI) visit (11). Information on fasting glucose, HbA1c, and glucose-lowering medication use was collected at the carotid MRI visit. For validation of incident self-reported diabetes cases, we excluded participants who did not attend both visit 4 and the carotid MRI visit, participants with self-reported diabetes at visit 4, and those who were missing data on fasting glucose, HbA1c, or medication use at the carotid MRI visit, leaving a sample size of 1,480 participants (including 258 with self-reported diabetes during annual follow-up telephone calls occurring between visit 4 and the carotid MRI visit).

Definition of self-reported diabetes

We classified persons as having self-reported diabetes if they answered “yes” to any one of the following questions: “Has a doctor ever said you have diabetes (sugar in the blood)?” (asked at visit 4 and during annual telephone calls); “Were any of the medications you took during the past 2 weeks for diabetes or high blood sugar?” (asked at visit 4); “Did you take any medications during the past 2 weeks for diabetes or high blood sugar?” (asked during annual telephone calls).

Reference definitions for diabetes

For validation of prevalent self-reported diabetes, we compared self-report with 2 reference definitions using information from visit 4: fasting glucose ≥126 mg/dL or diabetes medication use (definition 1) and fasting glucose ≥140 mg/dL or diabetes medication use (definition 2). For validation of incident self-reported diabetes between visit 4 and the carotid MRI visit, we compared self-report with 5 reference definitions using information from the carotid MRI visit: fasting glucose ≥126 mg/dL or diabetes medication use (definition A), fasting glucose ≥140 mg/dL or diabetes medication use (definition B), HbA1c ≥6.5% or diabetes medication use (definition C), HbA1c ≥7.0% or diabetes medication use (definition D), and fasting glucose ≥126 mg/dL or HbA1c ≥6.5% or diabetes medication use (definition E). Cutpoints for fasting glucose were based on current (126 mg/dL) and past (140 mg/dL) American Diabetes Association recommendations (9, 12). Cutpoints for HbA1c were based on current American Diabetes Association (6.5%) and current Department of Veterans Affairs recommendations (7.0%) (9, 13).

Reliability of self-reported diabetes

We assessed the reliability of the question: “Has a doctor ever said you have diabetes (sugar in the blood)?” during 9 years of annual follow-up telephone calls. We performed this analysis in 2 groups: 1) individuals who said “yes” to this question at visit 4 and who participated in all annual follow-up telephone calls after visit 4 and prior to the carotid MRI visit and who participated in the carotid MRI visit (n = 302 cases) and 2) individuals who said “yes” to this question at visit 4 and who participated in at least 1 annual follow-up telephone call after visit 4 and prior to the carotid MRI visit (n = 1,325 cases). We restricted the populations to persons who said “yes” at visit 4 because a diagnosis of diabetes is considered permanent.

Statistical analysis

We calculated the sensitivity (probability that persons will self-report diabetes, given that they fulfill the reference definition), specificity (probability that persons will not self-report diabetes, given that they do not fulfill the reference definition), positive predictive value (probability that persons will fulfill the reference definition, given that they self-report diabetes), and negative predictive value (probability that persons will not fulfill the reference definition, given that they do not self-report diabetes) for self-reported diabetes compared with each reference definition of diabetes. To assess the reliability of self-reported diabetes, we calculated the percent positive agreement (pairwise agreement between visit 4 and each follow-up telephone call) using information collected at the annual telephone calls that occurred between visit 4 and the carotid MRI visit. We used logistic regression to investigate possible predictors of high reliability, including the following variables measured at visit 4: age, field center/race, sex, education, family income, insurance, usual medical care, alcohol consumption, cigarette smoking, prevalent heart disease, family history of diabetes, hypertension, hypercholesterolemia, fasting glucose, and cognitive function. High reliability was defined as reporting a physician diagnosis of diabetes or diabetes medication use at visit 4 and during all annual telephone calls during follow-up.

All P values are 2 sided, and P < 0.05 was considered statistically significant. Analyses were performed by using Stata, version 11, statistical software (StataCorp LP, College Station, Texas).

RESULTS

Characteristics of the 10,321 participants at visit 4 and at the carotid MRI visit are shown in Table 1. At baseline (visit 4), the mean age was 63 years, 57% were female, and 20% were black. The prevalence of self-reported diabetes was 10% (n = 1,041).

Table 1.
Participant Characteristics at Visit 4 (1996–1998) and the Carotid MRI Visit (2004–2005), the Atherosclerosis Risk in Communities Study

The sensitivity, specificity, positive predictive value, and negative predictive value of self-reported prevalent and incident diabetes using the various reference definitions are shown in Table 2. The sensitivity of prevalent self-reported diabetes using definition 1 (fasting glucose ≥126 mg/dL or medication use) was lower than when using definition 2 (fasting glucose ≥140 mg/dL or medication use) (59% vs. 71%). The specificities were high (>95%) when both definitions were used.

Table 2.
Validation of Prevalenta Self-reported Diabetes at Visit 4 (1996–1998) and Incidentb Self-reported Diabetes Status (Between 1996–1998 and 2004–2005), the Atherosclerosis Risk in Communities Study

There was a similar pattern for incident self-reported diabetes when definition A (fasting glucose ≥126 mg/dL or medication use) was used compared with definition B (fasting glucose ≥140 mg/dL or medication use). The sensitivity for incident self-reported diabetes was lower with definition A (62%) than definition B (80%), but specificity was similar (89% vs. 87%). Comparing definition C (HbA1c ≥6.5% or medication use) with definition D (HbA1c ≥7.0% or medication use), we found the sensitivities to be 59% and 64%, respectively, and the specificities were 87% and 85%, respectively. Both the sensitivities and specificities with HbA1c definitions were lower than when fasting glucose definitions were used. Definition E (fasting glucose ≥126 mg/dL or HbA1c ≥6.5% or medication use) had the lowest sensitivity (56%), the highest specificity (91%), and the highest positive predictive value (55%) of the 5 incident reference diabetes definitions.

Table 3 shows the reliability of self-reported diabetes defined by the question: “Has a doctor ever said you have diabetes (sugar in the blood)?” The percent positive agreement (pairwise agreement between visit 4 and each follow-up telephone call) was similar among those who participated in all annual follow-up telephone calls (range: 92.7%–95.4%) and those who participated in at least 1 annual follow-up telephone call (range: 93.6%–95.9%). This suggests that the participants who had complete follow-up did not lead to an overestimation of reliability. Overall, 237 (78.5%) of the 302 participants with diabetes at visit 4 who took part in all annual follow-up telephone calls reported diabetes during every annual follow-up telephone call (i.e., high reliability). Of the remaining 65, 46 (70.8%) failed to report diabetes on only 1 of the 9 annual follow-up phone calls. In our analyses of predictors of high reliability, participants with coronary heart disease were more likely to report with high reliability compared with persons without heart disease (odds ratio = 6.02, 95% confidence interval: 1.16, 31.30). No other factors examined were significant predictors of high reliability (all values for P > 0.05).

Table 3.
Prospective Reliability of the “Has a Doctor Ever Said You Have Diabetes (Sugar in the Blood)?” Question Among Participants Who Answered “Yes” to This Question at Visit 4 (1996–1998), the Atherosclerosis Risk in ...

DISCUSSION

Both prevalent self-reported diabetes and incident self-reported diabetes were 84%–97% specific and 55%–80% sensitive when multiple reference definitions were used. Self-reported diabetes performed well for identifying diabetes defined by fasting glucose, HbA1c, and/or medication use. The reliability of self-reported diabetes was >92% at all time points.

There are a number of studies that have shown high rates of confirmation of self-reported diabetes diagnosis based on information on medication, data from medical records, and other sources (47). By contrast, there have been few studies comparing self-reported diabetes with fasting glucose and HbA1c (14, 15). Our results are consistent with those of 2 prior studies reporting specificity and sensitivity for prevalent self-reported diabetes compared with fasting glucose ≥126 mg/dL or medication use (14, 15). Another study (16) validated prevalent self-reported diabetes status by using HbA1c ≥7.0% and medication data, reporting results similar to those in the present study. Although we were unable to compare prevalent self-reported diabetes with HbA1c values at visit 4 (HbA1c measurements were not available at this visit), the 98% specificity and the 85% sensitivity reported in this study are consistent with our comparison of incident diabetes with HbA1c ≥7.0% at the carotid MRI visit (16). Most prior studies have not used HbA1c nor assessed multiple reference definitions using different cutpoints of diabetes biomarkers.

The positive predictive values for self-reported diabetes were relatively low but ranged widely (14%–73%, depending on the reference definition). The difference in positive predictive values across our definitions is largely explained by differences in diabetes prevalence (e.g., prevalence ranged from 4.0% (59/1,480) using definition D to 17.3% (256/1,480) using definition E). The moderate sensitivities for self-reported diabetes (<70% for prevalent and <80% for incident self-reported diabetes) could represent participants who have not yet been diagnosed with diabetes by their physician. The impact of misclassifying these individuals as noncases depends on the study. A recent ARIC Study analysis compared the relation between traditional diabetes risk factors with 3 different diabetes case definitions. Magnitudes of association were lower for self-reported diabetes compared with definitions that also incorporated fasting glucose, but risk factors remained significantly associated with diabetes defined by self-report alone (17).

Certain limitations should be considered when interpreting the results of this study. Participants may not have brought all medications to the study visit, which could have resulted in an underestimation of the validity of self-reported diabetes. Additionally, test results conducted during the study are reported to the participants, and often to their physician, which could reduce the number of undiagnosed cases and overestimate estimates of sensitivity compared with the general population. HbA1c was not measured at visit 4; therefore, we were unable to validate prevalent self-reported diabetes against HbA1c. Our study also has a number of strengths. We were able to validate self-reported diabetes compared with multiple reference standards on the basis of levels of fasting glucose and HbA1c, as well as medication use. The annual follow-up telephone calls allowed us to also validate incident self-reported diabetes cases and to assess the reliability of the telephone question.

In summary, self-reported diabetes is 84%–97% specific and 55%–80% sensitive as compared with multiple reference definitions in this community-based population. Self-reported diabetes is >92% reliable over time. Our results contribute estimates of the error associated with the use of self-reported diabetes versus diabetes defined by biomarkers and medication use. The acceptability of defining diabetes by self-report will depend on the goals of the specific study.

ACKNOWLEDGMENTS

Author affiliations: Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland (Andrea L. C. Schneider, Elizabeth Selvin); Welch Center for Prevention, Epidemiology, and Clinical Research, Johns Hopkins University, Baltimore, Maryland (Andrea L. C. Schneider, Elizabeth Selvin); Division of Epidemiology and Community Health, University of Minnesota, Minneapolis, Minnesota (James S. Pankow); and Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina (Gerardo Heiss).

The Atherosclerosis Risk in Communities Study is carried out as a collaborative study supported by National Heart, Lung, and Blood Institute contracts (HHSN268201100005C, HHSN268201100006C, HHSN268201100007C, HHSN268201100008C, HHSN268201100009C, HHSN268201100010C, HHSN268201100011C, and HHSN268201100012C). A. L. C. S. was supported by National Institute of Diabetes and Digestive and Kidney Diseases training grant T32 DK062707. E. S. was supported by National Institute of Diabetes and Digestive and Kidney Diseases grants K01 DK076595 and R01DK089174 from the US National Institutes of Health.

Conflict of interest: none declared.

REFERENCES

1. Brancati FL, Wang NY, Mead LA, et al. Body weight patterns from 20 to 49 years of age and subsequent risk for diabetes mellitus: the Johns Hopkins Precursors Study. Arch Intern Med. 1999;159(9):957–963. [PubMed]
2. Rimm EB, Chan J, Stampfer MJ, et al. Prospective study of cigarette smoking, alcohol use, and the risk of diabetes in men. BMJ. 1995;310(6979):555–559. [PMC free article] [PubMed]
3. Folsom AR, Kushi LH, Anderson KE, et al. Associations of general and abdominal obesity with multiple health outcomes in older women: the Iowa Women's Health Study. Arch Intern Med. 2000;160(14):2117–2128. [PubMed]
4. Okura Y, Urban LH, Mahoney DW, et al. Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure. J Clin Epidemiol. 2004;57(10):1096–1103. [PubMed]
5. Kriegsman DM, Penninx BW, van Eijk JT, et al. Self-reports and general practitioner information on the presence of chronic diseases in community dwelling elderly. A study on the accuracy of patients' self-reports and on determinants of inaccuracy. J Clin Epidemiol. 1996;49(12):1407–1417. [PubMed]
6. Kehoe R, Wu SY, Leske MC, et al. Comparing self-reported and physician-reported medical history. Am J Epidemiol. 1994;139(8):813–818. [PubMed]
7. Haapanen N, Miilunpalo S, Pasanen M, et al. Agreement between questionnaire data and medical records of chronic diseases in middle-aged and elderly Finnish men and women. Am J Epidemiol. 1997;145(8):762–769. [PubMed]
8. Centers for Disease Control and Prevention. National Diabetes Fact Sheet: National Estimates and General Information on Diabetes and Prediabetes in the United States. Atlanta, GA: Centers for Disease Control and Prevention, US Department of Health and Human Services; 2011.
9. American Diabetes Association. Diagnosis and classification of diabetes mellitus. Diabetes Care. 2010;33(suppl 1):S62–S69. [PMC free article] [PubMed]
10. The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. The ARIC investigators. Am J Epidemiol. 1989;129(4):687–702. [PubMed]
11. Wagenknecht L, Wasserman B, Chambless L, et al. Correlates of carotid plaque presence and composition as measured by MRI: the Atherosclerosis Risk in Communities Study. Circ Cardiovasc Imaging. 2009;2(4):314–322. [PMC free article] [PubMed]
12. American Diabetes Association: clinical practice recommendations 1996. Diabetes Care. 1996;19(suppl 1):S1–S118. [PubMed]
13. US Department of Veterans Affairs/Department of Defense. VA/DoD clinical practice guideline for the management of diabetes mellitus. Version 4. Washington, DC: US Department of Veterans Affairs/Department of Defense; 2010. (http://www.healthquality.va.gov/diabetes/DM2010_SUM-v4.pdf. ). (Accessed November 11, 2011)
14. Huerta JM, Tormo MJ, Egea-Caparrós JM, et al. Accuracy of self-reported diabetes, hypertension and hyperlipidemia in the adult Spanish population. DINO study findings. Rev Esp Cardiol. 2009;62(2):143–152. [PubMed]
15. Lima-Costa MF, Peixoto SV, Firmo JO, et al. Validity of self-reported diabetes and its determinants: evidence from the Bambuí study. (In Portuguese) Rev Saude Publica. 2007;41(6):947–953. [PubMed]
16. Goldman N, Lin IF, Weinstein M, et al. Evaluating the quality of self-reports of hypertension and diabetes. J Clin Epidemiol. 2003;56(2):148–154. [PubMed]
17. Bielinski SJ, Pankow JS, Rasmussen-Torvik LJ, et al. Strength of association for incident diabetes risk factors according to diabetes case definitions: the Atherosclerosis Risk in Communities Study. Am J Epidemiol. 2012;175(5):466–472. [PMC free article] [PubMed]

Articles from American Journal of Epidemiology are provided here courtesy of Oxford University Press