Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Aging Clin Exp Res. Author manuscript; available in PMC 2012 December 28.
Published in final edited form as:
Aging Clin Exp Res. 2012 August; 24(4): 370–376.
PMCID: PMC3531895

Prevalence of Diabetes Treatment Effect Modifiers: the External Validity of Trials to Older Adults

Carlos O. Weiss, MD, MHS, Assistant Professor of Medicine,1 Cynthia M. Boyd, MD, MPH, Associate Professor of Medicine,1 Jennifer L. Wolff, PhD, Associate Professor of Health Policy & Management, and Bruce Leff, MD, Professor of Medicine1


Background and Aims

Potential treatment effect modifiers (TEMs) are specific diseases or conditions with a well-described mechanism for treatment effect modification. The prevalence of TEMs in older adults with type 2 diabetes mellitus (DM) is unknown. Objectives were to (1) determine the prevalence of pre-specified potential TEMs; (2) demonstrate the potential impact of TEMs in the older adult population using a simulated trial; (3) identify TEM combinations associated with number of hospitalizations to test construct validity.


Data are from the nationally-representative United States National Health and Examination Survey, 1999–2004: 8,646 Civilian, non-institutionalized adults aged 45–64 or 65+ years, including 1,443 with DM. TEMs were anemia, congestive heart failure, liver inflammation, polypharmacy, renal insufficiency, cognitive impairment, dizziness, frequent mental distress, mobility difficulty, and visual impairment. A trial was simulated to examine prevalence of potential TEM impact. The cross-sectional association between TEM patterns and number of hospitalizations was estimated to assess construct validity.


The prevalence of TEMs was substantial such that 19.0% (95%CI: 14.8–23.2) of middle-aged adults and 38.0% (95% CI: 33.4–42.5) of older adults had any two. A simulated trial with modest levels of interaction suggested the prevalence of TEMs could nullify treatment benefit in 3.9–27.2% of older adults with DM. Compared to having DM alone, hospitalization rate was increased by several combinations of TEMs with substantial prevalence.


We provide national benchmarks that can be used to evaluate TEM prevalence reported by clinical trials of DM, and correspondingly their external validity to older adults.

Keywords: applicability, comorbidity, clinical trials as topic, type 2 diabetes mellitus


Type 2 Diabetes mellitus (DM) is the 4th leading cause of death due to chronic disease in the US,(1) and its increasing prevalence promises future growth in DM-related mortality and complications and provides strong motivation to improve the translation of findings from trials into clinical practices.(2, 3) A major barrier to the translation of treatments into practice is dearth of evidence describing the extent to which results from clinical trials may be applied to the broader population. It is suspected that adults who participate in clinical trials are younger, healthier, on average, than people seen in most clinical practice settings.(4, 5) This situation is thought to be acute in older adults, who as a group experience more health status heterogeneity compared to younger age groups.(6)

Prior research to examine the external validity of clinical trials has evaluated sociodemographic characteristics or comorbidity counts that may not convey the information most needed to effectively guide clinical decisions. Potential treatment effect modifiers (TEMs) are a subset of multimorbidity indicators that are specific clinical diseases or conditions with a well-described mechanism for treatment effect modification. A different distribution of TEMs, not sociodemographic variables, in the target population than in the trial would raise serious concerns regarding external validity.(7, 8) Selection of people with or without TEMs into trials may be planned, as a result of study inclusion and exclusion criteria(911) or unplanned, due to healthy volunteer biases.(12, 13) Therefore, ignoring TEMs in the reporting of clinical trials is a real basis for concerns about their external validity. The presence of would TEMs affects the applicability of trials to older adults, making them a priority according to a 2008 National Institute on Aging conference.(14) The prevalence of TEMs is currently unknown in the population of middle-aged (age 45–64 years) or older (age 65+ years) adults with DM.

This study draws upon a United States nationally-representative sample to estimate the prevalence of TEMs among older adults with DM. We illustrate the potential impact of TEMs for reported treatment outcomes in the general population using a simulated trial of a treatment to lower glycosylated hemoglobin (HbA1c) in DM. To assess construct validity, we then measure the association of patterns of TEMs with the number of hospitalizations in adults with DM.



The U.S. National Health and Nutrition Examination Survey (NHANES) is a nationally-representative study of non-institutionalized civilians designed to study the prevalence of diseases and conditions. Study details are publicly available.(15) We joined three survey waves from years 1999–2004 including 8,654 individuals aged 45 and older.

Participants were asked “have you ever been told by a doctor or other health professional that you have diabetes or sugar diabetes?” They were also allowed to answer that they had borderline diabetes. People who responded borderline were counted as having DM if they also took insulin or a pill for diabetes, or, suffered from retinopathy, a lower extremity ulcer that took more than 4 weeks to heal or numbness or tingling in the hands or feet due to diabetes. To determine whether under-reporting of DM might substantially affect our results, a sensitivity analysis was conducted to identify individuals who did not report diabetes but met any of the following criteria favoring specificity: fasting glucose >140, non-fasting glucose >200 mg/dL or HbA1c >7.6. These criteria were deliberately more stringent than usual diagnostic criteria because they were applied only to people who failed to report DM by the main criteria.

TEMs describe a subset of factors that contribute to complex health status in older adults.(16) We examined two categories of TEMs. Physiologic TEMs were defined as factors that contribute to complex health status in adults and were selected from a review of precautions and contraindications(17) of pharmacologic treatment of DM (Table 1). For continuous variables, groupings were pre-specified as follows. Anemia was defined as a serum hemoglobin <12 g/dL in women and <13 g/dL in men.(18) Congestive heart failure (CHF) was identified through a question asking “has a doctor or other health professional ever told you had congestive heart failure?” High ALT was identified as a high serum alanine aminotransferase (ALT) >40 SI in women and >45 in men. Polypharmacy was defined as taking ≥5 medicines daily, following a previously established cut-point(19) and inspection of prescribed medications and over-the-counter analgesics used daily. Supplements, vitamins and minerals were not counted. Warfarin use was identified. Low GFR was defined as a low glomerular filtration rate (GFR) <60 mm/L calculated using the MDRD equation(20).

Table 1
Potential Treatment Effect Modifiers.

The second category, health status TEMs, were defined as person-level factors that modify DM treatment. Cognitive impairment was ascertained through performance on the Digit Symbol Substitution test in 1999–2002,(21) defined as refusal or inability to perform the practice test for cognitive reasons or a result in the lowest quintile (<30 correct answers in 120 s). Participants who reported dizziness or imbalance lasting ≥2 weeks or for an unknown duration, or, difficulty with falls in the last year were categorized as dizziness. Participants who reported that mental health was “not good” for ≥14 of the past 30 days were categorized as having frequent mental distress in NHANES waves 2001–2004.(22) Mobility difficulty was defined using self-reported difficulty walking mile or up 10 steps without equipment.(23) Visual impairment was ascertained as self-reported extreme difficulty reading newsprint or seeing up close in people age 50+ years or examined visual acuity worse than 20/50 in the better eye.

Potential confounders included physician visits (≥2 visits last year), gender, education (≥12 years), low income (<$20,000 for household) and race (Black). These variables were considered potential confounders based on a priori information regarding their association with DM overall and with hospitalization.

Participants were asked whether they experienced an overnight hospital stay in the last year, other than an emergency room. Those who replied yes were also asked how many times this occurred in the last year. Self-report of this outcome has been shown to agree well with claims.(24)

Statistical Analysis

Analyses were pre-specified and weighted with sampling weights that account for sampling strategy and survey non-response. Differences between subjects with and without DM were compared using either a two-sided t-test with unequal variances or a chi square test. Binomial Wald 95% confidence intervals are reported. Age categorization groupings were made following convention and after confirming that relatively small numbers above age 85 limited analyses in that group. A pattern analysis was conducted to determine the prevalence of clinically meaningful mutually exclusive combinations of TEMs.(25) For patterns present in >2% of people with DM, we modeled the relative rate of hospitalization in people with DM and the pattern of TEMs compared to people with DM and no TEMs using a multivariable negative binomial regression model. Final models were adjusted for physician visits since gender, education, income and race were not significant predictors after accounting for age, physician visits, DM and TEMs. We report a Bonferroni adjustment to maintain a type I error probability of 0.05 for the set of related tests. Analyses were carried out in Stata version 11 software.(26) Based on initial findings, we also performed an ad hoc analysis of the distribution of age, DM status and high ALT as discussed below.

Drawing from 1,249 participants with DM and measured HbA1c, we simulated a simple trial(27) to examine whether the observed prevalence of TEM patterns could influence estimated treatment effectiveness in a general population. The purpose of the simulation study was to understand the operating characteristics of real-world patterns of potential TEMs under several scenarios. DM participants were randomly placed in a treatment or placebo arm. A treatment lowering HbA1c by 2 points on average was simulated. We then examined four levels of interaction between TEMs and the simulated treatment, ranging from small to modest (a factor of 0.1, 0.2, 0.3 and 0.4 for each modifier). For example, at the second level of interaction a person with 2 physiologic TEMs had his or her simulated in HbA1c moved toward null by a factor of (0.2 * 2). We assumed TEMs influenced HbA1c in the opposite direction as treatment, i.e. interrupting treatment and moving effect towards null, because we examined TEMs based on described mechanisms for blocking the ability to be prescribed or comply with a DM treatment. We assumed that each TEM acted equivalently and that modification was linear. In this way we examined under which conditions the average treatment response would no longer be a lower HbA1c for subgroups defined by the observed number of TEMs and levels of effect modification.


Overall NHANES response rates were 76–80% throughout the specified years. Of 8,654 participants age 45+, 7,883 completed the laboratory examination and 7,272 had no missing data on specified covariates and were included in a complete case analysis. A total of 1,443 people with DM age 45+ years with sampling weights were representative of approximately 12,333,000 people with DM in the US. People with DM were older, less often White, less highly educated and had lower household income than their counterparts without DM (results not shown).

Compared to middle-aged adults with DM, older adults with DM were more likely to be female, White, have health insurance, have a low income and have access to health care but were less likely to have completed >12 grades of education (Table 2). These variables were examined as potential confounders of hospitalization, but after accounting for TEMs only age and access to health care were significantly associated with the outcome and were used in the final models.

Table 2
Baseline Demographic Characteristics in Diabetes Mellitus, Overall and By Age Group: NHANES 1999–2004.

Age-stratified prevalence of TEMs was examined among people with DM (Figures 1a and 1b). Relative to adults ages 45 to 64, prevalence of TEMs was higher among individuals who were 65 or older, most notably for polypharmacy and low GFR. High ALT and frequent mental distress were less prevalent in old age. The prevalence of having any 2 or any 3 physiologic or health status TEMs was high, such that 19.0% (95%CI: 14.8–23.2) of adults ages 45–64 and 38.0 % (95%CI: 33.4–42.5) of adults ages 65+ had any 2 physiologic TEMs and 16.1% (95%CI: 10.2–22.1) of adults 45–64 and 40.9% (95%CI: 35.3–46.5) of adults ages 65+ had any 2 health status TEMs. [Figure 2 about here]

Figure 1
Prevalence of Physiologic (a) and Health Status (b) Potential Treatment Effect Modifiers among Adults with Diabetes Mellitus, by Age Group: NHANES 1999–2004
Figure 2
Simulated Randomized Controlled Trial with Real Distribution of Treatment Effect Modifiers among Adults with Diabetes Mellitus, NHANES 1999–2004

Example of Potential Impact by Observed Prevalence of TEMs on Treatment in Older Adults

A simulated randomized, controlled trial of a treatment lowering HbA1c by 2% on average was performed (Figure 2). At a low level of treatment effect modification there was average benefit, or lowering of HbA1c, regardless of numbers of TEMs. However, at small or modest levels of effect modification the benefit was cancelled in 3.9–27.2% of older adults with DM. Furthermore, because TEMs are correlated with increasing age, treatment effect was modified with age at an effect modification level of 0.2 or higher (p-value for interaction term = .001 for 0.2 and <.001 for 0.3 or 0.4). [Table 3 about here]

Table 3
Prevalence in DM and Rate Ratios for Number of Hospitalizations for Mutually Exclusive Patterns of Potential Treatment Effect Modifiers: NHANES 1999–2004.

Clinical Patterns of TEMs among People with DM and Association with Hospitalization Rate

Mutually exclusive clinical patterns of TEMs were examined and those present in >2% of people with DM are shown in Table 3. Among people with DM, TEMs were absent in 39.0% (95% CI: 34.5–43.6) and 37.7% (95% CI: 28.9–46.5) for physiologic or health status modifiers, respectively. Most clinical patterns of diabetes and physiologic TEMs were associated with a substantially higher number of hospitalizations. Hospitalization rates within the last year among study participants with DM were greater by a factor of 5.53 (95%CI: 3.06–10.0) among people with both CHF and polypharmacy, compared to people with DM but no physiologic TEMs. Fewer patterns of health status treatment effect modifiers were highly prevalent.

Adding laboratory-based criteria for DM to account for unreported DM as described in Methods resulted in 111 additional cases of DM. Of the unreported cases, 67.4% were middle-aged adults. Results were not altered consistently, substantively nor significantly by adding unreported DM to the sample (results not shown).


This study provides insight regarding the prevalence and potential impact on the care of older adults of type 2 diabetes-specific potential treatment effect modifiers (TEMs) - diseases or conditions with the potential to modify the effects of diabetes treatments. The external validity of results from well-performed randomized, controlled clinical trials is jeopardized when trial participants differ from clinically relevant target populations with respect to attributes that modify treatment effect.(7, 8) Drawing from a nationally representative sample of community-dwelling adults, we found potential TEMs were common in older adults with DM, as high as 51% in the case of polypharmacy. A simple simulation of a randomized, controlled trial with modest levels of treatment effect modification interrupting treatment demonstrated the potential for 3.9–27.2% of older adults with DM to experience no treatment benefit due to the prevalence of these TEMs. Because TEMs are positively correlated with older age, the attenuation of treatment benefit was greater with increasing old age. Although it is useful and important to describe the prevalence of TEMs individually, a person with DM may have several at once. Therefore we examined TEM patterns, taking into account the presence or absence of all TEMs simultaneously. The majority of even middle-aged adults ages 45 and older living with DM were found to have at least one; nearly 1 in 3 middle-aged adults had at least two, TEMs. Several TEM patterns of were associated with elevated rates of hospitalization relative to having DM alone.

By studying TEMs we hope to inspire improvements in reporting, and future design, of trials. Trials report on potential TEMs in their participants irregularly. For example, a review of selection criteria for the UKPDS 33 study demonstrates that the landmark study used CHF, renal function and visual function as exclusion criteria, but it did not report summary statistics for these TEMs nor other TEMs.(28) Unplanned and unexamined selection effects in the trials may be one of the reasons some trials are thought in retrospect to be wrong about treatment effect.(29) Our findings suggest that a simple table reporting the presence of physiologic and health status TEMs across age strata would benefit our understanding of a trial’s external validity. We observed that not all potential TEMs are more prevalent in older age: an elevated ALT and frequent mental distress were instead lower. Such findings can occur for a variety of reasons, including survivor effects and the inappropriateness of diagnostic strategies used at younger ages for older adults. It cannot be assumed that all conditions are more prevalent in older age.

These study results may also inform the design of future medium and large size trials of DM using more expansive inclusion criteria. The ability of larger, simple trials to find subgroups within which treatment will work depends on the ability to examine TEMs. Some trials may not make an effort to ensure external validity because of the up-front costs and time required to ensure representation of subgroups. Moreover, there can be debate about how to appropriately account for heterogeneous treatment effects.(30) However, these considerations should be weighed against the benefits that accrue from increasing statistical power by including people at higher risk of the outcome, in addition to the benefit of improved external validity to older adults.

The study findings here should be considered in light of several limitations. Results were limited to potential TEMs chosen primarily for their direct impact on DM treatments and do not include many conditions, such as hearing impairment or urinary symptoms, that may influence adherence.(31) It is also possible that information bias affected these results since people with two chronic diseases may have more contact with health professionals and thus have a third or more diseases identified. This would increase the proportion with two or more, as opposed to one or two, TEMs in our pattern analyses. In addition, in this study some conditions such as CHF were ascertained only via self-report, which can have poor diagnostic accuracy when compared to physiologic testing. For example, a study comparing self-report to structured medical record review in older women with disability found a kappa of 0.48 for CHF.{Simpson, 2004 #2277} This limitation results from the desire to achieve a representative sample, which entails ascertainment with methods that are not burdensome to participants, do not take a long time and are not prohibitively costly. Another limitation of NHANES is that it does not measure all health status TEMs, such as depression. Given the cross-sectional nature of the data, we cannot tell whether lower estimated prevalence for some diseases at age 65 years and older result from a cohort effect or attrition earlier in life. Our simulation study was simplistic and did not explore a large range of conditions nor complex interactions, rather a small set of conditions chosen on the basis of similarity to real-world conditions. The simulation only focused on those TEMs thought to interrupt treatment. As with all simulations its validity is uncertain without attempted replication using external data. These analyses are cross-sectional and only general construct validity was assessed, since hospitalization is the result of many factors. The purpose here is to inform the design and reporting of future trials that, by virtue of their focused and experimental design, can directly test for treatment effect modification according to these TEMs as well as according to TEMs that are important regardless of age such as disease severity and adherence.

Based on a nationally-representative survey, we find potential TEMs to be associated with DM and highly prevalent among adults, especially older adults, (6)with DM in the United States. Based on their distribution among people with DM, we estimated through a simple simulation that TEMs have a potential to alter treatment effect in real practice in a substantial number of cases. Several clinical patterns of TEMs were associated with an increased number of hospitalizations. Future trials of DM should consider how to ensure adequate external validity during the design phase, measure and report the presence of TEMs and consider when there is sufficient scientific knowledge to pre-specify hypothesis of treatment effect modification.


This work was supported by the Robert Wood Johnson Foundation (CW and CB); the Johns Hopkins Bayview Center for Innovative Medicine (CB); and the National Institutes of Mental Health (grant number K01MH082885, JW). The researchers acted independently and the sponsors played no role in the design, and conduct of the study; collection, management, analysis and interpretation of the data; nor preparation, review or approval of the manuscript. All authors have completed the Unified Competing Interest form at .


1. Jemal A, Ward E, Hao Y, Thun M. Trends in the leading causes of death in the United States, 1970–2002. JAMA. 2005;294:1255–59. [PubMed]
2. Woolf SH. The meaning of translational research and why it matters. JAMA. 2008;299:211–213. [PubMed]
3. Clancy CM, Slutsky JR. Commentary: a progress report on AHRQ's Effective Health Care Program. Health Serv Res. 2007;42:xi–x. [PMC free article] [PubMed]
4. Gale EA, Beattie SD, Hu J, Koivisto V, Tan MH. Recruitment to a clinical trial improves glycemic control in patients with diabetes. Diabetes Care. 2007;30:2989–92. [PubMed]
5. Vijan S, Kent DM, Hayward RA. Are randomized controlled trials sufficient evidence to guide clinical practice in type II (non-insulin-dependent) diabetes mellitus? Diabetologia. 2000;43:125–30. [PubMed]
6. Maddox GL. Aging differently. Gerontologist. 1987;27:557–64. [PubMed]
7. Longford NT. Selection bias and treatment heterogeneity in clinical trials. Stat Med. 1999;18(12):1467–74. [PubMed]
8. Piantadosi S. Clinical Trials: A Methodologic Perspective. Hoboken, New Jersey: John Wiley & Sons, Inc; 2005.
9. Fortin M, Dionne J, Pinho G, Gignac J, Almirall J, Lapointe L. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4:104–8. [PubMed]
10. Bartlett C, Doyal L, Ebrahim S, Davey P, Bachmann M, Egger M, et al. The causes and effects of socio-demographic exclusions from clinical trials. Health Technol Assess. 2005;9:iii–iv. ix–x, 1–152. [PubMed]
11. Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297:1233–40. [PubMed]
12. Gross CP, Mallory R, Heiat A, Krumholz HM. Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med. 2002;137:10–6. [PubMed]
13. Pinsky PF, Miller A, Kramer BS, Church T, Reding D, Prorok P, et al. Evidence of a healthy volunteer effect in the prostate, lung, colorectal, and ovarian cancer screening trial. Am J Epidemiol. 2007;165:874–81. [PubMed]
14. Boyd CM, Ritchie CS, Tipton EF, Studenski SA, Wieland D. From Bedside to Bench: summary from the American Geriatrics Society/National Institute on Aging Research Conference on Comorbidity and Multiple Morbidity in Older Adults. Aging Clin Exp Res. 2008;20:181–88. [PMC free article] [PubMed]
15. National Health and Nutrition Examination Survey [database on the Internet] [cited October 1, 2010]. Available from:
16. Boyd CM, Leff B, Wolff JL, Yu Q, Zhou J, Rand C, et al. Informing Clinical Practice Guideline Development and Implementation: Prevalence of Co-existing Conditions Among Adults with Coronary Heart Diseases. J Am Ger Soc. 2010 [in press] [PMC free article] [PubMed]
17. [Accessed December, 2010.];Micromedex 1.0 (Healthcare Series) [database on the Internet] 2010 Available at:
18. Woodman R, Ferrucci L, Guralnik J. Anemia in older adults. Curr Opin Hematol. 2005;12:123–8. [PubMed]
19. Ziere G, Dieleman JP, Hofman A, Pols HA, van der Cammen TJ, Stricker BH. Polypharmacy and falls in the middle age and elderly population. Br J Clin Pharmacol. 2006;61:218–23. [PMC free article] [PubMed]
20. Levey AS, Coresh J, Greene T, Stevens LA, Zhang YL, Hendriksen S, et al. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med. 2006;145:247–54. [PubMed]
21. Wechsler D. Wechsler Adult Intelligence Scale. 4. Upper Saddle River, New Jersey: Pearson Education; 2008.
22. From the Centers for Disease Control and Prevention. Self-reported frequent mental distress among adults--United States, 1993–1996. JAMA. 1998;279:1772–3. [PubMed]
23. Chaves PH, Garrett ES, Fried LP. Predicting the risk of mobility difficulty in older women with screening nomograms: the Women's Health and Aging Study II. Arch Intern Med. 2000;160:2525–33. [PubMed]
24. Wolinsky FD, Miller TR, An H, Geweke JF, Wallace RB, Wright KB, et al. Hospital episodes and physician visits: the concordance between self-reports and medicare claims. Med Care. 2007;45:300–7. [PMC free article] [PubMed]
25. Weiss CO, Boyd CM, Yu Q, Wolff JL, Leff B. Patterns of prevalent major chronic disease among older adults in the United States. JAMA. 2007;298:1160–2. [PubMed]
26. StataCorp. Special Edition Stata Statistical Software. 10.2. College Station, TX: 2010.
27. Tubach F, Giraudeau B, Ravaud P. The variability in minimal clinically important difference and patient acceptable symptomatic state values did not have an impact on treatment effect estimates. J Clin Epidemiol. 2009;62:725–8. [PubMed]
28. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33) UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998;352:837–53. [PubMed]
29. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887–92. [PMC free article] [PubMed]
30. Sun X, Briel M, Busse JW, Akl EA, You JJ, Mejza F, et al. Subgroup Analysis of Trials Is Rarely Easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials. Trials. 2009;10:101–11. [PMC free article] [PubMed]
31. Piette JD, Kerr EA. The impact of comorbid chronic conditions on diabetes care. Diabetes Care. 2006;29:725–31. [PubMed]