PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Med Care. Author manuscript; available in PMC Jun 1, 2013.
Published in final edited form as:
PMCID: PMC3351695
NIHMSID: NIHMS348646
A Comparison and Cross-Validation of Models to Predict Basic Activity of Daily Living Dependency in Older Adults
Daniel O. Clark, PhD,1,2,3 Timothy E. Stump, MA,4 Wanzhu Tu, PhD,1,2,4 and Douglas K. Miller, MD1,2,3
1Indiana University Center for Aging Research, Indiana University School of Medicine
2Regenstrief Institute, Inc., Indiana University School of Medicine
3Department of Medicine, Division of General Internal Medicine and Geriatrics, Indiana University School of Medicine
4Department of Biostatistics, Indiana University School of Medicine
Corresponding Author: Daniel O. Clark, PhD, Associate Professor of Medicine, Regenstrief Institute, Inc, 410 West 10th St., HS 2000, Indianapolis IN 46202, daniclar/at/iupui, (317) 423-5600, (317) 423-5653 FAX
Background
A simple method of identifying elders at high risk for Activity of Daily Living dependence (ADL) could facilitate essential research and implementation of cost-effective clinical care programs.
Objective
We used a nationally representative sample of 9,446 older adults free from ADL dependence in 2006 to develop simple models for predicting ADL dependence at 2008 follow-up and to compare the models to the most predictive published model. Candidate predictor variables were those of published models that could be obtained from interview or medical records data.
Methods
Variable selection was performed using logistic regression with backwards elimination in a two-thirds random sample (n=6,233) and validated in a one-third random sample (n=3,213). Model fit was determined using the c-statistic and evaluated vis-à-vis our replication of a published model.
Results
At 2-year follow-up, 8.0% and 7.3% of initially independent persons were ADL dependent in the development and validation samples, respectively. The best fitting, simple model consisted of age and number of hospitalizations in past 2 years, plus diagnoses of diabetes, chronic lung disease, congestive heart failure, stroke, and arthritis. This model had a c-statistic of 0.74 in the validation sample. A model of just age and number of hospitalizations achieved a c-statistic of 0.71. These compared to a c-statistic of 0.79 for the published model. Sensitivity analyses demonstrated model robustness.
Conclusion
Models based on widely available data achieve very good validity for predicting ADL dependence. Future work will assess the validity of these models using medical records data.
Keywords: Older Adults, Activities of Daily Living, Models of Care
One-fourth of the older adult population accounts for two-thirds of Medicare expenditures.(1) Persons with multiple chronic conditions (MCCs) who need basic activity of daily living (ADL) assistance are most likely to be high cost beneficiaries.(2) In 2007, the Agency for Healthcare Research and Quality (AHRQ) co-sponsored meetings that set research priorities for older adults with MCCs.(3) Underlying the research priorities were two related objectives—reducing Medicare expenditures and maintaining independence at home. Meeting either of the AHRQ objectives will require identifying individuals within the older adult population who are at elevated risk for dependency.
To be of greatest value, the risk identification model itself should be valid, low cost and have potential for wide application. Obtaining performance-based data on a population of older adults presents challenges and significant costs.(4) Models based on risk factors that can be obtained by interview or existing medical records data are likely to be more cost-effective. Existing theory and models indicate that demographic characteristics, prior health services use, and chronic disease diagnoses are all possible predictors of ADL dependency(1, 2) and these are variables that could be obtained by interview and often from medical records data.
For this report we replicated the most valid published model(5) which we refer to as the benchmark model, developed somewhat simpler models, and compared the predictive values of the benchmark and simple models. Analyses were based on a nationally representative sample of adults aged 65 or over who reported no ADL dependency in 2006. The outcome was ADL dependency status at the 2008 follow-up.
Sample
Data came from the Health and Retirement Study (HRS). The HRS is a panel study with a multistage area probability sample.(6) The HRS sample is nationally representative of community dwelling older adults. There were 9,966 self-respondents who were aged 65 years or over at the 2006 interview and independent in all ADLs (eating, dressing, bathing, transferring, toileting). For 458 of 574 respondents who died prior to the 2008 follow-up interview 2008 ADL dependency status was available from proxy “exit” interview data. We excluded 520 respondents who had missing data on one or more variables of interest leaving 9,446 for our analyses.
Measures
Outcome
The study outcome is ADL dependency. Dependence is defined as receiving help from another person to complete the activity.(7) In the HRS, persons or their proxy were asked whether, because of a health or memory problem, they received assistance in ADLs. Consistent with recent literature,(5) we included five basic ADLs; bathing, dressing, eating, toileting, and transferring from a bed to a chair. We defined ADL dependency as a report of receiving personal help (use of assistive devices is not considered dependence) of any type in any one of the five ADLs.
Predictors
Although the benchmark model included physical and cognitive impairments and ADL disabilities we elected not to include these as candidate variables for the simple model because they are rarely available in electronic medical records data or medical charts. Included as a candidate variable in the simple models but not in the benchmark model is prior hospitalization. In addition to often indicating an acute deterioration in health,(8) there is evidence that the hospitalization itself can accelerate ADL dependency and is thus an independent risk factor for ADL dependency.(9)
Age, gender, number of hospitalizations, total number of nights in hospital since last interview (2004), and chronic disease were candidate variables for the simple models. Chronic disease was based on a report that a doctor had told the respondent that he/she has the condition. The eight candidate conditions are listed at the bottom of Table 2. We included individual chronic disease rather than a count of diseases because a recent analysis of trajectories of ADL dependencies showed that mean number of chronic diseases does not differentiate trajectories.(10)
Table 2
Table 2
Two Year Incident Activity of Daily Living Dependence Regressed on Baseline Benchmark Model and Simple Models, Development Cohort (n=6,233)
For replication of the benchmark model, we included the nine ADL dependency predictor variables reported in Covinsky et al., 2006. The nine variables are shown in Table 2. On all variables, the HRS 2006-2008 wording and response options are the same as those of the 1993-1995 Assets and Health Dynamics of the Oldest Old survey used by Covinsky et al., 2006.
Analyses
Replication of the benchmark model and development of the simple models were completed on a random two-thirds (n=6,233) of the sample and validation of the models was carried out using the remaining one-third (n=3,213). To replicate the benchmark model, we used a logistic regression model. Development of the simple model was performed using logistic regression with backwards elimination. The model started with age, gender, number of hospitalizations, total number of nights in hospital, and the eight available chronic diseases as possible predictor variables and eliminated variables one at a time to improve fit. We also explored an even simpler model that did not include chronic disease diagnoses. Comparison of the models was accomplished using c-statistics.(11) As a general rule of thumb, a c-statistic of 0.7 to 0.79 is considered very good and 0.8 or greater is considered excellent.(12) To evaluate agreement (i.e., models agree that a subject is high or low risk) across benchmark and simple models in classifying individuals as high risk we used the Kappa method. (13)
We conducted several analyses to determine the robustness of the models. In these analyses, we assessed: 1) impact of attrition from death by estimating a model with a combined outcome of incident ADL or death using the development cohort, 2) alternative models using the development cohort that examined age alone, age and chronic diseases, and the benchmark model with hospitalizations included, and 3) model fit excluding persons 90 years or older.
The development and validation cohorts were similar on all variables (see Table 1). Table 2 shows results of ADL prediction models in the development cohort. At 2-year follow-up, 498 (8.0%) had developed ADL dependence. Replication of the benchmark model shows a c-statistic of 0.79, which is the same as that reported by Covinsky et al., 2006 using data from 1993-95. Results of the backwards elimination analyses for the simple model indicate that age, hospitalizations, and five chronic disease diagnoses give the best fitting model. The c-statistic in that model (using the model’s regression coefficients) is 0.76. Eliminating disease indicators from the simple model leaves just two variables; age and prior hospitalizations. The c-statistic drops to 0.73 in that model.
Table 1
Table 1
Descriptive Statistics for Development and Validation Samples.
Two hundred thirty six (7.3%) in the 1/3rd validation sample developed ADL dependence. As shown in Table 3, the c-statistic for the benchmark model is the same as in the development samples (0.79). Compared to the development sample, the c-statistics for the simple models are slightly lower in the validation sample—0.74 and 0.71 for the models with and without chronic disease diagnoses, respectively. The parameter estimates for variables in the simple models are similar across the development and validation samples with the exception of age 90 years or over.
Table 3
Table 3
Two-Year Incident Activity of Daily Living Dependence Regressed on Baseline Benchmark Model and Simple Models, Validation Cohort (n=3,213)
Based on rounded odds ratios from the development cohort (Table 2), we assigned points (0 for reference group, 1 for odds ratio of >1 and <1.5, 2 for odds ratio of ≥ 1.5 and < 2.5, etc) for model variables. Scoring is shown in Table 4. Compared to the models based on actual regression coefficients, this simple scoring strategy did not result in any change in the c-statistic of the benchmark model. There is a modest decline for the simple models. In Table 5 we show the distribution of incident ADL dependency in the validation cohort for each of the models using the variable scoring that is shown in Table 4. In both the benchmark model and the simple model including disease, 28.7% and 29%, respectively, of respondents with a score of 8 or more were ADL dependent at follow-up. Eight or more was identified by Covinsky et al. for the benchmark model as an optimal cut-point for clinical decision-making. With cut-off scores of 8 or more, the Kappa statistic showed 0.39 agreement between the benchmark and simple model with disease (not shown). Agreement between the benchmark and the simple model without disease was 0.27 while the Kappa between the two simple models was 0.68 (not shown).
Table 4
Table 4
Scoring1 for the Baseline Benchmark and Simple Model Variables and Model Fit (c-statistic) for Two-Year Incident Activity of Daily Living Dependence Regressed on Baseline Benchmark and Simple Model Variables Using the Variable Scoring Shown, Validation (more ...)
Table 5
Table 5
Two-Year Incident Activity of Daily Living (ADL) Dependence by Baseline Benchmark and Simple Model Scores, Validation Cohort (n=3,213).
Sensitivity Analyses
The c-statistics for the combined death and incident ADL outcome models were 0.75 and 0.72 for simple models with and without chronic disease, respectively; very close to those reported in Table 2. Analyses of age alone showed a c-statistic of 0.70 for predicting ADL dependency. Adding chronic disease to age improved the model fit to 0.75. Including hospitalizations in the benchmark model increased the c-statistic for the benchmark model by 0.01 (0.79 up to 0.80). Finally, re-estimating the simple models excluding persons over 90 years of age decreased the c-statistics to 0.74 (with disease) and 0.70 (without disease).
We sought to identify simple ADL dependency prediction models. Although the predictive validity of the simple models is lower than that of the benchmark model, the results do indicate a very good fit for the simple models. A scoring system developed from the odds ratios of the simple models retained a very good fit to the data. About 30 percent of persons scoring eight or more on the simple model with disease or on the model without disease went on to develop ADL dependency; whereas 10%-14% of those scoring six or less did so. Twenty-nine percent of those scoring eight or more on the benchmark model experienced incident ADL. Although the percent identified as high risk at a cutoff of eight is similar in the benchmark and simple model with disease, the Kappa statistic showed only fair agreement(13) between these models. Future work might explore alternative cut points, particularly identification of optimal cut points for subgroups.
Replication of the benchmark model in the 2006-2008 data shows clinical utility that is very good and very similar to that found in its original development based on data from 1993-1995. The benchmark model has a c-statistic that is better than the best simple model (0.79 vs. 0.74, respectively) but it requires multiple measures of functional status that are not available in common medical records.
Although all of the models shown in this report were developed using survey data, having eliminated certain variables from the benchmark interview model, we have created simple models for which scores could be created from medical records data. Although standardization of electronic medical records (EMR) has not yet been achieved,(14) EMRs are expanding and will be increasingly used for health care decisions.(15) In fact, the Centers for Medicare and Medicaid have distributed $73 billion dollars in incentive payments to be made from 2011 to 2015 for organizations that meet EMR usage criteria.(16) EMR data are not generally based on self-report so replication using actual EMR data on clinical populations need to be done and may produce different results.
For an individual clinician, the value of these brief instruments may be in raising awareness of the relative importance of certain states and conditions in risk for dependency (compared to other potential risk factors). However, for health care systems, administrators, policy makers, and researchers, these brief models are likely efficient, valuable screening tools to identify a high risk subset of older adults within a population. For example, applying a score of four or more as a positive screen to the simple model that includes chronic disease would identify 36% of the population as elevated risk and capture 67% of the incident ADL cases (calculated from data shown in Table 5). Within the 36% elevated risk subgroup, additional evaluation via interview,(5) brief assessment,(17) or comprehensive geriatric assessment(18) might be used to tailor interventions to high risk individuals. Comprehensive geriatric assessments of those at elevated risk may be the ideal approach given evidence that collaborative care interventions can be cost saving in high risk elders.(19)
In discussion above, we have used the cutoff examples of both 4 and 8; someone aged 90 or over would have a minimum score of 15 or above in the two simple models. Thus, it may be most efficient to assume that persons 90 years of age or over are a high risk group. Other indicators in the scoring system are helpful in older adults less than 90 years of age and our sensitivity analyses provided supporting evidence for the validity of the model in the 65 to 90 year age group.
Although self-report hospitalizations have been shown to correlate well with actual hospitalization records,(20, 21) there are other limitations to the measure of hospitalizations available in the HRS. It is an all-cause indicator and the effect of hospitalization on risk for dependency may vary by type of hospitalization. Some hospitalizations are for procedures that can improve ADL capacity (e.g., joint replacements), while others are for acute deteriorations in health. Hospitalization rates also vary by region, urban versus rural communities, and community socioeconomic level.(22) Thus, the role of hospitalization in incident ADL may differ by region or community due to differing admission criteria.
The baseline data we have used to inform the predictor variables are based on self-report data. Validation of self-reported chronic disease diagnoses against medical records data has shown good to excellent agreement (23-25) but confirming this in the context of these models would be important. Also, we were only able to evaluate chronic disease diagnoses contained in the HRS and this did not include chronic renal failure. It is unclear whether this would have a significant effect on results. Gill et al, 2010 showed that dependency trajectories are highly heterogeneous for persons with organ failure.(10) ADL dependency at two year follow-up was based on a proxy report for 7.5% of the sample. Proxy respondents have been found to overestimate the amount of hours given to ADL assistance but overestimates of need for ADL assistance are less apparent.(26) The intraclass correlation between older adults and their proxies on reports of need for assistance in seven instrumental ADLs was found to be 0.85 in a study of hip fracture patients.(27)
For some of the reasons noted above, validation in clinical populations using clinical data is needed but these analyses suggest that a simple approach to identifying elders at elevated risk for ADL dependency may be possible, particularly as EMRs become widely available. Identifying such high risk elders is a necessary step in implementing models of care that have been shown to improve outcomes and reduce costs of care for vulnerable older adults.(18) Thus, in the context of further assessment and interventions, ADL risk assessment tools could contribute to the goals of reducing Medicare expenditures and maintaining independence at home.
Acknowledgment
The authors wish to thank Steven R. Counsell for comments on an earlier draft. The authors take sole responsibility for the content of this final manuscript and have no conflicts of interest to report. All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. The project described was supported by Award Numbers R01 AG031222 and P30AG024967 from the National Institute on Aging to the IU Roybal Center. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging or the National Institutes of Health.
Supported by National Institute on Aging grants P30 AG024967 and R01 AG031222.
Footnotes
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
1. Congress of the United States: Congressional Budget Office In: High-cost Medicare beneficiaries. Office CB, editor. Congress of the United States; Washington, DC: 2005.
2. Lewin Group Individuals Living in the Community with Chronic Conditions and Functional Limitations: A Closer Look. U.S. Department of Health and Human Services; Washington, DC: 2010.
3. Norris SL, High K, Gill TM, et al. Health care for older Americans with multiple chronic conditions: a research agenda. J Am Geriatr Soc. 2008;56:149–159. [PubMed]
4. Bonk J. A road map for the recruitment and retention of older adult participants for longitudinal studies. J Am Geriatr Soc. 2010;58(Suppl 2):S303–307. [PubMed]
5. Covinsky KE, Hilton J, Lindquist K, et al. Development and validation of an index to predict activity of daily living dependence in community-dwelling elders. Med Care. 2006;44:149–157. [PubMed]
6. Hauser RM, Willis RJ. In: Survey Design and Methodology in the Health and Retirement Study and the Wisconsin Longitudinal Study. Waite L, editor. Aging, Health, and Public Policy; 2004.
7. Gill TM, Robison JT, Tinetti ME. Difficulty and dependence: two components of the disability continuum among community-living older persons. Ann Intern Med. 1998;128:96–101. [PubMed]
8. Boyd CM, Xue QL, Simpson CF, et al. Frailty, hospitalization, and progression of disability in a cohort of disabled older women. Am J Med. 2005b;118:1225–1231. [PubMed]
9. Boyd CM, Xue QL, Guralnik JM, et al. Hospitalization and development of dependence in activities of daily living in a cohort of disabled older women: the Women’s Health and Aging Study I. J Gerontol A Biol Sci Med Sci. 2005a;60:888–893. [PubMed]
10. Gill TM, Gahbauer EA, Han L, et al. Trajectories of disability in the last year of life. N Engl J Med. 2010;362:1173–1180. [PMC free article] [PubMed]
11. Spitznagel EL, Gohagan JK, Sessions DG. Graphic representation of logistics regression models. Proceedings of the Ninth Annual Conference of SAS Users Group International. 1984:870–873.
12. Ohman EM, Granger CB, Harrington RA, et al. Risk stratification and therapeutic decision making in acute coronary syndromes. JAMA. 2000;284:876–878. [PubMed]
13. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360–363. [PubMed]
14. Mishmari A. Blumenthal on EMRs: Debate “raging” over competition vs. standards Innovations in Cardiovscular Interventions (ICI) Meeting; Tel-Aviv, Israel: Israel Life Science Industry. 2010.
15. FierceHealthcare Information in EMRs to be Health Industry’s Most Valuable Asset in Five Years, Finds PricewaterhouseCoopers. FierceHealthcare; New York, NY: Oct 1, 2009.
16. Center for Medicare and Medicaid, (CMS) EHR Incentive Programs. Baltimore, MD: 2010.
17. Sarkisian CA, Liu H, Gutierrez PR, et al. The Study of Osteoporotic Fractures Research Group Modifiable risk factors predict functional decline among older women: a prospectively validated clinical prediction tool. J Am Geriatr Soc. 2000;48:170–178. [PubMed]
18. Counsell SR, Callahan CM, Buttar AB, et al. Geriatric Resources for Assessment and Care of Elders (GRACE): a new model of primary care for low-income seniors. J Am Geriatr Soc. 2006;54:1136–1141. [PubMed]
19. Counsell SR, Callahan CM, Tu W, et al. Cost Analysis of the Geriatric Resources for Assessment and Care of Elders Care Management Intervention. J Am Geriatr Soc. 2009;57:1420–1426. [PubMed]
20. Reijneveld SA. The cross-cultural validity of self-reported use of health care: a comparison of survey and registration data. J Clin Epidemiol. 2000;53:267–272. [PubMed]
21. Roberts RO, Bergstralh EJ, Schmidt L, et al. Comparison of self-reported and medical record health care utilization measures. J Clin Epidemiol. 1996;49:989–995. [PubMed]
22. AHRQ [Accessed June 20, 2010];Preventable Hospitalization: Executive Summary. 2010 [This Fact Book examines one critical area of health care quality: potentially preventable hospitalizations.] Available.
23. Bergmann MM, Byers T, Freedman DS, et al. Validity of self-reported diagnoses leading to hospitalization: a comparison of self-reports with hospital records in a prospective study of American adults. Am J Epidemiol. 1998;147:969–977. [PubMed]
24. Bombard JM, Powell KE, Martin LM, et al. Validity and reliability of self-reported arthritis: Georgia senior centers, 2000-2001. Am J Prev Med. 2005;28:251–258. [PubMed]
25. Martin LM, Leff M, Calonge N, et al. Validation of self-reported chronic conditions and health services in a managed care population. Am J Prev Med. 2000;18:215–218. [PubMed]
26. Cotter EM, Burgio LD, Stevens AB, et al. Correspondence of the functional independence measure (FIM) self-care subscale with real-time observations of dementia patients’ ADL performance in the home. Clin Rehabil. 2002;16:36–45. [PubMed]
27. Magaziner J, Zimmerman SI, Gruber-Baldini AL, et al. Proxy reporting in five areas of functional status. Comparison with self-reports and observations of performance. Am J Epidemiol. 1997;146:418–428. [PubMed]