|Home | About | Journals | Submit | Contact Us | Français|
In the Study of Osteoporotic Fractures (SOF), 18.5 % of incident hip fractures identified in Medicare Fee-for-Service claims data were not reported to or confirmed by the cohort. Cognitive impairment was a modest risk factor for false-negative hip fracture ascertainment via self-report.
Prospective cohort studies of fractures that rely on participant self-report to be the initial signal of an incident fracture could be prone to bias if a significant proportion of fractures are not self-reported.
We used data from the SOF merged with Medicare Fee-for-Service claims data to estimate the proportion of participants who had an incident hip fracture identified in Medicare claims that was either not self-reported or confirmed (by review of radiographic reports) in SOF.
Between 1/1/1991 and 12/31/2007, 647 SOF participants had a hip fracture identified in Medicare claims, but 120 (18.5 %) were either not reported to or confirmed by the cohort. False-negative hip fracture ascertainment was associated with a reduced modified Mini-Mental State Exam (MMSE) score (odds ratio 1.31 per SD decrease, 95 % C.I. 1.06–1.63). Point estimates of associations of predictors of incident hip fracture were changed minimally when the misclassification of incident hip fracture status was corrected with use of claims data.
A substantial minority of incident hip fractures were not reported to or confirmed in the SOF. Cognitive impairment was modestly associated with false-negative hip fracture ascertainment. While there was no evidence to suggest that misclassification of incident hip fracture status resulted in biased associations of potential predictors with hip fracture in this study, false-negative incident fracture ascertainment in smaller cohort studies with limited power may increase the risk of type 2 error (not finding significant associations of predictors with incident fractures).
Many large cohort studies with fracture outcomes including the Framingham Study, the Iowa Women’s Health Study, the Study of Osteoporotic Fractures (SOF), the Women’s Health Initiative, and the Osteoporotic Fractures in Men (MrOS) study rely at least, in part, on participant self-report to be the initial signal that an incident fracture may have occurred. While confirmation of self-reported fracture events via radiology reports can verify true fracture events and identify false-positive self-reports, this process does not account for incident fractures that are not reported to the cohort. In addition, confirmation of self-reported fracture events is not possible without access to medical records, and this may be another cause of false-negative ascertainment of true fracture events. False-negative ascertainment of fracture events and the degree to which participant characteristics are associated with both fractures and false-negative ascertainment of those fractures conceivably could introduce bias into cohort studies.
While some previous studies have found self-report to be accurate, these studies have focused on false-positive hip fracture events (a positive self-report of hip fracture with no fracture documentation in medical records) and have not estimated underascertainment of hip fracture [1, 2], included a younger study population , sampled only a small random subset of those not reporting fractures to assess underreporting [3, 4], or have had only very small numbers of hip fracture cases and thus, were limited with respect to statistical power [5, 6].
Two studies comparing hip fracture self-reports against national fracture registries in Denmark  and Iceland  found, respectively, that 41 % and 61 % of hip fractures in the registry were not self-reported by participants in survey studies who were specifically queried about their fracture history. Women aged 65 years or older participating in the Iowa Women’s Health Study were queried about incident hip fractures every 2 years, and hip fracture incidence by self-report was 2.61 per 1,000 patient years, whereas the incidence of hip fracture identified through inpatient Medicare claims for the same participants was 4.2 per 1,000 patient years . Medicare claims to verify and/or identify incident hip fractures in cohort studies is an attractive alternative to medical record review, since incident hip fractures can be identified with high accuracy through inpatient Medicare claims. The positive predictive value of a discharge diagnosis of hip fracture (ICD-9 code 820.0×) from an inpatient stay is reported to be 98 % [1, 10].
The SOF is unique in that participants are queried about incident fractures via postcard once every 4 months (with a response rate of 95 % through 2007). Hence, we hypothesized that the proportion of incident hip fractures that were either not reported to or confirmed by the cohort study would be very low. Using data from SOF linked to the available Medicare claims of SOF participants, our first aim was to estimate the proportion of hip fractures identified in Medicare administrative claims data that were also identified through the cohort study (e.g., self-report with confirmation by review of radiographic reports). The second aim was to examine whether the associations of known predictor variables with incident hip fractures identified and confirmed by the cohort study were altered when those participants with false-negative hip fracture ascertainment were reclassified as having had an incident hip fracture once their claims history was known.
From 1986 to 1988, 9,704 women at least 65 years old participated in the first examination of the prospective SOF. Women were recruited from population-based listings at four metropolitan centers in the United States (Baltimore, Minneapolis, Pittsburgh, and Portland, Oregon). The methods of recruitment have been described previously . After the baseline examination, subsequent examinations were conducted approximately every 2 years.
Since outpatient Medicare claims are not available prior to 1991, we tried to match inpatient and outpatient Medicare claims for the period January 1, 1991 to December 31, 2007 for all SOF participants actively enrolled during all or part of this time period. Because >80 % of Portland participants were recruited from Kaiser Permanente Northwest and were enrolled in Medicare Advantage (MA) (and therefore did not have detailed claims data in Centers for Medicare and Medicaid Services [CMS] files), we excluded all Portland participants (n=2,421) from our analyses. We additionally excluded those who died (n=272) or terminated (n=45) from SOF prior to 1/1/1991. Among the 6,965 actively enrolled, surviving women from the other three SOF sites as of 1/1/1991, enrollment in MA during the period 1/1/91 through 12/31/07 varied from 4 to 14 % of SOF participants at Baltimore, 40–50 % of participants at Minneapolis, and 0 % in 1991 rising to 30 % of participants at Pittsburgh as of 2007 (Fig. 1).
A finder file with one record for each with a social security number (SSN) or Medicare ID (Health Insurance Claim [HIC] number) was submitted to and used by CMS to match SOF participants in their data files. We used an algorithm similar to those previously described [12, 13] to confirm that social security or HIC number and at least two of three components of data of birth (month, day, and year) and gender agreed between SOF data and CMS claims data. By these criteria, 6,420 (92.2 %) were successfully matched to their Medicare claims, and of these, 6,003 were enrolled in FFS Medicare, both parts A and B, for at least 1 month during the period 1/1/1991 through 12/31/2007. Among these women, 5,008 women, who attended the second SOF visit in 1989–1990 had femoral neck BMD measured and had not reported a hip fracture prior to 1991 to the SOF cohort, comprised the overall study population for these analyses.
In SOF, incident fractures were identified by follow-up contact with participants or proxies every 4 months by mailed postcards or phone, with a 95 % response rate to these queries through 2007. Incident hip fractures were confirmed by physician adjudication of radiographic reports. Incident hip fractures were identified in the Medicare claims MedPAR file by a primary or secondary discharge ICD-9 diagnosis code of 820.0× or 733.14. Because 733.14 can include hip fractures due to pathology other than osteoporosis (such as malignancy), we did secondary analyses excluding those cases with a primary diagnosis of 733.14 (n=13) and those cases with only a secondary diagnosis of hip fracture (n=12).
At the SOF baseline examination, participants were queried about self-rated health status, height at age 25, and history of stomach surgery, diabetes, and Parkinson’s disease. Cognition was assessed by a modified Mini-Mental State Examination (MMSE) [14, 15], and depth perception was assessed. As described in previous publications [16, 17], at the SOF visit 2 examination, height, weight, contrast sensitivity for vision, and femoral neck bone mineral density (BMD) using dual energy x-ray absorptiometry (DXA) were measured. Participants were queried about clinical fractures since age 50, history of chronic obstructive pulmonary disease, current smoking status, current use of thyroid medications, and social support (living alone). The ability to stand at least five times from a chair without using one’s arms was evaluated by direct observation. Participants’ usual walking speed over a 6-m distance was measured in meters per second. The change of weight (kilograms) from age 25 to the time of each examination was calculated from self-reported body weight at age 25 and measured weight at visit 2. Body mass index (BMI) was calculated as the ratio of current weight (kilograms) to height at age 25 (in meters) squared. Participants also completed assessments of executive functioning using the Trails B test, Independent Activities of Daily Living (IADL’s), and depressive symptoms using the Geriatric Depression Scale.
A cross tabulation was performed of the proportions of SOF participants who had incident hip fractures reported to and confirmed by the cohort versus those who had incident hip fractures identified in Medicare claims. True-positive hip fracture ascertainments were defined as hip fractures identified in Medicare claims that were also self-reported to SOF and confirmed by review of reports of x-rays taken within 31 days of the first hip fracture admission date in claims. False-negative hip fracture ascertainments were a first hip fracture identified in claims with no self-reported and confirmed hip, pelvis, proximal femur, or distal femur fractures on follow-up postcards in SOF within 31 days of the fracture date in claims. In this analysis, we restricted the study population to those 702 women who had a claims-identified fracture and who were concurrently enrolled in SOF at the time of their hip fracture, so that there would be full potential for the fracture to be reported to and confirmed in SOF.
We estimated the associations of false-negative hip fracture ascertainment compared to true-positive hip fracture ascertainment with the variables that had been associated with incident hip fracture risk in prior published studies from SOF [16, 18] because the greatest potential for bias would be for those specific variables associated with both incident hip fracture status and false-negative ascertainment of these fractures. Candidate variables were age at the second SOF visit, 6-m walk time, chair stand speed, MMSE score, living with others, BMI, height at age 25, prior clinical fracture since age 50, self-reported health status, IADL’s, smoking status, femoral neck BMD, study site, tertiles of follow-up time period (<7 years, 7–11 years, ≥12 years), and an interaction term between study site and follow-up time.
The multivariable-adjusted associations of predictor variables with false-negative hip fracture ascertainment were estimated with logistic regression. Age, study site, follow-up time period, an interaction term between study site and follow-up time period were forced into the models and then other candidate predictors with an unadjusted association with false-negative hip fracture ascertainment at a p value <0.2 were simultaneously added to the full multivariable model.
To estimate the bias that may be present in the estimated age- and femoral neck BMD-adjusted associations of predictors with incident hip fractures, we compared the point estimates and confidence intervals of these associations from four sets of proportional hazards models, adjusted for study site, follow-up time period, and the interaction between study site and follow-up time period. For the first set (Aim 2a analyses), the dependent variable was incident hip fracture identified and confirmed in SOF between January 1, 1991 and December 31, 2007. Individuals were censored at death, termination from the cohort (typically at participant or family request or after repeated failed attempts to contact them), incident hip fracture, or end of study period (12/31/2007). For the second set of models, the dependent variable was incident hip fracture identified in either SOF or Medicare claims from the Baltimore, Minneapolis, or Pittsburgh sites. Individuals were censored only at incident hip fracture, death, or end of study period (12/31/2007), since hip fractures might be identified in Medicare claims after termination of participation from SOF. We used the comparisons between these two sets of regressions to examine practically how the estimated associations between predictors and incident hip fracture may be altered when a cohort study such as SOF is able to augment cohort data with FFS Medicare claims data. We included predictors that had been previously reported to be associated with or nearly associated with incident hip fracture [16, 18].
However, because a significant proportion of the incident hip fractures were reported to SOF during time periods when participants at the three sites (Minneapolis, Baltimore, or Pittsburgh) were enrolled in MA, the first two regressions still both misclassify those enrolled in MA who have a hip fracture and do not self-report it or do not have it confirmed by the cohort. The comparison of those two models does not fully address the issue of how large the bias may truly be from false-negative ascertainment hip fracture self-reports. Therefore, the two regressions described above were repeated including only those 4,248 active SOF participants who were alive and enrolled in FFS Medicare as of 1/1/1991, such that hip fractures identified in Medicare claims would be a better “gold standard” against which we could compare hip fractures identified through self-report and confirmed by the cohort study (Aim 2b analyses). Individuals were censored at death, incident hip fracture, the end of the study period, or enrollment into MA.
Of the 5,008 women in the study population for these analyses, there were 655 women with a SOF-adjudicated hip fracture and 704 women with a claims-identified hip fracture during follow-up, who did not have a fracture detected in SOF prior to 1/1/1991. Ninety-one women with a SOF-adjudicated hip fracture did not have a Medicare claim for that fracture largely because they were enrolled in MA at the time of their fracture and are excluded from these analyses. One hundred sixty-six women had a Medicare claim for hip fracture that were not reported to and/or confirmed in SOF.
Of the 704 hip fracture claims, 28 occurred after termination from the SOF study (Fig. 2a). From the remainder of 676, we excluded 29 women for which review medical record radiographic data collected in SOF revealed that the presence of a proximal femur fracture was uncertain or that the fracture location was the pelvis or distal femur.
Of the remaining 647 hip fractures identified in Medicare claims, 527 (81.5 %) were reported and confirmed in the cohort study (e.g., true positives), yielding a false-negative hip fracture ascertainment rate of 18.5 % (Table 1). Of the 120 false-negative hip fracture ascertainments among participants actively enrolled in SOF at the time of the hip fracture claim, 105 were not self-reported to the cohort and 15 were due to the inability of the cohort to obtain confirmatory radiographs or to data recording error.
Compared to the 527 women with true-positive hip fracture ascertainment, the 120 women with false-negative hip fracture ascertainment had higher femoral neck BMD, slightly lower MMSE scores, and were marginally more likely to be living with others (Table 1). After multivariable adjustment (including adjustment for age, study site, follow-up time period, and the interaction of site and follow-up time (Table 2), only lower MMSE score (odds ratio 1.31, 95 % CI 1.06–1.63 for each standard deviation decrease in score) was independently associated with false-negative hip fracture ascertainment. The c-statistic of this regression was 0.72, indicating that only a modest proportion of the phenomenon of false-negative hip fracture ascertainment was explained by the predictors in these models. A sensitivity analysis excluding 13 participants with a hip fracture diagnosis of 733.14 and 12 participants with only a secondary diagnosis of hip fracture showed virtually identical results (data not shown).
Between January 1, 1991 and December 31, 2007, 655 of 5,008 women in our analytic cohort self-reported a first hip fracture and had that fracture confirmed by SOF (Fig. 2b). First hip fractures were identified for an additional 166 women in claims that were not reported to or confirmed in SOF (some of whom experienced their hip fracture after terminating from the study). In general, the age-, and femoral neck BMD-adjusted point estimates and confidence intervals for associations of predictors with incident hip fractures reported to SOF were not substantially altered when the 166 individuals with a Medicare claim for hip fracture that had not been reported to and confirmed by SOF were reclassified as having had an incident hip fracture (Table 3). The age-adjusted rate of incident hip fracture was 0.00876/patient-year using confirmed self-reports of hip fracture only and was 0.01128/patient-year when the additional 166 women with a Medicare claim for hip fracture were reclassified as having an incident hip fracture.
When the analysis was restricted to those 4,248 women continuously enrolled in FFS Medicare from January 1, 1991 onward (such that all hip fractures could be detected in claims), 472 of 602 hip fractures identified in claims were self-reported to and confirmed by the cohort and 130 additional hip fractures were identified in claims (Fig. 2b). In general, the age- and femoral neck BMD-adjusted point estimates and confidence intervals of associations of predictors with incident hip fractures reported to SOF again were minimally changed when the additional 130 women with a Medicare claim for hip fracture were reclassified as having had an incident hip fracture (Table 4). Sensitivity analyses excluding 13 participants with a hip fracture diagnosis of 733.14 and 12 participants with only a secondary diagnosis of hip fracture showed virtually identical results (data not shown).
In the absence of administrative claims or study design with a case-finding mechanism that allows for capturing comprehensive medical record data on all participants, prospective cohort studies generally rely on self-report as the initial signal that a participant may have experienced an incident-adverse health event such as a fracture. In spite of intense, frequent follow-up in the SOF prospective cohort study to solicit self-reports of incident fractures, 18.5 % of incident hip fractures documented in Medicare claims were either not reported by cohort members or not confirmed by the study. A few of the hip fracture events identified only in Medicare claims are instances of miscoding of distal femur and pelvis fractures. However, we found that only 19 of 702 women (2.7 %) with first hip fractures identified in claims data had the fracture confirmed to be at a different skeletal site based on review of x-ray reports collected as part of the fracture adjudication process in SOF.
Impaired cognitive status, specifically a lower MMSE score, was associated with a higher likelihood of false-negative hip fracture ascertainment. Other variables that we postulated would be associated with false-negative hip fracture ascertainment, such as depression, inability to rise from a chair without use of the arms, and slower walk speed, were not significantly associated with false-negative hip fracture ascertainment. Models that include a large number of variables that together constitute a rich phenotypic description of the SOF population explained only a modest proportion of the phenomenon of false-negative ascertainment of hip fracture.
With misclassification (predominantly underascertainment) of a substantial proportion of outcome events, there is the potential for loss of statistical power in the estimation of associations between candidate risk factors and incident fractures. Additionally, there is potential for bias for variables that are associated with both the outcome itself and false-negative ascertainment of that outcome. Nonetheless, when available claims data were used to reclassify those with a Medicare claim for hip fracture undetected in the cohort data as truly having had a hip fracture, there was only a slight narrowing of the 95 % confidence intervals for estimated associations of predictors with incident hip fracture, and point estimates of the associations between predictors and incident hip fracture were not substantially altered. The only modest exception was MMSE, the only multivariable-adjusted significant predictor of false-negative hip fracture ascertainment that we could find. The point estimate of the association of MMSE with SOF-identified incident hip fractures was only 1.09 and insignificant, but the estimated excess risk nearly doubled (HR 1.17) and became significant after hip fracture status reclassification using claims data.
These results show that the vast majority of estimated associations of predictors with incident hip fractures from the SOF cohort have been reported accurately in prior publications. Moreover, the process of reviewing radiographic reports for those who do self-report a fracture in SOF identifies false-positive self reports, ensuring a specificity of 100 % for incident fracture identification in SOF. However, large numbers of women in the SOF cohort experienced an incident hip fracture and the potential for missing significant associations between predictors and hip fracture would be greater in smaller cohort studies with a lower number of outcome events.
The use of claims data in the Medicare population is not a complete solution to this problem due to the costs of obtaining, processing, and analyzing claims and the fact that reliable claims for those enrolled in MA plans are not generally available for research purposes from CMS. There is, of course, the possibility that claims data could be made available through collaboration with various health care organizations that offer MA health plans.
There are other important limitations to our study. SOF enrolled only Caucasian women until the late 1990s, and our results may not be applicable to men or non-Caucasian women. Our data is not applicable to other outcomes events such as cardiovascular or neoplastic diseases that were not rigorously ascertained by SOF. The study did not systematically record if a family member or other proxy filled out the 4-month follow-up questionnaires, and hence, we cannot ascertain if proxy reports were associated with false-negative hip fracture self-reports. Finally, we were not able to obtain medical records to confirm the hip fractures identified only in Medicare claims data that were not self-reported; however, the false-positive rate of the claims data was very small (approximately 3 %) for those hip fractures in claims data for which SOF did have medical records available.
In conclusion, false-negative ascertainment occurred in 18.5 % of women with incident hip fractures followed over a 17-year period even when self-report was frequently and vigorously pursued. Cognitive impairment is modestly associated with false-negative hip fracture ascertainment via self-report. In a cohort study with a large sample size and a high rate of completion of follow-up contacts, there may not be any significant bias from this incident hip fracture misclassification, but in smaller less robustly powered studies, there may be a risk of type 2 error (missing associations of predictors with incident hip fracture) if self-report is used to identify incident fractures. In these instances, supplementing self-report with another source of fracture event identification such claims data should be considered. Further studies of the proportion and predictors of false-negative ascertainment of other fracture and nonfracture health events are needed in order to better understand the causes of and impact of this phenomenon in epidemiologic cohort studies.
Source of funding The Study of Osteoporotic Fractures (SOF) is supported by the National Institutes of Health funding. The National Institute on Aging (NIA) provides support under the following grant numbers: R01 AG005407, R01 AR35582, R01 AR35583, R01 AR35584, R01 AG005394, R01 AG027574, and R01 AG027576. Dr. Curtis receives support from the NIH (AR053351) and the Agency for Healthcare Research and Quality (R01HS018517). The purchase of the linked CMS administrative claims data was supported, in part, by a grant to Dr. Curtis from the Arthritis Foundation.
Disclaimer The views expressed herein do not necessarily represent the views of the Department of Veterans Affairs or the United States Government.
Conflicts of interest None.
J. T. Schousboe, Park Nicollet Institute, Minneapolis, MN, USA. Division of Health Policy and Management, University of Minnesota, Minneapolis, MN, USA.
M. L. Paudel, Division of Epidemiology, University of Minnesota, Minneapolis, MN, USA.
B. C. Taylor, Division of Epidemiology, University of Minnesota, Minneapolis, MN, USA. Center for Chronic Diseases Outcomes Research, Department of Veterans Affairs Health Care System, Minneapolis, MN, USA. Department of Medicine, University of Minnesota, Minneapolis, MN, USA.
B. A. Virnig, Division of Health Policy and Management, University of Minnesota, Minneapolis, MN, USA.
J. A. Cauley, Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA, USA.
J. R. Curtis, Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, AL, USA.
K. E. Ensrud, Division of Epidemiology, University of Minnesota, Minneapolis, MN, USA. Center for Chronic Diseases Outcomes Research, Department of Veterans Affairs Health Care System, Minneapolis, MN, USA. Department of Medicine, University of Minnesota, Minneapolis, MN, USA.